VMware vSphere ‘Whitebox’ Server Lab Setup – Part 1
Posted on 07.Dec 2010 by Ray Heffer in Business & IT, Home Lab, Tutorials, VMware, VMwareIntroduction
Way back when VMware VI3 was released in 2006 (doesn’t time fly!), I built a home-brew lab server for ESX 3.0 and used it partly to study for my VCP exam. That particular machine is now my home theatre PC (HTPC) as it wouldn’t stand a chance of running VMware vSphere, so here is my mission to build a whitebox VMware vSphere lab server. I must also give credit to Simon Seagrave and Simon Gallagher their vSphere lab server articles which have inspired me to do something about it and build a vSphere lab at home. Simon has lots of great articles on building a vSphere lab, and I urge you to visit his site.
vSphere Lab Environment Topology:
Reasons for a home lab?
Throughout my entire IT career I have used equipment at home to help study for certifications and enhance my technical knowledge. As an IT professional you’ll not only need a lab to study for your certifications, but you will need a test-bed to experiment and try things for the first time. There have been occasions as a technical consultant where a fairly niche customer requirement presents itself and is something that requires very specialist skills. Having the flexibility to remotely connect to my home lab and test a procedure or new technology gives you the confidence to carry out that piece of work in front of the customer, even if it’s the first time you’ve done it. We all have to start somewhere, and there is a first time for everything… better that it’s in a test environment than a live production system though!
Ever since I first started in IT back in 1997, I have always had a home server, lab, test-bed or whatever you want to call it. I remember installing NT4 on a home built server in 1997, promoting it to a PDC and then configuring WINS and DHCP. I even had a couple of Cisco 2500 routers so I could emulate a serial connection between two subnets, and at the time I felt that I was at the cutting edge. For the past 5 years, VMware has made it a whole lot easier to create test and development platforms. VMware Workstation is the saviour of many developers and server administrators as it allows them to very easily create virtual servers for testing, experimenting or learning, whether it’s on their laptop or home PC.
The most important factor of having a home-lab for me personally has been the freedom to access a broad range of server technologies that I can do whatever I like, from build to destruction.
Server Choice
I desperately wanted to avoid buying a branded server from HP or Dell, because to get anything near the specification of my whitebox would cost a fortune. Not only that but the ML115 G5 is becoming very hard to get hold of, unless you find a bargain on Ebay. The later ML115 G6 is reported to work, but the B110i storage controller won’t work. I find that these type of servers can also be very noisy and my wife doesn’t understand the novelty of these things when they’re powered on in the spare bedroom!
Officially you would need to buy a server that is listed on the VMware HCL but thanks to VM-Help.com they have put together a whitebox HCL which has the Asus Rampage II motherboard on the whitebox HCL list. Also my intention is to build a system that I can put to other uses in the future such as a gaming PC, should later versions of vSphere not be compatible for any reason.
So for me the decision is clear… my own whitebox vSphere server!
My VMware vSphere ‘Whitebox’ Server Kit List
MOTHERBOARD: Asus Rampage II Extreme Intel X58 1366
CPU: Intel i7 930 2.8GHz Socket 1366 8MB L3 Cache
RAM: 12GB OCZ Gold Low Voltage (6x2GB) DDR3 PC3-16000C10 2000MHz Triple Channel
NETWORK: Intel Pro/1000 MT Dual port 1GbE (the two onboard GbE NIC’s don’t work with vSphere)
SSD: OCZ 120GB Vertex 2E SSD 2.5″ SATA-II Read = 285MB/s, Write = 275MB/s 50,000 IOPS
GRAPHICS: Asus HD 4350 512MB DDR2 DVI VGA HDMI Out PCI-E Low Profile Graphics Card
PSU: Antec TruePower New 650W Modular PSU
OTHER: LiteOn IHAS124-19 24x DVD±RW DL & RAM SATA Optical Drive
CASE: Antec Plusview 1000AMG
Network Switch: Linksys SLM2008
Thanks to Techhead for pointing this out. This is ideal for my home lab environment as it has 8 x 10/100/1000 GbE ports and supports 802.1q VLAN segmentation, and Jumbo Frame support (9KB). It even supports PoE (Power over Ethernet) from the first port, although I doubt I’ll use that. It is an awesome switch and pretty cheap at around £70.
Total Cost: £970 (inc VAT and Postage)*
*Best price I could find for all components listed as of December 2010
To help understand what features my setup will support, I’ve put together this table…
Features & Hardware Requirements
| Feature | Hardware Requirement | Meets Requirement? |
| vSphere | 64-bit processor Intel-VT AMD-V |
Yes |
| VMDirectPath | Intel-VTd AMD IOMMU |
No |
| DPM | Wake-on-LAN, Server IPMI or HP iLO | Yes |
| Dynamic Voltage & Frequency Scaling | Intel Enhanced Speedstep AMD Enhanced PowerNow |
Yes |
| Fault Tolerance | Intel’s Core 2, Core i7 AMD’s 3rd Generation Opteron family |
Yes * |
| VMotion | Similar CPU family (not cross vendor) | Yes |
| Enhanced VMotion Capability | Intel FlexMigration AMD-V Extended Migration |
Yes |
* Fault tolerance would need more than one ESX host in an HA cluster, and a processor that supports virtualisation (VT), however the virtual ESX hosts will not provide VT therefore Fault Tolerance cannot be used.
So here is the case I’m using, it’s an Antec Plusview 1000 AMG which is larger than I would normally use, but as it was just sitting in my garage I decided to put it to good use. It will also sit quite happily in my spare room under the desk, and the only fans I am intending to use in this case are the CPU fan and obviously the built-in PSU fan. Other than those two the graphics card and motherboard are silent which means that my wife won’t get too upset!
In part 2 I’ll show you the hardware build, configuration and then the installation of ESXi 4.1 and vCenter.
Update: I have had issues with the Asus Rampage II Extreme detecting 8GB instead of 12GB memory, see this link for further information.
Thanks for reading! Feel free comment below
asus rampage II, esx, esxi, i7, intel, lab, quad core, server, VMware, vsphere, whitebox
[...] on from part 1 of my VMware vSphere ‘whitebox’ sever series, this video provides an overview of the [...]
Hi, i’m also going to build my own system.
I see that you only have sdd, but do you know if the raid is a software raid or not. Is there a change that it’s also supported?
The motherboard has 6 x SATA II ports and can be configured with RAID 0, 1, 5, 10 (so it’s hardware). I will be adding a 1TB HDD to the system soon so I’ve got somewhere to store my ISO’s and other data partitions for virtual machines.
Hi Ray,
this was just what I was looking for!
It gave me the courage to go on with my plan to buy an ASUS Rampage III Extreme.
The rest of my shopping list:
CPU: Intel Core i7-950
Memory: GEIL 24 GB DDR3-1366 Hex kit (9-9-9) i’d liked to have the DDR3-1066 7-7-7 but they were out of stock.
Harddisk: OCZ Solid State Drive 120 GB
DVD: Sony OPTI-Arc DVD re-writer (the cheapest i could find)
It turned out very well, the onboard NIC works fine (there is only one Intel Gigabyte NIC onboard of the Rampage III) and the memory does it’s job fine as well.
In the past I have tried to get RAID 5 working on a ICH10 mobo, but i never got it working and from what I’ve read on the forums I think you will need a real hardware RAID-controller board to get RAID working in ESX(i).
At the moment my new server is connected via iSCSI to a volume on a Synology DS508, but that turned out to be quite unstable. I’ve just changed the set-up (got rid of the software iSCSI and installed a Broadcom NIC with iSCSI initiator capabilities directly coupled to a Vmkernel interface) and hope it’s a little bit more stable now
Looking forward to part 3 of your story,
Cheers
Hi,
You’ve got a nice setup there
I’ve just added a short part 3, and I say it’s short due to my lack of time over the Christmas and New Year break. That said, I will be adding a section on configuring Openfiler and maybe another video.
Ray
Hi,
I’m glad you like the article!! Part 3 is up, but there should be more to come next month!
I hope it has been useful to you.
Ray
Hi Ray,
Your post is an excellent help, my name is David and I’m writing from Spain.
I will need a semi-professional machine to make job-labs in my home with ESXi.
I have one provisional configuration, but I still have some doubts about hard components. It’s possible with your experience you can help me.
Motherboard: Tyan 7002 (S7002GM2NR-LE), link to motherboard manual http://www.tyan.com/manuals/S7002_Manual_v100.pdf
CPU: Intel Xeon E5620
Memory: 4×4 16GB 1066MHz ECC REG, from eBay http://cgi.ebay.es/16GB-4X4GB-FOR-TYAN-S7002AG2NR-S7002G2NR-LE-/330512922339?pt=LH_DefaultDomain_0&hash=item4cf41af6e3#ht_1934wt_905
Heat sink: Noctua NH-U12P SE2
Harddisk: 2 x Seagate ST31000528AS 1 TB (Barracuda 7200.12)
Power supply: Thermaltake Toughpower XT 875W (EPS12V 2.91)
Case: Chieftec CA-01B-B-SL (EATX)
My doubts are:
1. Are this components compatible with the motherboard and ESXi? I wouldn’t like to have problems with the RAM.
2. Is the power supply appropiate? I select this one with standar EPS12V 2.91, but I’m not sure.
3. Finally, this motherboard needs case with factor SSI-CEB, my selection is EATX but is quite large, Is the case EATX appropiate?
In your opinion, Need I some else components?
Thanks and best regards,
David.
The first thing I would do is check the whitebox HCL (http://vm-help.com/esx40i/esx40_whitebox_HCL.php#MB) but your motherboard isn’t listed. I would be surprised if it works with ESXi, but there is mention for Tyan boards here (http://communities.intel.com/thread/9435) which may help?
My lab server required a separate Intel E1000 NIC as the on-board 1Gbps NIC’s are not supported on ESXi.
Keep us posted on your setup!
Ray
Hi Ray,
It’s true that “Tyan 7002″ isn’t listed in the HCL, but in the manufacturer’s website we can see this motherboards are VMware Certified up to ESXi 4.0
http://www.tyan.com/product_SKU_spec.aspx?ProductType=MB&pid=682&SKU=600000027
http://www.vmware.com/resources/compatibility/detail.php?device_cat=server&device_id=10447
http://www.tyan.com/(A(mHS6qz3_WUfhuR-nAquodomI73WyTpkf2Gi_1W5d-kRtEzb5UUlKLZ21ZmCmpcro_nm5TB1vS8gecs99Qut_CZdn_q8_6mE6Dela1pa_t4Q1))/vmware/VMware_Certification.pdf
This should be correct.
I have problems with the power supply, I don’t know if my selection is appropiate. And other question, motherboard supports RAID5, recognize ESXi RAID5? I think yes.
Thanks and regards,
There will be no guarantee’s it will work, but I don’t see why not. Your PSU seems good to me, mine is only 650watt and works a treat
Hello Ray. I only have expereince with vmplayer and vmfusion but would like to get started with the larger scale products. I enjoyed viewing your 3 part series on building a home lad and have a few questions if you have a moment. What would be the proper VM software to start with, ESXi 4.1 or ESXi5? Would I need to start with a server class machine? In part 2, you install the esxi to the portable usb stick, can i ask the purpose of the usb install please? Thanks
Hello,
I am lucky to come across this page as it helped me a lot to prepare for my VMware 5.0 study;
I am building the following server as my lab env and I wonder if it is suitable; please assist and advise?
The server specifications are as follows:
CPU: Intel Core i7 3930K
Motherboard: MSI x79A-GD65
RAM: 32GB Kit 1600 (8GB * 4) G.Skill RIPJAWS-Z CL10
Hard drive: 1TB Western Gate Black
Many thanks in advance;
Kind regards,
Ayman Ouda
Hi Ayman,
Even i am looking to build a server for VMware 5.0 study. Did your above specifications work for you? I am still undecided on the hardware and looking for someone who has currently build a whitebox.
Hi Ray,
Thank you for your time and continued assistance to the community,
I have attempted an ESXI whitebox installation on a Rampage 11 board with the following specs but I am hardly going past the F11 Agreement acceptance stage since I get an error hardware compatibility. I understand you have successfully configured your Rampage 11 for installation. I have installed an Intel Gigabit card in place of the onboard Marvel cards which are incompatible so was able to go past that verification stage. Perhaps I need to tweek my BIOS settings a little bit. Please help advise.
System Settings :
Processor a Main Circuit Board b
3.33 gigahertz Intel Core i7 975
64 kilobyte primary memory cache
256 kilobyte secondary memory cache
8192 kilobyte tertiary memory cache
64-bit ready
Multi-core (4 total)
Hyper-threaded (8 total) Board: ASUSTeK Computer INC. Rampage II Extreme Rev 2.xx
Serial Number: 101089480000042
Bus Clock: 133 megahertz
BIOS: American Megatrends Inc. 1704 12/30/2009
Drives new – drive encryption Memory Modules c,d
500.00 Gigabytes Usable Hard Drive Capacity
264.14 Gigabytes Hard Drive Free Space
HP DVD Writer 1170d ATA Device [Optical drive]
3.5″ format removeable media [Floppy drive]
Secure Guard USB Device [Floppy drive]
Hitachi HDP725050GLA360 [Hard drive] (500.11 GB) — drive 0, s/n GEA564RJ234LXA, rev GM4OA5CA, SMART Status: Healthy
Secure Guard USB Device [Hard drive] (4.12 GB) — drive 1 10232 Megabytes Usable Installed Memory
Standard Dual Channel PCI IDE Controller NVIDIA GeForce GTX 275 [Display adapter]
Gateway FPD1530 [Monitor] (14.9″vis, s/n 4082, June 2003)
Communications new – connection speed & status Other Devices
↓ Intel(R) Gigabit CT Desktop Adapter
↓ Marvell Yukon 88E8056 PCI-E Gigabit Ethernet Controller
Connection Speed: 1 Gbps
↓ Marvell Yukon 88E8056 PCI-E Gigabit Ethernet Controller #2
Connection Speed: 1 Gbps
That setup should work fine, you have an i7 processor, is VT / virtualization enabled in the BIOS? The Intel dual 1Gb network adapter is fine, same as mine. I installed ESXi on to a USB pen-drive, are you doing the same?
Hi Ray,
Finally got it running. Yes VT/ virtualization was already enabled in the BIOS. It appears like it was the drivers for the optical DVD Writer that were failing. I have attached a USB Mini RW CD-ROM and installed to a pen drive and all looks good so far.
Thanks again,
Norbert
Hey mate, where you able to install ESXi5 on your Asus Rampage II Extreme with the Intel X58 chipset??
I have same Intel chipset but on another mobo vendor though and I get a “Relocating modules and starting up the kernel” message and then nothing, install stays there forever…:(
Thx,
Didier
Hey Didier,
Yes i successfully installed my ESXi 4.1 but not Ver.5 to my Rampage II Extreme.I found the fault to be the DVD ROM and after switching to a drive with Universal drivers all went well. Perhaps you could try with another drive or better still install from a pen drive.
Good luck,
Norbert
Howdy would you mind sharing which blog platform you’re using? I’m looking
to start my own blog in the near future but I’m having a tough time deciding between BlogEngine/Wordpress/B2evolution and Drupal. The reason I ask is because your layout seems different then most blogs and I’m looking
for something unique. P.S My apologies for getting off-topic
but I had to ask!