I configured a purpose-built server for hosting virtual machines in a home virtualization lab. This page explains the build and some things I learned in the process.
A competent IT professional needs a way to safely experiment and prototype. That requires some appropriate hardware to use for a virtual-machine hosting platform.
Knowing this, I recently did a lot of tedious research and plenty of intensive contemplation about HomeLab servers and what a proper virtual-machine environment for hands-on experimentation requires. The goal was build a good server that has as many desirable features as possible but doesn't cost any more than necessary.
Happily, the diligent effort paid off. For a very reasonable price you can build an efficient and capable HomeLab Server that will provide years of dependable VM-hosting service.
Traits of a Server
This Page's Outline
Here are a few attributes of a well-built HomeLab server. This section will explain why workstation-grade hardware is far superior to conventional PC hardware when you need a server but don't want a noisy and cumbersome rackmount system in your house.
- Software Compatibility
- Choose components with a strong preference toward compatibility. VMWare ESXi, FreeBSD, Linux, Windows, and probably even Mac OS will run well on this machine.
- High Availability
- Workstations and servers are more durable than PCs. They're designed so they can operate under a continuous day-and-night workload. Durability is especially important for depreciated (bargain-priced) off-lease computers that are out of warranty.
- Expandability
- More expansion slots, more memory slots, and dual-CPU capability all add value to your server. High maximum memory capacity is especially important.
- ECC RAM
- ECC (Error-Correction Code) memory is beneficial because it provides data integrity. Enterprise servers have ECC memory because data integrity is more important for servers than for PCs. If a single memory bit erroneously flips in your office-suite document the consequences are not likely to be critical. On the other hand, if a bit flips when the computer is writing a file to your encrypted storage volume the consequences could be more significant.
- CPU Features
- More cores and more on-CPU cache memory add value to your server. Built-in Features like AES New Instructions are also beneficial. i-Series processors have desktop-oriented features whereas Xeon CPUs are built specifically for workstations and servers.
- Energy Efficiency
- Components with this often-overlooked quality enable your machine run cooler, quieter, and possibly more reliably as a result. It will also run longer on UPS battery power.
Without a doubt your server should be made from a workstation rather than a PC. There are reasons why new workstations cost so much more than new PCs.
Precision T5500 Workstation
The machine I chose is a robust and scalable Dell Precision T500 Workstation. It's a super-duty compact-and-quiet desktop workstation for professional technical and scientific applications. A machine like this opens up a vast number of possibilities for running VMs.
I first discovered the T5500 because I was searching for a used machine with at least 8GB of ECC RAM to use as a FreeNAS or NAS4Free fileserver. This machine can do that and a whole lot more (to say the least!).
T5500 Highlights:
- Accepts 55XX (Gainestown) and 56XX (Westmere-EP) Xeon quad- and hexa-core CPU(s)
- Ttri-channel DDR3 ECC Registered RAM (six slots; second-CPU option adds three more)
- Virtualization Features
- Intel Virtualization Technology (VT-x)
- Virtualization for Directed I/O (VT-d)
- Extended Page Tables (EPT), a.k.a. Second Level Address Translation (SLAT)
- Storage Features
- Two well-ventilated internal drives bays for 3.5" hard drives
- Additional drives can be adapted into 5.24" drive bays and the 3.5" bay.
- Expansion Features
- Dual-CPU Capable (requires second-CPU Riser)
- A 64-bit PCI-X (PCI Extended) slot
- Plenty of PCI-E (PCI Express) slots
- A PCI slot. How quaint.
- Efficient (85% or 88%) 875W power supply
T5500 Lowlights:
- SATA II 3.0Gb/s host controller (current is 6.0, better for SSDs)
- Broadcom 5754 Gigabit Ethernet (Would have preferred Intel)
A 650 volt-amp UPS is more that adequate to provide always-on power.
CPU Upgrade
The T5500's CPU socket is a "Socket LGA 1366". It accepts 55XX-series ("Gainestown") and 56XX-series ("Westmere-EP") efficient-performance CPUs. Early T5500's came with 55XX CPUs and later models came with 56XX CPUs.
An optional second-CPU socket takes the form of a riser assembly, which also adds three additional RAM slots. The riser assembly is three parts, the riser itself (F623F), a heatsink, (W715F) and a fan (F306F).
56XX-series CPUs are more desirable. They're more energy-efficient, they have reduced latency in virtualization, and they provide AES New Instructions (AES-NI) that dramatically improve throughput for disk encryption and other cryptographic operations.
Inexpensive Xeon L5630 and L5640 CPUs are available that minimize power consumption for cooler, quieter, and less-expensive operation. X5650 and X5660 processors use more power but have more grrr.
RAM Upgrade
The main motivation to get this machine was because it supports ECC (Error-Correction Code) RAM -- and plenty of it. ECC RAM is strongly recommended for best results with the ZFS filesystem, especially if encryption is enabled.
The best way to expand the capabilities of a virtual-machine server is usually by adding RAM. You can install up to 72GB of memory in a T5500 according to Dell -- more according to the Internet.
Fortunately, even though high-grade ECC RAM was pricey when T5500s were new, high-quality name-brand sticks of compatible memory are available at very reasonable prices now. In fact, because of supply and demand, used server/workstation RAM is less expensive than used PC RAM.
Recommended name-brands: Hynix, Samsung, Micron
If you need lots of RAM, 24GB kits are available as 3x8GB kits. These higher-capacity 8GB modules cost more per GB. The premium is 70%, still not bad at under $4/GB. Filling all the slots (three 3x8 kits) in a dual-CPU system gives you the full rated capacity of 72GB of RAM.
Combining different RAM (module capacities, voltages, single-rank with dual-rank, etc.) in the same system can be tricky. The system will perform differently based on the combination of modules you use.
FWIW, I chose to remove the RAM that came with mine and install three low-voltage 8GB modules, opting for slightly better speed and better expandability. A less-expensive option (typically by about $30) would have been to install six 4GB modules. Installing three low-voltage modules rather than six standard-voltage modules also saves power and runs cooler.
The T5500 accepts "PC3L-10600R" (1.35V Registered ECC) or "PC3-10600R" (1.5V Registered ECC) modules. For best results, install double-ranked (2R) modules in sets of three, where the voltage of all installed modules matches -- either all 1.5V or all 1.35V modules.
Consult the Precision T5500 Service Manual for installing RAM sticks in the optimum manner.
Hard Drives
The second-best way to expand the capabilities of a virtual-machine server is usually by adding storage space.
The Precision T5500 has two internal 3.5-inch hard drive bays. It's possible to add more in the 5.24" drive bays. If necessary, external (eSATA) drives can be added as well.
New hard drives specifically designed for network attached storage (NAS) are recommended.
Network Adapter
| Brand | Part # |
|---|---|
| Intel | EXPI9402PT |
| HP | NC360T |
| Dell | X3959 |
| IBM | 39Y6128 |
Dual-Port Intel Pro/1000 PT NICs
Recent-model Intel Pro/1000 PT server network adapters are a safe bet. They're commonly re-branded and sold by server manufacturers under various model names.
Single- dual- and quad-port models are widely available. Intel's part numbers are EXPI9400PT, EXPI9402PT, and EXPI9404PT respectively. At current prices, expect to pay roughly $8 per port.
Graphics
Most T5500 machines have nVidia Quadro video cards installed. Graphics capability can easily be improved by swapping in a more-modern (5 years newer) and more energy-efficient GPU card.
For example, if your T5500 has a Quadro FX 580 card, common benchmark results predict that a well-chosen new card will provide 37% faster 2D performance while consuming half as much power at less than 19W TDP rather than 40W. DirectX support goes from 10 to 11 and OpenGL support jumps from v3.3 to v4.5. Bonus features are: 3D performance is 2.5X as fast and it has twice as much RAM.
SATA Adapter
SATA III 6.0Gb/s capability can be added in a PCI-E slot. For spinning drives this won't make much, if any, difference. For fast SSDs you're wise to go with SATA III.
Optimum HomeLab Build
This is an annotated list of the build I ultimately chose. I started with a $357 base configuration (plus hard drives), with a couple of upgrades soon to follow.
Price Summary
These prices current as of late October, 2015.
Initial Build
| Component | Price |
|---|---|
| Dell T5500 | $175 |
| CPU Upgrade | 28 |
| Upgrade to 24GB | 78 |
| 100GB SLC SSD | 70 |
| Quad-port NIC | 30 |
| Total | $381 |
Storage
For my build I already have three 1TB desktop hard drives I can use. Here are minimum prices for drives. New, (not refurbished) name-brand drives are recommended.
Lowest-price NAS drives (pairs):
| Capacity | 2X | 3X |
|---|---|---|
| 1TB | $74 / 1TB | $111 / 2TB |
| 2TB | 110 / 2TB | 165 / 4TB |
| 3TB | 130 / 3TB | 195 / 6TB |
| 4TB | 300 / 4TB | 450 / 8TB |
| 5TB | 340 / 5TB | 510 / 10TB |
Next Upgrades
Near-future upgrade components (prices may vary, obviously):
Phase One - Add CPU and another 24GB RAM:
| 2nd-CPU Riser Assy. | $99 |
| Upgrade to 48GB | 78 |
| Total | $177 |
Phase Two - Always-on Power, Upgrade GPU:
| 600VA UPS | 50 |
| Upgrade GPU | 50 |
| Upgrade to 72GB | 78 |
| Total | $178 |
A storage upgrade will likely intervene in there somewhere. We'll see how it goes with storage.
If more CPU power is necessary, a CPU Upgrade from dual-L56230 to dual-L5640 (currently $138 for a pair) goes from 8 to 12 cores and each core is a bit more powerful. Overall benchmark performance increases by 44%.
Links
These are just links (meaning they're not "affiliate" links).
Features that stand out:
- 4 Cores (8 vCPUs @ only 5W TDP per vCPU)
- 2nd (matching) CPU in hand
- 18GB ECC RAM
- Support for three 3.5" drives instead of two
- 4-display Support
T5500 Workstation
T5500 with three 3.5" drive bays, Quad-display graphics, 6GB 0f RAM:
The machine appears to have come from the factory with three spinning hard drives installed. The third hard drive is in the lower 5.25" drive bay, below the optical drive and above the 3.5" bay. The SSD, in 3.5" a mounting kit should be able to fit into the 3.5" bay with only an additional SATA data cable because a SATA power cable is provided.
CPU Upgrade
Two Low-power 56XX Quad-core CPUs with hyperthreading and AES New Instructions:
I have a thermal paste kit. If you don't, this premium kit is a very good choice.
Thermal Paste kit for CPU installation:
This quad-core CPU consumes only 40W TDP, meaning the existing fan is over-spec'd (a good thing) and I'll use high-grade Arctic Silver thermal compund when I swap out the chip (an even better thing).
Only one of the CPUs in the matching set will be installed right away. The second one will be added, along with more RAM, when budge permits. Those parts currently would cost $126 to add right now.
RAM Upgrade
24GB of RAM (six 1.5V modules):
Specify Hynix or Samsung brand-name modules. Micron would be fine, too.
24GB of RAM (three 1.35V modules):
- Samsung 8GB 2Rx4 PC3L 10600R 1333MHz 240 Pin Memory M393B1K70DH0-YH9 | eBay
- US $25.95 x3 = 77.85
Enterprise SSD
Boot, ZIL, and L2ARC drive:
This enterprise/industrial-grade SLC drive is a beast. List price was nearly $1,000 when it was new in 2010. It's designed for high IOPs and harsh environments, neither of which will happen in my HomeLab server. It sips 2.2W under load in a server and .125W at idle.
It will be used as the boot drive, for swap space, for a filesystem journal (ZIL, or "ZFS Intent Log") and for a high-performance filesystem cache (L2ARC, or "Level 2 Adjustable Replacement Cache"). Planned allocation for those are: 60GB, 12GB, 8GB and 15GB respectively. 5GB will be unused. The ZIL improves performance and data-integrity. The L2ARC improves disk I/O performance.
Network Ports
Quad-port NIC:
This is a sixth-generation Intel Pro/1000 PT card. As far as I can determine, it's compatible with virtually every operating system and hypervisor.
Phase One Upgrade
A second-CPU riser and 12GB more RAM will be purchased in the (hopefully near) future.
Second-CPU Riser assembly:
The assembly is three parts: Riser, Heatsink, and Fan. Along with another 12GB of RAM, it adds.
- 2nd CPU (d'oh)
- 12GB more RAM for a total of 72GB
Phase Two Upgrade
A should-have item and a nice-to-have will round out the build, probably with another 12GB RAM for 36GB total.
Clean, always-on Power (contributes to data integrity):
Much more recent, much faster, much more energy-efficient GPU(http://www.videocardbenchmark.net/compare.php?cmp[]=1640&cmp[]=2896"NVS 420 vs. GT 720 - Passmark"|benchmark):
Other Notable Links
Some T5500 links:
- Product Page
- Setup Guide (Service Manual) and User's Guide
- Drivers and downloads
- Brochure (pdf)
- Specsheet (pdf)
Some other parts links:
New-old-stock RAID-Specific HDD (need at least two):
These drives are probably a step above the least-expensive drives in quality.
SATA 3.0 adapter with 2 internal and 2 external (eSATA) ports:
This would theoretically speed up the SSD. Cards like these seem to use only two lanes of the PCI-Express bus. Best I can tell "PCIe 2.0 x2" limits throughput to below SATA III 6.0Gbps, but still it's faster than SATA III at 3.0 Gbps. Marvell 88SE9235 chipset appears to be a good choice. YMMV.
eSata III (6Gbps) Dock with cooling fan:
- Amazon.com: eSATA III or USB 3.0 External Docking Station with FAN
- $58.97 & FREE Shipping
This peripheral can be used to add a fourth drive, either as a Hot Spare or possibly to expand storage.
USB 3.0 Adapter - full-height, 7 ports including two internal:
- Amazon.com: Inateck Superspeed 7 Ports PCI-E to USB 3.0 Expansion Card ]: Computers & Accessories
- $25.99 & FREE Shipping
This card uses a Fresco Logic FL1100EX chipset, which I believe is a good one. As with hte SATA card, YMMV because it's hard to determine which chipsets are compatible with various operating systems and hypervisors.
Q & A
What's the very least expensive build you'd recommend?
An adequate single-CPU T5500 with a decent CPU (55XX will work, 56XX preferred), 12GB of RAM, two cheap 1TB drives from eBay ($74) and a 600VA UPS ($50) comes at around $300.
Do I really need a UPS?
Computers are sensitive electronic devices that tend not to tolerate blackouts, surges, and brownouts. If data integrity is important for your server a UPS is cheap insurance. At minimum you should install a good surge protector, which costs roughly half as much as the UPS does.
Where do cheap new 2TB and 3TB NAS drives come from?
You can find new 2TB NAS drives on eBay and 3TB NAS drives on eBay for low prices. Be sure to select a vendor with high ratings and a lot of reviews.
How much would faster CPUs cost?
You can substitute a pair of 6-core L5640 CPUs for $92 more. According to benchmarks you'll get a 40+% overall performance boost (and a 50% energy-consumption boost).
How much would more RAM cost?
6 GB more (36GB total) would cost $27 more, or 18GB more (48GB total) for $95 more.
How much would more storage space cost?
Western digital makes WD<n>0EFRX "Red" drives all the way up to 6GB.
Will this beast work for something like transcoding video files?
Certainly! That's what workstations are made to do. If you'll be doing a lot of that, consider installing a whopper video card or two. The two video-card slots each support up to 150W of power draw.