ESXi whitebox server

I usually have access to an ESX box at work where I can run multiple VMs and virtual routers for labbing and testing. I’ve also wanted one at home. It’s nice to be able to quickly spin up VMs when needed without always running them through my laptop.

While virtual routers don’t need lots of resources, I did want a beefy machine as there are a few servers I’d like to get running that need lots of CPU power.

Requirements

  • 32GB RAM capable with ECC
  • Fast CPU with at least 4 physical cores
  • Quiet
  • Small
  • OOB (ilo/IPMI/Etc)
  • Low power

Specifically these are the things I don’t need:

  • Optical drive
  • Hard drive
  • GPU

The point of the box is to sit in the corner with only power and network connected. If anything went wrong, I don’t want to have to connect a monitor to it. I’m also not running any tasks requiring video output on the server itself. All VMs will be logged in via SSH.

I already have a Synology DS411 which will provide an iSCSI connection to the ESX server. Hence no need for internal hard drives.

My initial build was going to be built around an Intel i7 4790. However the i7 doesn’t support ECC ram and it also has a built-in HD4000 GPU which I don’t need.

Parts list

I ended up going for the Intel Xeon E3-1230v3. 4 cores, 8 threads, all the virtulisation support I need. It has no built-in GPU. It supports up to 32GB ECC RAM. Intel have released a newer 1231v3, but I couldn’t find a good price for it in the UK and all it gives is an extra 100MHz which I’m not fussed about.

RAM is quite pricey at the moment. While I wanted 32GB, I’ll start with 16GB for now and add another 16GB when prices drop.

For the motherboard, I went with the SuperMicro X10SLL-F. It supports both the CPU and RAM and also has built-in IPMI. The board has two onboard Intel NICs, i217LM and i210AT. The board also has an on-board VGA card. I’m not going to use that, but it will be handy if I can’t log into both the server and IPMI. It also has an a-type USB slot on board which is quite handy as you’ll see later.

For the PSU, I wanted both silent and efficient. I don’t need a huge amount of wattage either as I have no GPU. I ended up with the Seasonic G-360. 80-Plus certified and very quiet.

Part of the problem with an ESXi whitebox is ensuring that VMWare recognises all your components. I did extensive research in order to ensure this was the case. There are a couple of things I hit upon, but they were easily fixed.

Final part list:

  • Intel Xeon E3-1230 V3
  • SuperMicro X10SLL-F
  • 2 X 8GB Crucial DDR3 PC3-12800 ECC Unbuffered
  • Seasonic G-360 PSU
  • Aerocool Dead Silence Gaming Cube Case
  • Old 1Gb USB flash drive

Building and installing

The SuperMicro board has a dedicated IPMI port and so I can do the entire install remotely. I’ll mount the ISO over the network, and all do all the config this way. This is the screen you see when first logging onto IPMI web interface:
Capture
I decided to install Vmware itself on a USB stick. What’s nice about this motherboard is that it has a USB port on the motherboard itself, meaning no external USB key required. This keeps it a bit neater.

The SuperMicro has two external NICs, one Intel 217 and an Intel 210. I’ve installed VMware 5.5 update 2 and the I210 Intel card is supported out of the box. No need to hack any drivers into the ISO. I’m more than happy with one NIC for now so I’ve no need to try and get the 217 working.

Once VMWare was installed, I created a 300GB iSCSI LUN from my Synology and attached VMWare to that. The install and set up really was painless.

Vmware shows my system as:
system

Virtual devices

I wanted to start a basic lab, so I have the following VMs running in my lab:
VMs

With all my VMs running, I see hardly any CPU and quite a bit of RAM usage as I expected:
resource
For now the RAM amount is fine. As I ramp up the lab and prices drop, I’ll add another 16GB to the system.

Power usage

As I wanted this to be low power, I’ve done full wattage readings on power usage.

  • Server off, IPMI on – 3.7 Watts
  • Server on, no VMs running – 23 Watts
  • Server on, all lab VMs running – 34 Watts

Not at all bad. In another post I’ll show the Synology power draw as well as the power draw if all VMs are using full CPU. I’ll also go over how I automate my VMs starting and shutting down.