Finally Finished - 'All-in-One' ESXi Server

F

fatguy

Weaksauce
Joined
Apr 25, 2005
Messages
93
Here's my finished "all-in-one" ESXi / NAS server. Thanks a ton to the folks in these forums for helping me through some issues with the build!

Intel Xeon 3440 Quad Core Processor (w/HT)
SuperMicro X8SIL-F Motherboard (dual nics, IPMI)
16GB ECC DDR3 Memory
LSI SAS 9211-8i HBA (PCI-e x8)
8GB Flash Drive (ESXi Boot)
2 x 1TB Drives (ESXi Datastore)
Fractal Design R3 Case

I have a NexentaStor Appliance running as a VM, with 'pass-thru' access to the HBA, with the following ZFS volumes:

2 x 1TB Drives (zfs mirror, 885GB actual)
6 x 1.5TB Drives (zfs raidz, 6.63TB actual)[/QUOTE]

I've added some pictures of the finished build because I really enjoyed looking at other builds when deciding what to do...

server-3.jpg

server-4.jpg

server-2.jpg

server-1.jpg
 
How are the noise levels? I'm building a NAS on much more modest hardware using an existing Antec case, and its going to be hard finding a suitable place in my apt for it. Is that sound deadening foam on the door?

What else do you use the box for besides file storage?
 
MrCrispy said:
How are the noise levels? I'm building a NAS on much more modest hardware using an existing Antec case, and its going to be hard finding a suitable place in my apt for it. Is that sound deadening foam on the door?

What else do you use the box for besides file storage?
Click to expand...

The server sits under my desk and I don't notice it at all. It does produce some noise, but all the fans are 1000 rpm / 15 dba, and the case is lined with sound dampening material. I picked this case primarily for quiet, dust-free cooling on the drives.

In addition to the NAS vm, I'm running a network services vm (dns, dhcp, ddclient, openldap), vpn (openvpn), devhub (git, confluence, jira, crucible/fisheye, hudson), public web/app server vm (photo and home movie sharing), and a postgres database vm.
 
Last edited:
Great looking NAS!

Just wondering, are the 1TB drives on the LSI 9211-8i or the Intel 3420?
 
pjkenned said:
Great looking NAS!

Just wondering, are the 1TB drives on the LSI 9211-8i or the Intel 3420?
Click to expand...

4 x 1TB drives in total - 2 of them are at the very bottom of the case and are on the 9211-8i (pass-thru to the Nexenta VM) running as a ZFS mirror volume. The other 2 (at the top of the case) are directly attached to the X8SIL-F, running as my primary and backup datastores.

I was thinking about adding in a raid card for ESXi boot + datastore redundancy, but I stuck with this setup and use a script to snapshot all my VMs every night and copy the snapshots to the secondary datastore. When I need something better - like for my database VM - I use a NFS-mounted datastore, which is served by the NAS.
 
Database said:
Very nice wiring sir! You should come do mine! :)
Click to expand...

thanks - this was actually the first build where I really spent extra time on cable management. Big complaint though - the Corsair power supplies are wired up assuming a top-mounted PSU position, so all of the sata cables are inverted, which led to some tricky wiring on for the drives.

got a pm about hard drive temps - the drives hang around 33C pretty much all the time.
 
what kind of performance are you getting on transfers to and from the nas. I'm thinking about doing the same thing but with a lower budget and curious about the performance I can expect.

How did you set up the ESXi storage pool? My idea was to run ESXi off a flash drive, NAS vm off a seperate disk and the rest of the VMs stored in a pool inside the NAS.
 
yankees9920 said:
what kind of performance are you getting on transfers to and from the nas. I'm thinking about doing the same thing but with a lower budget and curious about the performance I can expect.

How did you set up the ESXi storage pool? My idea was to run ESXi off a flash drive, NAS vm off a seperate disk and the rest of the VMs stored in a pool inside the NAS.
Click to expand...

Transfer rates are good, both with CIFS/SMB and NFS. From a windows machine on my network, I'm moving files back and forth at 80-100MB/s (gigabit network, GS108 switch). If you have a more scientific test you'd like me to run, let me know.

I'm booting from a flash drive, but have the datastores on dedicated disks. (Originally my ESXi server was diskless, I suppose at this point I could lose the flash drive.) I thought seriously about throwing in a small-ish SSD instead just to hold my NAS VM, and have all other VMs live on a datastore provided by the NAS.

The problem I ran into was that the NAS vm is not really my core VM - it relies on the network services VM for DNS and LDAP. I didn't like this circular dependency, especially when I was tweaking Nexenta or playing with other NAS appliances (FreeNAS, EON, OpenFiler). Every time I had to reboot my NAS all my other VMs went away.
 
You must log in or register to reply here.
Back
Top