ESXi Home server

Need some advice (had posted my questions in storage but it's probably better suited here.)

Need to create an all in one home server inside a norco 4224 case.

Needs to be able to connect those 24 drives (HP SAS expander?)
And run them as one large drive pool (ZFS?) with shares (NSF/SMB?)
Need to be able to serve multiple XBMC players connected simultaneously (NFS?)
Also needs to transcode media on the fly to androids/ps3/xbox (Plex or Mezzmo)
Would like it to Automatically backup up to 4 Windows 7 machines (WHS2011)

Looks like I need either 2 or 3 VM's:
WHS2011
one VM for nas duties
on VM for transcoding

I guess the nas and transcoding could be combined into one linux VM but what distro?

And the main thing I have no real clue about.
What hardware?
Other than the HP SAS expander I've got no Idea where to start.

ESXi compatibility and Vt-d support in an affordable package that can handle the streaming and transcoding tasks?

Ok, a few things here to think about...

If you do go the ZFS route you need to realize what you are signing up for. ZFS is an excellent file system for this sort of thing but it absolutely LOVES RAM and you really need to feed it huge amounts if you want to get the desired performance out of it.

Also on the software front you are a bit limited--the only good way to go ZFS is going to be Nexenta (which has a free edition but with 24-drives you are going to be way over the limit) so that leaves you with OpenIndiana or FreeNAS. I'd probably recommend FreeNAS 8 as the most approachable way to go here, but as I said with ZFS you usually want 1GB of RAM for every 1TB of storage so factor that in.

It might be nice to try and find one OS to rule them all for this but with ZFS your options are very limited. So probably the most straightforward way to go is ESXi for the base, FreeNAS 8.3 for the storage server, and then a CPU heavy transcoding server for Plex. How much hardware is totally dependent on how many streams you are going to run at the same time and at what bitrates.

You may very well be able to get away with a uniprocessor workstation class board in the Xeon E3 family which should do fine as long as you aren't going nuts with a ton of HD video streams at once. So I'd probably get a decently clocked Xeon E3 quad core and a motherboard with sufficient RAM sockets to hold the amount of memory you are going to need. If you need to go bigger, you can always look at a dual Xeon E5 setup but that's going to start hitting your budget hard.

Also it would be good if you can find a motherboard with at least two gigabit ports (make sure they are Intel controllers) and then maybe even consider adding another 2-4 gigabit ports if you have a router that can handle it. Teaming is a perfectly acceptable way to remove a bandwidth bottleneck, especially if you are streaming 1080p without much compression (MakeMKV -> Disk?) you are going to need that extra bandwidth.

Ok, here is one way to go that won't bust the bank but lets you hedge your bets.

Start with this motherboard: SUPERMICRO MBD-X9DRL-3F-O

It's not super expensive at $369 and it includes the following that will be helpful for you:

1. 3x Gigabit Ethernet ports (Intel Controller)
2. Support for up to 256GB of ECC Registered RAM
3. 2x CPU sockets for E5 processors

Note that you don't need to use both processor sockets at the same time, so you can start with one processor and add a second if you find you aren't getting the throughput on the transcoding you need.

For the processors I'd probably go with this: Xeon E5-2620

Not the fastest clockspeed wise but for what you are doing it should be enough. You get six cores and twelve threads @ 2.0GHz base (2.5GHz boost) at a very reasonable 95W and you can always add a second chip for a reasonable $420. Don't let the clockspeed worry you too much, what you really need for transcoding and I/O throughput is cores not raw per-core speed.

For memory I'd get a couple of these: Crucial 24GB (3 x 8GB)

That will give you 48GB of memory with the option to bump it to 64GB should you choose to by adding two more sticks. Plenty for what you need, just make sure you get ECC AND Registered memory for this board.

Altogether you could get the starter kit of 1x E5 processor, motherboard and 48GB of RAM for around $1000. Not bad at all and certainly a drop in the bucket compared to what your storage array is going to cost.

Thanks much thekernel. I did not know I could start with one CPU and add another as needed. That really simplifies the decision.
As for the software, I do not need ZFS. But I've heard its good. I could just as well use UnRaid in a VM is that's possible. I would be using WHS2011 as the one OS except it doesnt support more than 4 cores and only one processor, which may not be enough once all the media players are up and running. The board you linked seems to have some negative reviews. Especially concerning the BIOS, would i be better served going to a board like this in a slightly higher price range (with 16 drive support built in no less) SUPERMICRO MBD-X9DA7-O

Edit: I believe i should go with the SUPERMICRO MBD-X9DA7-O, with 24gb of ram to start. Running ESXi. 1 HDD dedicated to holding VM OS's. 3VM's Freenas (With all other drives passed to it. WHS that will backup to a network share (in the freenas VM)
And a ubuntu server (something I've wanted to setup for a while now) which will run the plex server and any other linux services that I may need.

I'm assuming you were planning to go with the free version of ESXi 5?

You have to be careful that the free version has a hard limit of 32gigs of RAM which means the hypervisor will not booth if more than 32 gigs is installed/enabled.

You can purchase the Essentials kit at $495 which I believe removes the RAM limitation, but restricts the socket count to 2 (which still works for you).

Well that's incredibly annoying. I'm not sure I really want to spend another 500 dollars just for ZFS functionality. In fact I think I'd be much happier with a UnRaid VM rather than ZFS through FreeNAS.

ESXi is not the only virtualization solution you can use. True, it's certainly the easiest when it comes to remote management but since you seem partial to Windows Server you might want to consider Hyper-V server which is free and does not have any such limitations. It's a free bare-metal hypervisor and you don't even need to run Windows Server to use it (although a caveat: I've never actually spent much time with it so I can't recommend it personally). If you want more information on it, this would be a good place to start.

You can also consider options based on Xen if you like, although your mileage may vary there. All the big shops use Xen for their IaaS and PaaS offerings, but they have a lot more development resources to throw at it than you do. However, it's still a solid hypervisor and you can take a look at the Xen Cloud Platform offering to see if it meets your needs.

A few other things to consider as well:

1. Since you seem to want to run Windows Server you could ditch the idea of Windows Home Server (it's a dead end anyway) and go with Windows Server 2012 and the ReFS file system. ReFS may not be quite as enterprise oriented as ZFS (well...it's enterprise oriented but it doesn't have some of the more exotic features of ZFS) but it's a huge improvement over just about anything else out there.
2. You could probably consolidate to one OS if you were willing to go the Windows Server 2012 route since it will run Plex server just fine.
3. I really don't recommend unRAID. Either use FreeNAS 8.3 if you want to go on the free side or use Nexenta (free if you are within the storage limits, otherwise commercial). If you don't want to go the ZFS route though, I really think ReFS is a better option.

One other advantage of Windows Server 2012...because it supports Hyper-V at the OS level as well as the baremetal version you could always just load that as the OS on the machine and see how you like it...then you could spin up VMs at your leisure without having to blow away the underlying OS. I would probably try to consolidate this into one OS instance if I were you, and Windows Server 2012 does have a lot of advantages.

I wouldn't personally go that route because my primary machine is a Macbook Pro Retina so I get pretty screwed on the remote virtualization management tools thing. But if you are already a Windows user it's definitely a great option.

Windows server 2012 was not something i was familiar with, it seems like it suits all of my needs in one OS.
Edit: On closer inspections it seems prohibitively expensive software for a home server.
The problem with ZFS it seems is you can't add drives to a storage pool that's already been created. Making it useless for me as I will be adding drives one by one and never more than 1 at a time.
I think it might be better to put my nas into it's own box with it's own hardware and just run OI/Napp-IT on that. Then setup a server for transcoding on one of my always on W-7 pc's until i setup a proper ESXi box to run WHS(or server2012) and Ubuntu (to play around with and run plex) at a later date. I do have 22u of free rack space to put systems in, so i suppose making an All in One isnt really necessary.

Upon further inspection of ZFS i've decided i will not use it. The fact taht you cannot use different sized drives in an array (without limiting all drives to the lowest) clinches it for me. ReFS in server 2012 is far too expensive (i dont really want to spend much on software.)

I believe my best options are drivebender running on WHS 2011, or unRaid. For the NAS anyway.
ESXi for the Dual CPU server which will run my transcoding service, backup service, sabnzbd/sickbeard/headphones/couchpotato, and perhaps pfsense and a UTM software.