New ESXI Server and Storage Setup, Thoughts

[cmacnichol]

Hello all,

I just got a new server and am trying to figure out the best way to set it up. Trying to do it properly this time rather then having data spread across all kinds of places.

The Goal here is to have a DIY SAN connected to the server over Fiber where I can dole out space with VMDK's. I am just not sure that is the best method to do this.

So here is what Hardware I have or what I will be getting soon.

HP DL360 G5 (Soon to be retired, ESXI 5.5)
HP DL380 G7 (E5620, 64gb RAM, New Server)
HP N40l Microserver (8GB ECC RAM, Severely Underpowered, Running WHS 2011)

Various 1.5 and 1 TB Hard Drives (Currently Failing) installed in the N40L as a drive pool with Drive Bender

4 x 3TB Western Digital RED Drives (New Purchase)
2 x 4gb Fiber HBA's (Not Yet Used)
4 x 450 GB SAS Drives (Located in DL360)
4 x 43 GB SAS Drives (Unused)

My Current setup has data split between the DL360 and the Microserver. I have a Linux VM Running Plex, Sickbeard and Sanbzd as the Microserver could not handle the transcoding anymore. All New downloaded data is saved to the internal SAS storage then later moved manually to the Microserver.

There are various other VM's setup as well for other things, but none are as data heavy as the Plex VM.

Now here is what I was thinking. Using the 4 x 3 TB drives in a ZFS Pool running Freenas on the Microserver.

This DIY SAN, would be linked to the DL380 via a 4 GB Fiber Connection.

The DL380 with ESXI would be running something like Freenas or Openmediavault (Suggestions?) to host Plex and Sickrage. Might even split these into separate stand alone Linux VM's, not sure the best route. I could then have different VMDK's created for each type of data off of the ZFS pool which is shared over the Fiber link.

Somewhere in there I would need to have Windows File sharing as well. I also had the thought to use some kind of caching software to write to the 10K SAS Drives then to the 5400 drives on the fiber target.

Thoughts, Comments?

Am I heading in the right direction or is there a simpler way to do this?

 

[w00t69]

With shared storage setup with freenas you can do whatever the fuck you want sir. You'll be able to move things off and onto it as you please. The fiber connection is overkill for 4 WD Red's but that's ok. If you can get more RAM into the microserver and enable the cached mode in freenas that will drastically speed up short transfers and overall performance. You are definitely on the right track. Everything past you setting up the shared storage is personal preference.

 

[ToddW2]

Why FC for 4Gb/s when you can do 4Gb/s copper for $50.

If I was doing something for higher-performance SAN why not 40Gb/s? I wouldn't run a SAN on FC just to 'do it' if performance was nearly identical if not the same to a 4port ethernet NIC.

Here's some info I've gotten recently on my own new ZFS NAS, iSCSI & Media Serving project.

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

https://blogs.oracle.com/roch/entry/when_to_and_not_to

http://www.napp-it.org/doc/downloads/napp-in-one.pdf
"iscsi is done via comstar in the nappit webui, there are a heap of guides out there to step you through it if it looks confusing. "

http://mediatomb.cc/

I'm testing ZFS w/Napp-IT in an ESXI VM. My VMs will run off SSD and the ZFS RADZ2 w/5x5TB RED+.

I went back and forth about putting the NAS in a VM (Dual E5-2620 w/192gb RAM) or on baremetal which was only an E3-1220v2 (No HT) 32gb RAM. For now I'm trying ESXI VM, but still may go to the NAS but potentially in the future with an E5 Haswell or newer if my storage needs change/grow. I don't need a lot of space myself, but I have various performance needs depending on what I'm doing.

Todd

 

[cmacnichol]

That is a lot of good information, thank you both.

As for why FC, mostly because I already have the parts sitting around. Just ordered the 4 3TB RED Drives from the newegg special on Sunday.

My problem here is the microserver it seems. The N40L is limited to 8GB of RAM so I cannot even expand that. Will that make such a big difference? Might have a Core 2 DUO with 16GB of RAM I can use, power consumption on that and size is a lot bigger though.

I do have a spare 64GB SSD sitting around, could I use that for CACHE in Freenas? Is it worth it?

My biggest concern now is design of the storage pool. The Microserver, if that is what end up using to host the drives, has a pretty pitiful CPU in it. Hence why I am not really going to use it for much anymore.

The plan was to let the stand alone freenas box handle the ZFS pool and backup while the freenas VM would offer all of the services, taking advantage of the higher CPU and Memory of the server. Hmm, could I use the server memory to cache stuff going to\from the VMDK on the Stand Alone Freenas?

 

[cmacnichol]

I am still interested in thoughts on my above post, but I just had something else occur to me.

I might be making this more complicated then it needs to be. Currently, my Server has about 1.2 TB of Storage in 10K SAS Drives. Thinking about it, I cannot see myself using that much for the VM's themselves.

My biggest storage hog is, of course, Media files. I am wondering if for simplicity it would be easier to pass the Fiber HBA through to a Virtual Freenas Instance, which would use the servers more powerful specs, and treat the Disks in the Stand Alone Freenas as basically Direct Attached Storage. This would also let me use the Servers memory for Deduplication. If I ever needed to expand this storage, I can just offer more Raidz arrays to the storage pool over the fiber link.

So, can a Freenas install act as both an Initiator and a Target for a Fiber link? Is this possible? Or is my original idea above a better idea. It kind of gets away from my DIY SAN Idea, but as I said, I am not even sure if that is the best way to do it.