![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Ok, I'm looking for some input on a new NAS setup for my home office, I'll go over my current setup, space requirements, and my thoughts for a solution. Any feedback is more than welcome...
First, I want to keep the cost reasonable, less than $1000 for sure before storage... Also keeping in mind that this is a 3 to 4 year solution with Seagate's recent announcement of 60TB drives being on the market by 2016, I don't want to go crazy and build too much storage on 3Gb and 6Gb technology with 10TB to 60TB on the not to distance horizon.
So, today I have 20TB's of raw RAID5 storage, on my network most of this is using Synology 4 disk appliances, plus a very small file server. The synology boxes run 4 2TB WD 5400rpm drives, for a total of 24TB's of storage, these are great little boxes, and have handled drive failures very well over the last couple of years, with no data loss. Having so many separate mount points spread across the network is a bit of a pita.
My file server running mirrored 1TB 7200 WD hard drives, running on a T100 Dell, with 2 Intel 1gb nics. This system is my biggest headache on the planet today, constantly slowing my network down, and the biggest reason I'm looking to upgrade to a bigger better NAS.
A little bit more about how the storage is used... first, I have a small XCP (Xen Cloud Platform) cloud, with a Dell XS23-SB server with 64 megs of memory, this runs my primary production VM cloud, currently I have about 45 VMs running on this system using the file server via NFS, we're in the process of adding a second XS23-SB in the not to distance future, and adding another 50-100 VMs (hence the reason to upgrade storage). I have 2 Dell T110's running a small test VM cloud also running XCP, and a single Dell T110 running vmware esxi 5, running some odd and end VMs, and for building VMs to be migrated to remote colo DC's.
My network is fairly straight forward, and should be robust enough to handle this additional load. I have a Dell 6024 L3 switch acting as the brains of my network, this allows me to seperate the home network, and the work network. The rack where all of the storage equipment will sit is connected to the L3 switch via 2 1gbit fiber connections to a Dell 2724 .
The XS23's have 4 'blades' in them each with 2 1gbit NICs one of which is dedicated to VLAN access for the VM's, the other acts as a management/storage network. These ports are currently set to 9K jumbo frames.
Sorry to go into all of this detail, but I feel all this background information will give folks a better feel for what my data storage needs are. Also, I'm very comfortable using both Windows and Linux based operating systems, including messing with the command line, but I want a stable solution that largely does not require a lot of care and feeding.
So, that explains the network, and my current storage, let me explain the different types of data I have, I'm going to break this down into different 'datasets'.
dataset #1 - VM data stores, XCP can handle either NFS or iSCSI, VMware I believe is limited to NFS, this data needs to be much faster, it will have 8 systems with single 1gbit nic's banging away at it 24/7 plus a test cloud with 2 1gbit nics, and the vmware esxi box. Right now I'm getting about 400mbits out of the current system at peak, this is obviously way to slow. Today I move about 200-250 GB of data in a 24 hour period, I'd like to build for triple that in a 24 hour period with the new system. So say 750+ GB every 24 hours. I currently use thin provisioning, and since I have a number of windows and linux VMs, a lot of data is duplicate data... I figure I need 4TB of usable space if I stick with thin provisioning (before data de-duplication), or 8TB if I need to do thick provisioning for speed(again before de-duplication).
dataset #2 - This is VM backup images, 2 times a week we run a script, that exports the VMs as .xva files, these are large single file backups of the VMs, ranging in size from 4GB to 50 or 60 GB each, 2 per week, and we keep them around for 4 weeks. Today this represents about 3TB of data, obviously this data screams for de-duplication and would drastically save on needed disk space. Obviously as we grow the number of VM's this space will also grow, so figure 6-9 TB's of pre-de-duplicated backups. Speed isn't much of an issue here, as long as we can spool out the entire backups over a 24-36 hour period, without impacting the cloud's performance, we only read from this data once a month to make an offsite backup, and if we have to restore a VM for some reason. I know somebody is going to suggest using snapshotting, that just isn't an option because of administrative controls issues... I know how much slower running a program that exports data from a NAS and then copies that data back to the same (or another) NAS, but we need the .xva files instead of snapshots.
dataset #3 - PC Backups, once a day we move some PC backups to central storage, this is a lot of very small files, maybe 400-600GB of storage almost never accessed, and most done over slow VPN links. But probably doesn't lend itself to deduplication.
dataset #4 - Software and odds and ends storage, not sure data de-duplicaton would help much, but storage of linux/windows iso's, software, etc... Today *maybe* 250-300GB total.
Everything below here is for personal use
dataset #5 - This dataset is made up of avi, mp4, and mkv files, the range in size from 350 megs to 5 or 6 GB, no need for data de-duplication. write speed is largely not an issue, read speed needs to be in the 6-10 MB range, we have 4 rooms with media players so just enough to play 4 1080p's direct via CIFS shares. Today this is about 2.5TB of disk usage,
dataset #6 - This is a critical dataset, it's mostly pictures and home movies, today this is well under 100GB of storage, it will slowly grow over time, little need for data de-duplication, this dataset gets rsync'd to Amazon S3 once a week right now to automatically move the data offsite.
So, I was hoping to build 1 monster NAS/SAN for the entire storage, sometimes I worry about putting all my eggs in 1 basket. But, eher si what I'm thinking:
OpenIndiana with ZFS.
I have a couple of these SAS enclosures from a previous project: http://www.unixsurplus.com/product/...d-bay-sas-sata-ssd-san-array-35-nas-jbod-good
They're only 3 GB/s SAS/SATA, but I think this should be fast enough.
I don't have a front end for the enclosures thought, I'd like to stick to rack mounted, if at all possible. I was thinking maybe one of these two boxes:
http://www.unixsurplus.com/product/rackable-asus-kfsn4-dual-dual-core-2214-32gb-2u-server
or
http://www.unixsurplus.com/product/rackable-nm46x-dual-dual-core-2214-32gb-1x-250gb-2u-server
I have experience with sgi/rackable systems and they're pretty good for older tech system. Not too loud for my garage, and with 32 gb of ram should help with ZFS memory caching and de-duplication. I'm not sure how compatible they would be with OpenIndiana, and would be limited to just 2 onboard 1gbit nic's since the single PCI expansion slot would have to hold an HBA card. Not sure if this will present a major bottle neck or not.
If these won't work with OI, then I'd end up having to build a rackable case system to link to the enclosure above. I'm open to suggestions on cases, cpu, and motherboard specs, something that can hold a lot of inexpensive memory would be great.
For an HBA card I was thinking this:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118101
But, I'm open to suggestions, I just need something that I can connect to the enclosures I have access to.
Anything else you can think of?
As for the pools:
pool1 - VM live storage, starting with a mirrored pair of 7200rpm 3TB drives, with a 60gb MLC SSD for l2arc. As I need more space I'll add pairs of 7200rpm 3TB drives. de-duplication turned on.
pool2 - VM backups plus pc backups, and software storage, RaidZ2 with 6 2tb 7200 RPM drives to start, adding 6 2tb drives as vdev's as there is a need for more space. de-duplication turned on.
pool3 - media storage and personal data (pictures/home movies), 10 5400rpm 2tb drives in RaidZ2, I'll still move the personal stuff to S3 every week for offsite safety. No de-duplication, just nice slow data storage.
pool4 - 4 7200rpm 1tb drives, stripped for temp space.
Ok, let me have it, what am I doing completely wrong? Where am I missing the boat?
Thanks for the feedback.
First, I want to keep the cost reasonable, less than $1000 for sure before storage... Also keeping in mind that this is a 3 to 4 year solution with Seagate's recent announcement of 60TB drives being on the market by 2016, I don't want to go crazy and build too much storage on 3Gb and 6Gb technology with 10TB to 60TB on the not to distance horizon.
So, today I have 20TB's of raw RAID5 storage, on my network most of this is using Synology 4 disk appliances, plus a very small file server. The synology boxes run 4 2TB WD 5400rpm drives, for a total of 24TB's of storage, these are great little boxes, and have handled drive failures very well over the last couple of years, with no data loss. Having so many separate mount points spread across the network is a bit of a pita.
My file server running mirrored 1TB 7200 WD hard drives, running on a T100 Dell, with 2 Intel 1gb nics. This system is my biggest headache on the planet today, constantly slowing my network down, and the biggest reason I'm looking to upgrade to a bigger better NAS.
A little bit more about how the storage is used... first, I have a small XCP (Xen Cloud Platform) cloud, with a Dell XS23-SB server with 64 megs of memory, this runs my primary production VM cloud, currently I have about 45 VMs running on this system using the file server via NFS, we're in the process of adding a second XS23-SB in the not to distance future, and adding another 50-100 VMs (hence the reason to upgrade storage). I have 2 Dell T110's running a small test VM cloud also running XCP, and a single Dell T110 running vmware esxi 5, running some odd and end VMs, and for building VMs to be migrated to remote colo DC's.
My network is fairly straight forward, and should be robust enough to handle this additional load. I have a Dell 6024 L3 switch acting as the brains of my network, this allows me to seperate the home network, and the work network. The rack where all of the storage equipment will sit is connected to the L3 switch via 2 1gbit fiber connections to a Dell 2724 .
The XS23's have 4 'blades' in them each with 2 1gbit NICs one of which is dedicated to VLAN access for the VM's, the other acts as a management/storage network. These ports are currently set to 9K jumbo frames.
Sorry to go into all of this detail, but I feel all this background information will give folks a better feel for what my data storage needs are. Also, I'm very comfortable using both Windows and Linux based operating systems, including messing with the command line, but I want a stable solution that largely does not require a lot of care and feeding.
So, that explains the network, and my current storage, let me explain the different types of data I have, I'm going to break this down into different 'datasets'.
dataset #1 - VM data stores, XCP can handle either NFS or iSCSI, VMware I believe is limited to NFS, this data needs to be much faster, it will have 8 systems with single 1gbit nic's banging away at it 24/7 plus a test cloud with 2 1gbit nics, and the vmware esxi box. Right now I'm getting about 400mbits out of the current system at peak, this is obviously way to slow. Today I move about 200-250 GB of data in a 24 hour period, I'd like to build for triple that in a 24 hour period with the new system. So say 750+ GB every 24 hours. I currently use thin provisioning, and since I have a number of windows and linux VMs, a lot of data is duplicate data... I figure I need 4TB of usable space if I stick with thin provisioning (before data de-duplication), or 8TB if I need to do thick provisioning for speed(again before de-duplication).
dataset #2 - This is VM backup images, 2 times a week we run a script, that exports the VMs as .xva files, these are large single file backups of the VMs, ranging in size from 4GB to 50 or 60 GB each, 2 per week, and we keep them around for 4 weeks. Today this represents about 3TB of data, obviously this data screams for de-duplication and would drastically save on needed disk space. Obviously as we grow the number of VM's this space will also grow, so figure 6-9 TB's of pre-de-duplicated backups. Speed isn't much of an issue here, as long as we can spool out the entire backups over a 24-36 hour period, without impacting the cloud's performance, we only read from this data once a month to make an offsite backup, and if we have to restore a VM for some reason. I know somebody is going to suggest using snapshotting, that just isn't an option because of administrative controls issues... I know how much slower running a program that exports data from a NAS and then copies that data back to the same (or another) NAS, but we need the .xva files instead of snapshots.
dataset #3 - PC Backups, once a day we move some PC backups to central storage, this is a lot of very small files, maybe 400-600GB of storage almost never accessed, and most done over slow VPN links. But probably doesn't lend itself to deduplication.
dataset #4 - Software and odds and ends storage, not sure data de-duplicaton would help much, but storage of linux/windows iso's, software, etc... Today *maybe* 250-300GB total.
Everything below here is for personal use
dataset #5 - This dataset is made up of avi, mp4, and mkv files, the range in size from 350 megs to 5 or 6 GB, no need for data de-duplication. write speed is largely not an issue, read speed needs to be in the 6-10 MB range, we have 4 rooms with media players so just enough to play 4 1080p's direct via CIFS shares. Today this is about 2.5TB of disk usage,
dataset #6 - This is a critical dataset, it's mostly pictures and home movies, today this is well under 100GB of storage, it will slowly grow over time, little need for data de-duplication, this dataset gets rsync'd to Amazon S3 once a week right now to automatically move the data offsite.
So, I was hoping to build 1 monster NAS/SAN for the entire storage, sometimes I worry about putting all my eggs in 1 basket. But, eher si what I'm thinking:
OpenIndiana with ZFS.
I have a couple of these SAS enclosures from a previous project: http://www.unixsurplus.com/product/...d-bay-sas-sata-ssd-san-array-35-nas-jbod-good
They're only 3 GB/s SAS/SATA, but I think this should be fast enough.
I don't have a front end for the enclosures thought, I'd like to stick to rack mounted, if at all possible. I was thinking maybe one of these two boxes:
http://www.unixsurplus.com/product/rackable-asus-kfsn4-dual-dual-core-2214-32gb-2u-server
or
http://www.unixsurplus.com/product/rackable-nm46x-dual-dual-core-2214-32gb-1x-250gb-2u-server
I have experience with sgi/rackable systems and they're pretty good for older tech system. Not too loud for my garage, and with 32 gb of ram should help with ZFS memory caching and de-duplication. I'm not sure how compatible they would be with OpenIndiana, and would be limited to just 2 onboard 1gbit nic's since the single PCI expansion slot would have to hold an HBA card. Not sure if this will present a major bottle neck or not.
If these won't work with OI, then I'd end up having to build a rackable case system to link to the enclosure above. I'm open to suggestions on cases, cpu, and motherboard specs, something that can hold a lot of inexpensive memory would be great.
For an HBA card I was thinking this:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816118101
But, I'm open to suggestions, I just need something that I can connect to the enclosures I have access to.
Anything else you can think of?
As for the pools:
pool1 - VM live storage, starting with a mirrored pair of 7200rpm 3TB drives, with a 60gb MLC SSD for l2arc. As I need more space I'll add pairs of 7200rpm 3TB drives. de-duplication turned on.
pool2 - VM backups plus pc backups, and software storage, RaidZ2 with 6 2tb 7200 RPM drives to start, adding 6 2tb drives as vdev's as there is a need for more space. de-duplication turned on.
pool3 - media storage and personal data (pictures/home movies), 10 5400rpm 2tb drives in RaidZ2, I'll still move the personal stuff to S3 every week for offsite safety. No de-duplication, just nice slow data storage.
pool4 - 4 7200rpm 1tb drives, stripped for temp space.
Ok, let me have it, what am I doing completely wrong? Where am I missing the boat?
Thanks for the feedback.