100tb.....it must be the Pedo-byte server
I'm sure the FBI would be interested in that lol
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Build-Log: 100TB Home Media Server
- Thread starter treadstone
- Start date
I'm sure the FBI would be interested in that lol
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
Thanks
Friends of mine were kind enough to do that for me already. I still can't lift anything too heavy (as in this server) with my right arm. Although I did and can carry the boxes with the HDD...
The server is now mounted in a 42U Dell server rack and I am working or trying to figure out how I want to setup the storage pool.
Dang!
Are they on to me already?
- Joined
- Jun 27, 2001
- Messages
- 15,814
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
I'd love to post them, but first I need to get them recovered from an old 80GB Maxtor drive that died. It was in my wife's computer until one day it would no longer spin up. It contained pictures of my kids and some of the 'fun' projects I build over the years.
Worst part was that I had just bought an external backup drive to backup all of the pictures and documents the day before it died
Back when the unit was in use, I had reached just about 200 Blu-ray disc. So there was a LOT of room left in it.
Anyway, the unit itself is mainly disassembled now and used as spare parts for other projects (like a 30 x 72 inch routing table to route Plexiglas/Lexan type neon signs).
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
Here are some configurations I have tried on my server so far and I listed some of the pro and cons for each of the configurations:
(Please note that I only listed a few pro and cons below, I am sure there are others...)
RAID 5:
Configuration:
RAID 6:
Configuration:
RAID 50:
Configuration:
RAID 60:
Configuration:
JBOD:
Configuration:
Spanned Volume (Windows Server 2008 R2):
Configuration:
Now since I am not after access speed and I am not too concerned about backing up the data from this server since this server will be used mainly for serving Blu-ray movies that I am backing up from my originals, I am starting to wonder if I really need any RAID at all!
My main focus is shifting more towards power consumption at this point. Out of all the above RAID configurations, the RAID 5 setup (3 groups of 16 drives) has the lowest power consumption, followed by the RAID 6 setup (2 groups of 24). The RAID 50 and RAID 60 setups use the same amount of power since all drives are powered up and being accessed at the same time.
My original idea/intend was to setup the system in a RAID 50 configuration.
However, I think that the ideal setup for my requirements would be to have one large volume across all drives in the storage pool and the OS would start with the first drive and once full, move on to the next drive and so on. If a drive should fail, I will loose the movies contained on that single drive at which point I would replace the drive and re-rip the movies from my originals. All my movies are in a database that is kept on a different system, so it's fairly easy to figure out what was on the damaged drive. This way the power consumption would be kept to a minimum as all the drives that are not being accessed can stay in standby/sleep mode, or in other words, for the majority of the time, only a single drive in the drive pool would be active. Another benefit of this setup would be that the drives have less wear and tear so the life expectancy should be higher than any of the above RAID configuration setups.
In regards to an OS, currently I am leaning towards Windows Server 2008 R2.
Now I had looked into spanned volumes in Windows, however the issue here is that first of all Microsoft only allows a maximum of 32 drives in a single volume and secondly and more importantly, if one drive fails the entire volume is destroyed, so it's kind of like RAID 5 or 6 but without the redundancy!
From the looks of it, this would be more along the lines what I am looking for especially since it appears that only a single drive is active during a single file read or write request, however I do NOT like the part where I would loose ALL of the data on that volume if a drive bites the dust!
I've been incredibly busy in the last few month and don't have as much time as I would like to spend on this server. I REALLY need to get this done, so I hope that some of you have some good suggestions for me.
Is there such a file system/storage solution that allows for multiple drives (in my case 48 x 2TB drives) to be configured as a single volume but access only a single drive at a time and in case of a damaged or failed drive does NOT bring down the entire volume?
(Please note that I only listed a few pro and cons below, I am sure there are others...)
RAID 5:
Configuration:
- 3 groups of 16 drives in RAID 5 configured as 3 independent volumes
- Total drive capacity per RAID 5 group: 32TB
- Total drive capacity for 3 RAID 5 groups: 96TB
- Available volume capacity per RAID 5 group: 30TB (16 x 2TB drives = 32TB - 1 x 2TB parity drive = 30TB)
- Maximum available capacity per volume: 30TB
- Total available storage capacity: 90TB
- Access speed
- Data redundancy (1 parity drive per RAID 5 set)
- HIGH power consumption!
- 16 drives active to access a single file!
- If two drives within a single RAID 5 group fail, that particular volume (up to 30TB) is no longer usable!
- 1 drive is 'lost' to parity per RAID 5 set (2TB per set, 6TB total)
- Moderate amount of heat generated by the active drives
RAID 6:
Configuration:
- 2 groups of 24 drives in RAID 6 configured as 2 independent volumes
- Total drive capacity per RAID 6 group: 48TB
- Total drive capacity for 2 RAID 6 groups: 96TB
- Available volume capacity per RAID 6 group: 44TB (24 x 2TB drives = 48TB - 2 x 2TB parity drives = 44TB)
- Maximum available capacity per volume: 44TB
- Total available storage capacity: 88TB
- Access speed
- Higher data redundancy (2 parity drives per RAID 6 set)
- HIGHER power consumption than the 3 groups of RAID 5 configuration!
- 24 drives active to access a single file!
- If three drives within a single RAID 6 group fail, that particular volume (up to 44TB) is no longer usable!
- 2 drives are 'lost' to parity per RAID 6 set (4TB per set, 8TB total)
- Moderate amount of heat generated by the active drives
RAID 50:
Configuration:
- 3 groups of 16 drives in RAID 5 contained in a single RAID 0 group configured as one large volume
- Total drive capacity per RAID 5 group: 32TB
- Total drive capacity for RAID 50: 96TB
- Available volume capacity per RAID 5 group: 30TB (16 x 2TB drives = 32TB - 1 x 2TB parity drive = 30TB)
- Maximum available capacity per volume: 90TB
- Total available storage capacity: 90TB
- Single large volume
- Access speed
- More 'available' space than RAID 60 configuration
- Data redundancy (1 parity drive per RAID 5 set)
- Extremely HIGH power consumption!
- ALL 48 drives active to access a single file!
- Higher rate of failure probability as all drives are in use for any kind of access
- If two drives within a single RAID 5 group fail, the entire volume (up to 90TB) is no longer usable!
- Lower protection than RAID 60 configuration
- 1 drive is 'lost' to parity per RAID 5 set (2TB per set, 6TB total)
- Highest amount of heat generated by the active drives
RAID 60:
Configuration:
- 2 groups of 24 drives in RAID 6 contained in a single RAID 0 group configured as one large volume
- Total drive capacity per RAID 6 group: 48TB
- Total drive capacity for RAID 60: 96TB
- Available volume capacity per RAID 6 group: 44TB (24 x 2TB drives = 48TB - 2 x 2TB parity drives = 44TB)
- Maximum available capacity per volume: 88TB
- Total available storage capacity: 88TB
- Single large volume
- Access speed
- Higher data redundancy (2 parity drives per RAID 6 set)
- Extremely HIGH power consumption!
- ALL 48 drives active to access a single file!
- Higher rate of failure probability as all drives are in use for any kind of access
- If three drives within a single RAID 6 group fail, the entire volume (up to 88TB) is no longer usable!
- Less 'available' space than RAID 50 configuration
- 2 drives are 'lost' to parity per RAID 6 set (4TB per set, 8TB total)
- Highest amount of heat generated by the active drives
JBOD:
Configuration:
- 48 drives configured as independent volumes
- Maximum available capacity per volume: 2TB
- Total available storage capacity: 96TB
- LOWEST power consumption!
- One drive active to access a single file
- Single drive failure does NOT impact any other drive/volume!
- Longer drive life expectancy due to lower wear and tear
- All drives available for storage!
- Lowest amount of heat generated by the active drives
- No data redundancy
- 48 individual volumes!
Spanned Volume (Windows Server 2008 R2):
Configuration:
- 2 groups of 24 drives each configured as two spanned volumes
- Maximum available capacity per volume: 48TB
- Total available storage capacity: 96TB
- LOW power consumption!
- One drive active to access a single file
- Longer drive life expectancy due to lower wear and tear
- All drives available for storage!
- Lowest amount of heat generated by the active drives
- No data redundancy
- Single drive failure brings down the entire volume (up to 48TB)!
- 32 drive limit per volume!
Now since I am not after access speed and I am not too concerned about backing up the data from this server since this server will be used mainly for serving Blu-ray movies that I am backing up from my originals, I am starting to wonder if I really need any RAID at all!
My main focus is shifting more towards power consumption at this point. Out of all the above RAID configurations, the RAID 5 setup (3 groups of 16 drives) has the lowest power consumption, followed by the RAID 6 setup (2 groups of 24). The RAID 50 and RAID 60 setups use the same amount of power since all drives are powered up and being accessed at the same time.
My original idea/intend was to setup the system in a RAID 50 configuration.
However, I think that the ideal setup for my requirements would be to have one large volume across all drives in the storage pool and the OS would start with the first drive and once full, move on to the next drive and so on. If a drive should fail, I will loose the movies contained on that single drive at which point I would replace the drive and re-rip the movies from my originals. All my movies are in a database that is kept on a different system, so it's fairly easy to figure out what was on the damaged drive. This way the power consumption would be kept to a minimum as all the drives that are not being accessed can stay in standby/sleep mode, or in other words, for the majority of the time, only a single drive in the drive pool would be active. Another benefit of this setup would be that the drives have less wear and tear so the life expectancy should be higher than any of the above RAID configuration setups.
In regards to an OS, currently I am leaning towards Windows Server 2008 R2.
Now I had looked into spanned volumes in Windows, however the issue here is that first of all Microsoft only allows a maximum of 32 drives in a single volume and secondly and more importantly, if one drive fails the entire volume is destroyed, so it's kind of like RAID 5 or 6 but without the redundancy!
From the looks of it, this would be more along the lines what I am looking for especially since it appears that only a single drive is active during a single file read or write request, however I do NOT like the part where I would loose ALL of the data on that volume if a drive bites the dust!
I've been incredibly busy in the last few month and don't have as much time as I would like to spend on this server. I REALLY need to get this done, so I hope that some of you have some good suggestions for me.
Is there such a file system/storage solution that allows for multiple drives (in my case 48 x 2TB drives) to be configured as a single volume but access only a single drive at a time and in case of a damaged or failed drive does NOT bring down the entire volume?
nitrobass24
[H]ard|DCer of the Month - December 2009
- Joined
- Apr 7, 2006
- Messages
- 10,466
Sounds like you want WHS, but it has a 32 drive limit
blargsauce
n00b
- Joined
- Mar 19, 2010
- Messages
- 8
unraid would be good, but that also has a drive limit of I believe 20.
The only system that I know does this is Linux LVM, and I only know this from watching how it behaves; I noticed that when accessing a specific file it would only access the drive(s) that actually had the data.
So, if you were using Linux, I believe you could set up a 48-drive storage pool using LVM, and minimise your disk accesses. If a drive fails (or better yet starts to fail), you simply remove it from the pool (which moves its data to another part of the pool), pull the drive, then add its replacement. It would be a good idea to keep a copy of SpinRite 6.0 lying around too...
That's the system I'm familiar with; I've never touched Windows Home Server or UnRAID. NextentaStor with ZFS may also be worth a look - ZFS also has storage pool capabilities, but I'm not familiar with them. Others here probably are.
EDIT:
I know you stated that you don't a have a lot of time to play around with this, but take your time nonetheless. 96TB of storage will not benefit from hasty decisions.
zer0gravity
[H]ard|Gawd
- Joined
- Dec 8, 2008
- Messages
- 1,131
Just found this build....just have to say....
legendary
legendary
@OP
Update re: my experience with LVM. The behaviour I expressed was on an LVM volume that had been created and filled, then expanded further. That may explain why it pulled data of some disks and not others. All I can say is "test, test and test again".
...are not [H]ard.
Update re: my experience with LVM. The behaviour I expressed was on an LVM volume that had been created and filled, then expanded further. That may explain why it pulled data of some disks and not others. All I can say is "test, test and test again".
...are not [H]ard.
I was thinking about this also. One dead motherboard = one dead server until it gets replaced. Can't really do much redundancy with a component like that.
Unless you build two 100TB servers...and have each one as part of a RAID 1 iSCSI target...
Seriously though, dead motherboard = dead computer in almost any scenario.
epimetheus
Gawd
- Joined
- Jun 20, 2004
- Messages
- 815
Umm... I think you missed the point of this whole forum. This build may not rank that high in the corporate world, but for a home media server (which is what it is), it rocks.
Edit: To keep this post from being OT, WHS sounds like the appropriate operating system for the storage pool capabilities. Maybe possible to run 2 WHS's in VM's to work around the 32 drive limit? It would be a little more intensive on the housekeeping side of things as far as file management goes, but it could work.
Last edited:
I agree with the above, I too was going to say just split the machine into two virtual WHS and maintain them separate. You don't even need to have duplication turned on, however unless your really quick at ripping hundreds of discs, I would save myself time and shed some storage to redundancy.
Either that, or invest in a bluray mega changer that will auto-rip for you because a drive is going to fail at some point, and you will lose tons of data
Unless you build two 100TB servers...and have each one as part of a RAID 1 iSCSI target...
Seriously though, dead motherboard = dead computer in almost any scenario.
Yeah, but like you said, "unless you built two."
Most major SANs/NASs are not one single server system.
Well, there's no getting around it to be honest. If a RAIDed disk fails, replace it: data stays intact. If a duplicated disk fails replace it: data stays intact. If a memory DIMM fails, replace it: data (usually) stays intact.
If a motherboard fails...cry, and keep crying until you get a replacement. Or spend more money and build a cluster.
cleric_retribution
Limp Gawd
- Joined
- Oct 20, 2006
- Messages
- 274
You should really look into using an OS with ZFS support. It was made for this type of application.
You can create "raidz" (raid5) or "raidz2" (raid6) groups of disks - say 8 disks per group -- then you can then create one large storage "pool" out of all of your disk groups.
So, for example, you could have 6 groups of 8 disks in raidz, along with a couple "hot" spares and you then have the ability to sustain up to 6 concurrent disk failures before you lose data from your "pool".
If a drive fails from any group, a hot-spare will automatically be inserted into the disk group and it will be "scrubbed" (rebuilt) to ensure there's no data corruption. Also, If more than 1 disk fails out of a particular group, you do not necessarily lose all the data from that group, depending on how you've configured your ZFS replication options for your storage pool (which can be done on a per directory or file basis, for important data)
If you want an easy way to manage your ZFS, as well as use the server for sharing over the network to just about anything, I'd recommend using FreeNAS
If you want the latest ZFS module support, you'd have to install FreeBSD 8.x or OpenSolaris (I'd recommend FreeBSD), but you'll be going command-line for management, and manual configuration of file-sharing software like samba and netatalk, etc.
I'm using FreeNAS to manage a ~10TB storage pool comprised of 2x raidz2 groups of 6 1TB drives, and share all my media, files, and backups across my network with it. All managed via a nice web interface, and very customizable.
I know this will not meet your power usage requirements, but for this much storage and disks, IMO power usage should be expected and redundancy should be required. I have my disks set to a 30-minute spin-down time, so for most of the day when I am away / at work, the disks are offline (my computers all sleep too). When something accesses the file server, disks all spin up and run. This has kept my power usage to a minimum.
You can create "raidz" (raid5) or "raidz2" (raid6) groups of disks - say 8 disks per group -- then you can then create one large storage "pool" out of all of your disk groups.
So, for example, you could have 6 groups of 8 disks in raidz, along with a couple "hot" spares and you then have the ability to sustain up to 6 concurrent disk failures before you lose data from your "pool".
If a drive fails from any group, a hot-spare will automatically be inserted into the disk group and it will be "scrubbed" (rebuilt) to ensure there's no data corruption. Also, If more than 1 disk fails out of a particular group, you do not necessarily lose all the data from that group, depending on how you've configured your ZFS replication options for your storage pool (which can be done on a per directory or file basis, for important data)
If you want an easy way to manage your ZFS, as well as use the server for sharing over the network to just about anything, I'd recommend using FreeNAS
If you want the latest ZFS module support, you'd have to install FreeBSD 8.x or OpenSolaris (I'd recommend FreeBSD), but you'll be going command-line for management, and manual configuration of file-sharing software like samba and netatalk, etc.
I'm using FreeNAS to manage a ~10TB storage pool comprised of 2x raidz2 groups of 6 1TB drives, and share all my media, files, and backups across my network with it. All managed via a nice web interface, and very customizable.
I know this will not meet your power usage requirements, but for this much storage and disks, IMO power usage should be expected and redundancy should be required. I have my disks set to a 30-minute spin-down time, so for most of the day when I am away / at work, the disks are offline (my computers all sleep too). When something accesses the file server, disks all spin up and run. This has kept my power usage to a minimum.
Last edited:
I like ZFS, but it isn't OS independent... and it doesn't run on Linux (which I hate to say it, but it is a lot more widely used than OpenSolaris and FreeBSD).
cleric_retribution
Limp Gawd
- Joined
- Oct 20, 2006
- Messages
- 274
Hardware RAID is fine, and I never said the OP should ditch the controllers. Just a suggestion on filesystem to achieve redundancy and performance for this level of storage, because I really think it'd be stupid to run 100TB of storage with no redundancy.
ZFS is designed to run on high-performance SAS or SATA backplanes and controllers.
If he wanted, the OP can run it over his existing RAID cards by passing them through JBOD (wasteful, I know)
raidz / raidz2 performance will usually beat anything but the most high-end hardware RAID5/6 configurations due to the variable width striping, the centralization of the filesystem and logical volume management, and the elimination of non-full-stripe-width writes (by-product of filesystem/volume integration, which is not possible with hardware RAID array and a regular OS).
This is why, IMO for a "home" application such as this, ZFS makes much more sense from a price/performance or price/reliability standpoint on a pair of decent SAS/SATA controllers for a few hundred dollars, rather than $1k+ for hardware RAID cards. (not that price is really an issue in this case
)There's alot of good info on RAIDZ that outlines exactly how it works @ http://blogs.sun.com/bonwick/entry/raid_z
RAID doesn't address the consistency of the data on the drives themselves, something that ZFS does.
As for speed:
http://blog.nominet.org.uk/tech/2007/10/15/quick-zfs-performance-numbers/
Using a single treaded dd instance, 531MB/s is achieved on a Thumper which much fewer drives than this system.
also
http://www.markround.com/archives/35-ZFS-and-caching-for-performance.html
But with some more serious hardware. Regardless, it shows that the limit in speed isn't ZFS itself.
Both of those benchmarks you reference have problems with RAM cache. Read the text and the comments for more information.
As a rule of thumb, find out the size of the RAM cache, and make sure you write 10 times that amount of data in the test. Also, be sure to use random data unless you are sure that there is no compression somewhere in the pipeline.
By the way, one thing I have been wondering about is what happens if you run ZFS with an SSD dedicated to the log, and the SSD fails. I saw a report of this, and it apparently took down an entire file server and resulted in data loss. Anyone else have any experience with this?
As a rule of thumb, find out the size of the RAM cache, and make sure you write 10 times that amount of data in the test. Also, be sure to use random data unless you are sure that there is no compression somewhere in the pipeline.
By the way, one thing I have been wondering about is what happens if you run ZFS with an SSD dedicated to the log, and the SSD fails. I saw a report of this, and it apparently took down an entire file server and resulted in data loss. Anyone else have any experience with this?
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
I was considering WHS, but only the V2 release but since it's not out yet and I really need to get this finished, I guess I have to look for a different OS...
To be honest, I have never really looked at or considered UNRAID. But from what I have heard others talk about here, it doesn't seem to be the solution for me either.
Well, what I am after is to have one large volume of 90TB+ for my movie storage, hence my original idea of RAID50 (90TB) or RAID60 (88TB). Now due to the cons I posted before with either one of those setups, I have since decided against a RAID50 or RAID60 setup.
Unfortunately, I am not too familiar with Linux, although my younger brother is quite the guru on a few different Linux flavors, so I guess I need to ask him over for a beer or something and get his input on this...
ZFS is definitely something I need to have a closer look at!
Thanks!
Yes, I am aware that EVERYBODY has a 1PB server in their basement
I have yet to see any 'off the shelf solution' that handles 1PB in a SINGLE enclosure!
I am aware that this build is far from 'cost' effective, that wasn't my goal here. as stated before, I wanted all of the drives in a single enclosure and that's what this build gave me.
Besides, everybody has different idea's of how they would build such a server...
I had a quick look at LVM and it is definitely something I will have to look at in more detail.
I also agree with your 'Current off the shelf solutions... are not [H]ard.' statement
That is true for most equipment. Believe me, I know. I design communication equipment for a living and quite a few of my customers ask for redundancy on a new design/project. And no matter how far you push redundancy, you will always have a single point of failure somewhere.
In any case, I don't really need redundancy anyway since this server is used as a storage medium to hold all of my Blu-ray movies. So even if there is a drive that failed and I can no longer access the movies on the drive, I just replace the defective drive and backup all the movies from the original Blu-ray discs again.
Unless you build two 100TB servers...and have each one as part of a RAID 1 iSCSI target...
Seriously though, dead motherboard = dead computer in almost any scenario.
No need for a second 100TB server (unless I run out of space
). I don't need redundancy, see my previous comment above.I was considering WHS, but only V2 not the original WHS. But running two WHS instances in a VM environment, makes things way more complicated than necessary and will still not give me a single large volume...
Again, I don't need redundancy. I have close to 500 Blu-rays right now and will be buying/adding a whole bunch more once I get them transferred onto the server.
If you go back a few posts, I had posted about a little project I build a few years back. A 2000 disc Blu-ray changer (I should actually call it a disc handler). Anyway, the problem with disc changers is that they can usually only play a single disc at a time and are extremely slow in changing discs.
I wanted everything in one box, hence this build. I had a 'distributed' setup before that I am retiring shortly (well, I have already removed more than half of my servers from my network).
I purchased two additional WD20EADS drives (when I purchased all of the storage drives) as spares in case any of the drives in the storage pool fail. I actually ended up with two of the same motherboard too (got the second one from newegg as an open box item). So I actually do have a spare motherboard in case the other one dies...
I was considering FreeNAS as well as OpenFiler as an alternative OS and I may go an have a second look at them again. For now I am toying around with Windows Server 2008 R2.
As I stated so many times before, I don't really need redundancy since I have the original discs in case I need to get the lost data back. This server is intended to give me instant access to all my movies rather then having to go through hundreds of cases to find the movie I am interested in watching...
I think I set my spin-down timer to 10 minutes.
As to power usage, I just configured the drive pool as 3 groups of 16 drives in RAID 5 again and during the initialization phase, the system was drawing way over 900 watts! Mind you, they were ALL active at the same time, but still that's way too much...
Unfortunately, that's true!
Hardware RAID is fine, and I never said the OP should ditch the controllers. Just a suggestion on filesystem to achieve redundancy and performance for this level of storage, because I really think it'd be stupid to run 100TB of storage with no redundancy.
ZFS is designed to run on high-performance SAS or SATA backplanes and controllers.
If he wanted, the OP can run it over his existing RAID cards by passing them through JBOD (wasteful, I know)
raidz / raidz2 performance will usually beat anything but the most high-end hardware RAID5/6 configurations due to the variable width striping, the centralization of the filesystem and logical volume management, and the elimination of non-full-stripe-width writes (by-product of filesystem/volume integration, which is not possible with hardware RAID array and a regular OS).
This is why, IMO for a "home" application such as this, ZFS makes much more sense from a price/performance or price/reliability standpoint on a pair of decent SAS/SATA controllers for a few hundred dollars, rather than $1k+ for hardware RAID cards. (not that price is really an issue in this case
There's alot of good info on RAIDZ that outlines exactly how it works @ http://blogs.sun.com/bonwick/entry/raid_z
I did run some tests where the RAID controller was in JBOD mode (basically just as a SATA HBA). I will be looking at ZFS again, maybe even this weekend if I find the time for it. One of the major issues for me is that I am not too familiar with Linux (or FreeBSD in this case). So I will need some help to figure this out...
I couldn't care less for speed. This server was build for storage capacity not for throughput speed. As long as the server is capable of streaming at least two simultaneous Blu-ray movies, that's all I need in terms of throughput/bandwidth. If I was going for throughput speed, I would have picked a different motherboard, more memory and different Ethernet controllers.
Interesting, do you happen to have a link to that report?
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
Here is one of my configuration setups with Windows Server 2008 R2.
I configured the drive pool as 3 groups of 16 drives each in RAID5. The initialization took 27 hours and 40 minutes on two of the three groups and the group that was split between the two SAS expanders took 27 hours and 46 minutes.
This setup gives me a total of 94TB of available storage space, however the storage pool is still dived among 3 x 30TB volumes. There is also a system volume of 60GB for the OS and other programs, 1.94TB (the remaining space of the 2TB drive) that I will be using for temporary recorded TV storage and another 2TB volume for my music collection.
I tried to extend one of the 30TB volumes but windows complained with an error message. Something along the lines of that the clusters are larger than what it can support. It was late and so I didn't bother with it too much so I just configured each as a simple volume and grabbed these screen shots.
I need to play with and setup Windows Server 2008 R2 to go into sleep mode when not in use and see how this particular setup would work for me (or not whatever the case may be)...
I configured the drive pool as 3 groups of 16 drives each in RAID5. The initialization took 27 hours and 40 minutes on two of the three groups and the group that was split between the two SAS expanders took 27 hours and 46 minutes.
This setup gives me a total of 94TB of available storage space, however the storage pool is still dived among 3 x 30TB volumes. There is also a system volume of 60GB for the OS and other programs, 1.94TB (the remaining space of the 2TB drive) that I will be using for temporary recorded TV storage and another 2TB volume for my music collection.
I tried to extend one of the 30TB volumes but windows complained with an error message. Something along the lines of that the clusters are larger than what it can support. It was late and so I didn't bother with it too much so I just configured each as a simple volume and grabbed these screen shots.
I need to play with and setup Windows Server 2008 R2 to go into sleep mode when not in use and see how this particular setup would work for me (or not whatever the case may be)...
I own 3 Sony 400 disc DVD changers and a 400 disc CD changer. I was about to buy another 400 disc DVD changer since my DVD collection consists of over 1450 DVDs. But those things are DAMN slow and as you mentioned only allow you to play one disc at a time. The server route gives you more flexibility. I even designed a 2000 disc changer myself. It was capable of playing two discs at a time. The unit required a computer as the playback device since it was only a mechanical transport system and had two blu-ray disc drives at the bottom of it. It was quite neat and the kids always enjoyed watching the little arm go up and down to grab a disc and drop it into the drive trays. They played with it so much that it actually broke down
Somehow it jammed up and I think one of the motors burned out. Never got around fixing it after that since I had just started to put everything onto HDD anyway. That was back at the beginning of 2008 when I build my HTPC. Shortly after that, there were rumors that Sony might release a blu-ray disc changer, but they didn't 'really' come out with a useful design until just recently. They had a neat design with the 200 disc firewire connected DVD changers and I had considered buying a few of those and modding them by replacing the DVD drive with a blu-ray drive, but by that time, Sony had stopped making them. A shame because those looked neat, although for the size of them, they should have had a higher capacity.
Anyway, I'm hoping to get the server online in the next few days. I did manage to finally getting it moved into the basement a few days back
ahh.. sony's wonderful changers..
I have blown the first sony 200 dvd changer to market (got people thought I was nuts when I bought that monster), the 300 flipper (it would, at the push of a button, flip the disk for you to read both sides), the 300+1 and now I blew the 400 about 5 months ago.
along the way one of them, and I am not 100% sure which one, I think the 400, decided to make horried perfect circle scratchs/buffed marks about 7/8ths of the way from center out on about 60% of the disks before the drive reader deicided to stop working. So I have been spending way to much time recoving DvDs the last month
so, now, I am doing what you are. I have had it with Sony's changers
the irony of all these changers is I blew the reader, either the motor to move the drive head, or the laser stopped working. the carasol worked perfectly on all of them till they landed in the junk heap
Last edited:
houkouonchi
RIP
- Joined
- Sep 14, 2008
- Messages
- 1,622
My core 2 duo 2.66 Ghz system running opensolaris + zfs can do about 450 megabytes/sec writes and 800 MB/sec reads (well over gigabit which is about 115 megabytes/sec). That being said about 3.2% of disk space is lost to ZFS which will be over 1TB in this case.
I would really highly suggest against using raid5 at all. In my oppinion 15 drives is too many for raid5 and I would feel much safter with 2x raid6 even better 3x raid6 than 3x raid5 if it was my machine. Raid 6 has a lot of advantages that aren't just about having any two drives fail at one time.
Also with ZFS there is raidz3 (tripple parity) which is the only thing to do it AFAIK. I remember there was a bit more of a write hit (300 megabytes/sec or something) but again I am maxing out my CPU when I am writing as the the crc and parity calculations is using multiple cores.
I would really highly suggest against using raid5 at all. In my oppinion 15 drives is too many for raid5 and I would feel much safter with 2x raid6 even better 3x raid6 than 3x raid5 if it was my machine. Raid 6 has a lot of advantages that aren't just about having any two drives fail at one time.
Also with ZFS there is raidz3 (tripple parity) which is the only thing to do it AFAIK. I remember there was a bit more of a write hit (300 megabytes/sec or something) but again I am maxing out my CPU when I am writing as the the crc and parity calculations is using multiple cores.
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
@ houkouonchi:
I am going to reconfigure the server again. The 3 x RAID5 setup was just for testing one configuration option. I don't really want to run the server in any kind of RAID mode anymore anyway...
Does anyone know if the processor on the ARC-1680 series still monitors the SMART data of any connected drive and will email the necessary information if the controller has been configured in JBOD mode?
I am trying to figure out if it makes sense to keep the ARC-1680i or switch to a HBA only type controller?!?
I am going to reconfigure the server again. The 3 x RAID5 setup was just for testing one configuration option. I don't really want to run the server in any kind of RAID mode anymore anyway...
Does anyone know if the processor on the ARC-1680 series still monitors the SMART data of any connected drive and will email the necessary information if the controller has been configured in JBOD mode?
I am trying to figure out if it makes sense to keep the ARC-1680i or switch to a HBA only type controller?!?
nitrobass24
[H]ard|DCer of the Month - December 2009
- Joined
- Apr 7, 2006
- Messages
- 10,466
Treadstone,
I have been thinking about this for a while and if power efficiency is your primary concern, then heres what i would do if I were in your shoes.
Run your RAID card, or get a Supermicro SASLP and run the drives in JBOD mode.
Install Server 2008 R2 w/ Hyper-V role.
Run two instances of WHS, Hyper-v is the only virtualization platform that has a really good implementation for Direct Disk Access. Yes WHSv1 limits you to 32 drives, but if you only have 500 BRs right now 64tb will go a long way. So you would end up with 1 64tb jbod pool and could run a second WHS vm for the remaining disk.
Plus with WHSv2 being released at the end of this year, you *should* be able to switch and use a single drive pool. Currently in Beta there is not a drive limit.
I have been thinking about this for a while and if power efficiency is your primary concern, then heres what i would do if I were in your shoes.
Run your RAID card, or get a Supermicro SASLP and run the drives in JBOD mode.
Install Server 2008 R2 w/ Hyper-V role.
Run two instances of WHS, Hyper-v is the only virtualization platform that has a really good implementation for Direct Disk Access. Yes WHSv1 limits you to 32 drives, but if you only have 500 BRs right now 64tb will go a long way. So you would end up with 1 64tb jbod pool and could run a second WHS vm for the remaining disk.
Plus with WHSv2 being released at the end of this year, you *should* be able to switch and use a single drive pool. Currently in Beta there is not a drive limit.
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
Nitro, that's what I was thinking too, however I am still not convinced that I really need to run one OS on top of another. Granted, WHS does have some neat features, however I'd rather stick with just one OS (e.g. Server 2008 R2).
Do you happen to know if I switch the Areca controller into JBOD mode if the controller will still monitor the HDDs SMART information and email me in case there is a problem?
If it doesn't do that, then there is no real benefit for me to keep the ARC-1680i...
I was looking at the AOC-USAS2-L8i as a possible alternative to the ARC-1680i.
Do you happen to know if I switch the Areca controller into JBOD mode if the controller will still monitor the HDDs SMART information and email me in case there is a problem?
If it doesn't do that, then there is no real benefit for me to keep the ARC-1680i...
I was looking at the AOC-USAS2-L8i as a possible alternative to the ARC-1680i.
nitrobass24
[H]ard|DCer of the Month - December 2009
- Joined
- Apr 7, 2006
- Messages
- 10,466
I do not have an answer for you on the Areca.
I would not get that SM card because it is designed for a UIO slot so it will not mount properly in your case without modification (which im sure you are comfortable doing) but there are other alternatives.
The only reason i suggest running WHS on Server 2008 is because in 2008 there is not a good way to pool your drives. Yes you can do spanning, but if you lose one drive you will lose everything.
There is not a performance hit, that i have seen with running WHS on Hyper-v (thats what I do) and in this situation if you lose a drive you only lose the data on that one drive.
Unfortunately there just is not an ideal off-the-shelf platform for your situation at this time, that I am aware of.
I would not get that SM card because it is designed for a UIO slot so it will not mount properly in your case without modification (which im sure you are comfortable doing) but there are other alternatives.
The only reason i suggest running WHS on Server 2008 is because in 2008 there is not a good way to pool your drives. Yes you can do spanning, but if you lose one drive you will lose everything.
There is not a performance hit, that i have seen with running WHS on Hyper-v (thats what I do) and in this situation if you lose a drive you only lose the data on that one drive.
Unfortunately there just is not an ideal off-the-shelf platform for your situation at this time, that I am aware of.
See http://support.microsoft.com/kb/140365/EN-US/
You'll need 16kb cluster sizing or more to have >32TB partitions
treadstone
Gawd
- Joined
- Jan 25, 2010
- Messages
- 567
Alpha servers
I remember those, used to have one too, ages ago Thank you xnoodle, I'll have a look at it...
ZFS versions below version 19 can not remove log devices. That means if the log fails; you lose access to the filesystem.
To regain access; you would need to run ZFS version 19 or higher. Likely that means ZFS version 24 which will be in FreeBSD 9.0 or the upcoming OpenSolaris 2010 stable release.
Log devices can be separate from the main vdev if you need the steady write throughput. They can even mirror them if you need redundancy on the log device level. But otherwise, using an SSD as log device is not very beneficial. It's 100% sequential access. SSDs are best used as cache device, or to serve iSCSI targets from used as system drive; dealing with random access I/O.
Well if you need any guidance with ZFS or FreeBSD i may be of help. In another thread on this subforum, i am working at building a nice web-interface to ZFS on FreeBSD. I also written some guides to install all this from the very start:
http://hardforum.com/showthread.php?t=1521803
Running FreeBSD and ZFS is a lot different than running Windows with NTFS; both have distinct advantages and disadvantages. Probably the biggest disadvantage of the ZFS approach is that you would not be familiar with the underlying OS. A web-interface can help with that, but still something to keep in mind.
I'll save you a longer reply for now, but there's alot to be said about ZFS of course. Much information can also be found in this thread:
http://hardforum.com/showthread.php?t=1500505
Good luck with your sleek project!