• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Need suggestions for 3.2T storage solution

R

Rix2357

[H]ard|Gawd
Joined
Sep 15, 2001
Messages
1,274
Yes...you read that right. I am looking to integrate a system with 3.2Terabytes of storage. I'm basically looking for products that would allow me to use a drive as 3.2T under Win2k/XP. The main thing is that DVR programs have to be able to write to what would be recognized under 2k/XP as an ordinary drive. My first thought was a device that could hold 8 400GB drives in an external housing and have it linked by external SATA to the regular computer. Perhaps someone might know some more about other products? Any info regarding this would be very much welcome.
 
There are many external harddrive arrray devices, but I haven't ever seen one that plugs into the host via SATA, almost all talk to the host using a SCSI interface, even though the array's harddrives could be SATA. We have a few of these units at work, when I go back monday I could get their manufacturers names.
 
I have a PC-2100A Lian-Li case that can hold 12 drives. I have a 3ware 9500S-8 SATA card that can, obviously, hold eight drives. I have it populated with 5 Maxtor DiamondMax10 300GB drives for 1.2TB of RAID 5. If you got the 12 drive version of the RAID card you could fill up that case and not have to go external. You could get 3.3GB of RAID 5 with that setup.
 
As I recall, NTFS only suports single volumes up to 2TB
I also semi-recall that there might be an enterprise plugin from M$ that allows for bigger volumes but its just like a haze on the side of my mind that keeps bugging me.

Douglite should chime in for the above

You can however mount larger network folders as a local drive under 2000/xp
when you get above 2tb of space your getting into the area of enterprise storage where most storage solution's are network based and not centered around NTFS.
some of the linux filesystems support sizes of 128TB and above

Hardware wise, a 3ware card with a bunch of 400gb drives will do you pretty well
the more expensive 12port card that 3ware has allows for dynamicly increasing volume size so you can add on extra drives later as you want
 
You can go above 2TB if you use dynamic disks, however I strongly recomend against it.
 
A) I hope you've looked at tape backup systems - that's the only hope you have of preserving data if something catastrophic happens to this array. Otherwise, I would strongly consider RAID6 (with two rotating parity schemes allowing the array to lose two members and remain alive) for such a massive arrray - it is quite possilbe for a drive to go down during a rebuild prompted by the failure of a different drive, then your array is totally hosed if you're using RAID-5. Rebuilding arrays upwards of a TB on ATA drives takes hours, sometimes over a day to complete. You will need 9 400GB drives in RAID-5 or 10 in RAID-6 to achieve 3.2TB.

B) NTFS is capable of handling volumes well beyond 2TB, into petabytes and exabytes in fact (16 exabytes less 1KB is the absolute maximum of NTFS). However, a conventional Master Boot Record can only address a partition up to 2TB in size - this is not a limitation of Windows, and will affect Linux volumes as well. On Windows, you must use dynamic disks to circumvent this limitation. I'm sure it's possible on Linux as well, but I'm not a *nix expert. Also, many RAID controllers have a 2TB cap on the size of an array - you will need to look at controller documentation to find one that does not have this limitation. Once you remove the MBR limitation and are free of controller limitations, then you're good to 16TB with the default allocation unit size of 4KB with NTFS, and making the AUs bigger will allow you to go much farther than that, but you will lose NTFS compression and the ability to run some utilities.

C) You need a quality PSU for such a setup. PC Power and Cooling is the standard by which all other PSUs fall short. Zippy/Emacs makes rock solid high wattage PSUs too. Head to the Power Supply Forum to learn more.
 
Hmm, I would rather an external chassis to house the drives as opposed to the Lian-Li suggestion. The reason being is that I do expect some of the drives to fail and I would rather be able to have them in 5.25" bays to be swapped out.

Another one I supposed is some sort of rackmount chassis with a lot of 5.25 bays. I would need 9 5.25" bays because a DVD burner needs to be included in addition to the 8 400GB drives. I think I only need to provide 3.2TB before applying the raid 5 array, since the purchase order specs doesn't say if it's 3.2TB accounting for a raid setup. I'm going to check on that anyways, but seems like just a matter of buying a device with some more drives to go with it.

I don't mind if the interface between the computer and the storage device is SCSI or propietary. As long as it works with IDE or SATA drives. After thinking about it, SCSI or some controller based card would probably be the best and have the least loading on the CPU as the DVR also has to compress the video files.

I know you can mount network shares as a drive, but I would rather not use NAS simply because I would have to setup a small isolated network with the stuff and that is just another layer of complexity that decreases reliability of the overall system.

To tell you the truth, all this storage is a bit silly IMO since this array would hold well over 2 years with the DVR system currently available, but the customer demanded a 3.2TB storage unit. If they are willing to fork over the cash, who am I to say no? eh?
 
Your only choice here is SATA, pure and simple. 300GB SCSI drives are $800+ a pop, and you will need 11 of them. I don't think he wants to shell out $9000 on hard drives.

Hot swap is absolutely necessary. You will have to rebuild, often. Virtually any SATA controller with enough ports for this job is also going to have a hardware accelerated XOR engine for RAID-5. Even if you go with an 8 drive RAID-5 array, you will want a hot spare so that a rebuild can commence immediately after a drive fails.

I also hope that this customer understands that he must have a tape backup system for this, or be prepared to accept the periodic loss of everything on the array. Backups for an array this large will be a full time job. Really, someone should be watching it all the time to feed it more hot spares.
 
Hmm, any manufacturers that come to mind as far as these drive array go? I can probably get most computer products through ASI which is my computer parts distributor. I had asked them first but they gave me a link to Infrant Technologies which was more of a NAS type device and while I'm sure I could get it to work, I would also like to know other solutions.

Really, the only two methods that I know of are:

a) external drive bay array linked to computer with some sort of controller card

or

b) some super big rackmount device where I can put in a 3ware SATA card.

If anyone knows of an alternative c) or d) let me know, manufacturer names would really help.

I'm just trying to do as much research over the weekend since ASI isn't open until Monday.

I'm also going to look at the tape backup. I did some rough calculations and
400GB > 400,000MB

Say transfer rate is 40MB/s

So 10,000 seconds to rebuild an array or 167 minutes or approx. 3 hours. I'll also factor in that I don't get a transfer rate of 40MB/s and get a lousy 10 MB/s. Then that should only take 12 hours. Correct me if I'm wrong anywhere. I am assuming that all of these devices should be able to rebuild the array on the fly while still being able to maintain operation of the RAID array?
 
DougLite said:
B) NTFS is capable of handling volumes well beyond 2TB, into petabytes and exabytes in fact (16 exabytes less 1KB is the absolute maximum of NTFS). However, a conventional Master Boot Record can only address a partition up to 2TB in size - this is not a limitation of Windows, and will affect Linux volumes as well. On Windows, you must use dynamic disks to circumvent this limitation. I'm sure it's possible on Linux as well, but I'm not a *nix expert. Also, many RAID controllers have a 2TB cap on the size of an array - you will need to look at controller documentation to find one that does not have this limitation. Once you remove the MBR limitation and are free of controller limitations, then you're good to 16TB with the default allocation unit size of 4KB with NTFS, and making the AUs bigger will allow you to go much farther than that, but you will lose NTFS compression and the ability to run some utilities.
Click to expand...

Yeah that sounds more like it
I knew there was some sort of 2gb limitation but something didn't feal right in saying it was NTFS.

for the size of array he is talking about a tape backup unit would run $4-8k for the drive software and tapes

for that price it would probobly cost the same or less to be running a Raid 15/16 setup with mirrored raid5 arrays (275x10[drives]+750[controller]+500[powered enclosure])

backing up 3.2gb+ of space is just NOT cheap nomatter how you want to do it

If he is going to be looking at investing 4k into something like this
going a small step further and buying an existing comercial soln might be better off in the long run
 
DougLite said:
I also hope that this customer understands that he must have a tape backup system for this, or be prepared to accept the periodic loss of everything on the array. Backups for an array this large will be a full time job. Really, someone should be watching it all the time to feed it more hot spares.
Click to expand...

From what Rix2357 is saying the array is going to have a very low level of usage
just streaming data onto and off of it
its not going to be a hevily accessed database server running some of the largest forums on the internet

with that said
I highly doubt he will have more than 1 drive failure over the corse of 2 years
if he has any
If he was using a raid 6 system and had 2 hot spares it would be slightly overkill reliability considering the planed usage
 
3.2TB requires just over 37MB/sec to build in 24 hours (3.2*1000*1000=3200000, that's the size of the array in MB, then divide by 60 for seconds, 60 for minutes, and 24 for hours. If anything happens to any of the members in those 24 hours, you're back to square one. That's the time for the initial build. So yes, expecting the build to take "all day" is quite realistic.

Rebuilds hopefully won't take that long, but will still be "hours." Any IT manager or admin that has had to deal with such situations will tell you that it's possible to lose a member during a rebuild, forcing them to go back to the tapes. If he needs that much space, he needs backups. If he doesn't need backups, he doesn't need that much space.

I'd go with the 12 port 3Ware, have at least one hot spare at all times, and once again, I think RAID-6 is a good idea. As has been pointed out, this array will not be seeing heavy usage, but that doesn't mean bad things can't happen.
 
One more question on this before I let this die. What would be the negatives of going with a PCI raid card of 8 or 12 channels vs. PCI-X. I'm expecting at most 2MB/s of constant data writes for normal operation.
 
Rix2357 said:
Hmm, I would rather an external chassis to house the drives as opposed to the Lian-Li suggestion. The reason being is that I do expect some of the drives to fail and I would rather be able to have them in 5.25" bays to be swapped out.

Another one I supposed is some sort of rackmount chassis with a lot of 5.25 bays. I would need 9 5.25" bays because a DVD burner needs to be included in addition to the 8 400GB drives.
Click to expand...
I have twelve hard drives and two DVD burners mounted in a Stacker case. There are several 3-in-2, 4-in-3, and 5-in-3 drive modules can can pack more drives into a smaller space. Using hot-swap back planes makes replacing drives easier, but the standard Stacker modules provide for better cooling (but you have to pull the whole module to swap out failures).

However, for a reliable system that size, I'd also look at putting the DVD-R in a 1U server sitting on top of a 14-bay rackmount NAS box.
 
Zamboni said:
I have twelve hard drives and two DVD burners mounted in a Stacker case. There are several 3-in-2, 4-in-3, and 5-in-3 drive modules can can pack more drives into a smaller space. Using hot-swap back planes makes replacing drives easier, but the standard Stacker modules provide for better cooling (but you have to pull the whole module to swap out failures).

However, for a reliable system that size, I'd also look at putting the DVD-R in a 1U server sitting on top of a 14-bay rackmount NAS box.
Click to expand...

For what he wants, I think it is technically a SAN box not a NAS box.

A NAS is Network Attached Storage, the NAS would have an Ethernet connection.

A SAN is a Storage Area Network, it can have a SCSI, or FibreChannel connection.

SANs are usually used for things like Cluster Servers so that both Cluster nodes share the same data drives. But they can also just be used as a large or high performance drive expansion for one server.

==>lazn
 
http://www.nstor.com/default.asp?bdy=/product/sata/4700_ov.asp

This is what I would recomend. An external box is much better because you now have dedicated cooling for the drives as well and dual reduant power supplies which are the most often failed component. In general the systems are designed to function as one unit and will almost always work better then some piece meal solution cobbled together by an amatuer compared to something designed by professional engineers.
 
We use a company called AXUS for our storage devices for our clusters. I provide hardware support and do R&D on these devices and many others and find that AXUS's "DemonRAID" line is fairly stable and provides most, if not all, the features that you are requesting. Unfortunately, they are Taiwan-based; we purchase these in bulk directly from Taiwan. You might look at StorCase which is a division of Kingston. These devices are very similar in features and should be readily available to U.S. customers.
 
A Sun toaster is my suggestion. At my dad's workplace, they use these everywhere. Nice boxes, if I do say so myself. At this capacity, it's probably best to get the array from a retailer rather than build it yourself. If a disk dies, Sun is the one who has to replace it, not you.

This box, with 4.8TB, costs $21k. And needs a $1700+ FC adapter. This project will not run cheap.
 
You must log in or register to reply here.
Back
Top