• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Advice needed from ... experts

B

BigFred

n00b
Joined
Apr 27, 2012
Messages
6
Hi everybody,
this is my first post on this forum, and I would like to get some advice from those of you who are experts in Hi-performance storage systems.
I'm not really in IS or related business, I own a facility in Paris where hi-def (in fact, 4k def) video and film editing is done.
My team (6 people) are in need of powerful Storage solutions. We create between 2 and 4 Tbytes of data per day, most of them being 4k video (very VERY large files, typically 50 to 600 Gbytes and above). We don't need these files to be secured anyway or kept for a long period, they are all delivered to our customers quickly and can be disposed of quickly when processed.
But I'm now on a new project where this amount of data will be generated daily and will have to be shared between several workstations (basically, one for 'capturing', one for color correction, one for image restoration, and one or more for customer delivery including video encoding etc...).
Existing system is based on 2Gig Fiber Channel drive bays, shared through a Qlogic 8GB switch. This system could be sufficient if pushed further, but we lack the internal know-how on performance tuning, and would need to replace disk enclosures with much better one. BUT we all find that software needed to manage Fiber Channel (we use Metasan now) is too complex in our environment (we use W7 64bits only, with some servers under 2008R2).

Now the question is : to migrate on higher bandwith architecture, would it be better to choose Infiniband or SAS switching instead of FC ? Of course cost will be a real concern and I'm also open to other solutions as suggested by those of you who faced these problems.

My 'favorite' solution would be to buy a LSI 6160 Sas switch and to share through it some RAID arrays between the workstations. The fact is that here in Paris, we lack experts in this area and no one I'm in touch with is able to help me design this network. The last guy (who is official reseller of LSI in Paris) told me that through SAS switches we would only be able to share JBOD areas, not RAID0 or Raid6 enclosures. This would be a showstopper since disk performance would be way too low, but it is exactly the opposite of what I understand from LSI whitepapers currently available.

Would someone here be able to help ?

Cheers from paris,
Fred
 
Hi Kristof,

Thanks for your links.
But the fact is I already hired 'consultants' when the existing network based on FC was deployed. At the cost of a kidney. And 2 years after, my tech platform is almost out of date and clearly under what I'd expect considering the money I spent :mad:.
You linked to TigerTechnology, and we use one of their products (MetaSan) which is crap in terms of performance and maintenance of the network.
Hence, I'd rather rely on 'industrial', or more IT-oriented solutions than on 'video specialists' :confused: who oversell their solutions and are not reliable.

Thanks again,
Fred
 
I'd look to 10g ethernet from the workstations to a NAS, it's simple and most non-IT people can understand ethernet. Then a lower end storage setup from someone like NetApp, EMC, Oracle, or one of the many other storage vendors out there. It's likely the storage would use SAS to connect the disks to the controllers. So something like:

Workstations --> 10g Ethernet Switch --> NAS Controllers --> SAS disks

4 TB a day isn't actually too bad if as you say you aren't really keeping it. If you need to keep 4 days you'd be at 16 TB which would be roughly 50x 600 Gig SAS disks assuming you mirrored. Lots of disk shelves are around 24 disks so two of them gives you a nice round 48 which is close enough.

So you grab a handful of 10g ethernet cards for the workstations, a 10g ethernet switch, a low end storage controller, and a couple disks shelves (you may even be able to reuse the shelves you have). You could consider adding some SSDs or flash to the controller to improve speed but your work loads sound like they are likely large sequential actions so cache may not help too much.

For example here is an Oracle solution we'd discussed recently in another thread:

https://blogs.oracle.com/si/entry/7420_spec_sfs_torches_netapp

36 TB exported data able to handle 270k IOPS for $430k.
 
some suggestions

1. you need a fast network
I would use 10 GBe. Latency may be not as good as FC but you need bandwith.
10 GBe is quite cheap and very easy to manage (plug and run)

2. you need a fast storage hardware. With large files and multi-user, you need high sequential values and best I/O with large capacity.

I would use a 8 core storage server with as much RAM as possible (64 GB+, better 128 GB+) and 7200k disks (SATA are good enough) without SAS expander (use as much SAS controller as needed). I would use a 16-24 bay storage server

I would use ZFS datapools. ZFS is very easy to manage even and will inform you about data problems, is fast, easy to manage and has to problem with ultra large files

I would build pools from 3x mirror, not only because of data security but because of best reachable read (and write performance). ex. a pool from 15 or 21 disks (+ hotfix) organized in 3 x 5 disks or 3 x 7 disks. If you use 4 TB disks, you have up to 20 - 28 TB capacity with unique performance.

You can even do snapshots of multi-TB disks and ZFS. If you need more capacity, you can either use more disks or use 2 x mirrors.
You can share your storage via iSCSI, SMB, NFS or AFP.

I would use at least two servers with enough free bays at the second server to move the disks to the second box in case of oroblems (enough free bays), optionally use an expander for this empty case.

You can use Oracle or Nexenta. (beside EMC and netapp)
You can even use OpenSource and whiteboxes, look at OpenIndiana and SuperMicro.
The last case gives you a soluton with network costs of a few thousands and storage costs between 5k and 10k Euro per box.
 
So I work in IT... for the video industry (we make equipment, much of it 4k capable). I don't personally manage arrays that are used for video, but I know we constantly struggle to find equipment that can actually move 4k, real-time, non-stop.

The issue is, in video, you can't have spikes in IO. You have to sustain very high bandwidth, never dipping below your target speed. For example, if you're working with uncompressed 4k you might need up to 3GBps. Even if you're working with some of the standard compressions, you probably need 450MBps or better, each user! Double that if you need the array to do incoming and outgoing at the same time.

You're talking about extremely high-end and high-demand storage. A lot of storage options can give you that type of bandwidth some or most of the time, very few can promise ALL of the time.

That said, if you're trying to do it on the cheap, hire Gea to build you a ZFS monster. I can't personally vouch for it's ability to sustain the speed 100% of the time, but I plan to test it out one of these days and maybe build an array or two for work.
 
obrith said:
So I work in IT... for the video industry (we make equipment, much of it 4k capable). I don't personally manage arrays that are used for video, but I know we constantly struggle to find equipment that can actually move 4k, real-time, non-stop.

The issue is, in video, you can't have spikes in IO. You have to sustain very high bandwidth, never dipping below your target speed. For example, if you're working with uncompressed 4k you might need up to 3GBps. Even if you're working with some of the standard compressions, you probably need 450MBps or better, each user! Double that if you need the array to do incoming and outgoing at the same time.

You're talking about extremely high-end and high-demand storage. A lot of storage options can give you that type of bandwidth some or most of the time, very few can promise ALL of the time.

That said, if you're trying to do it on the cheap, hire Gea to build you a ZFS monster. I can't personally vouch for it's ability to sustain the speed 100% of the time, but I plan to test it out one of these days and maybe build an array or two for work.
Click to expand...

ZFS scales quite well with number of vdevs and RAM for read and write caching. (I suppose I/O is the problem, not sequential performance with multi user read/write the same time as well as file sizes in TB and the option to increase capacity to use, so 128 bit ZFS is the solution with no reachable file size or partition size limits). If you have enough RAM and mirrored spindels, you should be able to reach what is needed. In the extreme, you can use a mainboard with 512 GB RAM where you can keep (nearly) the whole file in RAM on your storage. With 3 x mirrored discs you have also n x number of total disks sequential read performance and n/3 total write performance. with good enough I/O (n/3 of total disc number) . (You can even improve I/O with SSD and hybrid storage) . If performance is not good enough or space is low: just add some spindels. SuperMicro has cases for 72 x 2,5" WD Raptors (max 1 TB) and 10k RPM.

The main advantage: It just works. Everyone with basic IT knowledge can handle during use and during problems (If you have a spare box to import your disks/ pool on problems). And I do not think, you need a ZFS monster, just a fast enough quite simple NAS/SAN.
 
Last edited:
_Gea said:
ZFS scales quite well with number of vdevs and RAM for read and write caching. (I suppose I/O is the problem, not sequential performance with multi user read/write the same time)
If you have enough RAM and mirrored spindels, you should be able to reach what is needed. In the extreme, you can use a mainboard with 512 GB RAM where you can keep (nearly) the whole file in RAM on your storage. With 3 x mirrored discs you have also n x number of total disks sequential read performance and n/3 total write performance. with good enough I/O (n/3 of total disc number) . (You can even improve I/O with SSD and hybrid storage) . If performance is not good enough: just add some spindels. SuperMicro has cases for 72 x 2,5" WD Raptors (max 1 TB) and 10k RPM.

The main advantage: It just works. Everyone with basic IT knowledge can handle during use and during problems (If you have a spare box to import your disks/ pool on problems). And I do not think, you need a ZFS monster, just a fast enough quite simple NAS/SAN.
Click to expand...

Yep. I know it's entirely possible. Most certainly on paper... but what I was getting at is you need a pretty high margin above what you "need" to make sure you NEVER drop below your target speed (often in the gigabytes per second for 4k video streams). I have several of your appliances (which we're going to donate some money to your project sometime soon), but just because my appliances can hit 800MB/sec read/write doesn't mean it can anywhere near sustain it well enough for video. Averages and peaks are not good enough for video. The only numbers that matter are minimums.

Most of the arrays we have capable of 4k are several SSDs striped (RAID0) across two controllers, again striped (RAID00), or RAID50's in the same setup.

In any case, I think one of his best options if he wants to keep it cheap is to look at ZFS and try to scale it to where he needs... and since he's a lot closer to you than most of us, I figured I'd point him at you :p
 
If you need this to be inexpensive, you want something like ZFS or multiple high end RAID controllers.

I would suggest 10000 or 15000RPM SAS disks in RAID10 or maybe parity RAID configurations, with multiple bonded 1G ethernet channels, or 10G ethernet. There are some [relatively] inexpensive ways of doing 10G/20G/40G over infiniband as well compared to 10G ethernet, but you can't have long runs of fiber channel or patch panels.

If you have money to spend [say $100,000+] you should look into HP Lefthand equipment. It will scale to the speeds and capacity that you need, is relatively easy to manage once its setup, and as you grow your storage cluster the system gets faster.
 
obrith said:
Most of the arrays we have capable of 4k are several SSDs striped (RAID0) across two controllers, again striped (RAID00), or RAID50's in the same setup.

In any case, I think one of his best options if he wants to keep it cheap is to look at ZFS and try to scale it to where he needs... and since he's a lot closer to you than most of us, I figured I'd point him at you :p
Click to expand...

Raid-0 SSD are heavy regarding the costs if you need TBs. Second problem is there is no Trim with SSD and Raid, so they loose performance.

I use SSD as well for performance sensitive VM's so I know this is an option.
I have not tried, but I am quite sure, you can reach whats needed with enough spindles.
(I would avoid Raid-O, bad situation if one disk of 20+ fails with a total data loss and your customer is waiting for the nearly finished video) . I would use mirrors, as much as possible, best 3 x mirrors for ultimate read performance or 2 x mirrors for a more balanced relation for read/writes. AND as much Ram as possible to avoid disk use at all. If you use (in extreme) 73 WD Raptors on 5 x 16 port LSI SAS 16 ports controllers in a 36 x striped Raid 1 config and 128 GB+ GB RAM, you should be able to edit 4k Videos by much more than 6 users without worry of data loss - Raid-50 is not an option at all (Raid 50 is equal to a ZFS stripe of two Raid-Z1). I would use a stripe of 7 up to more than 30 striped mirrors.

And be prepared of problems with any solution: You need two systems, best is to use mainstream hardware and as simple as possible overall solutions (what I have learned in more than 30 years in IT)

I would: either spend 30000+ Euros and look for a system house offering one ready to use solution or
look for a ZFS experienced IT consultant with connections to a SuperMicro Hardware reseller to build two redundant systems, optionally with more RAM or disks when performance is too low with the initial lower cost setup. You will not get higher performance anywhere with any Linux, Windows or Apple based solution (I have not forgotten any)
 
Last edited:
Very, very interesting replies here. Thanks to everyone to try and imaginate which problems we are facing.
In fact, some of you imagine bigger problems than those existing : We do not capture 4k video files in realtime. If this were the case, we would have to face 10's of daily Tbytes instead of a few terabytes ... We are capturing 4k video files from a film scanner, and not in real time. This is done on local raid array (Raid0) onto the capture workstation which is fine. And one of my biggest problems is that this workstation, by nature due to software used to manage the scanner, can not be configured to access a standard network. No way. So this workstation is standalone and produces daily terabytes of data, which are then mirrored on a 2Gig FC SAN (24 TB storage, which is enough in terms of size but way too low in perf).
The fact we cannot include the main capture workstation into a 'standard' network or a domain, but only in a 'workgroup', could look strange to you but it means we face 'sharing' problems with every other workstation.
Thus we do not need realtime anywhere (and it would of course not run with our current configuration) so such 'hi-end' bandwith and IOPS are not necessary, my question was more to get the best 'Plug-in' storage system with our constraints, and with better performance than the 2G FC we have.

10GigE would be an option, but I expect this to cause problems with network configuration. SAS switching is, from my point of view, much more 'storage based' which will be comparable with FC, is this wrong ?
 
BigFred said:
Very, very interesting replies here. Thanks to everyone to try and imaginate which problems we are facing.
In fact, some of you imagine bigger problems than those existing : We do not capture 4k video files in realtime. If this were the case, we would have to face 10's of daily Tbytes instead of a few terabytes ... We are capturing 4k video files from a film scanner, and not in real time. This is done on local raid array (Raid0) onto the capture workstation which is fine. And one of my biggest problems is that this workstation, by nature due to software used to manage the scanner, can not be configured to access a standard network. No way. So this workstation is standalone and produces daily terabytes of data, which are then mirrored on a 2Gig FC SAN (24 TB storage, which is enough in terms of size but way too low in perf).
The fact we cannot include the main capture workstation into a 'standard' network or a domain, but only in a 'workgroup', could look strange to you but it means we face 'sharing' problems with every other workstation.
Thus we do not need realtime anywhere (and it would of course not run with our current configuration) so such 'hi-end' bandwith and IOPS are not necessary, my question was more to get the best 'Plug-in' storage system with our constraints, and with better performance than the 2G FC we have.

10GigE would be an option, but I expect this to cause problems with network configuration. SAS switching is, from my point of view, much more 'storage based' which will be comparable with FC, is this wrong ?
Click to expand...

Just as a point of reference, here is a link to a number of systems that are building for a client specifically for 4k video http://hardforum.com/showpost.php?p=1038638054&postcount=15 and would meet your needs for front end systems, and then you just need a storage tank for the back end.
 
a few words more on budget :
I got to go on the cheap for now. Existing storage arrays are outdated but I could get back the disks (Standard SATA) which are a few months old, and reuse them.
I expected to be able to get rif of the FC switch, replace it with SAS switch or 10GigE switch, replace some HBAs with others an get it go. Since I also have to invest on new workstations (Typical workstation is an HP Z800, which will be replaced in short term by Dual Xeon E5 Workstations for better performance) - this will cost my little company almost what I could invest this year :mad:. I would say the 'networking' and storage' part of this has to stay under 15K $.
Of course I agree that this is a tiny bdget, but second hand is a valid option for some parts which should match the budget. A sas switch is around 3K$, which is rather cheap. Supposing i could connect this to second-hand SAS enclosures with Raid5 capabilities like Dell MD1200 or equivalent, with a few SAS HBAs plugged into workstations I would be able to enhance existing performance on the cheap. The question is : will it work as expected ? and will I be able to scale up the system later on to rise storage and performance ?
 
mwroobel said:
Just as a point of reference, here is a link to a number of systems that are building for a client specifically for 4k video http://hardforum.com/showpost.php?p=1038638054&postcount=15 and would meet your needs for front end systems, and then you just need a storage tank for the back end.
Click to expand...

A good point to you, you seem an expert compared to people I used to speak with in video here in Paris. But I expect this kind of setup to rise way ahead of my budget :eek:
 
BigFred said:
A good point to you, you seem an expert compared to people I used to speak with in video here in Paris. But I expect this kind of setup to rise way ahead of my budget :eek:
Click to expand...

Yes, you are looking about 40k a system for something like this, but you could definitely bring the price down by half by going with prosumer workstations as opposed to high-end servers, using third party RAM and going with something less than the Quadro 6000 SDI package.
 
BigFred said:
I'm almost convinced to set a big upgrade to Infiniband, with a switch like this one :
http://www.sandirect.com/qlogic-12200-36-port-40gb-qdr-infiniband-p-1371.html
and Qlogic HCAs.
Will replace my SAN FC boxes with low cost servers and plug into the IB network.
Is there anybody here to tell me which trouble I may be in when setting up this network ?

Thanks to everybody,
Fred
Click to expand...

If you do purchase that switch, for the extra $$$ I suggest you purchase the dual power supply model for redundancy/availability as well as the management module to make network management easier.
 
You must log in or register to reply here.
Back
Top