Looking for network transfer speed and some redundancy

I

ICE_9

2[H]4U
Joined
Feb 28, 2005
Messages
3,459
I would like to setup a RAID that will allow me to transfer data at an extremely fast rate to all the computers connected to it. I know my connection will be through a GB network card. I would like for that to be the slowest part of the connection between all the computers. I am currently looking at:

1 4x SATA2 card
4x SATA2 hard drives

Theory behind it:
SATA2 theoretically transfers at 300Gb per second. 4 drives would (in my head) transfer at 1200 Gb per second. Maybe I am crossing this with a RAID 0 setup. Also, the ability to rebuild as needed would be appreciated. I plan to keep an extra hard drive on tap at all times (was thinking raid 0+1 or 5+1 if it were ever possible.) Suggestions or corrections to me are appreciated.

Reason:
Want to saturate my Gb connection instead of it only doing 10% max on transfers to one system.:cool:
 
Gigabit ethernet will max out around 125 MiB/s.
Most HDDs are unable to sustain a transfer rate of 100 MiB/s. I would put the average sustained transfer rate for a consumer level HDD, when taken across the whole disk, somewhere between 50 and 70 MiB/s. Two consumer level HDDs should be able to almost saturate a single GbE connection.

The 3Gbps signaling rate SATA claims to allow up to 300MiB/s data transfer speeds. Four drives in a RAID-0 array would therefore be able to -theoretically- burst 1.2 GiB/s.

Anyway, what is your question?
 
If you run a decent array, you should have no problems saturating a gigabit network. If it helps you any, a good gigabit card only recieves 80% utilization, unless if you have some nice pieces of hardware...
 
Question is, I want to run a RAID setup that will enable the full use of the GB ethernet card for data transfer. I do not want the GB ethernet card slowed down by the hard drive sustained transfer rate. What setup would allow for this to be capable? Also, this is more networking stuff though, is it possible to use multipule ethernet cards and multiple arrays on one system to allow for like 3 arrays and 3 cards to run the transfers? I want as many transfers to be sustained as high as possible across one box.
 
You should be able to get your gigabyte network to run in full duplex mode allowing for 2 gigs a second transfer rate, your array though unless you put some decent money in it will probably be the slowest thing given you run in full duplex mode. Otherwise a pretty cheap card will saturate the network.

You can have as many cards in your system as you want, why not get an Areca 1231ML card and some speedy drives raptors or something to go with it.

As with running 2-3 connections not positive on but they do make 10gigbyte switches might want to consider getting one of those.

What do you need that kind of speed for? Streaming movies and music won't come close to using all of that, if it's only to copy files faster think of how often you are going to be moving files, i have a 2tb array and only really needed the speed when putting the files on it, other than that my 100mbit connection is working just fine
 
oplin said:
You should be able to get your gigabyte network to run in full duplex mode allowing for 2 gigs a second transfer rate
Click to expand...
In two directions. If you're only going in one, you're not going to exceed 1 gigabit.
oplin said:
What do you need that kind of speed for? Streaming movies and music won't come close to using all of that, if it's only to copy files faster think of how often you are going to be moving files, i have a 2tb array and only really needed the speed when putting the files on it, other than that my 100mbit connection is working just fine
Click to expand...
QFT. 100 mbit is enough to stream 2 HD movies; if you're doing more than that, okay ;) Gigabit isn't a waste of money, though - gigabit switches are cheap enough now, and 10x as futureproof as 100mbit switches. So if your future usage, or your need for speed fuel it, go for it.
 
Big reason is for a future LAN party setup. Need something that can server up 6.4 GB files quickly to lots of people, fast, so that everyone can patch and install quickly without having to wait 20 min to get the file from the server. That is pretty much it.:rolleyes:
 
ICE_9 said:
Big reason is for a future LAN party setup. Need something that can server up 6.4 GB files quickly to lots of people, fast, so that everyone can patch and install quickly without having to wait 20 min to get the file from the server. That is pretty much it.:rolleyes:
Click to expand...

There is some truth to this. Recently I had several friends over for a WoW party and it took each of them 15+ minutes to get the patch from my file server on a 100MB switch.

I swapped out the 100MB switch with a gig model and now get 1GB+ per minute. While that's not exactly spectacular I'm quite pleased with it. :p
 
ICE_9 said:
Big reason is for a future LAN party setup. Need something that can server up 6.4 GB files quickly to lots of people, fast, so that everyone can patch and install quickly without having to wait 20 min to get the file from the server. That is pretty much it.:rolleyes:
Click to expand...

Set up a local BitTorrent tracker. Ten to one they've got a torrent client on their machines anyways ( :rolleyes: ) (and uTorrent's easy to set up if they don't) and as long as you've got a gigabit switch they'll all be able to get the maximum data rate rather than 1/n of it.
 
ICE_9 said:
Big reason is for a future LAN party setup. Need something that can server up 6.4 GB files quickly to lots of people, fast, so that everyone can patch and install quickly without having to wait 20 min to get the file from the server. That is pretty much it.:rolleyes:
Click to expand...

Two iRAMs in RAID-0 and a dual-port intel GbE card should be a good setup for that. It would be good if your switch supported trunking. BitTorrent should be able to significantly improve times as well, for a lot less.
 
AH! See now I would not have thought of the bit torrent thing. It does require them to have it though. :( Gonna have to think on that one a bit. I think the IRAM is a good idea but still needs some work done. Would be really cool to have an SATA port and an Infiniband port on it. Something that Gigabyte can incorperate into their own mobos for screaming performance.
 
Infiniband? What for? It's not just 4 sata ports aggregated into one; Infiniband is originally a high-bandwidth low-latency interconnect for cluster applications. A 3.0gbps port wouldn't hurt, but then, it wouldn't help much for this either. 150 MB/s is more than gigE will sustain, capacity is more of an issue than speed here.

uTorrent is easy to put on a thumb drive; include a torrent file on the drive and it's self-contained. Make a batch script that opens the .torrent with uTorrent, select the local hard drive as the download location, and away it goes. When you take it out it's gone. It may keep settings in %appdata%, though; maybe there's a command-line client or something that doesn't venture outside its directory? Or allow direct file transfers but cap them with a firewall so BT gets priority?
 
You could also set up a multicast network, so that the single source can stream the 6.4GiB file to all listeners at once.
 
It's not a music file. It's things like the Fear Combat game which is 1.7GB by itself. The 30MB patch that goes with it. Wolfenstien Enemy Territory game, BF2 full patch, etc.

The batch file looks to be a good idea to run. I used to have a setup that would post a webpage from my file server with the BT links. That would be easier since it would only require the end user to surf for the page.

Infiniband would be great for transfering data and loading apps quicker on the system. Gigabyte iRAM is limited to SATA standard connection versus the RAMs actual transfer speed. Something to the effect of having a RAM drive on your computer that will allow for the fastest connection to the CPU and the RAM for ultra fast load times, quick boots, and such.
 
If you have a linux fileserver, it will cache whatever its sending.

So even if you have a HDD that is limited to 20MB/sec, ten people downloading the same 1.7GB file from a 2GB Linux server will completely saturate your gigabit link. Just a FYI.

My fileservers have a typical output of 45MB/sec each at my LAN. If everyone is pulling the same file, it will jump to over 100MB/sec. If I have only two to three users, they will saturate the gigabit link. From there is drops with the added users to a low point of 45MB/sec, that is with over 200 simultaneous connections.
 
200 connections? How many people are at that party?

Haven't really looked into the gigabit switches lately but throughput on them typically isn't all that great. So while the torrent idea would likely work a little better it would probably saturate the switch in one hell of a hurry. Unless of course you bought a really nice one.

Depending on how many users you have, an external USB/firewire HD(or two) might work well.

Personally when I need to move large files from one system to another I rip out my array, plug it into the other computer and start copying. Won't really work in your case unless everyone has easily accessible SATA ports.
 
Anarchist4000 said:
200 connections? How many people are at that party?
Click to expand...

Typically 50 or so, all gigabit network. My fileservers host over 3TB of content, so they get slammed for the entire LAN. People never go home with HDD space anymore.
 
ICE_9 said:
Infiniband would be great for transfering data and loading apps quicker on the system. Gigabyte iRAM is limited to SATA standard connection versus the RAMs actual transfer speed. Something to the effect of having a RAM drive on your computer that will allow for the fastest connection to the CPU and the RAM for ultra fast load times, quick boots, and such.
Click to expand...

Infiniband would be pretty expensive. HCAs are ~$200, and implementing the protocol for a small-volume product like an iRam would be a lot of work with not a lot of clients. That means a lot of engineering overhead.
 
You must log in or register to reply here.
Back
Top