Help with server hardware for large file transfers

G

gruvent

n00b
Joined
Jul 13, 2010
Messages
7
I need a little help selecting the correct hardware for a file server. I've decided to go with WHS along with a NORCO 3116 case.

The problem is that I have to move 30 to 50 GB single files to/from the server and I don't want to wait hours for the file transfer. I will be using SATA drives. I need help picking out a JBOD control cards (Sas/sata?) for the 16 bays,Motherboard, NIC.

Also, the server will be on a gigabit network.

The 10T+ builds on the forums where very helpful on this site, but alot of them did not have speed test.

Any questions please ask.

Thanks in advance!

Gruvent
 
Last edited:
Find a mobo with two PCI-E x16 slots, grab a PCI-E x1 Intel NIC, and two of these controllers and you should be set:
$100 - SuperMicro AOC-SASLP-MV8 PCI-Ex4 8 Port SATA Controller Card

Oh and four SFF-8087 to Multi-lane SATA Forward break cables as well:
$16 - 3ware SFF-8087 to Multi-lane SATA Forward Break-out Cable

I recommend this Intel NIC as a starter if the mobo you choose does not have an Intel NIC:
$30 - Intel EXPI9301CTBLK 10/100/1000Mbps PRO/1000 GT PCI-E x1 NIC OEM w/ 1 x RJ45
 
Danny Bui said:
Find a mobo with two PCI-E x16 slots, grab a PCI-E x1 Intel NIC, and two of these controllers and you should be set:
$100 - SuperMicro AOC-SASLP-MV8 PCI-Ex4 8 Port SATA Controller Card

Oh and four SFF-8087 to Multi-lane SATA Forward break cables as well:
$16 - 3ware SFF-8087 to Multi-lane SATA Forward Break-out Cable

I recommend this Intel NIC as a starter if the mobo you choose does not have an Intel NIC:
$30 - Intel EXPI9301CTBLK 10/100/1000Mbps PRO/1000 GT PCI-E x1 NIC OEM w/ 1 x RJ45
Click to expand...
Danny,

Thanks for the response! I did not have any info on the break-out cables. Assuming that a server using this hardware is setup correctly and running in a perfect enviroment, what file transfer speeds could I achieve with good 7200RPM Drives? (Just looking for a range)
 
Last edited:
gruvent said:
Danny,

Thanks for the response! I did not have any info on the break-out cables. Assuming that a server using this hardware is setup correctly and running in a perfect enviroment, what file transfer speeds could I achieve with good 7200RPM Drives? (Just looking for a range)
Click to expand...

Decent to Good 7200 RPM drives will get you 70-115MB/s real world in non-RAID configuration. Constant could be as high as 115MB/s in real world usage over a single gigabit link. If you want more speed than that, you need to look at link aggregation and multiple Intel NICs or something like an Intel Pro/1000 PT Dual or Pro/1000 PT Quad (I use the quads and they are great).

Also, if you don't have a motherboard, try getting dual onboard Intel 82574L NICs. They are simply great.
 
The server will only be part of the equation. If you are transferring large files (from the sounds of it you are talking about 30 - 50GB movie ISO images?!?), you also need to look at the system the files are currently stored on. You can have a super fast server but the transfer of the files can still be slow if your current system will not be able to serve up the data fast enough! Sure you can put fast drives into your server and maybe even RAID them together as well as add a quad GigE NIC into your new server and aggregate the links via an appropriate switch, but if your existing system only has a single GigE interface and say you have 5400rpm green drives, that won't do you much good. Look at your overall setup and consider all of the components involved before you spend your money, so you spend it wisely :)
 
Just a small note: if you would like the flexibility to move away from Windows and use ZFS in the future, getting another controller than the MV8 might be worthwhile. If you're certain you're never going to run a non-windows OS on there this shouldn't be a concern though. But good replacement controllers, that work great with both Windows and BSD/ZFS, would be:

Intel SASUC8I:
http://www.intel.com/products/server/raid-controllers/SASUC8I/SASUC8I-overview.htm

Supermicro USAS-L8i:
http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm

The MV8 uses a different chip and works horrible under Linux and BSD according to what i read. If you want some flexibility and leave the door open for another setup either now or in the future, then i suggest changing the MV8 to one of the above, or comparable offers.

Network aggregation might be worthwhile in your case; be sure to read up on how to implement that.
 
IMHO, the best motherboard for a storage server, be it WHS, FreeBSD/ZFS, Linux, Windows 2008, or whatever is the SuperMicro X8SIL-F. It has remote KVM (IPMI), 3 PCIe x8 slots, 6 SATA ports, ECC RAM, and 2 onboard Intel Gigabit NICs. It'll take a cheap Pentium G6950 CPU. Its a server motherboard so there's no odd issues with RAID or storage cards. There's onboard video so no problem there.
 
Yes it's a nice board, low power consumption too. But it has some drawbacks:
- Intel platform not the primary choice if you want compression and encryption (at least on FreeBSD). AMD has excellent products in this area, and also allows you to use ECC memory.
- only three pci-express ports (not useful for teaming a lot of pcie x1 gigabit network cards)
- high price if you do not require Intel NICs

But every option has drawbacks, ultimately it depends a lot on your situation and priorities. But it's certainly a great NAS motherboard to consider if the advantages outweigh the disadvantages.
 
Treadstone said:
...(from the sounds of it you are talking about 30 - 50GB movie ISO images?!?)
Click to expand...

I have to move around database backup files.

pjkenned said:
...you need to look at link aggregation and multiple Intel NICs
Click to expand...

Teaming,bonding or link aggregation is new to me. This looks like something I should implement as it looks like it will really help with transfer wait times for these large files. Sounds like having an Intel Pro/1000 PT Dual card in both machines will make a big difference. I think this is a good idea.

WHS Have any issues running/setup dual nic? Is it difficult to setup?

Finally, if a single nic in this system could get 70-115MB/s, what is possible with a dual nic on both machines? (Still using good 7200RPM drives on both systems)

sub.mesa said:
...The MV8 uses a different chip and works horrible under Linux and BSD according to what i read. If you want some flexibility and leave the door open for another setup either now or in the future, then i suggest changing the MV8 to one of the above, or comparable offers.
Click to expand...

Thanks, I will keep this in mind. I don't plan on moving to ZFS unless WHS Fails to preform.


lizardking009 said:
IMHO, the best motherboard for a storage server, be it WHS, FreeBSD/ZFS, Linux, Windows 2008, or whatever is the SuperMicro X8SIL-F. It has remote KVM (IPMI), 3 PCIe x8 slots, 6 SATA ports, ECC RAM, and 2 onboard Intel Gigabit NICs. It'll take a cheap Pentium G6950 CPU. Its a server motherboard so there's no odd issues with RAID or storage cards. There's onboard video so no problem there.
Click to expand...

Recomended hardware thats known to preform good is what im looking for. Also, sub.mesa, thanks for pointing out some of the drawbacks.
 
sub.mesa said:
Yes it's a nice board, low power consumption too. But it has some drawbacks:
- Intel platform not the primary choice if you want compression and encryption (at least on FreeBSD). AMD has excellent products in this area, and also allows you to use ECC memory.
- only three pci-express ports (not useful for teaming a lot of pcie x1 gigabit network cards)
- high price if you do not require Intel NICs
Click to expand...

Just two counterpoints - the X8SIL-F does support ECC memory with Xeon CPUs. If you "go big" and get something like a Xeon X3440 for around $200, which is much more than a AMD CPU granted, you can use ECC memory with the X8SIL-F including a total of six DDR3 slots with registered ECC DIMMs. Granted, at that point you are somewhere in the way overkill version of a NAS which is how I find myself building since I tend to build a box, then continue to find new things to run on it.

Just as a note, the OP is talking about 30-50GB single files. I'm willing to bet those are Blu-ray ISOs. If that is the case, losing a 50GB file has a value of $20 or so, and that is fairly hard to do since the blu-rays are fairly resilient to a bad bit or two. With pure media storage, ECC is probably less necessary because it costs more than the 1-2 max files/ year that will suffer without it.

Second, on PCIe NICs, it is not hard to get ahold of a Intel Pro/1000 PT Dual or Quad these days and there are two onboard Intel NICs plus a management NIC. The KVM over IP also means you are saving money by never having to hook up a keyboard, monitor, mouse, or CD/DVD drive.

One drawback not mentioned is that the X8SIL-F does have slightly higher power consumption than a consumer board because of the BMC/ KVM-over-IP features and extra onboard components. It is only a few extra watts, but it brings idle power consumption closer to AMD levels even with Clarkdale CPUs.
 
Thanks for the correction on the Xeon; i forgot it also works with (some) Xeons instead of just Core i3. So yeah ECC would still be an option. Also, i do believe some core i3 have hardware AES-decryption by now. That would be the holy grail for a high-performance (multigigabit) fileserver with compression and encryption. That makes the X8SIL-F even better, compared to the AMD-platform. But i still don't know if this works on FreeBSD/FreeNAS platform.

The KVM-over-IP (Marvell chip) features are optional, are they not? If you don't need them, the regular X8SIL may suffice; as john4200 pointed out, the X8SIL-F is the version that has IPMI/marvell chip. The power consumption of this board is very low considering its features; if we compare normal motherboards we have to include the consumption of NICs/SAS controller as well.

Since a direct comparison is hard, you can also take the approach of the visual eye: just looking at the chipset heatsink tells you this is a very power modest board. It also doesn't have 'overclock' VRM with 8 or more phases like many retail consumer boards do; which generally is bad for low idle power consumption. Basically, anything that consumes alot of power within a small space would need heatsinks. Chips without a heatsink are <1W.

And yeah three PCI-express might be enough:
2 controllers (16 drives + onboard controller)
1 dual-port Intel NIC

That gives you 2+2 intel gigabit ports, and a lot of disks. Actually i might place this option as best ZFS NAS option, indeed. :p
Would the Windows Intel link aggregation drivers allow you to team the onboard + addon dualport NIC together? Or does the teaming with Windows drivers only work per NIC (i.e. quad-port NIC required to get 4 gigabit) ?
 
Last edited:
gruvent said:
I have to move around database backup files.

WHS Have any issues running/setup dual nic? Is it difficult to setup?
Click to expand...
Tbh, if you are looking at 50GB databases then odds are this is not for home use. If it isn't for home use then I'd suggest to not screw around with enthusiast hardware and/or WHS for that matter. There are very few instances in which some thrown together whitebox solution is a good idea in a production environment.
 
sub.mesa said:
The KVM-over-IP (Marvell chip) features are optional, are they not? If you don't need them, the regular X8SIL-F may suffice; last time i checked there were two versions of this board.
Click to expand...

The -F indicates IPMI. There are versions without the -F.
 
gruvent said:
I need a little help selecting the correct hardware for a file server. I've decided to go with WHS along with a NORCO 3116 case.

The problem is that I have to move 30 to 50 GB single files to/from the server and I don't want to wait hours for the file transfer. I will be using SATA drives. I need help picking out a JBOD control cards (Sas/sata?) for the 16 bays,Motherboard, NIC.

Also, the server will be on a gigabit network.

The 10T+ builds on the forums where very helpful on this site, but alot of them did not have speed test.

Any questions please ask.

Thanks in advance!

Gruvent
Click to expand...

Like Danny said get a good PCI E Gb Ethernet
Note your trx rate is depends on your NIC
so it may be good to look at teaming the NIC to double up the bandwidth

Use 2x NIC or Dual Gb E card

That certain will increase the transfer rate

**
 
sub.mesa said:
And yeah three PCI-express might be enough:
2 controllers (16 drives + onboard controller)
1 dual-port Intel NIC

That gives you 2+2 intel gigabit ports, and a lot of disks. Actually i might place this option as best ZFS NAS option, indeed. :p
Would the Windows Intel link aggregation drivers allow you to team the onboard + addon dualport NIC together? Or does the teaming with Windows drivers only work per NIC (i.e. quad-port NIC required to get 4 gigabit) ?
Click to expand...

The Core i3 does not have AES-NI enabled while the Core i5-650 and above do. It is one of the features that people use. Frankly, I have <1GB on my home media server that I would want encrypted so it is not a huge deal for me.

Remember that PM about testing mesa storage? I was using the Supermicro X8SIL-F, 8GB ECC, X3460, 1068e, HP SAS Expander, and an Intel Pro/1000 PT ;-)

Sadly, about 20 minutes in, I started looking up the best 1068e driver, which somehow morphed to a full blown OpenSolaris implementation. I am not sure why that happened exactly, but it was pretty cool!

@gruvent - the 70-115MB/s figure is just what I have seen and use as a general rule. Dual NICs on both machines mean you need the target and host disks to do more than that since you can probably get 225-230MB/s max bandwidth from two NICs. The limiting factor with dual NICs can be disk transfer speeds. You either need RAID 0/5/6 (or equivalent) with traditional disks or FAST SSD's, on both ends, to read/ write that fast.

To give an example, I had 2x Intel X25-V 40GB's in Raid 0 on a dual NIC system. I pulled 221MB/s when transferring 20GB from the Raid 0 X25-V server to my OCZ Vertex 120GB Raid 0 array. I was clearly limited by NIC bandwidth. Conversely, I then tried doing the reverse, transferring the file back from the OCZ Vertex Raid 0 client to the Intel X25-V Raid 0 server. Transfer speeds were just over 70MB/s. The issue was the the OCZ Vertex drives had about 500MB/s of sequential read transfer speed, the transfer was then bottlenecked by a 230MB/s aggregated dual NIC setup, and then limited to the X25-V Raid 0 write speed of about 70MB/s. You can only transfer as fast as your slowest component.

For hard drives that are not in a raid configuration, there is a similar issue. The max speeds (i.e. outer edge of physical media reads) can be 130MB/s which is faster than GigE actually transfers, making one capped by the GigE bandwidth. However, if the file is located near the inner edge of the disk, transfer speeds may be 70MB/s making you capped by the hard drive not the GigE port. In the latter case, adding a second NIC would make no difference.

Hope that helped.
 
Thuleman said:
Tbh, if you are looking at 50GB databases then odds are this is not for home use. If it isn't for home use then I'd suggest to not screw around with enthusiast hardware and/or WHS for that matter. There are very few instances in which some thrown together whitebox solution is a good idea in a production environment.
Click to expand...


Your Correct. It's not for home use. However, I will be using it on a windows workgroup network that is the same size as a home network. (3 Users and a few servers). WHS looked good because it uses pools, appears to be simple and the OS manages the files for you. I don't have time to configure/move files around when drives fail. WHS looks simple to setup and walk away from. Also, I don't have an enterprise server solution budget.
 
I'm just curious, why push for PCI-E?

Code: 
Gigabit Ethernet (1000BASE-X)	1,000 Mbit/s	125 MB/s
PCI 32-bit/33 MHz		1,067 Mbit/s	133.33 MB/s
PCI Express 1.0 (x1 link)[31]	2,000 Mbit/s	250 MB/s

Nevermind, I just found why:

PCI bandwidth is shared to all devices on the bus. PCI Express uses dedicated links, each of which is faster than PCI
 
pjkenned said:
The Core i3 does not have AES-NI enabled while the Core i5-650 and above do. It is one of the features that people use. Frankly, I have <1GB on my home media server that I would want encrypted so it is not a huge deal for me.

Remember that PM about testing mesa storage? I was using the Supermicro X8SIL-F, 8GB ECC, X3460, 1068e, HP SAS Expander, and an Intel Pro/1000 PT ;-)

Sadly, about 20 minutes in, I started looking up the best 1068e driver, which somehow morphed to a full blown OpenSolaris implementation. I am not sure why that happened exactly, but it was pretty cool!

@gruvent - the 70-115MB/s figure is just what I have seen and use as a general rule. Dual NICs on both machines mean you need the target and host disks to do more than that since you can probably get 225-230MB/s max bandwidth from two NICs. The limiting factor with dual NICs can be disk transfer speeds. You either need RAID 0/5/6 (or equivalent) with traditional disks or FAST SSD's, on both ends, to read/ write that fast.

To give an example, I had 2x Intel X25-V 40GB's in Raid 0 on a dual NIC system. I pulled 221MB/s when transferring 20GB from the Raid 0 X25-V server to my OCZ Vertex 120GB Raid 0 array. I was clearly limited by NIC bandwidth. Conversely, I then tried doing the reverse, transferring the file back from the OCZ Vertex Raid 0 client to the Intel X25-V Raid 0 server. Transfer speeds were just over 70MB/s. The issue was the the OCZ Vertex drives had about 500MB/s of sequential read transfer speed, the transfer was then bottlenecked by a 230MB/s aggregated dual NIC setup, and then limited to the X25-V Raid 0 write speed of about 70MB/s. You can only transfer as fast as your slowest component.

For hard drives that are not in a raid configuration, there is a similar issue. The max speeds (i.e. outer edge of physical media reads) can be 130MB/s which is faster than GigE actually transfers, making one capped by the GigE bandwidth. However, if the file is located near the inner edge of the disk, transfer speeds may be 70MB/s making you capped by the hard drive not the GigE port. In the latter case, adding a second NIC would make no difference.

Hope that helped.
Click to expand...
Yes,
Helps out alot.

Thank you.
 
From what I have read about link aggregation for 1 host to a server you will not see much more than the throughput of a single link. This is because the 802.3ad does MAC hashing in most cases IIRC. If you are working with more than one host to the server you will see an increase in speed on the server side.
 
Physically moving the hard drive is the fastest way to move the data.

Put a removable hard drive case on each machine and just move the drive as needed,
 
@pjkenned: exactly my point in my previous post. His setup will be limited to the slowest link in the chain, which will most likely be his current machine he wants to transfer the files from (though this is speculation since the OP did not include any specifics of his current system)!
 
WHS is not for business environments, JBOD should also not be used for a business environment...

Are these databases already backed up somewhere or to some medium?

Is this server meant to serve people accessing the database?
 
Just a small nit: the X8SIL already has Dual Intel 82574L NICs on board. You don't really need to use up any PCIe slots for the stand-alone NIC cards if you are using this board. 3x PCIe x8 should be sufficient for most medium to large NAS applications.
 
the database is backed up by other methods. I just noted that because someone was wondering what kind of files were 30 to 50 Gigs. I have other large files like VM images. Just need a 2nd location to store files. This is not for public access. Just 2 or 3 people will be accessing the files.

I don't have to use WHS, but it's designed for both Home and Small Office use. So there is nothing wrong with using it in a business enviroment. (Small home or office link) What im looking for is something that will manage files and redundancy and is simple. I'm looking for simple, simple because if it goes down, I could quickly recover the files and set it up again without any suprises. something that if a drive fails, I could just quickly hot swap another drive in without downtime or data recovery.

ZFS sounds great, however unless there is a simple GUI interface, I don't really want to figure out all the new commands. Also, all the PC's are windows. So if I had to work with the files on the server because of some failure, there will be a learning curve on recovering the files.

I am a system builder, I've built many PCs. I've just never focused a PC build for file storage. I've already learned alot just from this thread like recomended dual NICs and Control cards that will preform. A system configured properly with good 7200RPM sata drives, using single gigabyte cards, I should see ~100MB/s. Anything faster is going to require dual nics and raids. I am fully aware of the the system speed will be bottlenecked at your slowest hardware. If I get the correct hardware combination for the file server, then I can start benchmarking the network and replacing bottlenecks.
 
Last edited:
Guess I will find out. Maybe I will buy the compatable ZFS hardware just incase i change my mind :)
 
Last edited:
The Core i5 is not supported in X8SIL/X8SIA Supermicro boards. It will work, but at each POST it will report an unsupported processor. You have to confirm that error or deactivate error confirmation completely in BIOS.

Further, VT-d will not work correctly with i5-650 in these boards despite the support for VT-d on both processor and board. I initially thought "not another board with buggy BIOS", but then I tried a X3460 and it worked fine.

AES-NI however works fine with an i5-650 in a X8SIA-F. I'm still not completly sure of the i3/i5 supporting or not supporting memory ECC functions on these boards. Is there a definitive way of determining if ECC functions are active or not? Dmidecode -t memory reports:

Code: 
Handle 0x0008, DMI type 5, 28 bytes
Memory Controller Information
        Error Detecting Method: 64-bit ECC
        Error Correcting Capabilities:
                Single-bit Error Correcting
        Supported Interleave: One-way Interleave
        Current Interleave: One-way Interleave
        Maximum Memory Module Size: 4096 MB
        Maximum Total Memory Size: 24576 MB
        Supported Speeds:
                Other
        Supported Memory Types:
                DIMM
                SDRAM
        Memory Module Voltage: 3.3 V
        Associated Memory Slots: 6
                0x0009
                0x000A
                0x000B
                0x000C
                0x000D
                0x000E
        Enabled Error Correcting Capabilities:
                Single-bit Error Correcting

@pjkenned:
What type of file transfer did you use to achieve 220 MB/s over an aggregate link? Was a switch involved or did you connect the computers directly? As far as I understood link aggregation will not stripe single TCP connections over multiple links normally.
 
Last edited:
That was the best I've ever seen with Hyper-V on two Pro/1000 PT ports with a RAID controller + SSD's. I honestly don't remember if the config at that point if it was through a switch or was a striaght through from server to client (I change the setup often). Not going to say that was typical, but it was the best case I've seen.
 
to my knowledge you must use a XEON processor to use ECC memory. It doesn't matter if the board supports it or not as it wont be able to use the ECC functions of the memory unless its a Xeon cpu and the core i5/i7 don't support ECC memory.

I am not 100% sure on this but about 90%.
 
HP and Lenovo sell workstations with i5-650 and ECC memory. I could not find anything in the Core i3/i5 datasheets or board manuals that would suggest them not supporting ECC features. What is limited to Xeon CPUs is registered memory.

From the datasheet:

The type of memory supported by the processor is dependent on the Intel 5 Series Chipset SKU in the target platform:
  • Desktop Intel 5 series Chipset platforms support non-ECC unbuffered DIMMs only and do not support any memory configuration that mixes non-ECC with ECC unbuffered DIMMs.
  • Server and Workstation Intel 3400 Series Chipset platforms support ECC unbuffered DIMMs. Workstation Intel 3400 Series Chipset platforms also support non-ECC unbuffered DIMMs. Workstation Intel 3400 Series Chipset platforms do not support any memory configuration that mixes non-ECC with ECC unbuffered DIMMs.

It seems that this is chosen arbitrarily by Intel (like their processor features) as the PCH is rather simple and has nothing to do with the memory on LGA1156 platforms. I'm going to plug the i5 into the X8SIA-F again (poor socket), as soon as I have a way of confirming working ECC functions :)

Regarding link aggregation with intel NICs, the driver lets you aggregate two NICs, as long as one of them is a server NIC. Tested with a one-port PRO/1000 PT Server and a one-port PRO/1000 PT Desktop adapter (which seem to only differ in the ROM contents and price). I did not test it with onboard NICs.
 
Last edited:
Regarding ECC support, this is an example

Dell PowerEdge T110 Server
Supported CPU - Xeon 3400 Series/PentiumG6950/Celeron1101/Core i3-530/Core i3-540
In the summary features, ECC is claimed for the server.

The relevant processors are fully capable of doing it. The deciding factor I think whether it is allowed to be programmed by BIOS to activate the support. Obviously the chipset must support ECC as well.

It appears that most non-tier-1 boards are not allowed to activate the support.

I could be 100% wrong and very happy to be proven wrong. :) Because I am also looking for in-expensive ECC support on Intel platform.
 
You must log in or register to reply here.
Back
Top