Building a NAS. PCI bandwidth question

[Valnar]

After looking at all the off-the-shelf NAS solutions like Thecus and Infrant, I decided to build my own. Infrant is too slow and Thecus has reports of stability and data loss issues.

I have an old PIII Tualatin 1.2Ghz box laying around and it would serve me well with Linux or Win2003. I plan on adding a gigabit Ethernet PCI card and LSI MegaRAID SATA 150-6. I want hardware RAID for a variety of obvious reasons, like online expansion and RAID level migration, takes cycles off the PIII, reliability, etc.

My biggest concern is both the GigE NIC and MegaRAID controller will both be on the PCI bus. I don't plan on using this box interactively - it will only sit in a corner - so I'm not so concerned with some of the speed hit I'll take. As long as it's somewhat faster than Fast Ethernet and below server-grade Gigabit Ethernet (and the Thecus box), it's the better alternative.

My two questions:

1) If the PCI bus were to get saturated by the RAID controller and GigE NIC simultaneously, will there just be a speed hit, or will there actually be a real problem? By problem, I mean data loss, box locks up, total meltdown, etc.

2) Does anyone have an idea of how such a box might run, with said PCI limitations?

-Robert

 

[Madwand]

There are tons of variables here, and it's impossible to predict the performance unless you have a very similar system, OS, client, usage context, etc.

The system shouldn't "melt down" or anything like that due to PCI bus contention, however, since you mentioned it, you might want to look at the 12V output of the PSU and think if it's going to have enough power to start and run however many drives you're planning for. Older systems tended to put the emphasis on 5V, and might not have enough on 12V if you start piling on the drives.

With a 1.9 GHz AMD K7 (2600+ Barton), and a cheap PCI SATA card, and a PCI-based GbE NIC, I was able to hit around 30 MB/s file transfer, using Vista.

Here's a little comparison graph:

The VIA raid was on-board. The single drive was on-board. IIRC, I used the on-board 3com/Marvell PCI NIC. JF means 9K jumbo frames.

 

[Valnar]

Ugh. 30Mb isn't what I expected. But thanks for the info.

Yah, the case, motherboard, etc are all fine. I was going to upgrade the PSU though. I think it's only 230W.

Robert

 

[Valnar]

Madwand (or anyone),

Getting a server based board with PCI-X (and pertinent CPU) is a little above my price range, but I may be able to fork over $$ for a desktop motherboard with an integrated gigabit NIC. I'll still have the SATA RAID controller bottleneck on the PCI bus, but if the GigE NIC is on the Intel CSA, like with a 865/875P chipset board, then it shouldn't contend with the PCI bus. Right?

How much of an improvement would be had?

-Robert

 

[unhappy_mage]

Getting a server based board with PCI-X (and pertinent CPU) is a little above my price range,

Dual Pentium 3 boxes with 64-bit 66 mHz PCI buses can be found on eBay for under $200 a system. Just a though.

but I may be able to fork over $$ for a desktop motherboard with an integrated gigabit NIC. I'll still have the SATA RAID controller bottleneck on the PCI bus, but if the GigE NIC is on the Intel CSA, like with a 865/875P chipset board, then it shouldn't contend with the PCI bus. Right?

Correct, as far as I can tell on my 875p board. I can get real gigabit speeds from it, but I don't have a pci storage adapter to tell if it's saturating the pci bus as well. As far as I can tell, though, it should work fine.

 

[Madwand]

How much of an improvement would be had?

This is a tough question. Even with all components off the PCI bus, we sometimes see posts asking "why don't my file transfers go above 25 MB/s". And then responses "That's normal, you need enterprise gear to get real gigabit speeds". And "Yeah, I've got enterprise gear, and I get 45 MB/s". Etc.

So for an unknown system, environment and usage pattern, I suggest aiming for something reasonable first, like 30 MB/s, and noting that at 3x effective 100 Mb/s performance, it's a big improvement. Hoping for 100 MB/s, etc., for an unknown system is generally an exercise in frustration. I'd say that the next jump is 50-60 MB/s, another somewhat commonly-achieved level of performance.

I haven't done much testing with PCI storage controllers because I've avoided them by design, so don't have good numbers for you. I guess that the performance will depend significantly on the nature of the controller. If it's a real hardware controller, then it'll be better than a hybrid or software controller, because more of the communication will be directly from the controller to the drives and not through the PCI bus.

The CPU also matters. People are used to dealing with file servers in 100 Mb/s terms, where the bottlenecks are clearly not in the CPU. Things change when dealing with gigabit level speeds. I've even saturated a 2 GHz Athlon 64 in a specific case with a RAID controller and an FTP server. Some NICs, storage controllers and even transfer software and security software have higher CPU utilization than others. But conversely, if all your components are top-notch and the OS and transfer software well-tuned, you could again see low or moderate CPU utilization, leading one to the conclusion that the CPU performance isn't so important -- which is fine for that setup, but not true for others.

With a NIC on a PCI bus, and the rest of the system off the PCI bus, and a good NIC, I've been able to exceed 100 MB/s real file transfers across certain systems. So there is hope for high-speed transfers with an on-board NIC and a PCI-based storage controller.

But, but... that test had PCI only on one end and RAID on both ends -- for high speed file transfers, bottlenecks on both ends matter; everything matters, etc.

 

[Valnar]

I'm going to start with my PIII box and save up for a better motherboard later. My need for a home Server will never go away, so I decided to invest in it.

Here are a couple that would to the trick. PCI-X and Gigabit Ethernet, with single desktop CPU support. No Xeon or Opterton needed.

http://www.supermicro.com/products/motherboard/P4/875/P4SCT.cfm
http://www.supermicro.com/products/motherboard/Xeon3000/3000/PDSMA+.cfm

For now, I'll get a good PCI-X RAID controller, and put one of these in it. Both can go to a new box eventually.
-Robert