Highest throughput possible... SAS v. SATA, SCSI controller, RAID 1+0? Bottleneck?

mbrownnyc

Weaksauce
Joined
Jul 2, 2008
Messages
108
Hello all,

I'm attempting to configure a server that will have the sole purpose of serving files to a group of PCs that are interconnected to the server over GigE.

This server will have two to three teamed (bonded+failover) NICs configured, offering a 3GBit/sec connection.

I definately want to go with RAID 1+0, but am unsure if I should go with SAS or SATA; as I'm having trouble finding benchmarks that pin one against the other. I'm sure SAS is faster, but would like to hear if it makes sense in this configuration.

Will 6 SATA drives, configured in a RAID 1+0, on a RAID controller with 512MB buffer, peak the throughput of a 3GBit/sec NIC? Or is going with SAS drives necessary?

Has anyone benchmarked SATA v. SAS?


Thanks!

Matt
 
From my limited experience max Raid0 throughput is the sum of the individual drives - some overhead.
Let's say your individual drive will support 80MB/s read, so with 6 drives you get 6 * 80 = 480MB/s - some overhead. If you need more pure throughput, you will need more disks.

Also worth considering is that throughput alone doesn't equal I/O performance. Some drives are significantly better than others when looking at multi-user I/O. However, that's the extend of my knowledge, on I/O. Looking forward to hear more about multi-user I/O from people who know this stuff. ;)
 
I'm curious how you calculated your load. If you have "two to three nics" with failover, how did you come to the calculation of 3gbit? If you are failing over, you would want to remain in operation, thus either your peak is 2gbit (or 1gbit) or your needs call for at least 4 nics.

Assuming you are counting a loss of one nic, you are looking at 2gbit, also, knowing that you will never peak out your 1gbit connection (most likley 900mbit utelization), you are looking at about 1.8gbit total throughput.

SAS and SATA will both do the job, it's going to depend what programs/type of files you have on those disks. You can saturate your connections with large files without any problems, but your problem is going to come into when you have a lot of disk activity... not so much your bandwidth.

Even with 6 drives, realistically you will most likley saturate your connection before you would cripple the array (but it will all depend on the type of files and the type of access).
 
Thanks for the replies guys:

Thuleman: thanks for reiterating the important point that there absolutely will be concurrent access to data on the drives (over SMB).

Ockie: I was considering HP's NIC "Teaming," which is failover+bonding. Therefore, ~3GigE from the switch (w. "link aggregation"), having four GigE hosts possibly saturating this ~3GigE connection. I'm trying to avoid having bottlenecks, and not rising above like $5000 for a solution.

Therefore, I figure, sustaining a 400MB read/write I/O (to saturate the reasonable 3GigE network connection) for several threaded processes on several PCs that are access the same logical disk:

Six 36GB (in RAID 1+0, SAS 15K) on RAID controller with 512MB cache.


Will the above solution remove "the HDDs" from being the cause of a bottleneck? If so, kick ass; I'd like to move on to CPU and mobo caches. Do you all think I can keep a solution under $5K?


This is the first time I'm venturing into this kind of analysis of a server, since my new job has these particular needs.


Thanks very much for your insight,

Matt
 
Don't use HP's proprietary teaming, use the standards based LACP. The HP server NIC drivers support it and it works great.

Redundancy on teamed network connections is not like RAID where you get N-1 capacity for RAID5 or N.5 on RAID1. If you have 4 connections you get 4GB/s. If one fails you are down to 3GB/s. Etc. You don't have a parity, mirror, anything like that.
 
Don't use HP's proprietary teaming, use the standards based LACP. The HP server NIC drivers support it and it works great.

Redundancy on teamed network connections is not like RAID where you get N-1 capacity for RAID5 or N.5 on RAID1. If you have 4 connections you get 4GB/s. If one fails you are down to 3GB/s. Etc. You don't have a parity, mirror, anything like that.

HP has two or three different teaming methods now, two of them being 802.3ad. 802.3ad teaming is also a Windows built-in function on Server 2003 and later, but has a high CPU penalty in Fast versus Slow. The problem is that all of these assumptions about Link Agg are wrong. You do not get 4Gb/sec unless you have 4x1Gb/sec or any combination thereof. Your maximum single stream throughput remains the maximum of one port of throughput.

EDIT: Somebody go get me some coffee. Initially wrote it as '4GB' and '1GB' - gha, NO. Absolutely not. GIGAbit not GIGABYTE.
 
Thanks for the great information guys.
Thanks for pointing out LACP.
I'll take a look at those articles a little later today when I have time.

AreEss:
You're saying that if I use LACP on the server, this would be bad. Does HP's own Teaming "technology" not offload processing to the CPU?


I'll take a look through that article, but I'm wondering how many drives would I need to have in a RAID 1+0 to get ~128MB/sec for one out of five hosts performing reads and writes? How could I even calculate this?
 
Thanks for the replies guys:

Thuleman: thanks for reiterating the important point that there absolutely will be concurrent access to data on the drives (over SMB).

Ockie: I was considering HP's NIC "Teaming," which is failover+bonding. Therefore, ~3GigE from the switch (w. "link aggregation"), having four GigE hosts possibly saturating this ~3GigE connection. I'm trying to avoid having bottlenecks, and not rising above like $5000 for a solution.

Therefore, I figure, sustaining a 400MB read/write I/O (to saturate the reasonable 3GigE network connection) for several threaded processes on several PCs that are access the same logical disk:

Six 36GB (in RAID 1+0, SAS 15K) on RAID controller with 512MB cache.


Will the above solution remove "the HDDs" from being the cause of a bottleneck? If so, kick ass; I'd like to move on to CPU and mobo caches. Do you all think I can keep a solution under $5K?


This is the first time I'm venturing into this kind of analysis of a server, since my new job has these particular needs.


Thanks very much for your insight,

Matt

A good controller and 6 of those SAS drives should have no problem.

This was my results on 4 SAS drives on Raid 10 using an Adaptec 3105 controller:

Keep in mind, this is RAID 10 with only 4 SAS 15K SAVVIO's... enough to already saturate two gigabit controllers. To saturate 3, it should be done with ease using 6 drives and a faster RAID method than what I used.

8MB


32MB
 
A good controller and 6 of those SAS drives should have no problem.

This was my results on 4 SAS drives on Raid 10 using an Adaptec 3105 controller:

Keep in mind, this is RAID 10 with only 4 SAS 15K SAVVIO's... enough to already saturate two gigabit controllers. To saturate 3, it should be done with ease using 6 drives and a faster RAID method than what I used.
<snip>

Thanks Ockie. That looks great. I wonder what the relationship between i/o times and drive additions in RAID 1+0. There must be a static amount, up to the speed of the controller?
 
Its worth mentioning that you are really talking about two different things here and someone mentioned it above before but I think it got lost in the shuffle. You are talking about sustained throughput when you are talking about saturating a 3Gbps connection but then you also want it to push that kind of throughput while being accessed by multiple users at once which is all about IOs not throughput. To do both at once you are talking about 15k drives with the highest throughput you can find on a good RAID card.

It sounds to me like this is going in an HP server being as you are discussing HP teaming. If thats true I can also tell you that you aren't going to do 360MBps sustained off of 6 15k Savvio drives in an HP server with anything less than the P600 controller (possibly higher). I can pull some benchmarks for you if you want, but last time I ran HDTACH on a RAID 10 with 6 x 15k Savios on a P400 the max speed was slightly less than 280Mbps. My guess is that the P400 controller just doesn't have the processor capability to max out the drives even with 512Mb of BBWC on it.
 
Do you have an accurate i/o profile from each client?


Gramps always said that you measure twice and cut once or you end up paying four times as much to do the job right.
 
<snip>You are talking about sustained throughput when you are talking about saturating a 3Gbps connection but then you also want it to push that kind of throughput while being accessed by multiple users at once which is all about IOs not throughput. To do both at once you are talking about 15k drives with the highest throughput you can find on a good RAID card.

<snip> HP server...<snip> I can also tell you that you aren't going to do 360MBps sustained off of 6 15k Savvio drives in an HP server with anything less than the P600 controller (possibly higher). I can pull some benchmarks for you if you want, but last time I ran HDTACH on a RAID 10 with 6 x 15k Savios on a P400 the max speed was slightly less than 280Mbps. My guess is that the P400 controller just doesn't have the processor capability to max out the drives even with 512Mb of BBWC on it.

If you have a P600 (w. 512 write-cache), benchmarks would be awesome. I appreciate the specific input and insight. I guess, straight-up I'm looking to find out how I can "max out" the 3Gbps network connection, by delivering throughput of 64MB/sec to six hosts [384MB/6 =] (give or take about 1000Mbps due to not true 1Gbps performance). Okay... so given the reservation of not true throughput of 128MB/sec performance of 1Gbps network card; let's say 3Gbps network connection perform at about 300MB/sec: Can we have 6 hosts with concurrent throughput to the disks at about 50MB/sec?



Do you have an accurate i/o profile from each client?

Gramps always said that you measure twice and cut once or you end up paying four times as much to do the job right.

What do you mean accurate i/o profile? I don't know what it would be, but the devs apparently are I/O hungry guys... I want to go back to them with a sort of utopian config for the fastest server possible, so we can sit and scale back from there.


I really appreciate everyone's input. You all can consider yourselves an asset to this community, which I'm happy to also be a leeching part of... :) Maybe, soon enough, I'll be in your place with my skillz and be more active.

Sincerely appreciated,

Matt
 
IMO the 'fastest server possible' if you are just looking for flat file storage and maximum speed over the network would be a NAS device, not an HP box with all the OS overhead, etc. We run Netapp for CIFS/SMB and it keeps it's network connections pegged all day long.

However, if you are looking to install applications, then disregard this post.
 
HP has two or three different teaming methods now, two of them being 802.3ad. 802.3ad teaming is also a Windows built-in function on Server 2003 and later, but has a high CPU penalty in Fast versus Slow. The problem is that all of these assumptions about Link Agg are wrong. You do not get 4Gb/sec unless you have 4x1Gb/sec or any combination thereof. Your maximum single stream throughput remains the maximum of one port of throughput.

EDIT: Somebody go get me some coffee. Initially wrote it as '4GB' and '1GB' - gha, NO. Absolutely not. GIGAbit not GIGABYTE.

AreEss:

I just re-read this thread and your post seems to touch on a very important point that i previously skipped over: that any link-aggregation does not combine speed of NICs, but only delivers failover. I can't imagine this is true. Please explain.
 
Back
Top