Best PCIE 1x SATAIII controller with PM

G

geant90

Limp Gawd
Joined
Aug 6, 2011
Messages
464
Hey guys,

I need some opinions of actual owners. What is the best (in your opinion) PCIe 1x SATA III Controller card that supports Port Multipliers.
Used in FreeBSD or linux. It can be sata or esata.

The one I buy would most likely be used in Nas4Free for a mITX ZFS and software RAID5 expansion so name some products!
 
I would say the best is a ASM1061 based card. Why does it have to be SATA3? I can't speak about FreeBSD support.
 
Why does it have to be SATA3?
Click to expand...

Although it will take many years before a single hard drive would have a real benefit from SATA3 I would expect that this would be desired if you are using port multiplication provided you have a port multiplier that does SATA3.
 
With port multiplication each drive has its turn so the maximum bandwidth of the connection doesn't come into play.
 
It depends on whether CBS or FBS is used. If FBS is negotiated, the PM can aggregate multiple data streams and better utilize the uplink. I do not know which PMs and SATA host controllers support FBS.
 
Well I use to use the adonics PM and now I am testing a new china brand which uses FIS. The PM is "blind" so lets say it negotiates all drives @ 1.5gbps which would allow 192MB. The average data drive transfer at average 80MB write. I dont have a specific plan of use just testing the PM. so if anything lets say SATAI speed of 1.5gbps gives me the bandwidth of 2 drives. So a single SATAIII port could give me 8 drives of bandwidth with no bottleneck IF the controller and PM negotiates @ SATAIII speeds which under freebsd it should. But I do not have the PM yet should be here in a week or two. And thats also skipping that pcie 1x is only 5gbps in dual direction. And what i would personally use it for ZFS or SWRAID under Freenas or Nas4Free in Raid5+ it would be as if it was CBS the way that RAID5/6 stores the data. I Do plan on testing SSD's in RAID0
 
Personally I have 4 1x cards with marvell controllers and 6 enclosures with marvell port multipliers, and clearly they take turns, if I try to use 2 drives at a time in the same enclosure they compete with each other. Sometimes under stress drives drop out entirely. I would never use those with any kind of software RAID/ZFS, it's a recipe for disaster.
 
Aesma said:
Personally I have 4 1x cards with marvell controllers and 6 enclosures with marvell port multipliers, and clearly they take turns, if I try to use 2 drives at a time in the same enclosure they compete with each other. Sometimes under stress drives drop out entirely. I would never use those with any kind of software RAID/ZFS, it's a recipe for disaster.
Click to expand...

FIS works fine and under any PM it would work fine as RAID5/Z they wont have to "compete" with each other taking turns if there under the same array since data is dished out in a sort "carddealer" way
 
I don't like PMs either, but drives dropping out seems to point to a different problem.
 
Mediasonic eSata III PCIe card HP1-SS3
supports FIS switching and PM works fine on external enclosure. Will order one of these cards I just wished it had internal sata aswell
 
geant90 said:
Mediasonic eSata III PCIe card HP1-SS3
supports FIS switching and PM works fine on external enclosure. Will order one of these cards I just wished it had internal sata aswell
Click to expand...
Don't believe everything you read.

Not really Mediasonic's fault ... except that they believed their supplier/partner, Hotway. Kudos to Astrotek who actually appears to know the difference between CBS & FBS and describes their ASM1061-based card correctly [link].

ASMedia themself only describes the ASM1061/2 as "supports Port Multiplier". (If it supported FBS, they would/should claim it.)

When you get your HP1-SS3 and test it, please do report back--I'd love to be wrong. But make sure that you know the difference btw FBS & CBS (and know how to test for it).
 
If its PCI-E 1.0 x1 then your not gonna get more than 200 megabytes/sec from it anyway so anything above SATA2 is a wash IMHO.
 
My P8Z77 WS uses an ASM1061 controller for the eSATA ports and a Marvell 9128 for the internal ports. I don't know why they made that decision, but I guess it's because of Marvell's RAID functionality. Most motherboard makers use Asmedia these days, I think they have less issues.
 
Well just thought it would interest some of you that I have a MB with SB850 SATAIII got a 1 to 5 SATA PM FIS switching and it negotiates drives at sataI 1.5gbps works right out the box with nas4free/freenas. 11 drive RAIDZ2 ZFS has not dropped a single drive since creating the array also tried softwareraid. I get 100MB+ read and write (im sure the 100+ write is jsut the RAM cache) speeds. no problem saturating gigabit network for you that dont know Gigabit=125MB so it never drops under 100MB is pretty bad ass SMB and FTP.
5400RPM Drives
1 PM=5 drives
1PM=4 Drives
2 Drives direct

Get my all my SMART data I'll let you know when its been up a week transferring.
 
Last edited:
Glad to hear that you arrived at an acceptable solution. It is especially good that you avoided using an add-on card, which is often a problem for PM usage (flaky drivers, etc.).

What PM enclosures are you using (full model # please)?

Also, I'm curious what the LOCAL transfer rate is. Could you run the following command :
Code: 
time dd if=[i]SomeBigFile[/i] of=/dev/null bs=256k count=4096
Thanks.

Note: SomeBigFile should be > 1GB (and in the RAIDZ2 array). Also, just run the command one time; don't do multiple runs and average the results--otherwise buffering will skew the answer.

--UhClem
 
Will do right after I run these last test with three RaidZ arrays. An external enclosure I use is Mediasonic HF2-SU3S2 which is JMB321 which does FIS switching on SB850 just fine. Im testing FIS switching with 3 separate array and still gigabit is my limiting factor geting 30-40MB per 3 transfer in SMB which we know is not the best transfer method. Even so that good performance.
 
Welcome to NAS4Free!
nas4free:~# time dd if=/mnt/RAIDZ2T/123456789 of=/dev/null bs=256k count=4096
4096+0 records in
4096+0 records out
1073741824 bytes transferred in 7.648855 secs (140379420 bytes/sec)
0.007u 0.701s 0:07.70 9.0% 28+3054k 8260+0io 0pf+0w

Is it not strange that it is exactly a bit over 1Gbit @ 133.9MB/s I find that pretty acceptable myself. I want to try this again when my other PortMultipliers come in and I test with 20 Drives. Also these are all WDGreen 5400RPM drives (with the exception of 4 WDBlacks 7200RPM) in the array. Once again I'm pretty damn satisfied as RAIDZ limits write speed to that of one drive but the read is the combined individual. Still= $30 steal. I cannot stop smiling XP

Also UhClem please share some more testing commands or line commands. Don't know many as I am out of my Windows environment. If you could link me to a man page as well.

Thanks
 
Also if anyone is interested I will be getting these in bulk for super cheap I might get them for 20 a piece If we want to get something good going
 
Also tried in striped (for those who don't know = RAID0 Equivalent )

nas4free:~# time dd if=/mnt/RAID0ZFSS/123456789 of=/dev/null bs=256k count=4096
4096+0 records in
4096+0 records out
1073741824 bytes transferred in 0.473970 secs (2265421795 bytes/sec)
0.000u 0.213s 0:00.47 44.6% 26+2880k 216+0io 0pf+0w

1GB file just under half a second
Whopping 2160.5 MB/s which is also 2.1GB/s
 
geant90 said:
... An external enclosure I use is Mediasonic HF2-SU3S2 which is JMB321 which does FIS switching on SB850 just fine.
Click to expand...
Not really ... Mediasonic does not understand Port Multiplier technology! (Recall earlier about the HP1-SS3 ... you should be ashamed :) [link].)

The JMB321 does NOT support frame/FIS-based switching (FBS), only command-based switching (CBS). The AMD SB850 is documented as supporting FBS; it also supports CBS (which is what you are making use of).

As you've stated (and I agree), for network access (GigE), the inefficiencies of CBS are mostly hidden. I'll say more in a follow-up to your performance numbers posting ...
 
geant90 said:
Welcome to NAS4Free!
nas4free:~# time dd if=/mnt/RAIDZ2T/123456789 of=/dev/null bs=256k count=4096
4096+0 records in
4096+0 records out
1073741824 bytes transferred in 7.648855 secs (140379420 bytes/sec)
0.007u 0.701s 0:07.70 9.0% 28+3054k 8260+0io 0pf+0w
Click to expand...
Yeah, that is about what I would expect for the config you described (5 in PM1; 4 in PM2; 2 internal), using CBS PMs. (I would have estimated 150-180 MB/s).

If you had FBS PMs AND they were controlled properly, you would have seen 425-475 MB/s (for that config).
Is it not strange that it is exactly a bit over 1Gbit @ 133.9MB/s
Click to expand...
Completely unrelated.
I want to try this again when my other PortMultipliers come in and I test with 20 Drives. Also these are all WDGreen 5400RPM drives (with the exception of 4 WDBlacks 7200RPM) in the array.
Click to expand...
With 20 drives/Z2 in 4 CBS PMs, I expect a LOCAL read test to be 300-360 MB/s, if you don't run out of CPU juice; though you'll probably only get 260-280 (based on the 140 with 2 PMs).
Still= $30 steal. I cannot stop smiling XP
Click to expand...
That is a true bargain! I paid $55-65 each for 3 SansDigital TR4Ms, but had to watch and wait for Newegg to do a sale price/limit 1. They are only 4-drive units, and have no USB, but they are FBS PMs (using a SiI3726).
Also tried in striped (for those who don't know = RAID0 Equivalent )

nas4free:~# time dd if=/mnt/RAID0ZFSS/123456789 of=/dev/null bs=256k count=4096
4096+0 records in
4096+0 records out
1073741824 bytes transferred in 0.473970 secs (2265421795 bytes/sec)
0.000u 0.213s 0:00.47 44.6% 26+2880k 216+0io 0pf+0w

1GB file just under half a second
Whopping 2160.5 MB/s which is also 2.1GB/s
Click to expand...
To quote: "If it seems too good to be true, it is!"
Most/all of that 1GB was in the buffer cache; your dd command never hit the array.
Also UhClem please share some more testing commands or line commands. Don't know many as I am out of my Windows environment. If you could link me to a man page as well.
Click to expand...
Sorry, my BSD-specific experience/knowledge is too old to be useful here [BSD2.7 in late 70s thru SunOS4 early 90s].
 
LIES! you have a bit more secrets? I'm just getting into FreeBSD so even my command line directory exploring is not to great. Its similar so I figured it out eventually.I'm still quite pleased with the performance and price Any other specific way I would test FIS? Thanks for the info and should I be running the dd command with a larger file? 10GB + etc?

For another test would 3 Drives Direct(Negotiated@S3) in RAIDZ with dd command, then same array through PM and dd would be a good test? To compare speeds
 
UhClem said:
Not really ... Mediasonic does not understand Port Multiplier technology! (Recall earlier about the HP1-SS3 ... you should be ashamed :) [link].)

The JMB321 does NOT support frame/FIS-based switching (FBS), only command-based switching (CBS). The AMD SB850 is documented as supporting FBS; it also supports CBS (which is what you are making use of).

As you've stated (and I agree), for network access (GigE), the inefficiencies of CBS are mostly hidden. I'll say more in a follow-up to your performance numbers posting ...
Click to expand...

:( Is there a command I can run that would say FIS or CBS is being used?
 
Also UhClem I sent you a PM earlier asking you for any other test/tips. I could really use your help if you can spare the time.
 
UhClem,
What about the Addonics AD5SAPM
http://www.addonics.com/products/ad5sapm.php
This says it supports FIS. Is there any way to check which switching it is using? I ran The same command with 3 Drive RAIDZ1 which I will post in the following post. I'm going to buy a Sil3726 5 x 1 PM (expensive) to test and compare.
 
So I ran a DD test with a single RAIDZ1 Logical Array with 3 Physical WD Black 2TB SATAII 7200RPM hard drives.
I created the array directly on the onboard SATA ports, transfered the file and then rebooted.
Ran the DD test reboot and then the JMB321 aswell as the AD5SAPM (Through a single SB850 SATA port).
Here are the results from fastest to slowest. The AD5SAPM should be FIS (claimed by website description but I'll wait until UhClem can confirm that) which is strange.
I did run multiple test but instead of average I just used the best result as there preety tight results.
If your wondering I did seperate files which would not use buffer.

Command: time dd if=/mnt/RAIDZ/123456789 of=/dev/null bs=256k count=4096

SB850 (NO PM) :05.169170 secs (207720349 bytes/sec) 198.09 MB/s
JMB321 :07.873807 secs (136368828 bytes/sec) 130.05 MB/s
Addonics AD5SAPM :11.748918 secs (91390698 bytes/sec) 87.15 MB/s
 
UnClem I saw your post on another forum. How can I run a random access sector test? I like to have the drives just as individuals UFS and try them individually then at the same time.
 
Is there a command I can run that would say FIS or CBS is being used?
Click to expand...
Not directly. But it is probably indicated in the BSD kernel message log. You should connect 3 drives to one JBM321 enclosure, AND also 3 drives to that AD5SAPM, and then reboot. Locate and examine the kernel message log, and see if there is any difference between the messages for each PM's initialization.
What about the Addonics AD5SAPM
Click to expand...
This is based on the SiI3726. It supports FBS or CBS, selected when initialized by the driver/controller.
geant90 said:
So I ran a DD test with a single RAIDZ1 Logical Array with 3 Physical WD Black 2TB SATAII 7200RPM hard drives.
I created the array directly on the onboard SATA ports, transfered the file and then rebooted.
Ran the DD test reboot and then the JMB321 aswell as the AD5SAPM (Through a single SB850 SATA port).
Here are the results from fastest to slowest. The AD5SAPM should be FIS (claimed by website description but I'll wait until UhClem can confirm that) which is strange.
I did run multiple test but instead of average I just used the best result as there preety tight results.
If your wondering I did seperate files which would not use buffer.

Command: time dd if=/mnt/RAIDZ/123456789 of=/dev/null bs=256k count=4096

SB850 (NO PM) :05.169170 secs (207720349 bytes/sec) 198.09 MB/s
JMB321 :07.873807 secs (136368828 bytes/sec) 130.05 MB/s
Addonics AD5SAPM :11.748918 secs (91390698 bytes/sec) 87.15 MB/s
Click to expand...
If the AD5SAPM was in FBS mode AND being controlled properly (by controller/driver), you should have seen 150-160 MB/s (for a 3-drive Z1). The fact that you only got about half that, suggests that you are very unlikely to see FBS working correctly on the SB850 under BSD; ie, don't count on it, nor make further investments.
[Note: buffering will never make things slower; only faster.]

Also, I don't believe the JMB321 number (130+ MB/s). Too high for a 3-drive Z1 in a CBS PM.

It is crucial to be able to get accurate test timings, in order to make correct assessments. This can only be done if you eliminate buffering effects. BSD (and ZFS) make this difficult/impossible. There is one possible solution I came across (credit Stefan Esser [link]), but haven't tested. You can try it out by doing:
Code: 
time dd if=/mnt/RAIDZ/123456789 of=/dev/null bs=256k count=4096
time dd if=/mnt/RAIDZ/123456789 of=/dev/null bs=256k count=4096
pushd /mnt/RAIDZ
umount /mnt/RAIDZ
popd
time dd if=/mnt/RAIDZ/123456789 of=/dev/null bs=256k count=4096
The 2nd test should be too fast. If the 3rd test time is believable, then the hack works, and you should use it for all future testing.
UnClem I saw your post on another forum. How can I run a random access sector test?
Click to expand...
??? I don't use BSD; search, or ask in a BSD forum. [I wrote my own benchmarking programs; they're not available.]

--UhClem
 
Well I tried another PCIE SATA host adapter. I will also try a completely different build featuring the ud3 990fx and its southbridge. Those commands did work and clear cache and the more realistic and happy results with the JMB321 I got in a 3 Drive RAIDZ array= 155MB/s
 
Also if it matter I did try using the drives individually as UFS and file transferring to 3 shares/drives at the same time with no slow downs and it shared the bandwidth. How ever its not like I can tell between the flickering if there taking turns but over gigabit the SMB transfer speeds dropped to about 30+ for all 3 preety much saturating GigE
 
geant90 said:
Those commands did work and clear cache
Click to expand...
That's good; will make testing useful/dependable.
and the more realistic and happy results with the JMB321 I got in a 3 Drive RAIDZ array= 155MB/s
Click to expand...
That's terrific!! [I am so pleased to be wrong.:)]
All of my testing/experience with CBS was with the SiI3726 (on three different controllers; all with similar results, such as you saw in that other forum posting.)

That means that the JMB321 is capable of achieving a throughput of ~240 MB/s (for sustained transfer; no seeking); about twice what I could get from the 3726 in CBS.

Note: 240 =~ (155 / 2) * 3
 
You must log in or register to reply here.
Back
Top