Intel SSD in RAID0 benchmarks

S

sub.mesa

2[H]4U
Joined
Feb 16, 2010
Messages
2,508
A thread you want to click! Yes i love them too. :)

Alright, so i ordered 5 Intel X25-V 40GB SSDs, put them in the server and created a RAID0. I don't have too much time at the moment, but i wanted to share my first benchmark:

Code: 
RAIDTEST on /dev/stripe/intelstripe - testing random read/write performance
concurrency     performance in I/O's per sec.   average
1 (read)        3691    3769    3794            3751
2 (read)        6702    6712    6709            6707
4 (read)        10716   10683   10667           10688
16 (read)       17173   17229   17206           17202
64 (read)       19274   19308   19260           19280
128 (read)      19262   19322   19302           19295
256 (read)      19312   19312   19335           19319

This benchmark is testing with 4KiB - 128KiB request size; so average a little above 64KiB. You can calculate the MB/s by multiplying the average request size with the IOps score. So our calculation would be (<iops> * 64) / 1000 = MB/s:

1-queue = 240MB/s
2-queue = 430MB/s
4-queue = 684MB/s
16-queue = 1100MB/s
64-queue = 1233MB/s
128-queue = 1234MB/s
256-queue = 1236MB/s

As you can see, it scales up to 64-queue where the maximum throughput is already achieved. This is not even sequential read; this is random read over the entire 'surface'. Though with a high queue enough it is just as fast as sequential read. I will post 512-byte and 4K random read scores later. Probably will make a nice article with lots of benchmarks. The fun thing is; benchmarks take alot less time. That's also dangerous as the scores fluctuate more if it runs shorter.

Hope you guys like this teaser - more to come! :cool:
 
I have been waiting for this since reading your post a few days ago, I got 3 of these drives for my new build but haven't had time to RAID or test them yet.
 
That would make one hell of a SQL store :D

When you have some time I would be curious to see some Write speeds (or are you using it for read cache or something?)
 
Sorry for lack of updates - i'll surely post alot more benchmarks. However, this takes time and currently i'm busy with work; i can only do this in my free time. However, feel free to suggest specific benchmarks or questions. :)

@shadow: i'm using the onboard nVidia 750a SLI 6 onboard AHCI-enabled SATA/300 ports. All 5 Intel SSDs are on this controller.

@Tau: i am using them for several purposes:
- central storage (NFS) for application data
- iSCSI images for my 5 workstations (including ability to use snapshots)
- read cache device for my huge RAID-Z arrays

In actual use it will see very little writes, about 98% read 2% write i think.

I will be using only 4 of the 5 SSDs for real - one SSD i will give to a relative; but i'll keep it for a week or so to do some benchmarks with 5 SSDs as well; just to see how well it scales. And it does scale! However, some benchmarks put alot of load on my CPU; and my humble AMD 2.0GHz quadcore (9350e - 65W first gen quadcore) gets 100% load when reaching 70.000 IOps. Other benchmarks pushed beyond that, though i can imagine if the disk is that fast, the CPU must keep up with actual processing of that data, too. Though really most stuff is just pushed to the network. I will be upgrading the server to something fancy when i think i'm ready for it; i hope for a nice 6-core + 32GB RAM + 10 gigabit ethernet; but prices will have to fall before i'll make that jump; it's pure luxery for me, and enjoyment. :)

I did some actual in use write tests by copying over some large data stuff from my old SSD to the new battery of four SSDs. The individual SSDs do 44MB/s write; clearly seen with gstat as ZFS commits the journal and puts 100% load on the SSDs. The old SSD was the bottleneck, with 140MB/s average read, and of course 140MB/s written to the SSDs. As expected, the write scales perfectly with RAID0; as this really is a RAID0 of four SSDs.

So far, i'm quite excited. I do think i would need a faster CPU to properly benchmark the power of 512-byte random read - it should get way over 70.000 IOps with the RAWIO benchmark that i used; but since it spawns so many processes (up to 256) it is really playing a huge overhead that is quite artificial i think. I haven't used all benchmark suites yet; i will post more benchmarks in a few days. :)
 
Awesome. I'm looking forward to seeing your results. Looks like you were lucky to get those Intel X-25V's. As of right now, I can't find a listing for them anymore on tigerdirect or newegg. Not even an out of stock listing. It may be awhile before they get more.
 
Heh - I even considered buying 8 of them, as i could get them rather cheap (under most webshop prices). But really i don't need it; it would only serve as fulfilling my appetite for benchmarking. :)

So i got 4 instead; figured i would upgrade when the newer Intel 6Gbps come out and still use these for other purposes (i.e. as dedicated ARCL2 cache). Really what i need most is more network bandwidth. I'll be trying to use lagg to team several cheaper gigabit cards and see what that gives me in total throughput. As the 5 workstations do I/O at the same time it would be nice if that all translates to at least 100MB/s per client; thus 5 gigabit NICs either dedicated or teamed with lagg driver. Surely it won't scale 100% - but right now it's like 20MB/s per workstation when the server has to push ~100MB/s over gigabit.

I used lagg before on older FreeBSD 7.x, but performance was bad due to one NIC being on PCI controller instead; that translates to higher latencies which prevent the lagg driver from scaling nicely, at least in my case. All this wouldn't be necessary if 10GBaseT NICs dropped to $100-150, of course.
 
Nice man, there was a big sale on those things on the retail side of one of the companys I do support for... $90 each CDN... I was tempted to pickup a few.
 
sub.mesa said:
Heh - I even considered buying 8 of them, as i could get them rather cheap (under most webshop prices). But really i don't need it; it would only serve as fulfilling my appetite for benchmarking. :)

So i got 4 instead; figured i would upgrade when the newer Intel 6Gbps come out and still use these for other purposes (i.e. as dedicated ARCL2 cache). Really what i need most is more network bandwidth. I'll be trying to use lagg to team several cheaper gigabit cards and see what that gives me in total throughput. As the 5 workstations do I/O at the same time it would be nice if that all translates to at least 100MB/s per client; thus 5 gigabit NICs either dedicated or teamed with lagg driver. Surely it won't scale 100% - but right now it's like 20MB/s per workstation when the server has to push ~100MB/s over gigabit.

I used lagg before on older FreeBSD 7.x, but performance was bad due to one NIC being on PCI controller instead; that translates to higher latencies which prevent the lagg driver from scaling nicely, at least in my case. All this wouldn't be necessary if 10GBaseT NICs dropped to $100-150, of course.
Click to expand...


Thats actually the same plan with the zfs box I have been playing with.... except I was figuring on 4x gigE lines should be enough, then just stagger the backups.
 
By the way; if we compare the scores with HDD we get:

SSD random read IOps qd=1: 3750 = (33.5x faster than HDD)
HDD random read IOps qd=1: 112
HDD RAID0 random read IOps qd=1: 118 (5% faster than HDD)

SSD random read IOps qd=64: 19300 (142x faster than HDD)
HDD random read IOps qd=64: 136
HDD RAID0 random read IOps qd=64: 256 (88% faster than HDD)

This is light random I/O as the average request size is 64KiB (mixed 4KiB - 128KiB). So the difference in performance (33.5x - 142x) with HDD will get bigger as the request size goes down. I still have to do proper random read 4K tests without bottlenecking my CPU. That test really would show an extreme difference between HDD and SSD.
 
I will also compare with the older OCZ Core2 (JMicron) SSD, which i used previously in RAID0. I will replace these and retire them; for usage elsewhere. But first i would like to know howmuch faster the Intel is than the OCZ Core2 SSD, especially regarding random I/O.
 
I'm setting up the storage system now, copying all my data over and configuring everything.

Then i tested performance over NFS mounted with ordinary onboard non-intel gigabit network to the server Intel NIC. The read performance of a 2.9GB file was 101MB/s measured with dd. That's higher what i got when using HDDs; about 90-95MB/s.

Will keep you guys informed of my progress. :)
 
sub.mesa said:
I'm setting up the storage system now, copying all my data over and configuring everything.

Then i tested performance over NFS mounted with ordinary onboard non-intel gigabit network to the server Intel NIC. The read performance of a 2.9GB file was 101MB/s measured with dd. That's higher what i got when using HDDs; about 90-95MB/s.

Will keep you guys informed of my progress. :)
Click to expand...

i hate you :(


im picking up Intel NICs from the shop today to play with to see if i can get CIFS up to 100MB/s
 
I never seen CIFS go that high; certainly would like to hear your results!

Will do more RAID0 benchmarks tomorrow; for now i'm going to setup this beast.

I've created BSD labels (like "fdisk" partitions but BSD-style) on the SSDs:
20GiB = for ZFS storage (both central data & iSCSI zvol images)
10GiB = for cache device for my huge RAID-Z
7GiB = remaining capacity (unused; zeroes)

Total = 37GiB = 40GB (1024 vs. 1000)

So for each SSD i sliced it in three. Then put a zpool RAID0 on all first chunks of each SSD; so 4 x 20GiB = 80GiB. The chunk for L2ARC cache is unused at the moment; will use it first to benchmark my SSDs.

So even though the SSDs are in use by ZFS, i can put conventional RAID0 with geom_stripe on the second chunks of each SSD (or only 2) and do benchmarking. Once i'm done, i write zeroes to that chunk and use it for real as cache device (RAID0-ed).
 
Just wondering if Intel has released drivers to support TRIM in raid yet? I haven't been keeping up to date.
 
Won't help you if you're on Linux or BSD; the Intel drivers are Windows drivers.

Both FreeBSD and Linux support TRIM on the device level (BIO_DELETE), but filesystem support may not be implemented. At least ZFS doesn't support TRIM yet and it would be difficult to do so due to ZFS design. UFS may already have TRIM; though i haven't checked that. Don't know details on the Linux filesystems either.

But you shouldn't be waiting for Intel; its something the filesystems have to implement.

I just make sure i leave the 7GiB per drive unused, that would be enough to keep it from degrading indefinitely.
 
Alright some days passed, still haven't done much benchmarks. But this one is nice:

iSCSI-on-ZFS-SSD.png


This is from my Ubuntu workstation, benchmarking its system disk which is an iSCSI volume mounted over the gigabit network on my FreeBSD server running ZFS + SSD + ZVOL.

Some small dips can be seen, but 98.9MB/s is very nice considering i'm using regular onboard gigabit and not a high-end Intel card. The access times are pretty low even though they are over the gigabit network. Locally there are under 0.1ms; but i settle for an average access time of 0.4ms over the network - not bad at all!

Right now i'm running 4 of my 5 workstations this way; on the SSDs in RAID0. This is what it looks like on the server-side:

Code: 
[root@mesa ~]# zpool status ssd
  pool: ssd
 state: ONLINE
 scrub: scrub completed after 0h2m with 0 errors on Sat May  8 15:22:36 2010
config:

        NAME             STATE     READ WRITE CKSUM
        ssd              ONLINE       0     0     0
          label/intel1a  ONLINE       0     0     0
          label/intel2a  ONLINE       0     0     0
          label/intel3a  ONLINE       0     0     0
          label/intel4a  ONLINE       0     0     0

errors: No known data errors

[root@mesa ~]# zfs list -r ssd
NAME             USED  AVAIL  REFER  MOUNTPOINT
ssd             40.1G  38.2G    21K  /ssd
ssd/V2             5G  39.6G  3.58G  -
ssd/V3             5G  39.6G  3.61G  -
ssd/V4             5G  39.7G  3.50G  -
ssd/V5          8.10G  42.4G  3.42G  -

Some other stuff hidden, the V2-V5 are the disk images used for iSCSI. So each of these actually contain a partition table and Ext4 filesystem that Ubuntu uses; but this is not directly visible on the server. Technically you could mount the filesystem on the server, though it should not be used at the same time with write-access enabled or you risk corrupting the filesystem. The REFER is what is actually being stored; 3.5GiB per workstation.

So far, i'm extremely happy with performance. Basically it feels like i have an SSD in each my workstations now; while in fact there are no local disks at all and the machines do everything over the network. I will be making a detailed howto on this setup; just to see if other people would want to run something like this, too. The snapshotting on system disks is great; snapshot it then do an update. If it ruined your system then simply rollback the snapshot on the server and reboot. Up and running in no-time. :)

As i continue setting up and migrating my work i'll have more time for benchmarks done locally. Still might take some weeks for me to put them online.
 
No, TRIM is not supported for SSD drives as part of a RAID array, even with the current firmware.

hhara said:
Just wondering if Intel has released drivers to support TRIM in raid yet? I haven't been keeping up to date.
Click to expand...
 
tanin ,,,,,that has been disproven. google around and you will see....it is only effective for non-member drives in raid mode. so if the drive is actually active in raid it will not work.
 
oh good to know.. so its better to buy 1 big ssd than raiding it currently then?
 
depends how devoted you are to uber speed. if you want to break the array every few months and reimage it, you will be fine with intels.
or you could get some of the OCZ line that have advanced garbage collection (like my 8 vertex for example) which is basically trim that works in raid. runs autonomous of the operating system as well, so you can use it with any OS. idle once a week for one hour and your set!!
 
so if i go wiht the OCZ with GC all u need to do is idle? So if i leave my pc on overnight not doing anything it will be good?
 
You must log in or register to reply here.
Back
Top