Interesting cheap JBODs available

@markL, if your link jumped from 4-wide to 8-wide that should be the dual-linking that we are looking for, you didn't notice any increase in bandwidth with the connection set up that way?
 
Sorry, I am not talking about these JBODs.. I was meaning what I see with my LSI and a different expander..

But I really want to get a few for work but I want to confirm everything is ok with LSI controllers because that is what I will be using them with. So kinda waiting on Packetboy :)
 
ah sorry, i misunderstood. Well i would need to tear my machine open but i can test it out with a 1068e based LSI HBA. The only problem is i'm not exactly sure where i can check the number of active links between the HBA and the expander, i can see the negotiated link speed, but not number of links.
 
Mr.J said:
arc-1880x_with_se3016-sas.png

arc-1880x_with_se3016_sas-info.png


No, I didn't use 2 cables as the labels on the enclosure says one of the "In", and the other is "Out". I had assumed it's for daisy-chaining. Let me know if this assumption is wrong.
Click to expand...

There is your problem! Your card is only running at x4 PCI-E (1.0) at that. You need x8 @ PCI-E 2.0 to get the full speed of the card (2+ gigabytes/sec on read) which is PCI-E 1.0 @ x4 will definitely limit you to ~500 MB/sec. Nothing to do with the expander.

Likely they label them like that but on most SAS expanders both can be used if you want. If you hookup two cables (well if you have them) can u see if ur links on SAS information changes from 4x3G to 8x3G? That chip is the same thing used in the HP SAS expander so probably compatible with areca.

Also assuming you fix the PCI-E issue with only one cable I think its likely you will bottleneck again at around 1 gigabyte/sec due to the fact its a 3 gigabit link and not 6 gigabit. Two connections should net you around 2000 megabytes/sec.
 
Packetboy - read the previous page... we already figured that have just labelled the product wrong on Ebay and it is only a 3G expander.
 
The 500MB/s limit was due to my pci-e bus. I had it at a physically 8x slot, but actually only runs at 4x. I moved the card to another slot (16x physical runs at 8x), and got a speed improvement to 650MB/s. I was hoping for a 2x improvement, but looks like only a 30% improvement. Looks like I now need to hunt for a different system/motherboard that has PCI-E 2.0.

arc-1880x_with_se3016_sas-pcie8.png
 
Mr.J said:
The 500MB/s limit was due to my pci-e bus. I had it at a physically 8x slot, but actually only runs at 4x. I moved the card to another slot (16x physical runs at 8x), and got a speed improvement to 650MB/s. I was hoping for a 2x improvement, but looks like only a 30% improvement. Looks like I now need to hunt for a different system/motherboard that has PCI-E 2.0.
Click to expand...

Is that still R0 or R6? 650mB/sec from 8 drives seems pretty good to me.. what are you expecting?
 
x8 PCI-E should be able to get in the 800-900 megabyte/sec range so that shouldn't be the limiting factor. Your old version of hdtune is only oging over the first 2TiB of the array though so u cant see if speeds drop down near the end as things get slower on the drives. Can you try something else like hdparm or a newer version of hdtune?
 
The previous screen shot (post #171) was R0. Attached below is R6 with 8 drives, from a newer version of HDTune. I don't know what to expect, that's the reason that I am trying to get some of you guys to post your performance stats. This is the first time I am using a SAS expander with a new card (arc-1880x). I had been using an internal Areca card with 8 drives running R6, and getting about 350MB/s.

READ:
arc-1880x_with_se3016-sas_raid6_8drives-read.png


WRITE:
arc-1880x_with_se3016-sas_raid6_8drives_write.png
 
Well from the curvature of the graphs it does appear to be peaking in the top 30% or so. Doesn't look necesarily like the card or exapnder is bottlenecking but hard to say without more disks.
 
Something is definitely limiting the throughput, but I am not able to pinpoint what it is just yet. Added 4 more drives to the array and put it back on R0. Throughput is still around 650MB/s for read, and 770MB/s for write. I am initializing the 12 drive array to R6 and see if it's the card has any impact on the performance or not (I don't think so as we are looking at R0 here). Also from looking at the curvature of the graphs, the drives don't seem to be a limiting factor now. I think I am expecting to see results close to what packetboy was getting in post #131.

READ (R0, 12 drives):
arc-1880x_with_se3016-sas_raid0_12drives_read.png


WRITE (R0: 12 drives):
arc-1880x_with_se3016-sas_raid0_12drives_write.png
 
Last edited:
OK, Got my chassis hooked up. Since i run my boxes in the storage closet in my room it is quite loud. would it mess with the firmware if i ran some quieter (but larger, cut out room for 140MM fans) slower RPM fans? those 80MM delta fans scream :s
 
Can the expander in this be replaced with a 6Gb/s expander? If so, which model would fit?

I'll still on the fence about getting one of these due to the performance numbers, but I do love the number of drives it can present.
 
brianmat said:
Can the expander in this be replaced with a 6Gb/s expander? If so, which model would fit?

I'll still on the fence about getting one of these due to the performance numbers, but I do love the number of drives it can present.
Click to expand...

I believe the problem with be powering the expander. Then 6G Chenbro expanders may fit physically, but it depends on the PSU outputs if you could connect it.. Same thing with the HP expanders.

May I ask - why do you need 6G? You're never really going to need more than the 800mB/sec people are seeing... and once you use it in 'real world' cases, you probably won't even hit 800mB/sec the same anyway..

The only time it would matter is if you were daisy chaining a bunch together... but if you were doing that, you would have to be using 6G drives.. and if you can afford 48+ 6G drives for storage.. I think you can afford more than $300 on a chassis..
 
MarkL said:
I believe the problem with be powering the expander. Then 6G Chenbro expanders may fit physically, but it depends on the PSU outputs if you could connect it.. Same thing with the HP expanders.

May I ask - why do you need 6G? You're never really going to need more than the 800mB/sec people are seeing... and once you use it in 'real world' cases, you probably won't even hit 800mB/sec the same anyway..

The only time it would matter is if you were daisy chaining a bunch together... but if you were doing that, you would have to be using 6G drives.. and if you can afford 48+ 6G drives for storage.. I think you can afford more than $300 on a chassis..
Click to expand...

Good question, and I guess the only valid answer is to determine whether the drive cage can really be upgraded going forward or not. I am looking to throw some SSDs in there for our SQL boxes at some point since the MD3000i doesn't support any SSD devices but even then we really will not be saturating any connections anyway.
 
Hive said:
OK, Got my chassis hooked up. Since i run my boxes in the storage closet in my room it is quite loud. would it mess with the firmware if i ran some quieter (but larger, cut out room for 140MM fans) slower RPM fans? those 80MM delta fans scream :s
Click to expand...

based on my observation:
140mm fan will not fit!...
120mm fan will fit perfecty

for 120mm you can use 4 1/2" Steel Hole Saw Drill Bit as I did :)
I installed 120mm dual bearing fan with ~2000-2600rpm (if my memory serves me well hehe).
 
brianmat said:
Good question, and I guess the only valid answer is to determine whether the drive cage can really be upgraded going forward or not. I am looking to throw some SSDs in there for our SQL boxes at some point since the MD3000i doesn't support any SSD devices but even then we really will not be saturating any connections anyway.
Click to expand...

Oh - I didn't realize you were looking at this for work. Make sure you note that there is no redundant PSU on this. I'm going to probably look at getting two for work and running a mirror with each side in a different chassis to give some faux PSU redundancy..
 
As an eBay Associate, HardForum may earn from qualifying purchases.
I've ordered 8 SFF 8087 to SFF 8088 cables from ebay merchant discussed here a few days ago...$26 a piece seems like a great deal...of course they are coming from Hong Kong so who knows how long that'll take.

I also broke down and bought an LSI SAS6160 SAS2 Switch...I'll temporary disconnect the drives from the SAS expander in these enclosures and wire the drives directly to 8 ports on the SAS6160 (which will negoitiate to 3Gbps). Then my OI server will connect to the SAS switch via 2 6Gbps wide ports on the switch.

I'm hoping that by eliminating the SAS expander, my LSI HBA and LSI Sas switch will now do SAS multiplexing and I'll be able to pump upwards of 4000MB/s.

I'm probably just dreaming.
 
Hey packetboy - I was looking at those but someone else pointed out that Monoprice has them even cheaper.. >_<

Anyway - Good luck with the SAS Switch setup - will be great to see your numbers if everything works out.
 
Awesome...I needed 8 more cables anyway...so should get the monoprice ones pretty quick and be able to do some testing late next week (I'm traveling the first part of the week).
 
stevebaynet said:
Crikey! that site has excellent prices on SAS cables, just bookmarked it. thnx!!
Click to expand...

I deal with them all the Time. The only sucky thing is that I am located in California, so taxes can kill it for me, But they are still cheaper with tax. Since they are local to me, I can do pick up or Ship it cheap for basically next day.

I purchase a set of SSF-8087 to SATA/SAS Break out cable from them, and they work great for me!
 
I received my box today - still waiting on HBA and cables.

Anyone had any luck with the management port?
 
Access to the management port is why I need the manuals. If you find out how to access it, please share.
 
Sorry to zombify this thread, but it seems more of these are available (again) for any interested parties.

Ebay item number 170786488319
 
packetboy said:
I've ordered 8 SFF 8087 to SFF 8088 cables from ebay merchant discussed here a few days ago...$26 a piece seems like a great deal...of course they are coming from Hong Kong so who knows how long that'll take.

I also broke down and bought an LSI SAS6160 SAS2 Switch...I'll temporary disconnect the drives from the SAS expander in these enclosures and wire the drives directly to 8 ports on the SAS6160 (which will negoitiate to 3Gbps). Then my OI server will connect to the SAS switch via 2 6Gbps wide ports on the switch.

I'm hoping that by eliminating the SAS expander, my LSI HBA and LSI Sas switch will now do SAS multiplexing and I'll be able to pump upwards of 4000MB/s.

I'm probably just dreaming.
Click to expand...

Wondering if you had a chance to test this yet and if so, what the results were?
 
ZFSLover said:
Wondering if you had a chance to test this yet and if so, what the results were?
Click to expand...

Looks like my dreams where crushed...I got all the cables..both from Monoprixe and Hong Kong. However, when I try to bypass the expander by using these cables with the Sas Switch it doesn't recognize any of the drives. Poking through the switch manual it states you can't connect drives *directly* to the switch (e.g. I presume you MUST use an expander).
 
Emulsifide said:
The enclosure in question is not what you've listed. We're working with the SE3016 which is the older brother of the one you've posted.
Click to expand...

Hmm, I thought earlier in the thread there was mention of the 1116 and the SE3016, and they were determined to be the same, guess not?

Anyone have the 1116 model?

Thanks!
 
You must log in or register to reply here.
Back
Top