Need Advice on my Setup - 24 disk DAS

Morazl

[H]ard|Gawd
Joined
Jun 13, 2003
Messages
1,147
I recently came into a ton of free hardware, which I will list out to get started so you know what we're working with.

Server:
HP ProLiant DL385
Single 2.8GHZ Opteron
4GB of RAM
2x 74GB 10k SCSI Disks for the OS (Mirrored)
Windows Server 2003 64-bit R2

Add-in Controller Cards:
Silicon Image SiI 3124 SATALink Controller (4x eSATA ports)
Silicon Image SiI 3124 SoftRaid 5 Controller (4x eSATA ports)
Smart Array 6i

Disk Enclosures:

2x Norco-4104 12-bay SATA disk enclosures

One has 12x 500GB SATA disks
The other has 8x 500GB SATA and 4x 1TB disks

What do I want to do with this?

All I want to use it for is torrenting and LAN video streaming (planning on using Tversity for that)

I'm making this post because I'm not sure what the best way to configure these volumes/cards, or if I even have the right cards that I'd ideally want. Now I'll give you some more detailed information about what I'm currently seeing.

Right now the server is hooked up to only one enclosure, through the SoftRaid5 Controller. I need 4 more eSATA cables to hook the other one up, so those are on the way.

When I go into disk management, this is what I see:

diskmanagement.png


Note: The 273GB disk is something I intentionally brought offline, in order to save a little power ;)

When I go into the properties of the SoftRAID 5 card, it detects all 12 disks:
softraid5.png


I've been into the RAID config on the Silicon Image card, and it really doesn't have many options - as well as shows some weirdness. It only ever shows 4 disks, so I ended up configuring it with no RAID in the Silicon Image BIOS. However, when I booted up as you can see, it shows 2x 1.5TB volumes as well as a single TB volume, is that how it's supposed to be? Also, I've ran hardware diagnostics on every single drive and weeded out the ones with issues, so that can be totally taken out of the equation.

Ideally the way I want it to work would be 2 volumes in Windows, one for each enclosure, and to be using RAID 5 for redundancy. I don't plan on backing this up, so I'm going to rely on RAID to be my backup (yes I know, RAID isn't a backup - but it's better than nothing?)

Oh and, this wouldn't be complete without a few pics..

front-2.jpg

back.jpg
 
correct me if I am wrong, but they should be infiniband connectors on the Norco.... not esata connections shouldn't they?
 
Silicon Image is the worst possible RAID setup, in my eyes. Wouldn't a resetup of your system/disks with something like FreeNAS make sense? That way, you can still use Silicon Image as you won't be using their crappy RAID drivers.

All this port multiplication likely will lower the speeds of your whole array to something like a single disk. Could you post an AS SSD and CrystalDiskMark benchmark screenshot?

I also don't see a central question in your post; anything in particular you want help or advice with? I'm assuming you have no backup of your data and you are using Silicon Image RAID as only protection for your data?
 
@stanza I just took a look again at my Newegg link - which says it's a DS-1240 and it's definitely not. It uses eSATA II cables. My bad for not catching that.

@submesa Yeah I'm sure these cards are total crap - as I said everything was free but originally purchased/setup by a half-assed IT company. I need to read up on FreeNAS, heard of it but have no idea what it is/does. I don't need huge speeds at all so that is not a concern - I would be satisified with even single-disk speeds. This thing will just be downloading torrents and streaming over LAN, which my desktop in sig is currently handling with a non-raid SATA disk setup without problems or performance concerns.

My central question is why are my disks showing up how they are? Are they showing up correctly? What is the *best* way to setup them up given the current hardware and the purpose that I want to use them for. I'm not saying I won't spend any money here, as long as it makes sense to purchase some different controller cards. Currently all my data resides on my desktop and that will be my backup until I get something figured out with this setup, and yes I'm using RAID as my "backup" in addition to my desktop.

Does that make sense? Thanks again for your reply.
 
FreeNAS looks good, my only question is - does this run as the operating system? I need a uPNP streaming program to serve the files that will be on there. Is it possible to do both? (I use Playback on OSX and Tversity on Windows)
 
Apparently the drivers are unable to do >2TiB arrays, so you will be limited to separate <2TiB arrays. Though write performance is bad indeed, my prime concern is with reliability; i expect such Silicon Image arrays to lose all data within 2 or 3 years of operation. Not due to disk failure; but due to design weaknesses/bugs in the drivers. It's very low-quality RAID that shouldn't be used for anything even remotely important.

On the other hand, you can opt to not use those Silicon Image RAID drivers if you choose a non-Windows OS like FreeNAS (which is freeBSD under the hood) or OpenFiler (based on Linux) or Nexenta (based on OpenSolaris). In this situation, Silicon Image would act as SATA controller ("HBA") only, so you get raw disk access. That means you will see each disk with its original capacity; just like without RAID. Now you can use software RAID to create a RAID array.

If you want RAID5, perhaps you should consider using ZFS instead. It works well with FreeBSD, Opensolaris, and some web-GUIs as well: Nexenta, FreeNAS (has issues with ZFS) and my Mesa web-interface.
 
ZFS is looking very very attractive. I have a FreeBSD box that I use as a webserver right now, but doesn't have any controller cards in it. My only concern is being able to stream video from it to PS3/Xbox 360... now how would I do that when I'm on FreeBSD or some other open source OS?
 
The same way you'd use your FreeBSD box for any other server, really.

I little bit of searching found this port which should do the trick: mediatomb. I don't use it, however, and I'd suggest that you read through the docs to make sure that it does what you want it to.
 
Awesome... my buddy setup the FreeBSD box and I don't know hardly anything about it. I'll have him setup mediatomb and we'll give it a shot. So my Silicon Image cards will be fine on FreeBSD with ZFS? (as far as reliability goes)
 
Last edited:
I would just plonk WHS on it, and setup the drives as NON raid.

That way you will end up with a bucket load of space... and some reliability (WHS folder duplication for important data)

ZFS will be slow as a dog with sil3124 cards and only 4gig ram

Or if you want to spend a little dollars

Grab yourself a cheap SAS card with external connections (say a Dell SAS 5/e or similar). And replace the e-sata connectors for the ones you originally had linked (infiniband)

then your choice opens up

Add more Ram and go for Opensolaris / FreeNAS etc
Leave the Ram and install WHS and it will be a lot faster than the sil3124 cards.
Keep your 2008 server idea and get a RAID card with external connections to the HDD Arrays
 
I was under the impression that if he got a bit more RAM (perhaps 6 GB - 8 GB), ZFS would run just fine, assuming he didn't go off and throw all the disks into the same raidz vdev. The port multiplication seems to be the biggest performance issue. Either way, WHS would have the same performance issues (in fact, I have a hard time believing that WHS would have better IO performance than ZFS - doesn't WHS just "extend" the disks, so the performance is limited to that of a single drive?)
 
I was under the impression that if he got a bit more RAM (perhaps 6 GB - 8 GB), ZFS would run just fine, assuming he didn't go off and throw all the disks into the same raidz vdev. The port multiplication seems to be the biggest performance issue. Either way, WHS would have the same performance issues (in fact, I have a hard time believing that WHS would have better IO performance than ZFS - doesn't WHS just "extend" the disks, so the performance is limited to that of a single drive?)

correct whs just spreads the data across several drives

zfs will (depending on zfs config level) write to several drives AT ONCE and stripe the data.
Which on the sil3124 type cards will kill performance

and AFAIK opensolaris doesn't have proper support for these cards. Freenas will, but then Freenas has config issues with ZFS (via the web interface) that is when you make say a raidz config, then want to add to it... you need to use the command line.... else troubles will appear

Don't forget you are talking port MULTIPLICATION here.... not several ports each with a drive attached.... so whatever speed you would get from RAIDing say 4 drives together.... will not happen, as you are talking to 4 drives down ONE port.....ie 1 port (that would normally talk to one drive... is now trying to talk to 4 drives....and if the drives are striped/RAIDed...then thats trying to talk to 4 drives at once down a port thats designed to talk to one drive.

Then when you get to command cue depths and such (NCQ etc)....thing go funny.

imagine say a software RAID.
Now this software raid writes / reads to several drives normally and the drive can queue up requests to a certain level....process the requests and hand them back to the (fake) controller "software raid"

which can get a bit busy when all those requests (for 4 drives) are passing down a cable designed for one drive;)
 
Stanza, really good points. I'm going to do some more thinking about this. My buddy is going to come down and we're going to see if we can get Free BSD going with ZFS and mediatomb - just to see how it works and if the performance is good enough. I'll certainly take your thoughts into consideration - it makes sense.
 
Last edited:
Update:

So far we go FBSD 8.1 installed and have attached the two 12 disk port multipliers with the ZFS filesystem (raidz1). Ended up using a different server, has 2x P4ish Xeon's @ 2.4GHZ with 4GB of RAM. Performance is much better than expected - it's hitting the disk speed limits. It's configured as one big pool, getting 70-100mb/s burst and on average 50mb/s via FTP protocol. Took a bit to figure out but SFTP has a bunch of overhead and could only do 22mb/s by using the Transmit SFTP client on OSX. Started copying 1.5TB to it last night and it's going smoothly!

Next step: Mediatomb setup!

gstat

Code:
dT: 1.070s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| fd0
    0      0      0      0    0.0      0      0    0.0    0.0| acd0
    0      0      0      0    0.0      0      0    0.0    0.0| da0
    0      0      0      0    0.0      0      0    0.0    0.0| da1
    0      0      0      0    0.0      0      0    0.0    0.0| da0a
    0      0      0      0    0.0      0      0    0.0    0.0| da1s1
    0      0      0      0    0.0      0      0    0.0    0.0| da2
    0      0      0      0    0.0      0      0    0.0    0.0| da3
    0      0      0      0    0.0      0      0    0.0    0.0| da4
    0      0      0      0    0.0      0      0    0.0    0.0| da5
   11    110      0      0    0.0    110   9349  120.7   97.8| ada0
   10     99      0      0    0.0     99   8532  127.6   97.2| ada1
   11    106      0      0    0.0    106   9010  117.7  104.5| ada2
   10    108      0      0    0.0    108   9189  123.6   97.1| ada3
   13    104      0      0    0.0    104   8989  128.7  103.1| ada4
   11     94      0      0    0.0     94   8110  108.5  102.4| ada5
   11     94      0      0    0.0     94   8110  105.3  101.9| ada6
   12     96      0      0    0.0     96   8322  117.4   94.3| ada7
   10    105      0      0    0.0    105   9040  128.8   95.8| ada8
   11     96      0      0    0.0     96   8428  133.5   91.4| ada9
    0    169      0      0    0.0    169  14463   58.2   81.4| ada10
    1    178      0      0    0.0    178  15217   48.3   85.7| ada11
   11     93      0      0    0.0     93   7950  118.2  100.2| ada12
   12    103      0      0    0.0    103   8960  107.7  101.8| ada13
   10    103      0      0    0.0    103   8960  116.5  104.4| ada14
   10     96      0      0    0.0     96   8468  122.7   95.7| ada15
    9     96      0      0    0.0     96   8389  115.6   95.0| ada16
    0    164      0      0    0.0    164  13955   56.0   81.7| ada17
   11    101      0      0    0.0    101   8589  124.6   95.6| ada18
   11    109      0      0    0.0    109   9491  127.0   98.8| ada19
   10    108      0      0    0.0    108   9451  118.8  104.8| ada20
   11    107      0      0    0.0    107   9268  125.9   97.1| ada21
   13     99      0      0    0.0     99   8509  136.4   97.4| ada22
    0    163      0      0    0.0    163  13864   55.9   78.8| ada23
    0      0      0      0    0.0      0      0    0.0    0.0| ufs/root
    0      0      0      0    0.0      0      0    0.0    0.0| da1s1b
    0      0      0      0    0.0      0      0    0.0    0.0| da2s1
    0      0      0      0    0.0      0      0    0.0    0.0| da3s1
    0      0      0      0    0.0      0      0    0.0    0.0| da5s1
    0      0      0      0    0.0      0      0    0.0    0.0| da2s1d
    0      0      0      0    0.0      0      0    0.0    0.0| da3s1a
    0      0      0      0    0.0      0      0    0.0    0.0| da3s1b
    0      0      0      0    0.0      0      0    0.0    0.0| da3s1d
    0      0      0      0    0.0      0      0    0.0    0.0| da3s1e
    0      0      0      0    0.0      0      0    0.0    0.0| da3s1f
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/4c6d7089a85f44b5
    0      0      0      0    0.0      0      0    0.0    0.0| da5s1d
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/4af370abc7d93b62
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/4af370cfd9ee120f
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/4ad90b3ce0892f29
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/4af4725f7c2bcba3
    0      0      0      0    0.0      0      0    0.0    0.0| ufsid/4c6d7089a85f44b5d
 
Back
Top