• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Storage Server Refresh...Suggestions?

S

SatanSkin

Weaksauce
Joined
Aug 16, 2004
Messages
79
I've decided to do a refresh of my current storage server. I am going to migrate my current backblaze pod over to a couple Norco 4224 boxes and am new to the SAS side of things. I have been reading through a few of the SAS expander threads and such, but a lot of them seem to be older posts and/or heavily Windows based. So here is my obligatory build thread and I'm looking for any advice or guidance you guys have.

From what I've read so far, I think I got the general idea. SAS Backplane > SAS Expander card > SAS HBA/Controller. Is that the correct flow of things?

My current setup:

Case: Backblaze Pod 45-bay
Mobo: Don't remember, Haven't pulled out the server to check, and it's offline at the moment.
CPU: AMD Phenom X2 Quad-Core something or other (it's older gear if you haven't noticed)
RAM: 8GB of whatever that board takes
OS: Ubuntu Server x64
Other: SnapRAID

I'm looking at moving the current internals over to a new Norco 4224 and using some other old gear laying around to fill the second 4224. I will be rebuilding them with new Sandy Bridge systems later as time/money/wife permits.

The new systems must have/be:
- Linux servers
- support 24 drives each
- support 3TB+ HDDs
- software RAID or SnapRAID
- spin down drives
- support SMART monitoring
- as cheap as possible without horrendous performance

I have seen/heard recommendations for the IBM M1015 flashed to IT firmware. Is there a particular one because I have seen them described as rebadged 9240-8i, 9220-8i, etc? And what exactly is the IT firmware for? Can it be flashed from within Linux or is it through a Windows based utility?

What are your suggestions for SAS expander and/or controllers/HBAs to pull this off and that will run smoothly with Linux? And while we're at it, any recommendations for easily monitoring drive health, etc in Linux? Something tells me there is something better than my scheduled SMART tests and emails. Though these are headless servers so something with a web interface or CLI based would be preferred.

Thanks guys.
 
Last edited:
Why are you moving away from the backblaze pod? Personally, I never liked the look of them (from the web articles I have read about them), but given that you already have one, you must have a better reason than "don't like it" to go to the trouble and expense of changing.

Also, why Sandybridge instead of Ivybridge?
 
For that many drives I'd suggest Xeon CPU, supermicro motherboard and ECC memory. Many, many threads on it.
 
Last edited:
JoeComp: Two reasons mainly. The first being that I never went through and properly did up a wiring harness for dual PSUs. I've been running off a single huge ass PSU and just been lucky; which I don't like or feel comfortable with anymore. The second being that about half of the storage is used for backups. I can break that half off into a second whole enclosure that can then be pretty much off most of the time. Just have it power on, sync backups, and go back off saving me a little more money on electricity. Can boredom and experience count as a third reason?

As for Sandy vs Ivy, mostly a money issue. I figure I'll be able to find cheaper parts for a SB build.

atbnet: I bought it from Protocase when they first started offering them. It was only the enclosure and I had to source out the rest. Since you bring it up, I'll probably be selling it afterward. If there's enough interest I might throw it up in the classified section.
 
Last edited:
There is only one IBM M1015 - grab one from ebay. For an explanation on IT mode firmware and why to crossflash: http://www.servethehome.com/ibm-serveraid-m1015-part-4/ Technically with linux you dont even need to crossflash since it'll deal with the ports as JBOD by default but its supposedly cleaner and provides much needed psychological contentment for the OCD crowd.

HP SAS Expander (32 ports) or Intel expander (24 ports) for the expander part

There is a lot of low cost commodity hardware these days for home oriented storage with high value-for-money ratios compared to a few years ago so its good times. Really the biggest challenge people face right now is simple stuff like which harddisks to buy because the consumer desktop drive segment has been in such flux with all the buyouts and product line changeovers and certain drives are less raid friendly than others. on the bright side linux tends to be very tolerant of some of the idiosyncrasies of certain desktop class drives like WD's that hardware raid controllers don't tolerate.

Also you might consider running ESXi as a base and virtualizing the storage instance since you'll have plenty of horsepower left for running other stuff on that box.

The correct signal flow with a Norco chassis is SAS controller -> SAS Expander -> dumb/passive Norco backplane (passthrough) -> STP (serial tunneling protocol) talking to SATA harddisk. While SAS expanders are technically daisy-chainable to additional expanders, its generally been proven more stable for our purposes not to go more than one SAS hop when talking to SATA drives. In practical terms that means you can hang two expanders off a single 8-port SAS2 controller, one expander per minisas connector.

So in my several "Norco clusters" each one consists of 2 x Norco 4224 cases, the bottom one has the main system/motherboard along with the 8-port controller (I run Areca 1880i's for hardware raid) and an HP Expander, then the top Norco 4224 is just a power supply, ATX power board and expander, with a SAS cable running between cases to link the 8-port controller with the second expander.
 
Last edited:
If you can live without having the two boxes connected, I recommend going with three IBM M1015 HBAs per box, and not dealing with the possible troubles of a SAS expander. As odditory said, the M1015 can work with linux in either IR mode as it usually comes (megaraid_sas driver) or IT mode if you flash it (mpt2sas driver).

As far as IVB vs. SB, I'm not sure there is much price difference. I'd just get the newer tech.

The Supermicro X9SCM-iiF motherboard (see thread in this forum) is an excellent server MB. If you add an Intel Xeon E3-1220v2 or -1230v2 CPU then you will probably have more power than you need, but it is always nice to have headroom! The equivalent SB CPUs (E3-1220v1, -1230v1) are only a few bucks cheaper, unless you are buying them used maybe you can find a deal.

I'd get at least 8GB of RAM (that motherboard can go up to 32GB if you need more).

I'm not clear on the relation between the boxes. Is one an exact backup of the other?
 
Odditory: Thanks for the info. I'll read up on the IT firmware there. I have another host dedicated to running ESXi so for these boxes I'm more concerned with lower power usage. It just needs enough balls to handle the storage duties and run a few web apps. Your setup does sound interesting for possible future expansion with the essentially empty boxes running as a sort of DAS.

Any specific reason to use ECC RAM?

JoeComp: Without having the two boxes connected how? I plan on them being two completely separate boxes so they won't be connected via any external SAS ports to each other. As for relation, there isn't much of one. One will house all of my media, host a few web apps and a MySQL server for HTPC clients; and the other will host backups for the rest of my network (all of my desktops, laptops, htpc's, servers, VMs etc). There will not be any backup of the media at the moment. Fault tolerance is currently handled via SnapRAID with two parity drives. Long term plan will be to eventually have a third box to sync all the media for a backup, but I can't afford that at the moment and it's "easy" enough to get my media back.

As for the IVB/SB issue, I haven't looked recently but just figured that the older tech would be cheaper and would suffice for what I need. When it is time to rebuild the box's internals, I will certainly look at both though.

Kristof: That's where I got mine. I bought it when they only offered the case bare.
 
Many reasons to use a xeon class cpu, and mobo and ecc mem when it comes to a serious home oriented storage server, can't type Em all now on my phone here but the more research you do the more you'll realize why it's become standard procedure for those of us with dozens or hundreds of drives and many TB. Especially beneficial with software raid on *nix since you're relying on cpu for calcs unlike
Hardware raid.

Patrick at serve the home . com will tell you the same thing he's reviewing these xeon class compenents constantly. We are not even talking about that much more than an SB setup. SUPERMICRO X9SCM-iiF or whatever the newest rev is, is the board you want. You're getting a shortcut from many hours of reading with that recommendation. Also less issues with storage specific pcie cards in super micro xeon class. The main people complaining in every new 'M1015 not working please help', or 'EXPANDER not working omg' thread are usually desktop class boards if that tells you anything. Storage component vendors are not testing with desktop boards and vice versa. Eventually incompatibilities get worked out or just worked around but meantime you're yanking your plank.
 
Last edited:
Tyan's S5510 is also worth a look, you get an extra NIC and XMP profile support over the Supermicro (Although XMP will be of no use if you for ECC - Although theoretically ECC sticks with XMP profiles could be made).
 
I found the X9SCM-F-O and X9SCM-O boards at newegg. Are those suitable replacements for your suggested -iiF board or is there something special about that board in particular?

I have actually been spending a lot of time on serve the home lately...
 
The X9SCM-iiF is the newest version of that board, but it is just in the process of being released and may only be available at superbiiz at the moment.

With the X9SCM-F, you risk getting a version with an older BIOS that will not work with Xeon E3v2 CPUs until you flash to a newer BIOS.

As I said, see the thread in this forum on the X9SCM-iiF. It has been active recently.
 
Thanks JoeComp. I have read through it and have it bookmarked so I can follow it. You were also right about the price difference being negligible on the IVB so I'll be going with it.

As of now, I am looking at the following for build out:

Case: Norco RPC-4224
Mobo: X9SCM-iiF
CPU: E3-1230v2
RAM: Kingston ECC KVR1333D3E9SK2/8G (or something else if that's not on the SM Compatibility list)
HP SAS Expander
IBM M1015

After reading up on it, it looks like flashing the M1015 to IT mode would just be easier even if I am running Linux. Thanks again for the help. I'll be posting pics once I actually get things ordered and built.

In the meantime, are any of you running Linux on your storage servers or are you all Windows based? If you are running Linux, care to discuss how you are managing/maintaining all of your drives and such?
 
Last edited:
Has anyone mentioned ZFS yet? Just making sure :)

when I get home I can post about my setup, in sig.
 
SatanSkin said:
In the meantime, are any of you running Linux on your storage servers or are you all Windows based? If you are running Linux, care to discuss how you are managing/maintaining all of your drives and such?
Click to expand...

I'm running linux on my server, and I use snapraid for my media files, and I have an mdadm RAID-1 on some of my non-media files (files that are either small or change frequently, and so are not optimal for use with snapraid).
 
extide: I looked at ZFS a little while back and decided it was not for me in this case. I'll try to remember most of the reasons why.
- I want as few drives up and spinning as possible at any given time
- There was something about upgrading, expanding, or replacing drives that I vaguely remember. But I didn't like that it would require multiple drives at a time to do
- I want to be able to pull the drive out and access the data anywhere else that I choose to mount it (useful for taking movies home to the parents' for the holidays)

Thanks though. I would still be interested in hearing about your setup though.

JoeComp: You doing anything special for monitoring the health of those drives? Or to simplify organization so that when one fails, you know exactly which drive to pull?
 
SatanSkin said:
JoeComp: You doing anything special for monitoring the health of those drives? Or to simplify organization so that when one fails, you know exactly which drive to pull?
Click to expand...

I monitor the health of the drives with smartctl. I wrote a script that monitors some of the raw values of SMART attributes, and also produces output like this:


Code: 
sdz R2C3 dos xfs 2.00 TB S3H9J1EZA53651 HD204UI SAMSUNG SpinPoint F4 EG (AFT)
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   100   100   051    -    0
  5 Reallocated_Sector_Ct   PO--CK   252   252   010    -    0
195 Hardware_ECC_Recovered  -O-RCK   100   100   000    -    0
197 Current_Pending_Sector  -O--CK   252   252   000    -    0
198 Offline_Uncorrectable   ----CK   252   252   000    -    0
199 UDMA_CRC_Error_Count    -OS-CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   -O-R-K   100   100   000    -    0

If the raw values of any of those are not zero, then I either replace the drive or start watching it carefully.

As for knowing which drive to pull, I apply a label to the drive when I install it. I used to only put the label in the filesystem, but since I am now adding new drives with gpt partitions, I also label the partition (recent linux kernels support partition labels as well as filesystem labels). You can list the drives by label like this:

Code: 
ls -l /dev/disk/by-label
*OR*
ls -l /dev/disk/by-partlabel (with recent linux kernels)

I have a Norco RPC-4224, with 6 rows and 4 columns of drives, so I label them r1c1, r1c2, ...r6c4
 
Last edited:
Hmmm... I'm running Ubuntu Server 12.04 with a 3.2.0-26 kernel and it doesn't look like I have that partlabel. Damn. That's pretty slick. I'll have to look into enabling that in my distro or just rebuilding the kernel with it. Or I can just start using them as the FS label instead.

Mind sharing your script that cuts the smartctl output to just those fields? That would sure make it easier to monitor.

Thanks.
 
SatanSkin said:
Case: Norco RPC-4224
Mobo: X9SCM-iiF
CPU: E3-1230v2
RAM: Kingston ECC KVR1333D3E9SK2/8G (or something else if that's not on the SM Compatibility list)
HP SAS Expander
IBM M1015
Click to expand...

Perfect. And don't forget 8 x SFF-8087 to SFF-8087 1.5M cables - the Norco ones on Newegg are fine. But sometimes when you get an HP Expander it'll include all those cables (just depends on the vendor) so maybe wait on those till you get the expander. Two will interlink the M1015 to the HP Expander (dual link for double the bandwidth between cards but technically one is all thats needed and plenty fast especially for a non-striped disk pool), and the other 6 will go to the Norco backplane. As for the Kingston memory I've never had a problem even if it wasn't on the HCL (HCL's are only as meaningful as how many makes/models a vendor has had time to test).

Also, if you were buying a new PSU, a Seasonic platinum doesn't suck, for extra clean power to those drives & components. People tend to go for a 700W PSU for a fully loaded Norco 4224, you could prob get away with less since you aren't striping your disks, but I like the extra headroom - all my Norco 24's have 700W but I do run mostly striping/hardware RAID (for now - slowly transitioning to non-striping)
 
Last edited:
SatanSkin said:
- I want as few drives up and spinning as possible at any given time
- There was something about upgrading, expanding, or replacing drives that I vaguely remember. But I didn't like that it would require multiple drives at a time to do
- I want to be able to pull the drive out and access the data anywhere else that I choose to mount it (useful for taking movies home to the parents' for the holidays)
Click to expand...

1. That's one irritant w/ ZFS (and hardware raid) for just storing movies & media - every disk spinning up just to access that one movie file. On the bright side, pool spindown seems to be supported now on OI + NappIt for example but its a bit mickey mouse since its not in the ZFS spec and you're setting the individual disks for spindown rather than setting it at the pool level.

2. Another irritant w/ ZFS at least for me coming from and being spoiled by one-disk-at-a-time hot expansion. ZFS vdevs don't expand, so to expand the pool you have to create an additional vdev (array) and so its costing you extra parity drives every time. essentially you're chaining multiple RAID5's or RAID6's into a RAID0. Or you can back up, reformat, recreate & restore but then you're pulling the table cloth and all the crap on it every time you want to add a few placemats.

3. There's a lot to be said for the extra benefits of non-striping in being able to remove and read a single disk elsewhere, from a recoverability standpoint. Murphy's law knows no bounds.

Bottom line, you're on the right track because the trend for home media storage in the next few years will be people moving away from striping schemes and toward non-striping schemes, and 3-4+ parity disks per 16-32 data disks won't be uncommon to see in builds since the software supports it. The software - Flexraid, Snapraid, disparity, and others just need to evolve a bit more since they're not as time tested as older striping solutions. Striping is a great performance and uptime multiplier but for simple storage & archiving of infrequently changing files it introduces unnecessary risk, technically speaking. Granted, the first big 48TB system I built 5 years ago this month with 48 x 1TB Hitachi's and 2 x Supermicro 24-bay cases and a couple of Areca controllers running everything RAID6, has has been infallible, and might disagree.
 
Last edited:
I think your #2 about ZFS was what I was thinking of in my #2. If I wanted to expand, I couldn't simply add a new 2TB drive. I would have to add 3 of them or so.

From my experience Flexraid still seems a little bit unpolished in Linux and is certainly not at a maturity worth paying for yet. From what I can tell, this is quite a different story on Windows. Snapraid has been great so far. It's very simple, straightforward, and abides by the age old *nix philosophy of do one thing great instead of many things half-assed. My only complaint is that I would love the ability to have more parity drives. Having only 2 parity drives for 20+ data disks just doesn't make me extra warm and fuzzy. I believe this is in the works though.

I will be reusing my current PSU in one of the new builds. I believe it was a 1kW PP&C PSU which I know will be quite overkill for only 24 drives, but it was intended for 46 drives. I will certainly take a look at the one you mentioned for future use though.

I have found a few M1015s and HP SAS expanders on ebay and amazon. Are there any speciality sites that stock these for really good prices that you have found?
 
SatanSkin said:
Hmmm... I'm running Ubuntu Server 12.04 with a 3.2.0-26 kernel and it doesn't look like I have that partlabel. Damn. That's pretty slick. I'll have to look into enabling that in my distro or just rebuilding the kernel with it. Or I can just start using them as the FS label instead.

Mind sharing your script that cuts the smartctl output to just those fields?
Click to expand...

By the way, you can also run `snapraid check` periodically if you want to verify the checksums on all your data (which will obviously fail if one or more of your data drives has gone bad).

Hmmm, according to distrowatch, Ubuntu 12.04 should have kernel 3.2, which I would have thought would be recent enough to have /dev/disk/by-partlabel, but I am running Archlinux with kernel 3.4. Maybe partition label support also depends on udev? Distrowatch says 12.04 only has udev 175, but I have 186:

Code: 
$ uname -r
3.4.4-2-ARCH
$ udevd --version
186
$ ls -l /dev/disk/
total 0
drwxr-xr-x 2 root root 2440 Jul 13 14:15 by-id
drwxr-xr-x 2 root root  600 Jul 13 14:13 by-label
drwxr-xr-x 2 root root  380 Jul 13 14:13 by-partlabel
drwxr-xr-x 2 root root  380 Jul 13 14:13 by-partuuid
drwxr-xr-x 2 root root 1000 Jul 13 14:15 by-path
drwxr-xr-x 2 root root  600 Jul 13 14:13 by-uuid

I did a websearch and found a Gentoo forum post that seems to be about udev 181 where /dev/disk/by-partlabel is present:

http://archives.gentoo.org/gentoo-dev/msg_1a1b7524002990f610a7b66b017917c7.xml

So perhaps you only need to update to udev 181 or better. I'm not sure how difficult that is on Ubuntu (could be side effects).

Here is my script for listing SMART attributes of all my sd* devices. If your drives are not /dev/sd*, then you will need to change that line. Obviously you will need to have `blkid` installed. My script also has a couple things that are somewhat specific to my system that you may (or may not) want to remove:

Code: 
#!/bin/bash

SCRIPT=`basename "$0"`

if [ `id -u` -ne "0" ] ; then
  echo "*** ERROR: $SCRIPT must be run as root"
  exit 1
fi

DEVS="`ls /sys/block/ | grep '^sd'`"

echo "------------------------------------------------------------------------------------"
echo -n "* = RAID member, "
echo "+ = mounted (at least 1 partition)"
echo "DEV_ FSLB  FS PLAB PTB  SIZE     SERIAL NUMBER     MODEL         /          FAMILY"
echo "===================================================================================="

for idev in $DEVS; do
  part=`blkid -p -o value -s PTTYPE /dev/"${idev}"`
  if [ -z $part ] ; then
    part="0PT"
  fi
  label=`blkid -p -n nolinux_raid_member -o value -s LABEL /dev/"${idev}1" 2>/dev/null`
  if [ -z $label ] ; then
    label="0LAB"
  fi
  partlabel=`blkid -p -o value -s PART_ENTRY_NAME /dev/"${idev}1" 2>/dev/null`
  if [ -z "$partlabel" ] ; then
    partlabel="0PLB"
  fi
  fs=`blkid -p -n nolinux_raid_member -o value -s TYPE /dev/"${idev}1" 2>/dev/null`
  if [ -z $fs ] ; then
    fs="0FS"
  fi

  mnt=`mount | grep "${idev}"`
  if [ -z "$mnt" ] ; then
    flag=" "
  else
    flag="+"
  fi
  raid=`blkid -p -n linux_raid_member -o value -s TYPE /dev/"${idev}1" 2>/dev/null`
  if [ ! -z "$raid" ] ; then
    flag="*"
  fi

  size="`smartctl -i /dev/"${idev}" | grep -i user | grep -i capacity | sed -rn -e 's_.*\[(.*)\].*_\1_p'`"
  serial="`smartctl -i /dev/"${idev}" | grep -i serial | grep -i number | sed -rn -e 's_Serial Number:\ *__p'`"
  model="`smartctl -i /dev/"${idev}" | grep -i device | grep -i model | sed -rn -e 's_Device Model:\ *__p'`"
  family="`smartctl -i /dev/"${idev}" | grep -i model | grep -i family | sed -rn -e 's_Model Family:\ *__p'`"
  if [ -z "$family" ] ; then
    family="$model"
  fi
  model="`echo ${model} | sed -r -e 's_(samsung|intel|wdc|hitachi|seagate)__ig'`"
  model="`echo ${model} | sed -r -e 's_   _ _'`"

  echo "${idev}${flag} ${label} ${fs} ${partlabel} ${part} ${size} ${serial} ${model} ${family}"
  smartctl -A -f brief /dev/${idev} | grep -e "ATTRIBUTE_NAME" -e "Raw_Read_Error_Rate" -e "Reallocated_Sector_Ct" -e "Current_Pending_Sector" -e "Offline_Uncorrectable" -e "UDMA_CRC_Error_Count" -e "Hardware_ECC_Recovered" -e "Multi_Zone_Error_Rate"
  echo
done


Here's another script I find useful for listing all my /dev/sd* drives in a compact table:

Code: 
#!/bin/sh

SCRIPT=`basename "$0"`

if [ `id -u` -ne "0" ] ; then
  echo "*** ERROR: $SCRIPT must be run as root"
  exit 1
fi

DEVS="`ls /sys/block/ | grep '^sd'`"

echo "------------------------------------------------------------------------------------"
echo "LABEL DEV PTB PLAB FS   SIZE     SERIAL NUMBER     MODEL         /          FAMILY"
echo "===================================================================================="
for ii in $DEVS; do 

  part=`blkid -p -o value -s PTTYPE /dev/"${ii}"`
  if [ -z "$part" ] ; then
    part="0PT"
  fi
  label=`blkid -p -n nolinux_raid_member -o value -s LABEL /dev/"${ii}1" 2>/dev/null`
  if [ -z "$label" ] ; then
    label="0LAB"
  fi
  partlabel=`blkid -p -o value -s PART_ENTRY_NAME /dev/"${ii}1" 2>/dev/null`
  if [ -z "$partlabel" ] ; then
    partlabel="0PLB"
  fi
  fs=`blkid -p -n nolinux_raid_member -o value -s TYPE /dev/"${ii}1" 2>/dev/null`
  if [ -z "$fs" ] ; then
    fs="0FS"
  fi
  mnt=`mount | grep "${ii}"`
  if [ -z "$mnt" ] ; then
    flag=" "
  else
    flag="+"
  fi
  raid=`blkid -p -n linux_raid_member -o value -s TYPE /dev/"${ii}1" 2>/dev/null`
  if [ ! -z "$raid" ] ; then
    flag="*"
  fi
  size="`smartctl -i /dev/"${ii}" | grep -i user | grep -i capacity | sed -rn -e 's_.*\[(.*)\].*_\1_p'`"
  serial="`smartctl -i /dev/"${ii}" | grep -i serial | grep -i number | sed -rn -e 's_Serial Number:\ *__p'`"
  model="`smartctl -i /dev/"${ii}" | grep -i device | grep -i model | sed -rn -e 's_Device Model:\ *__p'`" 
  family="`smartctl -i /dev/"${ii}" | grep -i model | grep -i family | sed -rn -e 's_Model Family:\ *__p'`" 
  if [ -z "$family" ] ; then
    family="$model"
  fi
  model="`echo ${model} | sed -r -e 's_(samsung|intel|wdc|hitachi|seagate|corsair)__ig'`"
  model="`echo ${model} | sed -r -e 's_   _ _'`"
  echo "${label}${flag} ${ii} ${part} ${partlabel} ${fs} ${size}  ${serial}  ${model} / ${family}"

done | sort
echo "------------------------------------------------------------------------------------"
echo "* = RAID member"
echo "+ = mounted (at least 1 partition)"
echo
 
Last edited:
Yep. I'm running 3.2 and udev 175. I bet it is a udev feature in a newer version. I'll work on getting that updated. Thanks for the scripts man! They should help out.
 
SatanSkin said:
I think your #2 about ZFS was what I was thinking of in my #2. If I wanted to expand, I couldn't simply add a new 2TB drive. I would have to add 3 of them or so.

From my experience Flexraid still seems a little bit unpolished in Linux and is certainly not at a maturity worth paying for yet. From what I can tell, this is quite a different story on Windows. Snapraid has been great so far. It's very simple, straightforward, and abides by the age old *nix philosophy of do one thing great instead of many things half-assed. My only complaint is that I would love the ability to have more parity drives. Having only 2 parity drives for 20+ data disks just doesn't make me extra warm and fuzzy. I believe this is in the works though.

I will be reusing my current PSU in one of the new builds. I believe it was a 1kW PP&C PSU which I know will be quite overkill for only 24 drives, but it was intended for 46 drives. I will certainly take a look at the one you mentioned for future use though.

I have found a few M1015s and HP SAS expanders on ebay and amazon. Are there any speciality sites that stock these for really good prices that you have found?
Click to expand...

While it is a bit mote of an investment in parity disks, I would suggest breaking your arrays into no more than 12-14 drives (including up to 2 parity drives) per array (especially if you are using 2TB+ drives) to save you insane rebuild times and the chance of losing everything in case of a significant failure (cue my obligatory RAID is not a backup yadda yadda speech.)
Also, if you plan on doing 1-drive-at-a-time upgrades (and not doing ZFS) then I would suggest doing hardware RAID6 with an Areca or LSI controller (If you are going to use Windows as the OS)
 
Mwroobel: Thanks for the input. As I mentioned in my OP, I won't be running hardware RAID and most likely not software RAID either. I also have a rather strict requirement for Linux OS instead of Windows.

However, your comment did give me an idea. Although snapraid only supports up to 2 parity drives at the moment, you can point it to a specific config file. So to workaround things, I could just split my drives across to config files allowing for up to 4 parity drives. It's a bit of a hassle, but it could be a solution until the dev adds support for more parity drives.
 
balance101 said:
Sorry for asking a noob question

I tho the M1015 only supports 16 devices -> http://www.redbooks.ibm.com/abstracts/tips0740.html

Say a HP SAS expander supports 36 drives.

Can you just use one M1015 -> SAS Expander? Thanks
Click to expand...

The documentation from the big vendors are frequently erroneous, or aren't written with home users in mind. The docs are written in the context of using an IBM product with other IBM products. They never intended for an M1015 to be used outside an IBM server, and they don't support it outside that context nor care. That's the whole point of this forum and others is to discuss how a product like that works "on the ground" meaning actual testing according to our needs. HP is no different, their HP expander docs are full or erroneous information and some of it is because it was written only with the scope of their product offerings, marketing or politics in mind.

HP expander is 36 ports (9 minisas connectors) but only supports 32 drives because one connection is used by controller interlink. And even then there are conditions that have to exist to go beyond 24 drives - such as SES2 being disabled on the controller. That tends not to be a problem though since most people are maxing out with a 24-bay case and there are practical reasons NOT to extend the remaining ports into another case and just keep a one expander per enclosure rule.
 
Last edited:
odditory said:
The documentation from the big vendors are frequently erroneous, or aren't written with home users in mind. The docs are written in the context of using an IBM product with other IBM products. They never intended for an M1015 to be used outside an IBM server, and they don't support it outside that context nor care. That's the whole point of this forum and others is to discuss how a product like that works "on the ground" meaning actual testing according to our needs. HP is no different, their HP expander docs are full or erroneous information and some of it is because it was written only with the scope of their product offerings, marketing or politics in mind.

HP expander is 36 ports (9 minisas connectors) but only supports 32 drives because one connection is used by controller interlink. And even then there are conditions that have to exist to go beyond 24 drives - such as SES2 being disabled on the controller. That tends not to be a problem though since most people are maxing out with a 24-bay case and there are practical reasons NOT to extend the remaining ports into another case and just keep a one expander per enclosure rule.
Click to expand...

thanks for your detail explanation.

HP SAS Expander is better deal than the Intel SAS Expander but if you're using a itx motherboard it might be a problem to power the HP SAS Expander

images


imagemagic.php


other members seems to order this to power their HP SAS Expander for their 2nd unit
 
Thanks, but my 2nd unit won't be used as DAS to expand the first one. It would be used as an entirely separate second server for backups.

Anyone have a good source for the M1015 or expander at real nice price?
 
balance101 said:
images


other members seems to order this to power their HP SAS Expander for their 2nd unit
Click to expand...

orly? imma have to look into that. ;)


ST3F said:
Oh ? ... are you really sure ?
What seems better in HP SAS Expander than the Intel ?
Cheers.
St3F
Click to expand...

Well he did say better deal, by which I assume he meant lower cost per port, which is true based on the going rate of each. Granted the HP won't negotiate SATA above 3Gbps, whereas the Intel uses an LSI chip that does support 6GBps SATA, but OTOH if we're talking spinning disks (which is what most people are buying an expander for) they're not even cracking 200MBps to saturate the 250MBps of SATA-II anyway. Also, if you are putting the Intel expander into a 24-bay case, well its only got room for 20 drives after the interlink to the controller, which means you have to either use the other 4 ports of the SAS controller or motherboard SATA. But for strictly a 24-bay DAS, the Intel isn't ideal for obvious reasons which is why I have HP's in mine. BUT the Intel does have MOLEX as an option for power instead of PCIe only like the HP. Again, all depends on intended use.

But if you were mixing SATA-III SSD's in with your then spinners then the LSI based expanders would obviously be favorable. All depends what you're going to do but neither is necessarily 'better' than the other. I run 8 of the HP's and 2 of the Intel's and based on a couple years usage of the Intel I still prefer the HP's. pound for pound.
 
Last edited:
odditory said:
Well he did say better deal, by which I assume he meant lower cost per port, which is true.
Click to expand...
If the HP has not the firmware 2.08... buying a SmartStorage P411 to flash the expander, is not a big deal at the end.

Intel RES2CV Family are more reliable with LSI cards and don't need PCie.

//

Other than power control board PE-2SD1-R10, another without PCIe slot from Supermicro : CSE-PTJBOD-CB1 (about 60 $)

thumb_uc_22313_z450_Supermicro_CSE-PTJBOD-CB1Powercontrolboard.jpg


++

St3F
 
So, I just placed my first orders for this buildout. Picked up:

Norco RPC-4224
SM X9SCM-IIF-O
Intel E3-1230v2
Kingston 8GB kit KVR1333D3E9SK2/8G
8 Norco SFF8087-SFF8087 Cables
120mm Fan Wall Bracket

Now I just need to find the right price on the M1015 and the HP SAS Expander. I'll reuse the PSU, HDDs, and Noctua fans I currently have. Then when I've got the first one all running smooth, it'll be time to duplicate.

By the way, if it looks like I am overlooking something please mention it. I always get the feeling like I'm forgetting something when I do a build.
 
SatanSkin said:
So, I just placed my first orders for this buildout. Picked up:

Norco RPC-4224
SM X9SCM-IIF-O
Intel E3-1230v2
Kingston 8GB kit KVR1333D3E9SK2/8G
8 Norco SFF8087-SFF8087 Cables
120mm Fan Wall Bracket

Now I just need to find the right price on the M1015 and the HP SAS Expander. I'll reuse the PSU, HDDs, and Noctua fans I currently have. Then when I've got the first one all running smooth, it'll be time to duplicate.

By the way, if it looks like I am overlooking something please mention it. I always get the feeling like I'm forgetting something when I do a build.
Click to expand...

If you are going to be rack mounting this make sure to get a good set of rails.
 
ST3F said:
Which one ?
Click to expand...

ebay has a bunch of M1015's, this one for under $100 delivered for example. ebay has used HP SAS Expanders for $180+ and new ones for $250+. You can also get them from Amazon affiliates for $250+ new, they usually have better return policies than ebay too.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Allright,
1. and the PSU ? ;)
2. have you a HP Smart Storage card like P411 to flash the firmware of the HP SAS expander ?

Cheers.

St3F
 
ST3F said:
Allright,
1. and the PSU ? ;)
2. have you a HP Smart Storage card like P411 to flash the firmware of the HP SAS expander ?

Cheers.

St3F
Click to expand...

Odditory is offering as a public service for a nominal shipping charge and a refundable paypal surety deposit a loaner P411 for anyone who needs a P411 so they can flash their HP SAS expander to the latest firmware.
 
Last edited:
You must log in or register to reply here.
Back
Top