Different drives not appearing on boot, Norco/SASLP/Expander

K

Kettchxxii

Gawd
Joined
Jun 13, 2007
Messages
651
Specs:

Windows Server 2008 R2
Norco 4220
SUPERMICRO X8SIL-F
Intel Xeon X3450
Kingston 4GB DDR3 1333 ECC Unbuffered Server Memory Model KVR1333D3E9SK2/4G
Corsair 550 PSU
SASLP into HP SAS Expander
Quad Port Intel Nic
640 WD Black
1 TB WD Black
500 GB 2.5' WD Blue
(2) 2 TB WD EARS Greens
(16) 2 TB WD EADS Greens

norco.jpg


The only way I have been able to get all 21 drives to show up properly is by booting the machine with only the Motherboard hard drives plugged in and one by one adding the expander drives. When I boot with all the drives installed I end up with two drives missing each time. These drives are always on the second and third rows, and vary as to which drive is missing. The image shows the methodology I used. In reboot 3 through 6 I only had two drives missing in this layout:

reboot 3: Row 2 Column 2, Row 3 Column 4
reboot 4: Row 2 Column 1, Row 3 Column 4
reboot 5: Row 2 Column 4, Row 3 Column 4
reboot 6: Row 2 Column 2, Row 2 Column 4

I am going to swap cables and see if these locations change, but it has only been happening in the second and third row, which seems to indicate a cabling issue.

Could this be a power issue as well? having all of the drives show up by adding them one at a time points in that direction as well. I also have a ARECA-1300ix that could be used for additional testing (replacing SASLP into Expander). The SASLP is running channel one into channel 9 (the highest internal port) on the Expander. I have replaced a few cables, but am starting to think I should just order a bunch of new 8087 cables and be done with it.

Any Ideas?
 
I would definitely have a look at your power supply. 550W for running 21 drives + the motherboard and I/O cards is a little weak!

I assume all of this ONLY happens during the first power up?
Or does this also happen when the system restarts (just a reset, power stays on and drives are basically still spinning)?
 
discovered this myself couple days back
backplanes of norco BEND! ... sometimes connectors slide bellow connections on drive, not making contact and keeping you puzzled for days
 
The power supply should be sufficient.
Click to expand...

I did do some testing with the current PSU running the motherboard and a 400 watt running the drives and it didn't seem to make a difference. That was when I was using the areca 1300-ix. See this post for details. Still 15% cashback on a corsair 1kw which brings it to the 150 mark after a rebate. Shipping ends up being next day.

discovered this myself couple days back
backplanes of norco BEND! ... sometimes connectors slide bellow connections on drive, not making contact and keeping you puzzled for days
Click to expand...

As for the ports, I did the reboots back to back to back. I think one I shut the unit down for a few minutes. If it was physical connections why would different drives fail to appear in disk management?

Still kinda left at either bad norco backplates, bad SAS cables, or Bad Power.

On a side note does anyone know a place I can get 8087 cables for less than 20 dollars a pop that I can be 100% sure they are amazing quality? All of the cables I have ordered over the last couple of months have been china ebayers. Maybe thats the issue, but 80 dollars for 4 cables off newegg hurts when I dont know thats the problem...
 
I think it may still be power related. Looking at your parts list above, I suspect that during boot-up (first power-up), your system reaches almost the power supplies maximum capacity. For example, I know that each WD20EADS drive will draw about 16W during spin-up and settle down to just around 6W when in use and just below 5W when idling. So add all this up and you easily get to 500W+...

Now I wouldn't go to a 1kW power supply as the power supply would not be very efficient with the load it has during normal operation. A good 650 to 750W power supply should be a good choice.

Usually, these kind of power supplies are most efficient when they have a 75 to 80% load after which they tend to drop off a few percent again. At 50% load or less you may see an efficiency of around 50 to 60%. This of course entirely depends on the power supplies implementation. You also do not want to run a power supply at it's maximum for extended periods of time as this will shorten it's expected lifetime. Just a few things to keep in mind :)
 
I would tend to agree with treadstone. Figure if you are pulling 336w (21 * 16 but the blue does not offset the blacks making 16w probably slightly low) at spinup for all drives, 85-95w for the CPU during some startup points. That's 431w without the mobo (which consumes 8w just for the IPMI feature), RAM, Mobo, lots of fans, SAS Expander, SASLP (no experience with those). You might be OK with a 550w PSU, but you are going to cut it very close at some startup points using all of that hardware.

Another thing to check is how banks 2 and 3 are wired. If they are pulling from the same 1 or 2 power lines, you may have something slightly overloaded.
 
Power supplies can handle brief loads over their capacity. I managed to run a E6600 and 21 drives off a 400w Corsair.
 
Which Corsair 550 PSU do you have? If its one of their single-rail designs (like the current 550VX) then I don't think the PSU should be your problem. Yes - you are running at the level edge of what it can handle during spinup, but it should be OK. Barely.

If it is an older split-rail design then you might be overloading one of the 12V rails and seeing the effects of the power-protection circuits on that rail kicking in. That would produce effects that exactly match your symptoms.

I absolutely agree that you should not solve this problem by throwing a 1KW PSU at it. Maybe as a troubleshooting step to eliminate power as your issue - but certainly not for ongoing use. Get yourself a good, solid 650W single rail PSU. Corsair's HX620 is nice but might be discontinued.
 
Last edited:
Most power supplies can handle brief overloads IF they are properly designed, some can't (read cheap brands). Is it recommended: No!

I've designed a variety of different switched mode power supplies. The largest one was a redundant 6000W -48V power system.

pjkenned touched on something I was going to mention in my previous post but forgot :)

It's not only the total amount of power that the power supply can supply, but it is also important to note how much power a particular supply can provide on the +12V rail(s) as this is the one that usually gets stressed the most during power-up in most desktop and server setups that utilize lots of HDD (mechanical not SSDs) as well as fans.

Another aspect the OP should look at (and again that is something pjkenned) touched on is if the power supply has multiple +12V rails and if that's the case, are they evenly distributed among the drives?

For example: If the power supply has 3 +12V rails but all of the drives are hooked up to two of them, the start-up load may create an imbalance in the power transformer and will cause some of the other rails out of regulation for a short period of time. It of course again all depends on how the power supply was designed and how well it can cope with overloads on individual rails.
 
its a VX550 corsair. Bought it off someone in the forum. My understanding is that this PSU is a single rail PSU.

I feel like I am looking at two options. Either buy 80 dollars worth of cables, which technically isn't backed by the whole different drives being unavailable after each boot, or going with a larger PSU. TigerDirect has a warehouse just across state lines and is running 15% BCB. If I were to buy a PSU what one would you recommend for this system? Or would you go for a cables first approach. Or better yet, I bought this case used from a respected forum member, but maybe I should just wait on the 4224?
 
There is no good reason that the VX550 can't run your setup. If you are daisy-chaining all 21 drives off one cable from the PSU then you might be overdriving the wires/connectors (and, frankly, risking a fire...). Buy the cables. Better yet - buy the wire and bare connectors and make exactly the cables you want - cable that fit your rig perfectly.
 
Last edited:
When I first put my system together I had this problem! In my case it was the cables - the connectors at the Norco backplane actually.

My setup is an Areca 1680i, HP SAS expander and Norco 4220 case (850W power supply). The cables I bought were fine but on some of the Norco backplanes they didn't latch in firmly due to a small amount of interference between the top of the connector and the metal support rails that run across the back of the drive bay area. Not every row was impacted because not every row has the metal rail near it.

I used a Dremel tool to add a bit of clearance and life was good.
 
Subscribed. And nervous.
I'm looking at an almost exact hardware purchase soon for a new Vail WHS. Received the Norco 4220 last week, have a PCP&C 750 for my p/s, and will purchase Supermicro X8SIL-F, Intel i3 530, Kingston 4GB DDR3 1333 ECC Unbuffered Server Memory Model KVR1333D3E9SK2/4G this pay. AOC-SASLP-MV8 x 2 would be next.

I am taking the Norco 4220 to work tonight to Dremel the support rails as GMcDonnell suggested. And remove the fan wall for measurement/replacement.
 
After some PMs with treadstone it looks as if I am getting cable errors...

(C7) Ultra DMA CRC Error Count

1 7/20/2010 9:42:28 AM 200 200 0 19
 
It says that those errors are generally caused by cable issues. I am guessing its a bad cable connection as others have mentioned.
 
I assume you have a couple of the same cables in your system?

Why don't you try to swap them between the controller and the backplane and see if either the problem moves with the cable to a different backplane (as in a bad cable) or goes away (as in a bad connection between certain plug and connector combination)!?!

If the problem stays on the same backplane even with a cable from a backplane that always works, that could mean that the connector or even the solder joints on the backplane connectors are bad, but at least that would rule out your cables!
 
I split the power so that the row 2-5 were not all on one cable. Now the fans and hard drives are split evenly across the two cables that provide the Molex connectors.

I also switched the SAS cables running to row 2 and 3 to rows 4 and 5 respectively, which caused the whole bottom row to not show up online (second channel on SAS Expander/Bottom Norco Row).

I then reseated that SAS cable and an additional drive did not show up on boot. The drive was the Vail VM hard drive (physically row 2 column 3/ Expander channel 1 drive 3).

Next I swapped the SAS cables attached to rows 4 and 5. This yielded only one missing drive (unchecked4 physically row 5 column 4/ expander channel 2 disk 4.

physically pulling unchecked4 and reattaching it gives me all 21 drives.

So I have every drive currently... but come next reboot I imagine I am going to have the same problem. Any Ideas?
 
Last edited:
it survived a restart!

all 21 drives were available after a direct reboot.

but after shutting it down for 2 minutes I lost two drives.

alright, lets install the areca-1300 and see what happens...
 
You should have at least one spare port on the SAS expander correct?

If so, move the cable for the 'problem' backplane/row to the spare SAS expander port and see if that helps. It might be a faulty expander port?!?

Although I would probably take the Norco case apart down to the backplane and inspect the connectors for any cold solder joints and also check how well the 8087 cable mates with the backplane connector!
 
yeah, lemme swap around some ports on the expander. Not having a tremendous amount of fun tonight.
 
Ok, here is where I am at...

norco2.jpg


Seems like every time I restart this thing I end up missing a drive or two. It seems as if each time this happens it is a random drive. Removing the drive that isn't appearing and reinserting it causes it to show up.

I am sitting at 21 drives after the process of removing and reinstalling new1 and new2 as shown in the picture above. The system is 100% stable at this point, but will likely lose a drive or two upon restarting. Any ideas for zeroing in on the issue?
 
The red/pink colored drives/row I assume is what is NOT working/showing up?

I also assume that your HP SAS expander is at 2.02?

It appears that whenever the expander is involved you have an issue?

What happens if you leave the two drives (New1 F: and New2 G: ) out and try to boot? Do all of the remaining drives show up?
 
treadstone said:
The red/pink colored drives/row I assume is what is NOT working/showing up?
Click to expand...

Correct, but in the second case illustrated in the image, upon taking out and reinserting they appear
I also assume that your HP SAS expander is at 2.02?
Click to expand...

I can't recall how I can check this as they expander is invisible to the OS and I didn't see anything in during boot. SynergyDustin assured me that I would get 2.02 or newer when ordered.

It appears that whenever the expander is involved you have an issue?
Click to expand...

I am sure if I hooked up 2 of the four rows to the SASLP they would all appear, but that would leave me short 2 rows. I picked up the SAS expander with the idea that I could expand to a second chasis down the road. Maybe life would be easier if I just got another SASLP and took the expander out of the equation.

What happens if you leave the two drives (New1 F: and New2 G: ) out and try to boot? Do all of the remaining drives show up?
Click to expand...

That is an interesting thought. Tomorrow I will reboot and see if any drives show up missing in windows, pull those drives and reboot. May as well see what happens then.


Thanks for taking the time to respond.
 
Also, don't know if it makes any difference but my drives are set to staggered startup. Hitachi 2TB and Seagate 750GB drives, via HP SAS expander and the 1680 controller.
 
GMcDonnell said:
Also, don't know if it makes any difference but my drives are set to staggered startup. Hitachi 2TB and Seagate 750GB drives, via HP SAS expander and the 1680 controller.
Click to expand...

My setup has staggered spin-up enabled too and based on the activity and error LED sequences I can see what drive the 1680 is sending the spin-up command to via SES2. However, according to Areca, and my own experiences with the 1680/HP SAS expander combination is that the HP expander doesn't handle this properly. All drives seem to spin right up at power on.

Are you sure they are powering up in sequence? Did you just enable it and figure that's how it now works or did you actually verify that they startup in sequence?
 
Kettchxxii said:
I can't recall how I can check this as they expander is invisible to the OS and I didn't see anything in during boot. SynergyDustin assured me that I would get 2.02 or newer when ordered.
Click to expand...

Well if you ordered them from Dustin, I'm sure they send you the 2.02 version.

I am sure if I hooked up 2 of the four rows to the SASLP they would all appear, but that would leave me short 2 rows. I picked up the SAS expander with the idea that I could expand to a second chasis down the road. Maybe life would be easier if I just got another SASLP and took the expander out of the equation.
Click to expand...

Life might be easier, but you also won't have the expansion capability down the road...

That is an interesting thought. Tomorrow I will reboot and see if any drives show up missing in windows, pull those drives and reboot. May as well see what happens then.

Thanks for taking the time to respond.
Click to expand...

NP
 
Dear OP,

I do not have actual experience with similar setup. However, you might try the following

By default, most SAS controllers support staggered spin-up. My reason for dianostic suggestion is suspect some wrong combination and the need to manually verify PSU relevancy.

1. For the WD drives, please select a few of them (better in same RAID-group) (do not use System-Drive), then use the jumper to set power management at the back of drive. Finally, connect them to on-board SATA ports. My understanding is WD has this special jumper position. I learn from forum members and It will enable staggered spinup the WD way. When OS finishes loading, the proper storage device driver will initiate disk startup request to the remaining relevant drives on standby one by one, thus reducing your power demand during boot.

2. My last test is to on-board SATA. After that I removed the jumper and the drives are back to normal.

3. If you wanted to test 1 above, --- "please connect them to on-board SATA" ---. As some noted, some problematic combinations with wrong controllers/settings may render the drives completely in-operable because the controller may send special low-level updates that permanently bound them to the controller, without it the drive cannot start.

4. If array connection, I think there's a time delay limit where it expects response upon power-up.

Finally : I understand you say HP SAS Expander not honoring staggered spinup, if you are willing to try (no guarantee of safety), set the WD power management jumper (PLEASE check the doc properly, don't set the wrong jumper) on the WD drives connected to HP SAS Expander and see how they behaves.

Again, apology, no guarantee of drive safety. disclaim all responsibility.
 
lightp2 said:
Dear OP,

I do not have actual experience with similar setup. However, you might try the following

By default, most SAS controllers support staggered spin-up. My reason for dianostic suggestion is suspect some wrong combination and the need to manually verify PSU relevancy.

1. For the WD drives, please select a few of them (better in same RAID-group) (do not use System-Drive), then use the jumper to set power management at the back of drive. Finally, connect them to on-board SATA ports. My understanding is WD has this special jumper position. I learn from forum members and It will enable staggered spinup the WD way. When OS finishes loading, the proper storage device driver will initiate disk startup request to the remaining relevant drives on standby one by one, thus reducing your power demand during boot.
Click to expand...

This feature is called PUIS (Power-Up In Standby). This will only work if the BIOS supports this feature. Not all motherboards/BIOS have or support this feature... yet (hopefully down the road more will). And unless his motherboard supports this feature, the BIOS will not be able to detect the drive(s) and hence the OS will have no knowledge of any drive actually being connected to the motherboard!

2. My last test is to on-board SATA. After that I removed the jumper and the drives are back to normal.

3. If you wanted to test 1 above, --- "please connect them to on-board SATA" ---. As some noted, some problematic combinations with wrong controllers/settings may render the drives completely in-operable because the controller may send special low-level updates that permanently bound them to the controller, without it the drive cannot start.

4. If array connection, I think there's a time delay limit where it expects response upon power-up.
Click to expand...

If they do time out, the OS is unable to detect them since during the BIOS scan the drive was not detected. This could be the case here too. The way you can usually figure out if this is the problem is to interrupt the BIOS HDD scan cycle by pressing CTRL-ALT-DEL to force the BIOS to start over again. This should give the HDDs enough time to spin-up and be ready and available. When the BIOS goes through its scan again, all drives should be available now and be recognized by the BIOS and hence the OS should be able to see them as well.

Finally : I understand you say HP SAS Expander not honoring staggered spinup, if you are willing to try (no guarantee of safety), set the WD power management jumper (PLEASE check the doc properly, don't set the wrong jumper) on the WD drives connected to HP SAS Expander and see how they behaves.

Again, apology, no guarantee of drive safety. disclaim all responsibility.
Click to expand...

I can't speak for the AOC-SASLP-MV8 controller he has, it may work on it. But I can tell you that the Areca controllers do NOT support the PUIS feature. Believe me I tested and tried it. I also asked Areca and they have no plans for the near future to implement this either! :(
 
With regards to the staggered spin-up, I tried splitting the load on the SASLP, with channel one taking row two and the expander only taking rows 3, 4, and 5. In this configuration the first channel drives (row 2) does not appear in windows. The SAS Expander also does not take any stagger order from the controller and just fires them all up when it gets the signal.

I am not entirely sure that this is a drive issue, although I will attempt to restart the computer 10 or so seconds into the boot to make sure that all the drives spun up. Maybe that will be the ghetto fix.

This setup does not require raid arrays so missing a drive at boot doesn't make my brain explode, so I don't think I am going to attempt to jumper the drives as the potential of trouble far outweighs the possible benefit. If I needed a stable RAID volume though...

At this point I am considering just ordering another SASLP in the short term and then contacting a few forum members here to see if they are interested in testing my SAS Expander and another controller I have laying around. Maybe some of my parts are defective...
 
The PUIS feature is not really RAID related. It's just another power saving method.

I think that the SASLP does not recognize the SATA drives on the first four ports because the controller has to talk to the expander in SAS mode and the controller most likely is not capable of having half it's ports in SATA and the other half in SAS mode. That's probably why your test with Row2 connected to port one of the SASLP controller failed!

The expander on the other hand can handle a mixture of different drives (however I think there are certain limitations as well).

I'm glad your brain stays in your head... wouldn't want to clean up that mess :)

You could still try the PUIS jumper on a single drive just to see if this feature is actually supported via the SASLP and or your motherboard :) ... I'm curious...

I'm sure there are always takers for 'spare' parts here :)
 
I've placed that jumper on some of my drives and unfortunately, they stay hidden from the controller as the drive doesn't seem to spin-up even tough the controller seems to send the spin-up command. Areca told me that the HP SAS expander apparently either doesn't pass on this command or is simply ignores it?!?

I guess I could try to connect a backplane directly to the ARC-1680i and jumper a few drives with this PUIS function enabled and see if it works without the HP SAS expander in-between.

But basically the jumper is there so that you don't have to store this power management feature via a command/setting in non-volatile memory on the HDD. You can enable or disable it at any time.
 
newegg just sent me a coupon code for free shipping on those damn cables. I bit on four. Couldn't hurt right?

thanks for the info on the power management. I am going to hopefully have some time today to run a few more tests on the system.
 
You must log in or register to reply here.
Back
Top