FreeBSD / ZFS - Constantly crashing after adding new vdev.

young_einstein

Weaksauce
2FA
Joined
Jun 20, 2010
Messages
102
Hi guys,

So I’ve got a box running ZFSGuru [0.1.9b3] that’s been absolutely rock solid since day 1 [which is over 12 months now], but a few days ago I added some more disks into the box, and ever since then it’s just been constantly crashing [IE - every 15-20 minutes].

The original vdev (which has never given me any problems) was 10 x Samsung HD204UI 2TB in RAIDZ2.

Now I’ve added another 5 x Hitachi HDS5C303 3TB in RAIDZ (same pool, just an additional vdev) and it all seems to have fallen apart for some reason.

Code:
Pool members	Status	Read errors	Write errors	Checksum errors
STORAGE         ONLINE	0		0		0
raidz2	        ONLINE	0		0		0
label/disk1	ONLINE	0		0		0
label/disk2	ONLINE	0		0		0
label/disk3	ONLINE	0		0		0
label/disk4	ONLINE	0		0		0
label/disk5	ONLINE	0		0		0
label/disk6	ONLINE	0		0		0
label/disk7	ONLINE	0		0		0
label/disk8	ONLINE	0		0		0
label/disk9	ONLINE	0		0		0
label/disk10	ONLINE	0		0		0
raidz1	        ONLINE	0		0		0
label/disk15	ONLINE	0		0		0
label/disk11	ONLINE	0		0		0
label/disk12	ONLINE	0		0		0
label/disk13	ONLINE	0		0		0
label/disk14	ONLINE	0		0		0

Problem is, this box has always “just worked” for me, so I’ve never really had to do any troubleshooting / fiddling with it. That was great while everything was running fine, but now that I’ve started having problems, I don’t have even the slightest clue where to begin in terms of troubleshooting this?

Here's the info on the ZFS status page if it's useful at all:

Code:
Status

Software	
Product	ZFSguru 0.1.9-beta3
Product build date	2011-Sep-16 @ 11:16 UTC
Distribution type	rootonzfs
System version	8.2-001 (checksum: b16729227805deb5ceb6b2eb0d9b55ea)
FreeBSD version	8.2-RC1
ZFS version	SPA: 15, ZPL: 4

Hardware	
Processor	Intel(R) Xeon(R) CPU E5506 @ 2.13GHz
Processor architecture	amd64 (64-bit)
Memory size	8 GiB, of which 12 GiB kernel memory (307.2 GiB max.)
Number of physical disks	16

Sensors	
CPU temperature	1: 46.0 °C   2: 46.0 °C   3: 45.0 °C   4: 45.0 °C  
Current CPU usage	0 %
Load averages	0.00, 0.10, 0.14
System time	Fri Oct 14 15:49:04 EST 2011
Uptime	11 mins

Any suggestions / advice ... because I'm starting to pull my hair out now :(
 
So I've stopped any major writing to the pool, and it seems to be running better. My uptime at the moment is just a little under 2 hours.

I'm still reading a fair bit from it, and that doesn't seem to be causing any problems, but it seems that as soon as I start doing any sustained writes, the whole system locks up and I have to do a hard reset.

Does anybody know if there are any issues involving the type/combination/arrangement of the drives I'm using that are known to cause problems?

IE - I've never read anything saying that seperate RAiDZ2 and RAIDZ vdevs won't play nicely in the same pool ... but who knows?
 
People have been mixing raidz2 and raidz and mirrors for a long time without problems. Maybe you could try to import your zpool from within Solaris 11 Express liveCD and see if it crashes?

Your behaviour is not normal. Could it be device driver error? Firmware of disks is buggy? Faulty hardware? Plug in everything again, maybe there is a cable loose or something?
 
People have been mixing raidz2 and raidz and mirrors for a long time without problems.

That's what I've always thought. My only concern would be whether adding 3TB disks into the mix might have introduced something weird?

Maybe you could try to import your zpool from within Solaris 11 Express liveCD and see if it crashes?

I'll add it onto my todo list ...

Your behaviour is not normal. Could it be device driver error? Firmware of disks is buggy? Faulty hardware? Plug in everything again, maybe there is a cable loose or something?

Who knows?

I've been using 3082E-R SAS HBAs which only support drives up to 2TB anyway, although the LSI website says that >2TB should still work fine, just the capacity will be limited to 2047GB.

I've just re-arranged the drives now so that the new 3TB disks are plugged directly into the MB sata ports [which DO support 3TB] so that's a plus at least, but I still don't know if that's going to make any difference to my stability problems.

I'll do a bit more testing now and see what happens ...
 
Well ... over 90 minutes now without it crashing. [*touch wood*]

Exactly the same OS/HDDs/HBAs as before. The only thing I've changed is moving the [new] 3TB disks onto the MB sata ports.

I'm going to leave it like this for at least another couple of days, just to make sure that it actually is stable, then I might try swapping it back again and seeing if the instability returns.
 
Seems that way. We'll see what happens when I get around to doing some more testing.

In the meantime ... if anybody else is using 3TB drives with ZFS, I'd love to know what controller(s) you're using and what your experience has been like.
 
I'm running a LSI 9201-16i with 8x 3TB Hitachi drives in OpenIndiana. I don't have any stability issues.
 
Good to know, because I've already got one of them on my shopping list for my next planned expansion [in a different chassis].
 
How do you connect 24 disks to the LSI card? Are you using an expander? Which one?

I don't have 24 disks. Only have 8 at the moment. I might just get a second LSI card and avoid the expander all together when the time comes to go past the 16 drive mark.

What size of PSU are you using? As mentioned in the previous post you may be overloading it. You could disconnect all of you parity disks (if you have a backup) and see if it lasts any longer.
 
I wonder if you are overloading the PSU on this system with the extra disks?

Good thought, but no.

I've got the box hooked up to a UPS so I can see exactly how much power it's drawing, and I'm still nowhere near maxing out the PSU.

FWIW I'm using a Seasonic X-750 [Gold +], so it's a good quality PSU as well.

Anyway ... good news is that I've been downloading flat out all night and the box still hasn't crashed on me yet.

That makes me happier knowing that I probably don't have any faulty hardware to be worrying about.

At the moment it looks like the worst case scenario is that I'll just have to invest in some new HBA's to handle the 3TB disks properly.
 
I've just spent the last couple of hours putting the pool through some pretty heavy I/O and it hasn't had any problems ...
 
How do you connect 24 disks to the LSI card? Are you using an expander? Which one?

I giggled :p

I've just spent the last couple of hours putting the pool through some pretty heavy I/O and it hasn't had any problems ...

Good to hear, I need to sell my LSI 1068e's as well. I've had the LSI 9201-16i in my cart a few times, just waiting for more funds to pull the trigger.
 
Back
Top