• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS resilver woes

P

packetboy

Limp Gawd
Joined
Aug 2, 2009
Messages
288
The only thing I don't like about ZFS...potentiallyi loooong resilver times:


Code: 
  pool: rz2pool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
 scrub: resilver in progress for 47h8m, 90.52% done, 4h56m to go
config:

        NAME                         STATE     READ WRITE CKSUM
        rz2pool                      DEGRADED     0     0     0
          raidz2-0                   DEGRADED     0     0     0
            c15t5000CCA221D6F23Ad0   ONLINE       0     0     0
            c15t5000CCA221DAD063d0   ONLINE       0     0     0
            c15t5000CCA221D42148d0   ONLINE       0     0     0
            c15t5000CCA221D6F22Dd0   ONLINE       0     0     0
            spare-4                  DEGRADED     0     0     0
              c5t5000CCA221DAD564d0  FAULTED      0     0     0  too many errors
              c5t5000CCA221D4210Ed0  ONLINE       0     0     0  1.58T resilvered
            c5t5000CCA221DAA15Cd0    ONLINE       0     0     0
            c5t5000CCA221D4204Dd0    ONLINE       0     0     0
            c5t5000CCA221D6824Ad0    ONLINE       0     0     0
            c5t5000CCA221DAA118d0    ONLINE       0     0     0
        spares
          c5t5000CCA221D4210Ed0      INUSE     currently in use
          c15t5000CCA221D4213Ad0     AVAIL   

errors: No known data errors

Resilver started this weekend due to a filed drive...however, server was basically useless during resilver operations (e.g. remote iscsi and nfs clients pounding on keyboards for 30-60 seconds sometimes)...VERY VERY VERY frustrating.

I started investigating and realized that I had these drives conected to an LSI 9211-8i HBA that was flashed with the Raid version firmware...and a very old version.

I didn't want to monkey around with flashing that on the fly so I updated the FW and BIO on a 9200-8e that I also have on this server and then connected my HP SAS expander to it using it's external SAS connector.

This didn't speed up the resilver appreciable, but made the server much more usable...not sure why...bottom line, if you're using LSI HBA's make sure you keep on top of your FW and BIOS upgrades.

Also, this highlights one thing that I don't think has been documented well and that is that while 8 drives in a raidz2 may optimize your pool for space, it may NOT be the most optimal for resilver operation..especially when the pool is full...as is the situation in my case (there's only 500GB free in this 12TB pool).
 
Wow, I suspect the 95% full condition doesn't help. I don't know any FS (including ZFS) that handles that very gracefully.
 
packetboy said:
Also, this highlights one thing that I don't think has been documented well and that is that while 8 drives in a raidz2 may optimize your pool for space, it may NOT be the most optimal for resilver operation..especially when the pool is full...as is the situation in my case (there's only 500GB free in this 12TB pool).
Click to expand...

The optimal number of drives for raidz2 is 10, not 8, which could also contribute to the slowness. That said I agree it's likely too full to be working optimally.
 
danswartz said:
Wow, I suspect the 95% full condition doesn't help. I don't know any FS (including ZFS) that handles that very gracefully.
Click to expand...

Raid 1 with a decent filesystem on top. A single raid Z, raid 5, or raid 6 array is not a high availably solution. Array performance goes to hell during a rebuild.
 
I've done resilvers on full (>90%) pools. I'm not sure that the degree of "fullness" causes an exponential increase in rebuild time... it seems to track fairly linearly for me... though I wouldn't go past 95% if I could help it more because I'm suspicious the free space remaining stat is not completely honest, like my car's gas gauge which accelerates towards empty. ;)

One thing to consider in a resilver is whether or not the faulted drive is still somehow "active" - i.e. during the resilver is ZFS trying to get data from it? During a recent rebuild my drive was dying mechanically and my estimated resilver time was probably about 200+ hours as the progress was being slowed by the repeated attempts to access the flaky drive. I manually took the drive offline and just rebuilt with it like the bad drive was completely missing and it went much faster. I had confidence the other drives were likely OK since I do scrubs every 3-4 weeks and RAID-Z2 meant I still had one spare drive just in case.
 
There is a way to get the pid for the resilver process and pbind it to a single proc. Might help let the system continue to operate while resilver.
 
kristof said:
Raid 1 with a decent filesystem on top. A single raid Z, raid 5, or raid 6 array is not a high availably solution. Array performance goes to hell during a rebuild.
Click to expand...

define 'decent filesystem'? i've used a number of different filesystems, and if you get over 95% it sucks. not just from array rebuilding, also fragmentation.
 
danswartz said:
define 'decent filesystem'? i've used a number of different filesystems, and if you get over 95% it sucks. not just from array rebuilding, also fragmentation.
Click to expand...

A modern filesystem that employs tactics to reduce fragmentation. Things like extents, multiblock allocation, delayed allocation, etc.

No filesystem will perform well on a small volume that's 95% full. However, for larger volumes 95% full can mean that there are still 500+ gigabytes free. Unless you're writing giant files, 500GB of free space should be enough for a smart filesystem to operate normally.

Array rebuilding on raid1 is a straight block copy so the filesystem used is irrelevant to the rebuild time.
 
I have a zfs disk which gets very slow above 90%, if i delete files, it Will be snappy again. All filesydtems get slow on 90%
 
Brutal...2.5 days to resilver.

Code: 
  pool: rz2pool
 state: ONLINE
status: The pool is formatted using an older on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on older software versions.
 scrub: resilver completed after 58h37m with 0 errors on Mon Apr 30 23:27:17 2012
config:

        NAME                        STATE     READ WRITE CKSUM
        rz2pool                     ONLINE       0     0     0
          raidz2-0                  ONLINE       0     0     0
            c15t5000CCA221D6F23Ad0  ONLINE       0     0     0
            c15t5000CCA221DAD063d0  ONLINE       0     0     0
            c15t5000CCA221D42148d0  ONLINE       0     0     0
            c15t5000CCA221D6F22Dd0  ONLINE       0     0     0
            c15t5000CCA221D4210Ed0  ONLINE       0     0     0  1.74T resilvered
            c15t5000CCA221DAA15Cd0  ONLINE       0     0     0
            c15t5000CCA221D4204Dd0  ONLINE       0     0     0
            c15t5000CCA221D6824Ad0  ONLINE       0     0     0
            c15t5000CCA221DAA118d0  ONLINE       0     0     0
        spares
          c15t5000CCA221D4213Ad0    AVAIL
 
The official recommendation is to keep disk usage 80-90% on ZFS. As I said, I have a single disk for storage and when I get above 90%, everything slows down to a crawl. Everything (on that disk), for instance, listing directories, copying files, etc. When I delete files, it gets fast again.
 
Its not about being full, as someone said earlier. If you have a large pool 5% could mean 100GB of storage. Fragmentation is then not a problem. However. A lot of snapshots and deleted snapshot will increase scrubbing and resilvering time.

I think I would copy all the data over to a new fresh ZFS pool every second year, just because my pool would become awfully slow to scrub/resilver. Block rewrite would probably solve this, but I don't think anyone is working with this yet. I guess by the time block rewrite is implemented, hammerFS would have replaced ZFS.
 
olavgg said:
I think I would copy all the data over to a new fresh ZFS pool every second year, just because my pool would become awfully slow to scrub/resilver.
Click to expand...
Ive never heard any ZFS user copy all data to another server. ZFS tries to minimize fragmentation in several ways. For instance, ZFS collects lot of writes in RAM, and then writes everything down in one sweep. This minimizes fragmentation.

I have an old 2TB storage disk that I am using heavily for torrents and other stuff that fragments the ZFS filesystem. The only time I have issues with performance is when disk utililzation goes over 90% full. As soon as I delete files everything is quick again. I have never noticed fragmentation.
 
You must log in or register to reply here.
Back
Top