• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFSonLinux 0.6.1 Released

olavgg said:
The problem with HW raid....how do you know? If i checksummed each datablock into a PostgreSQL server, then I could maybe find some errors. But doing that is way to complex and extremly difficult to keep in sync(if possible at all). Not to mention the lack of self healing That's is why I use ZFS for critical data.
Click to expand...

No need to checksum each block. A checksum over each file is enough to detect corruption. It takes one shell command to generate the initial checksums:

Code: 
find . -type f -exec sha256sum {} \; > SUMS.txt

And one command to check the files against them:

Code: 
sha256sum --check SUMS.txt

The latter can be put into a script that's run by cron each night (or however often you choose.) Another option is to use an intrusion detection package.

Also, if you use ncryptfs for file encryption you can tell it to do file checksums.
 
That doesn't work for "files" that are open. Databases, virtual servers and so on. But for static files it's a good solution.
 
dualathlon said:
One more reason to use ZoL under esxi, instead of OpenSolaris clones, is vmxnet3.
..

Vmxnet3 on OpenSolaris: can only run with mtu 1500, max speed around 160mbytes/s. E1000g could reach ~ 300mbytes/s with mtu 9000, tcp window size ...

However, it seems that esxi kernel port loving solaris network stack. My esxi server has better storage network perf with OpenIndiana nfs than that of Ubuntu. :-|
Click to expand...

You may edit /kernel/drv/vmxnet3s.conf in OpenSolaris alike systems where you can enable other MTU values. (OpenSolaris itself is too old, use Solaris 11.1 or OmniOS)
 
_Gea said:
You may edit /kernel/drv/vmxnet3s.conf in OpenSolaris alike systems where you can enable other MTU values. (OpenSolaris itself is too old, use Solaris 11.1 or OmniOS)
Click to expand...

Dear Gea,

I did edit vmxnet3.conf, and follow your tcp tuning link.
The last time i test on OI a7, it can receive data with mtu 9000 only, not the other direction.

I'll try omni in this weekend.
Best regards.
 
Well, I have upgraded my box to ZoL 0.6.1, from 0.6.0.98. All seems well. No pool version upgrade this time (however there was a few ZoL version updates ago).

Once I get some time I will be re-creating my ZFS array .. getting rid of the iSCSI layer (just using samba natively instead of NTFS on top of iSCSI), and also getting my drives aligned (I assume they are not aligned correctly, as they are AF 4K drives and I am getting TERRIBLE performance (~100MB/sec on a 6 disk RaidZ2). I just need to make sure I have all my backup locations nice and current, because I obviously will be nuking the whole thing in the process...
 
kac77 said:
I wouldn't expect this anytime soon. Linux itself doesn't have Unix/Windows ACL's. So it's going to be a while (if ever) for this feature.


This problem is actually shared with all of the other open source alternatives. ZFS under low memory will cause this scenario. However, the fix is setting an ARC limit and limiting resources on the VM host.
Click to expand...

Linux has posix ACL's and ACE's....
 
I am getting ultra poor performance on a raidz2 of 8 ~500GB drives. I am not sure why. I was expecting 20 to 30 times this sequential performance.

Code: 
datastore4 ~ # zpool status
  pool: zfs_test
 state: ONLINE
  scan: none requested
config:

        NAME                                                 STATE     READ WRITE CKSUM
        zfs_test                                             ONLINE       0     0     0
          raidz2-0                                           ONLINE       0     0     0
            ata-ST3500413AS_5VMR9FEJ-part5                   ONLINE       0     0     0
            ata-ST3500320AS_5QM2XCX9-part5                   ONLINE       0     0     0
            ata-ST3500320AS_9QM1X8B3-part5                   ONLINE       0     0     0
            ata-ST3500320AS_9QM23DTQ-part5                   ONLINE       0     0     0
            ata-ST3500413AS_9VMCXG9C-part5                   ONLINE       0     0     0
            ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5  ONLINE       0     0     0
            ata-ST3500320AS_5QM0LW3A-part5                   ONLINE       0     0     0
            ata-ST3500413AS_Z2AKZ3NX-part5                   ONLINE       0     0     0

Code: 
datastore4 zfs # pwd
/mnt/zfs
datastore4 zfs # dd if=/dev/zero of=outfile.bin bs=8M count=1000
1000+0 records in
1000+0 records out
8388608000 bytes (8.4 GB) copied, 395.556 s, 21.2 MB/s

Code: 
zfs_test    47.9G  3.48T      0     46     51  4.63M
zfs_test    48.2G  3.48T      0    178    102  18.3M
zfs_test    48.4G  3.48T      0    150      0  15.8M
zfs_test    48.7G  3.48T      0    275    102  25.3M
zfs_test    49.2G  3.48T      0    284      0  30.6M
zfs_test    49.4G  3.48T      0    240    102  24.8M
zfs_test    49.6G  3.48T      0    154    102  16.3M
zfs_test    49.9G  3.48T      0    163      0  16.5M
zfs_test    50.1G  3.48T      0    177      0  15.9M
zfs_test    50.3G  3.48T      0    158    102  16.8M
zfs_test    50.6G  3.48T      0    228      0  22.3M
zfs_test    50.8G  3.48T      0    187    102  18.9M
zfs_test    51.1G  3.48T      0    253    153  26.1M
zfs_test    51.4G  3.48T      0    178    102  18.6M
 
Last edited:
/dev/zero is a bad source, as zfs eliminates a lot of the actual writes. Try from actual data..
 
Agree that /dev/zero is not the best source, but i still would have expected you to see better in this simple dd bench.

Use Bonnie++ for a better bench.
 
danswartz said:
/dev/zero is a bad source, as zfs eliminates a lot of the actual writes. Try from actual data..
Click to expand...

Before the dd test I did rsync from a second mdadm raid6 array and I got the same crazy slow performance. From the leds on the raid cages it appears it writes for a little bit pauses for a few seconds then writes again.
 
danswartz said:
What do you get reading that big file back to the bit bucket?
Click to expand...

Code: 
datastore4 zfs # dd if=outfile.bin of=/dev/null bs=8M
100+0 records in
100+0 records out
838860800 bytes (839 MB) copied, 0.790562 s, 1.1 GB/s

Must be somewhat cached in ram. I will have to make a larger file than the 8GB of ram this xeon E31230 has..

Also I did a second test that is why its only ~800MB now and not 8.4GB
 
Robstar said:
Linux has posix ACL's and ACE's....
Click to expand...

I guess I wasn't too clear. Yes it does. However, there are different standards/versions. So far Linux does not support the more advanced ACL features found within Windows and Solaris, which I mentioned earlier.

Now Samba 4 is supposed to change some of this for CIFS based shares. However I haven't played around with this. Then there's the work found within NFS, but other than that, out of the box Linux does not support the full ACL structure of Windows or Solaris.
 
danswartz said:
What do you get reading that big file back to the bit bucket?
Click to expand...

Here is a bigger file. Still at 172 MB/s I would consider this a slow read for a double parity array on these drives on this system. mdadm + btrfs or ext4 would have been around 3 times as fast on the read the same drives.

Code: 
datastore4 zfs # dd if=/dev/zero of=outfile.bin bs=16M count=1000
1000+0 records in
1000+0 records out
16777216000 bytes (17 GB) copied, 775.237 s, 21.6 MB/s
datastore4 zfs # dd if=/mnt/zfs/outfile.bin of=/dev/null bs=8M
2000+0 records in
2000+0 records out
16777216000 bytes (17 GB) copied, 97.5358 s, 172 MB/s
datastore4 zfs #
 
Last edited:
You can drop the buffer cache with this command:

Code: 
# sync; echo 3 > /proc/sys/vm/drop_caches
If you do that before your read speed test, you can be sure that you will not be reading any data from the linux buffer cache.

Alternatively, you can try an iflag = direct read, but I am not 100% confident that will be an accurate representation of your read speed:

Code: 
# dd if=testfile of=/dev/null bs=8M iflag=direct
 
Keep in mind one (IMO of course) disadvantage of raidz* vs 'regular raid'. Because of how it works, in general you need to read from all N drives, so I don't expect the sequential throughput to be as good. If there is no essential data on that pool, I'd be curious to see you rebuild the pool as a 4x2 raid10 and repeat your tests...
 
If there is no essential data on that pool, I'd be curious to see you rebuild the pool as a 4x2 raid10 and repeat your tests...
Click to expand...
I could do that possibly tomorrow. There is nothing of value on the zfs filesystem.
 
JoeComp said:
You can drop the buffer cache with this command:

Code: 
# sync; echo 3 > /proc/sys/vm/drop_caches
If you do that before your read speed test, you can be sure that you will not be reading any data from the linux buffer cache.

Alternatively, you can try an iflag = direct read, but I am not 100% confident that will be an accurate representation of your read speed:

Code: 
# dd if=testfile of=/dev/null bs=8M iflag=direct
Click to expand...

Yes. That would work. I did not think of that... Too much going on here at once..
 
BTW The scrub performance is similar to the dd read.

Code: 
datastore4 zfs # zpool status zfs_test
  pool: zfs_test
 state: ONLINE
  scan: scrub in progress since Thu Apr 11 11:22:07 2013
    8.58G scanned out of 68.8G at 183M/s, 0h5m to go
    0 repaired, 12.47% done
config:

        NAME                                                 STATE     READ WRITE CKSUM
        zfs_test                                             ONLINE       0     0     0
          raidz2-0                                           ONLINE       0     0     0
            ata-ST3500413AS_5VMR9FEJ-part5                   ONLINE       0     0     0
            ata-ST3500320AS_5QM2XCX9-part5                   ONLINE       0     0     0
            ata-ST3500320AS_9QM1X8B3-part5                   ONLINE       0     0     0
            ata-ST3500320AS_9QM23DTQ-part5                   ONLINE       0     0     0
            ata-ST3500413AS_9VMCXG9C-part5                   ONLINE       0     0     0
            ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5  ONLINE       0     0     0
            ata-ST3500320AS_5QM0LW3A-part5                   ONLINE       0     0     0
            ata-ST3500413AS_Z2AKZ3NX-part5                   ONLINE       0     0     0

errors: No known data errors

I will see if I can rebuild the pool as a 4x2 raid10 later. I have medical imaging research to conduct.
 
brutalizer said:
Do you think hw-raid would have problems with this?
Click to expand...

I would say it depends on the implementation.There was a post a few weeks back where a guy was hot pulling an entire array where the hw raid controller rejected the whole array after the user later reconnected it causing the user a lot of work and pain to get the raid to just use the drives.
 
Yeah it would depend on the nature of the malfunction. If your PSU is randomly dropping current it could cause all sorts of *unpredictable* behavior. ANYTHING can happen. ZFS just happened to be able to recover my data in this instance. I may not be so lucky next time.
 
Okay. I hot removed a single disk of the 8 drive raidz2. Then added a spare. It resilvered but it will not allow me to remove the missing disk and will also not move from degraded. It tells me to use zpool replace but that fails with device busy.

Code: 
datastore4 ~ # zpool status zfs_test
  pool: zfs_test
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0 in 0h10m with 0 errors on Tue Apr 16 11:58:17 2013
config:

        NAME                                                   STATE     READ WRITE CKSUM
        zfs_test                                               DEGRADED     0     0     0
          raidz2-0                                             DEGRADED     0     0     0
            ata-ST3500413AS_5VMR9FEJ-part5                     ONLINE       0     0     0
            ata-ST3500320AS_5QM2XCX9-part5                     ONLINE       0     0     0
            ata-ST3500320AS_9QM1X8B3-part5                     ONLINE       0     0     0
            ata-ST3500320AS_9QM23DTQ-part5                     ONLINE       0     0     0
            ata-ST3500413AS_9VMCXG9C-part5                     ONLINE       0     0     0
            spare-5                                            DEGRADED     0     0     0
              ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5  OFFLINE      0     0     0
              ata-ST3500413AS_6VMPF5ZH-part5                   ONLINE       0     0     0
            ata-ST3500320AS_5QM0LW3A-part5                     ONLINE       0     0     0
            ata-ST3500413AS_Z2AKZ3NX-part5                     ONLINE       0     0     0
        spares
          ata-ST3500413AS_6VMPF5ZH-part5                       INUSE     currently in use

errors: No known data errors


datastore4 ~ # zpool replace zfs_test ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5 ata-ST3500413AS_6VMPF5ZH-part5
cannot open '/dev/disk/by-id/ata-ST3500413AS_6VMPF5ZH-part5': Device or resource busy
cannot replace ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5 with ata-ST3500413AS_6VMPF5ZH-part5: ata-ST3500413AS_6VMPF5ZH-part5 is busy

Any ideas?
 
kac77 said:
try "zpool replace -f "
Click to expand...

No different.

Code: 
datastore4 ~ # zpool replace -f zfs_test ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5 ata-ST3500413AS_6VMPF5ZH-part5
cannot open '/dev/disk/by-id/ata-ST3500413AS_6VMPF5ZH-part5': Device or resource busy
cannot replace ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5 with ata-ST3500413AS_6VMPF5ZH-part5: ata-ST3500413AS_6VMPF5ZH-part5 is busy
 
I also tried ommiting the new-device since the help says that is optional:

Code: 
datastore4 ~ # zpool replace -f zfs_test ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5
cannot open 'ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5': no such device in /dev
must be a full path or shorthand device name

If I try with the full path it fails (i believe rightfully so since that device no longer exists):


Code: 
datastore4 ~ # zpool replace -f zfs_test /dev/disk/by-id/ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5
cannot resolve path '/dev/disk/by-id/ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5'
 
drescherjm said:
No different.

Code: 
datastore4 ~ # zpool replace -f zfs_test ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5 ata-ST3500413AS_6VMPF5ZH-part5
cannot open '/dev/disk/by-id/ata-ST3500413AS_6VMPF5ZH-part5': Device or resource busy
cannot replace ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5 with ata-ST3500413AS_6VMPF5ZH-part5: ata-ST3500413AS_6VMPF5ZH-part5 is busy
Click to expand...

You can't replace the disk because it is in-use by the pool right now. You should just zpool remove the disk you removed.
 
You can't replace the disk because it is in-use by the pool right now. You should just zpool remove the disk you removed.
Click to expand...

I tried that however I can not get zfs to remove the disk.

Code: 
datastore4 ~ # zpool remove  zfs_test ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5
cannot remove ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5: only inactive hot spares, cache, top-level, or log devices can be removed
 
I will look at that however to me it is unacceptable if a drive totally dies I can not replace it.
 
extide said:
then just try zpool online like it says

zpool online zfs_test
Click to expand...

Code: 
datastore4 ~ # zpool status
  pool: zfs_test
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0 in 0h10m with 0 errors on Tue Apr 16 11:58:17 2013
config:

        NAME                                                   STATE     READ WRITE CKSUM
        zfs_test                                               DEGRADED     0     0     0
          raidz2-0                                             DEGRADED     0     0     0
            ata-ST3500413AS_5VMR9FEJ-part5                     ONLINE       0     0     0
            ata-ST3500320AS_5QM2XCX9-part5                     ONLINE       0     0     0
            ata-ST3500320AS_9QM1X8B3-part5                     ONLINE       0     0     0
            ata-ST3500320AS_9QM23DTQ-part5                     ONLINE       0     0     0
            ata-ST3500413AS_9VMCXG9C-part5                     ONLINE       0     0     0
            spare-5                                            DEGRADED     0     0     0
              ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5  UNAVAIL      0     0     0
              ata-ST3500413AS_6VMPF5ZH-part5                   ONLINE       0     0     0
            ata-ST3500320AS_5QM0LW3A-part5                     ONLINE       0     0     0
            ata-ST3500413AS_Z2AKZ3NX-part5                     ONLINE       0     0     0
        spares
          ata-ST3500413AS_6VMPF5ZH-part5                       INUSE     currently in use

errors: No known data errors

I got the drive back UNAVAIL but I still can not remove it from the array.

Code: 
datastore4 ~ # zpool remove zfs_test ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5
cannot remove ata-WDC_WD1001FALS-00J7B0_WD-WMATV1120865-part5: only inactive hot spares, cache, top-level, or log devices can be removed
 
It's entirely possible you've found a bug in ZoL. Can you post your tale to the zfsonlinux mailing list? There are definitely devs there who would be able to help you (and fix this if it's a bug...)
 
I have seen a similar post on git-hub doing a google search from many months ago that was closed. I will see about that later.
 
Please also post up the responses you get, as I would be interested in seeing what the solution turns out to be.
 
So spares work a little differently with ZFS. They only act as temporary replacements. Eg your spare is still listed under spares as 'INUSE' because it is temporarily taking spot of the missing disk. The array is waiting for you to get a perminatn replacement, it wants you to insert a new disk and do:
zfs replace zfs_test ata-WDC_whatever ata-NEWDISKYOUJUSTPUTIN

Once the permenant replacement is there the spare will become avaialble again incase a different drive fails.

If you want the spare to become a perminant replacement I think you need to do like:
zfs detach zfs_test ata-ST3500413AS_6VMPF5ZH-part5

That will remove it from list of spares so it can become a perminant replacmenet for missing disk and the unavalbe disk will go away and everythign will be how you expect.

Anwyays fairly sure this is how it works, I dont' really use hot spares.
 
You must log in or register to reply here.
Back
Top