• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS filesystem versions

TeeJayHoward

TeeJayHoward

Limpness Supreme
2FA
Joined
Feb 8, 2005
Messages
13,322
I run OmniOS on my main NAS and Solaris 11.1 on my backup NAS. As a result, I like to keep my drive pools at a lower, more compatible version. So, when I created my pool, I used a command like this:

Code: 
zpool create -o version=28 -O version=5 pool raidz2 disk1 disk2 disk3 etc
It worked beautifully. However, I'm used to creating subpools using the following command:
Code: 
zfs create pool/subpool

Notice an issue? The subpools don't have the "-o version=5" flag set, so they default to version 6! This makes them incompatible with anything except Solaris 11.1+.

Anyway, I'm in the process of adding a disk to my array. This involves ensuring that I have a current backup, destroying my current pool, creating a new one, and restoring from that backup. Well, my restoration was going at several KB/s... Needless to say, wholy inadequate for 13TB of data! I decided to just put both arrays in the same physical box, and copy that way. That's when I encountered an issue: The pool was incompatible!

"No worries," I think to myself, "I'll just upgrade the pool so I can copy, and then create a new, compatible pool later and back it up." So I do a "zpool upgrade". It successfully upgrades the zpool to version 5000. It then complains about being unable to mount a zfs version 6 filesystem! Crap. Now my data can't be accessed from EITHER system. 13TB of data (and the associated backup) are gone. I'm trying a couple of things right now to try and fix it, but I'm not hopeful.

TL;DR: Remember that you have to set the zfs filesystem version flag for each subpool you create, or else you lose your compatibility!
 
Whelp, I think I made it worse. The only OS which can handle my pool now is OmniOS' bleeding build.

Anyone know if they have any intention of supporting ZFS filesystem version 6 in a future release?
 
TeeJayHoward said:
Whelp, I think I made it worse. The only OS which can handle my pool now is OmniOS' bleeding build.

Anyone know if they have any intention of supporting ZFS filesystem version 6 in a future release?
Click to expand...

It is really a pity that we have now two incompatible ZFS lines of development.
Oracle as the major player would have the power to keep ZFS development consistent - but they are not willing to coorperate with the free fork Illumos...

What you can try
Import the pool in OmniOS and try a zfs send to Oracle -
not sure if it works...
 
_Gea said:
It is really a pity that we have now two incompatible ZFS lines of development.
Oracle as the major player would have the power to keep ZFS development consistent - but they are not willing to coorperate with the free fork Illumos...

What you can try
Import the pool in OmniOS and try a zfs send to Oracle -
not sure if it works...
Click to expand...
Unfortunately, I cannot import the pool in ANY OS that I know of. If I could, I could save decades of pictures. I can re-rip my videos... I don't think I have the originals for my photos. I had them on an "offsite storage" drive I gave to my mom a long time ago. Maybe if I'm lucky, she still has it.

ZFS version is 6 (Solaris only)
zpool version is 5000 (With OmniOS Bleeding-only features turned on)
 
If you are really stuck, can you maybe patch the pool with zdb to roll the upgrade back? Ugh, I feel your pain... Or maybe patch the filesystem version back to 5? If you are feeling bold, build a zfsonlinux setup with the version check disabled (or allowing 6 as 5?) Might work? What specifically did oracle change with FS version 6?
 
danswartz said:
If you are really stuck, can you maybe patch the pool with zdb to roll the upgrade back? Ugh, I feel your pain... Or maybe patch the filesystem version back to 5? If you are feeling bold, build a zfsonlinux setup with the version check disabled (or allowing 6 as 5?) Might work? What specifically did oracle change with FS version 6?
Click to expand...
ZoL would allow me to get past the ZFS filesystem check (If I could get it working, anyway!), but (AFAIK) it doesn't support the two features enabled with OmniOS' latest build, so I can't read any data off the drives.

Code: 
root@nas:~# zfs upgrade -v
The following filesystem versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS filesystem version
 2   Enhanced directory entries
 3   Case insensitive and SMB credentials support
 4   userquota, groupquota properties
 5   System attributes
 6   Multilevel file system support

For more information on a particular version, including supported releases,
see the ZFS Administration Guide.
 
I think I'm going to open an issue on the ZoL issue tracker to mention that it would be nice to disallow upgrading a pool if there are zfs filesystems whose version we don't support.
 
Greetings

I have a couple of questions and comments.

Firstly regarding "That's when I encountered an issue: The pool was incompatible! " Did this not ring alarm bells? ZFS at version 28 is perfectly capable of reading each and every version below it from 27 down to 1, if its incompatible then either you have an older OS that can't read version 28 because its highest version is lower than this (28) or as you have found out you were on a different fork, either way an upgrade would not do anything useful.

Secondly, I struggle to understand how OmniOS which could not understand version 6 of the filesystem (which is unique to Oracle's Solaris ZFS) can manage to "upgrade" a pool it cannot open, this would have to be incredibly sloppy programming by incompetent programmers to even let this happen.

Regarding "Notice an issue? The subpools don't have the "-o version=5" flag set, so they default to version 6! This makes them incompatible with anything except Solaris 11.1+." I also find it a bit strange that sub-pools don't inherit the parents ZFS and filesystem versions which I think they should.

Also I would like to know why do you use OmniOS's ZFS when the Solaris ZFS works very well, is there some other part of the OmniOS operating system itself that you use? Also if your transferring stuff from one pool to another then why not just get a couple of 10Gbe cards for moving the 13TB?

The only suggestions I can make is to contact the OmniOS people and ask them is there some way to roll back the upgrade on the pool concerned. Alternatively you are going to have to preserve the data on the hard drives concerned and forensically recover it yourself like these people have done, but its going to take a lot of effort to do this as you are going to have to become familiar with ZFS's on disk specification.

http://www.joyent.com/blog/zfs-forensics-recovering-files-from-a-destroyed-zpool

http://mbruning.blogspot.com.au/2009/12/zfs-data-recovery.html

https://www.youtube.com/watch?v=BIxVSqUELNc

http://www.eall.com.br/hp/Solaris/ZFS Internals.pdf

Good luck with your endevours as you are going to need it.

Cheers
 
Hobart, you are being a bit harsh. It's not that obvious that filesystems are a level below the pool and one would have to walk through them all checking for something like this. Keep in mind this code probably goes way back when there was one code base. The main reason for not using Solaris is it is closed, and you have no way to have bugs fixed and such without a paid maintenance contract. OpenZFS is constantly adding all kinds of new features and such, which you will not have any guarantee to get with Solaris 11.x.
 
HobartTas said:
Firstly regarding "That's when I encountered an issue: The pool was incompatible! " Did this not ring alarm bells? ZFS at version 28 is perfectly capable of reading each and every version below it from 27 down to 1, if its incompatible then either you have an older OS that can't read version 28 because its highest version is lower than this (28) or as you have found out you were on a different fork, either way an upgrade would not do anything useful.
Click to expand...
My pool was at version 28, filesystem 5. My subpool was at version 28, filesystem 6, which I didn't know. I didn't think too much about it. I knew that it would kill compatibility with my backup server (Solaris 11.1). I didn't care, as long as I got my data off. I could always just copy the data back... Or so I thought!

HobartTas said:
Secondly, I struggle to understand how OmniOS which could not understand version 6 of the filesystem (which is unique to Oracle's Solaris ZFS) can manage to "upgrade" a pool it cannot open, this would have to be incredibly sloppy programming by incompetent programmers to even let this happen.
Click to expand...
I agree.

HobartTas said:
Regarding "Notice an issue? The subpools don't have the "-o version=5" flag set, so they default to version 6! This makes them incompatible with anything except Solaris 11.1+." I also find it a bit strange that sub-pools don't inherit the parents ZFS and filesystem versions which I think they should.
Click to expand...
I agree.

HobartTas said:
Also I would like to know why do you use OmniOS's ZFS when the Solaris ZFS works very well, is there some other part of the OmniOS operating system itself that you use? Also if your transferring stuff from one pool to another then why not just get a couple of 10Gbe cards for moving the 13TB?
Click to expand...
I was just playing around with different OS. I wanted to try something different, and since I had a backup of my data, figured it couldn't hurt. Turns out I was wrong. As for the 10GigE... I have a gigabit network. I couldn't transfer files at 1MB/s. Something was going on. I had every intention of finding out what was going wrong, but my main priority was to back up my data!

HobartTas said:
The only suggestions I can make is to contact the OmniOS people and ask them is there some way to roll back the upgrade on the pool concerned. Alternatively you are going to have to preserve the data on the hard drives concerned and forensically recover it yourself like these people have done, but its going to take a lot of effort to do this as you are going to have to become familiar with ZFS's on disk specification.
Click to expand...
It's pretty well known that there's no way to roll back an upgrade. My data is pretty damned important to me, but not important enough to become a forensics expert. I've got enough going on in my life right now that I can't afford the time.

HobartTas said:
Good luck with your endevours as you are going to need it.
Click to expand...
Thanks!
 
TJH, I am confused. Re-reading things, it sounds like you are saying that you:

1. Created a pool v=28 under S11.1
2. Created a fs (not subpool) v=6 (default) under S11.1
3. Connected disks to OmniOS and did 'upgrade -a' (or some-such) to go to v=5000

Is this correct? If so, I am puzzled, as I can't get OmniOS to upgrade a pool that is not mounted. Was the filesystem maybe marked as not automount? Something else?
 
Here's what I did in a nutshell:

Solaris 11.1:
zpool create -o version=28 -O version=5 pool raidz2 disk1 disk2 disk3 etc
zfs create pool/I_AM_VERSION_6

Remove disks from Solaris 11.1, and put in OmniOS.
zpool import pool (Pool successfully imports. pool/I_AM_VERSION_6 doesn't. OmniOS tells me to upgrade.)
zpool upgrade pool

pool is now filesystem 5, zfs version 5000.
pool/I_AM_VERSION_6 is still filesystem 6. I don't know what the zfs version is, because you can't mount it on either OS.

You can no longer mount pool on Solaris 11.1 because of the ZFS version mismatch. You can not mount pool/I_AM_VERSION_6 on OmniOS because of the ZFS filesystem mismatch. Your data still exists, but you can't get to it.
 
Ouch, that sucks. Not that I would ever consider that possibility, but the forking of the ZFS versions made me go to the trouble to consolidate everything on ominos (I had a couple of old opensolaris boxes at clients and a couple newer omnios boxes other places) so I wouldn't run into issues down the road or get something confused.
 
TeeJayHoward said:
Here's what I did in a nutshell:

Solaris 11.1:
zpool create -o version=28 -O version=5 pool raidz2 disk1 disk2 disk3 etc
zfs create pool/I_AM_VERSION_6

Remove disks from Solaris 11.1, and put in OmniOS.
zpool import pool (Pool successfully imports. pool/I_AM_VERSION_6 doesn't. OmniOS tells me to upgrade.)
zpool upgrade pool

pool is now filesystem 5, zfs version 5000.
pool/I_AM_VERSION_6 is still filesystem 6. I don't know what the zfs version is, because you can't mount it on either OS.

You can no longer mount pool on Solaris 11.1 because of the ZFS version mismatch. You can not mount pool/I_AM_VERSION_6 on OmniOS because of the ZFS filesystem mismatch. Your data still exists, but you can't get to it.
Click to expand...


again
have you tried what j1immy did successfully in the same case?
http://hardforum.com/showpost.php?p=1040578893&postcount=8
 
_Gea said:
again
have you tried what j1immy did successfully in the same case?
http://hardforum.com/showpost.php?p=1040578893&postcount=8
Click to expand...
Unless I'm missing something, I'm stuck at step 5. I can't mount the pool, so I can't do a ZFS send. Even if I could, I don't think I had any snapshots set up.

TeeJayHoward said:
I can't import the pool at all, so I can't read it/do a ZFS send.
Code: 
root@nas:~# zpool import backup
cannot import 'backup': pool is formatted using a newer ZFS version: unsupported version
Click to expand...

edit: Aah, I see. Mount pool, not pool/BROKEN_SUBPOOL. Let me put together a OmniOS Bloody VM and see what happens.
 
TJ, not to be a jerk here, but it would help if you used the correct terminology. One imports a pool, and they are not sub-pools, but filesystems (datasets).
 
danswartz said:
TJ, not to be a jerk here, but it would help if you used the correct terminology. One imports a pool, and they are not sub-pools, but filesystems (datasets).
Click to expand...
Thanks! I was actually wondering what to call 'em. I'll use the correct terms from here on out.
 
Another nit that is VERY common outside the solaris ZFS community: people refer to 'having a ZIL' when in fact they mean 'having an SLOG device'. This drives solaris ZFS people nuts :)
 
danswartz said:
TJ, not to be a jerk here, but it would help if you used the correct terminology. One imports a pool, and they are not sub-pools, but filesystems (datasets).
Click to expand...

To add to this, pools are not mounted, file systems contained by the pool are mounted. A pool always contains at least one file system which is the root file system for the pool. There are no sub-pools only file systems contained by the pool.

It is possible to import a pool and not mount any of its constituent file systems.
 
_Gea said:
again
have you tried what j1immy did successfully in the same case?
http://hardforum.com/showpost.php?p=1040578893&postcount=8
Click to expand...

Code: 
root@fixit:~# ls -lah /backup/.zfs/snapshot/
total 24
dr-xr-xr-x   2 root     root           2 Aug 21 09:41 .
dr-xr-xr-x   4 root     root           4 Aug 14 17:34 ..

Doesn't look like I set up snapshots on my backup pool, unfortunately. Unless I'm missing something?

edit: Is it possible to take a snapshot now and move it over?
 
Last edited:
Looks like it let me take a snapshot of both backup and backup/storage.

Unfortunately, the snapshot is empty. Should I try a zfs send anyway?

Code: 
login as: root
Using keyboard-interactive authentication.
Password:
Last login: Thu Aug 21 04:58:59 2014
OmniOS 5.11     omnios-6962d80  July 2014
root@fixit:~# zpool import backup
cannot mount 'backup/storage': Can't mount a version 6 file system on a version 5000 pool. Pool must be upgraded to mount this file system.
root@fixit:~# zfs list -t snapshot
NAME                        USED  AVAIL  REFER  MOUNTPOINT
backup@testing              136K      -   290K  -
backup/storage@testing         0      -  12.5T  -
rpool/ROOT/omnios@install   268M      -  1.54G  -
root@fixit:~# ls -lah /backup/.zfs/snapshot/testing/
total 35K
drwxr-xr-x+ 3 root root 3 Aug 14 11:35 .
dr-xr-xr-x  3 root root 3 Aug 21 05:01 ..
drwxr-xr-x+ 2 root root 2 Aug 14 11:35 storage
root@fixit:~# ls -lah /backup/.zfs/snapshot/testing/storage
total 35K
drwxr-xr-x+ 2 root root 2 Aug 14 11:35 .
drwxr-xr-x+ 3 root root 3 Aug 14 11:35 ..
root@fixit:~# ls -lah /backup/storage/.zfs/snapshot/testing/
ls: cannot access /backup/storage/.zfs/snapshot/testing/: No such file or directory
root@fixit:~#
 
Last edited:
Code: 
Cached configuration:
        version: 5000
        name: 'backup'
        state: 0
        txg: 98112
        pool_guid: 3980262890953715659
        hostid: 842186808
        hostname: 'fixit'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 3980262890953715659
            children[0]:
                type: 'raidz'
                id: 0
                guid: 1130159151103149540
                nparity: 2
                metaslab_array: 30
                metaslab_shift: 37
                ashift: 12
                asize: 24004637229056
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 13474486044320785447
                    path: '/dev/dsk/c1t5000C5004E0A2CC0d0s0'
                    devid: 'id1,sd@n5000c5004e0a2cc0/a'
                    phys_path: '/scsi_vhci/disk@g5000c5004e0a2cc0:a'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 5573819435761518469
                    path: '/dev/dsk/c1t5000C500522B15A8d0s0'
                    devid: 'id1,sd@n5000c500522b15a8/a'
                    phys_path: '/scsi_vhci/disk@g5000c500522b15a8:a'
                    whole_disk: 1
                    create_txg: 4
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 4198804063194481999
                    path: '/dev/dsk/c1t5000C500537AF3E9d0s0'
                    devid: 'id1,sd@n5000c500537af3e9/a'
                    phys_path: '/scsi_vhci/disk@g5000c500537af3e9:a'
                    whole_disk: 1
                    create_txg: 4
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 18417867839860112832
                    path: '/dev/dsk/c1t5000C5005223EC89d0s0'
                    devid: 'id1,sd@n5000c5005223ec89/a'
                    phys_path: '/scsi_vhci/disk@g5000c5005223ec89:a'
                    whole_disk: 1
                    create_txg: 4
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 1089470385991005875
                    path: '/dev/dsk/c1t5000C5005334AA86d0s0'
                    devid: 'id1,sd@n5000c5005334aa86/a'
                    phys_path: '/scsi_vhci/disk@g5000c5005334aa86:a'
                    whole_disk: 1
                    create_txg: 4
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 10330363264372380124
                    path: '/dev/dsk/c1t5000C5006760A47Cd0s0'
                    devid: 'id1,sd@n5000c5006760a47c/a'
                    phys_path: '/scsi_vhci/disk@g5000c5006760a47c:a'
                    whole_disk: 1
                    create_txg: 4
                children[6]:
                    type: 'disk'
                    id: 6
                    guid: 3205134679639454644
                    path: '/dev/dsk/c1t5000C500523492F2d0s0'
                    devid: 'id1,sd@n5000c500523492f2/a'
                    phys_path: '/scsi_vhci/disk@g5000c500523492f2:a'
                    whole_disk: 1
                    create_txg: 4
                children[7]:
                    type: 'disk'
                    id: 7
                    guid: 16396885825294167808
                    path: '/dev/dsk/c1t5000C50053218101d0s0'
                    devid: 'id1,sd@n5000c50053218101/a'
                    phys_path: '/scsi_vhci/disk@g5000c50053218101:a'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

MOS Configuration:
        version: 5000
        name: 'backup'
        state: 0
        txg: 98112
        pool_guid: 3980262890953715659
        hostid: 842186808
        hostname: 'fixit'
        vdev_children: 1
        vdev_tree:
            type: 'root'
            id: 0
            guid: 3980262890953715659
            children[0]:
                type: 'raidz'
                id: 0
                guid: 1130159151103149540
                nparity: 2
                metaslab_array: 30
                metaslab_shift: 37
                ashift: 12
                asize: 24004637229056
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 13474486044320785447
                    path: '/dev/dsk/c1t5000C5004E0A2CC0d0s0'
                    devid: 'id1,sd@n5000c5004e0a2cc0/a'
                    phys_path: '/scsi_vhci/disk@g5000c5004e0a2cc0:a'
                    whole_disk: 1
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 5573819435761518469
                    path: '/dev/dsk/c1t5000C500522B15A8d0s0'
                    devid: 'id1,sd@n5000c500522b15a8/a'
                    phys_path: '/scsi_vhci/disk@g5000c500522b15a8:a'
                    whole_disk: 1
                    create_txg: 4
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 4198804063194481999
                    path: '/dev/dsk/c1t5000C500537AF3E9d0s0'
                    devid: 'id1,sd@n5000c500537af3e9/a'
                    phys_path: '/scsi_vhci/disk@g5000c500537af3e9:a'
                    whole_disk: 1
                    create_txg: 4
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 18417867839860112832
                    path: '/dev/dsk/c1t5000C5005223EC89d0s0'
                    devid: 'id1,sd@n5000c5005223ec89/a'
                    phys_path: '/scsi_vhci/disk@g5000c5005223ec89:a'
                    whole_disk: 1
                    create_txg: 4
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 1089470385991005875
                    path: '/dev/dsk/c1t5000C5005334AA86d0s0'
                    devid: 'id1,sd@n5000c5005334aa86/a'
                    phys_path: '/scsi_vhci/disk@g5000c5005334aa86:a'
                    whole_disk: 1
                    create_txg: 4
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 10330363264372380124
                    path: '/dev/dsk/c1t5000C5006760A47Cd0s0'
                    devid: 'id1,sd@n5000c5006760a47c/a'
                    phys_path: '/scsi_vhci/disk@g5000c5006760a47c:a'
                    whole_disk: 1
                    create_txg: 4
                children[6]:
                    type: 'disk'
                    id: 6
                    guid: 3205134679639454644
                    path: '/dev/dsk/c1t5000C500523492F2d0s0'
                    devid: 'id1,sd@n5000c500523492f2/a'
                    phys_path: '/scsi_vhci/disk@g5000c500523492f2:a'
                    whole_disk: 1
                    create_txg: 4
                children[7]:
                    type: 'disk'
                    id: 7
                    guid: 16396885825294167808
                    path: '/dev/dsk/c1t5000C50053218101d0s0'
                    devid: 'id1,sd@n5000c50053218101/a'
                    phys_path: '/scsi_vhci/disk@g5000c50053218101:a'
                    whole_disk: 1
                    create_txg: 4
        features_for_read:
            com.delphix:hole_birth
            com.delphix:embedded_data

Uberblock:
        magic = 0000000000bab10c
        version = 5000
        txg = 98122
        guid_sum = 4010277285652565933
        timestamp = 1408618921 UTC = Thu Aug 21 05:02:01 2014

All DDTs are empty

Metaslabs:
        vdev          0
        metaslabs   174   offset                spacemap          free      
        ---------------   -------------------   ---------------   -------------
        metaslab      0   offset            0   spacemap    332   free     361M
 
TeeJayHoward said:
Looks like it let me take a snapshot of both backup and backup/storage.

Unfortunately, the snapshot is empty. Should I try a zfs send anyway?
Click to expand...

Yes, try it to send to Solaris 11 as this is the only OS capable reading v6 filesystems.
At least J1mmy was able to regain his data this way.
 
I think the empty snapshot is misleading. Remember that nothing is in the snapshot as it shows until and unless you start changing things. So, give it a try...
 
It kinda looks like it's doing something?

backup:
Code: 
root@fixit:~# /sbin/zfs send backup/storage@testing | ssh 192.168.0.11 /sbin/zfs recv pool/storage
Password:
(It hangs here)

nas:
Code: 
root@nas:~# zpool iostat pool 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
pool         395G  21.4T     30     29  3.84M  3.55M
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0     97      0  2.92M
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0    182      0  6.13M
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0     97      0  2.93M
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0      0      0      0
pool         395G  21.4T      0     98      0  2.81M
^C
root@nas:~# zfs list
NAME                              USED  AVAIL  REFER  MOUNTPOINT
pool                              281G  14.9T   281G  /pool
pool/storage                      192M  14.9T   192M  /pool/storage

At under a meg a second, we're talking over 150 days worth of copying. Any way to speed it up?
 
I tried using netcat instead. Same slow speed. Damnit!

Code: 
root@nas:~# netcat -w 30 -l -p 1337 | zfs receive pool/storage
netcat: -w has no effect with -l

Code: 
root@fixit:~# zfs send backup/storage@testing | nc 192.168.0.11 1337
 
You might try using mbuffer or netcat instead of ssh, I think both are much faster.
Format would be something like:
With netcat:
sending: zfs send -R main/data1/BEC@may3 | nc 10.90.0.14 1337
recv: nc -w 30 -l -p 1337 | zfs receive main/data1/BEC

edit: same thought I guess..lol.
 
Got mbuffer compiled and installed on both machines. Same issue. I wonder if it's even related to zfs? Remember, this is the issue I had which caused me to move my HDDs in the first place!

Code: 
root@nas:~# mbuffer -s 128k -m 1G -I 1337 | zfs receive pool/storage
in @  0.0 KiB/s, out @  0.0 KiB/s, 27.8 MiB total, buffer   0% full

Code: 
root@fixit:~# zfs send backup/storage@testing | mbuffer -s 128k -m 1G -O 192.168.0.11:1337
in @  0.0 KiB/s, out @  0.0 KiB/s, 25.4 MiB total, buffer 100% full
 
TeeJayHoward said:
I wonder if it's even related to zfs?
Click to expand...
I'm gonna take a guess and say yes. I used netcat to transfer a file between the two systems at line speed. Damnit.

So what's next in the diagnostics?

edit: After work today, I'll pull the "pool" zpool and put the disks in the OmniOS system. I'll try a zfs send/recv without the network in the mix at all.
 
Last edited:
WOOHOO!

Code: 
root@omnios:~# time zfs send backup/storage@testing | mbuffer | zfs receive pool/storage
in @  189 MiB/s, out @  178 MiB/s, 6017 MiB total, buffer 100% full

Should be done by the end of work tomorrow.


edit: Should I be able to see any files while the zfs send is going? Or will they only be visible at the end? I'm guessing end, and that I'll have to mount pool/storage in order to see them?
 
Last edited:
I believe you won't see anything until the end, when it mounts the newly created filesystem(s).
 
You must log in or register to reply here.
Back
Top