Trying to understand ZFS

[svar]

Hi

Im geeking around with an FreeNAS on an virtual mashine and trying to make an working ZFS pool in terminal.
I use the "zpool create tank raidz ad6 ad8 ad10 ad12 ad14" command, and I got an pool that I can then use the command
"zfs create -V 10g tank/ufs", setup that over iscsi (in the webgui), mount it to my mac, format it and move files, but then I remove an disk to see if the raid is working, (I have to shut down the VM and reboot, can remove disks on the fly), the whole zpool is gone
If I reconnect the disk, I have to create the pool again
What Im missing?

 

[sub.mesa]

I tougt it was sonething like that.
For this testing Im using FreeNAS on Suns VirtualBox, and another funny thing is if I remove an disk in the config (to see if the raidz works) the whole pool is gone then I boot, but I guess thats an virtual box thing, and would not happen on an real box.

No virtualbox should work as well; i develop my project in virtualbox.

Could you start by telling me what you do exactly? I mean you create pool and then it is gone; was there ever a pool created? Did you see the pool with zpool status?

Likely the pool is either not mounted, or not imported. Try:

zpool import -a
zfs mount -a

In FreeNAS, I dont see any disks if I do all the zfs work from commandline

Logical; since FreeNAS uses MBR partitions. Generally you should follow the method of the GUI prefers/expects. Likewise my project doesn't list bare disks as available disks to create a pool from; you would need to format first. If you already have a pool with bare disks, the GUI should pick that up though. Any disk-related interaction might not work via the interface without formatting the disks via the interface.

As far as I see, I have no tools in FreeNAS to see the used/free space on a file.level, it only show the block-level, and that I guess is normal because FreeNAS never made an filesystem on the block, OSX did over iSCSI.

Yes; when using SAN protocol like iSCSI you don't really know anything about the filesystem/space in use. The server wouldn't see your files.

Edit:
Ok I yust found out that if I in FreeNAS use this comand direct (not using the webGUI)
"zpool create tank raidz ad6 ad8 ad10 ad12 ad14"
and remove an disk, the whole pool disapair.

It should not! It should be degraded if you lose one disk in RAID-Z configuration. What does zpool status output tell you?

I don't know FreeNAS well at all though; i can only guess on how it could be interfacing with FreeBSD OS instead.

 

[svar]

Here is what I do

1: Start off with an VM 6 drives (1 for os, 5 for data), clean install of the FreeNAS embedded mode.

2: Start the webgui to turn on SSH, and then jump into terminal on my mac.

3: "zpool create tank raidz ad6 ad8 ad10 ad12 ad14"

4: Zpool status, all seems fine:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad6 ONLINE 0 0 0
ad8 ONLINE 0 0 0
ad10 ONLINE 0 0 0
ad12 ONLINE 0 0 0
ad14 ONLINE 0 0 0

errors: No known data errors


(5): Shuting down, remove an drive, boot, "Zpool Status", all gone ("no pools available")

So I started all over again, down to step 4 and then:

5: zpool import -a
6: zfs mount -a

7: Shuting down, remove an drive, boot, "Zpool Status", all gone ("no pools available")


If I go thrue the web-gui, and add the disks, format them to ZFS-Storage-Pool, add an virtual device, add an device in managment, it all works as intended.

I can then run "zfs create -V 200g tank/ufs" and put it up in iSCSI but if I then try to reboot and remove an disk, FreeNAS at boot stops with the message
"Load NOP GEOM class", and seems to freez.

If I make an file on the pool instead and use it for iscsi, all is ok.

To me it seems like freenas is not ment to be setup in terminal.

 

[svar]

ok, I played around with the Mesa live CD, and found out that if I do the same here as in step one, same happens, pool gone.

Thanks to Mesa "find hidden pool" button, I can find the pool, but it show that data is damaged and I cant import it.

One thing that Mesa and FreeNAS has in common, and here is the key I think:
From the WebGUI, you CANT add an pool unless you format the drives.
If you do as I do, and create the pool on none formatted disks, it dont work if a disk is gone.

Tried now on FreeNAS to create an pool in terminal after I had formattet them as ZFS Storage Pool Device in webGui, and I got an pool.
I cant see the pool in the disk--managment tab, I think FreeNAS only show the disks/pools that has been added in the WebGUI here (mesa show the pools I add in terminal, so I guess its how FreeNAS gui is coded)

I then removed an disk, and I had to use "Zpool import tank" to get it back up, but it worked (same in Mesa).

If I only now can figure out the format command Mesa or FreeNAS use in the GUI and the set lable command, I think I can try this all in terminal.

 

[sub.mesa]

ok, I played around with the Mesa live CD, and found out that if I do the same here as in step one, same happens, pool gone.

Thanks to Mesa "find hidden pool" button, I can find the pool, but it show that data is damaged and I cant import it.

This is interesting! You might use a controller that is actually a FakeRAID controller, such as Promise, Silicon Image, JMicron, and so on. These overwrite the last 512 bytes.

To protect against this, use partitions which reserve space at the end. My ZFSguru project also does this, if you format disks with GPT, it reserves by default 1 megabyte at the end of the capacity of the disk.

This will make sure the controller BIOS won't interfere (overwrite) your data.

Also, it may be that you're hit by Gigabyte motherboards overwriting stuff on HDDs with a HPA, that acts as a 'software' version of their dual-BIOS feature; very annoying feature well explained here:
http://forums.overclockers.com.au/showthread.php?t=885212&page=3

One thing that Mesa and FreeNAS has in common, and here is the key I think:
From the WebGUI, you CANT add an pool unless you format the drives.

Correct; at least my project requires that you label the drive with a name. That makes it less easy to make mistakes that cause dataloss; for example adding the wrong disk to the pool which contained real data but was shown as available because you didn't import it yet; stuff like that.

I cant see the pool in the disk--managment tab, I think FreeNAS only show the disks/pools that has been added in the WebGUI here (mesa show the pools I add in terminal, so I guess its how FreeNAS gui is coded)

Correct; even if you don't use partitions but raw devices and set this up under CLI, under my project you would see the pool; even though you didn't follow the 'official' method. FreeNAS may very well be different; but i want my project to work in more diverse configurations as well. For example manually created pools.

I then removed an disk, and I had to use "Zpool import tank" to get it back up, but it worked (same in Mesa).

Remember that my LiveCD forgets your pool on reboot; the data isn't gone but you need to re-import it every time. Only the USB distribution remembers your pools, and of course ZFS-on-root installs. Generally the LiveCD should be used to perform a ZFS-on-root installation; after which you don't need the livecd anymore.

If I only now can figure out the format command Mesa or FreeNAS use in the GUI and the set lable command, I think I can try this all in terminal.

GEOM label:

glabel label MYDISK /dev/ad0

GPT is more complicated.

Could you try this:
1. create glabel disks; see if after reboot it shows as damaged etc, if it does, go to step 2
2. create GPT disks; format via my GUI; reboot and if it does NOT corrupt anymore, then you have something that overwrites the last portion of your HDDs.

What controller are you using? Any controllers running in RAID mode? Onboard RAID?

 

[svar]

Hi Mesa, and thanks again for youre reply

All I talk about in this thred is done in an VM, I now use Parallels desktop 6 for mac, but I alsa use Oracle VirtualBox, and none raid controllere there.

 

[sub.mesa]

So, aside from the LiveCD forgetting your pool (having to import everytime), does the ZFSguru web-interface give you problems with the pool across reboots? You said something about a corrupt pool when you tried to import, was that using the FreeNAS distribution? Did you write to the disks like create partition on existing pool?

 

[svar]

Hi

I used the 0.1.5, I now have dowloaded the 0.1.6 and installed it on disk.
Played around in webGUI, deleted disk and replaced it with an blanc one, had to use the Zpool replace comand in terminal to get it back online, so it seems to work

Somehow I think youre project is a lot better than FreeNAS, but I shure miss some stuff in GUI:

-iSCSI
-making of SAMBA users and shares

Ow, the start/stop/restart buttons at the service tab seems to not work.

 

[sub.mesa]

Those buttons on the Service page have no functionality attached to them, that page will be rewritten soon.

iSCSI support is also a must; expect this 0.1.7 or 0.1.8.

Replacing failed disks isn't yet supported via the GUI; this would be a must too. I'll focus on this kind of core features after the benchmarking feature is done.

Samba shares should already work, samba users is more difficult; this may be much later in development cycle. User management is kind of tricky. For now i focus on a single user. Though nothing prevents you tuning some stuff outside the web-gui, at least until there is proper support for it in the GUI.

Does creating samba shares work for you with 0.1.6 fresh ZFS-on-root installation? Make sure not to have any USB sticks of older Mesa releases plugged in; otherwise it would default to the older USB stick instead of 0.1.6 livecd. ;-)

Cheers.