FreeNAS increaing storage problem:

Yes and please tell us the output of the follow

# zpool status -v

# zpool list
 
In PuTTY, just select text then it is copied to your clipboard automatically. Then paste in the postbox here on OCP. :)
 
Looks like maybe in order to import the destroyed pool I may need to destroy the pool with the unknown status, just a though.

Will this be too drastic?
 
Last edited:
In PuTTY, just select text then it is copied to your clipboard automatically. Then paste in the postbox here on OCP. :)

Last login: Mon May 25 16:48:05 2009
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.

Welcome to FreeNAS!

freenas:~# freenas:~#
freenas:~#: Command not found.
freenas:~#
freenas:~# zpool status -v
no pools available
freenas:~# #zpool status
#zpool: Command not found.
freenas:~# zpool list
no pools available
freenas:~# zdb -l /dev/da0
zdb: Command not found.
freenas:~#
 
Alright FreeNAS doesn't have the zdb library; its stripped down to be smaller in size.

Maybe a stupid question, but have you tried rebooting the system? I would be curious if that changes anything to the command output (zpool status being the most important one; that should list your array).
 
Alright FreeNAS doesn't have the zdb library; its stripped down to be smaller in size.

Maybe a stupid question, but have you tried rebooting the system? I would be curious if that changes anything to the command output (zpool status being the most important one; that should list your array).

Yes, I usually turn the FreeNAS box off at night when I am not using it.
 
Looks like maybe in order to import the destroyed pool I may need to destroy the pool with the unknown status, just a though.

Will this be too drastic?

Could this work or will it make matters worse?
 
How important is the data on your ZFS volume? If you really want it back, i can do it remotely if you set up a separate freebsd system disk, then i login and checkout your disks in native FreeBSD 8.

You could use most of the how-to guides on FreeBSD installation on my website, then open port 22 (ssh) on your router and forward it to the freebsd machine, then contact me discretely with the username and password. You can watch what i'm doing with the 'watch' utility; and we could keep in contact using IRC.

It would be some work on your end to set this up and connect only the FreeBSD system disk and the ZFS disks, while you have a second (windows?) based system to remain in contact and use PuTTY so you're logged in just like me and watch what i'm doing. :)

It does imply you trust me, of course. I wouldn't need to actually see your data, though. Just want to get the pool back up and mounted. Figure out what may have went wrong.
 
How important is the data on your ZFS volume? If you really want it back, i can do it remotely if you set up a separate freebsd system disk, then i login and checkout your disks in native FreeBSD 8.

You could use most of the how-to guides on FreeBSD installation on my website, then open port 22 (ssh) on your router and forward it to the freebsd machine, then contact me discretely with the username and password. You can watch what i'm doing with the 'watch' utility; and we could keep in contact using IRC.

It would be some work on your end to set this up and connect only the FreeBSD system disk and the ZFS disks, while you have a second (windows?) based system to remain in contact and use PuTTY so you're logged in just like me and watch what i'm doing. :)

It does imply you trust me, of course. I wouldn't need to actually see your data, though. Just want to get the pool back up and mounted. Figure out what may have went wrong.

It looks like I have just efed my data anyways :eek:, by deleting the unknown pool and trying to import, that diden't work so I have added a new pool from the two virtual devices and that probably killed the data.

The data wasn't that important a loads of movies that took ages to rip to the hard drives, I do get a zpool status though now:

$ zpool status
pool: NAS
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
NAS ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad4 ONLINE 0 0 0
ad6 ONLINE 0 0 0
ad8 ONLINE 0 0 0
ad10 ONLINE 0 0 0
raidz1 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0

errors: No known data errors

Thanks for your offer, really appreciate it, it looks like I went into drastic mode before I read your post...bummer.

This has now made made re-think the whole FreeNAS thing, I have to go with another os probably, I have found that there is a lot of information outhere to set up FreeNAS but if you try to do anymore and you come to a stumbling block...the only thing I was trying to do was to expand my pool with another vdev.

Ah well lesson learned and all that.

I really appreciate your help sub.mesa and vraa of course, bye the way can you let me know your website address,you never know I may learn a little about FreeBSD.
 
Yes, this is definitely a serious bug/issue on FreeNAS' part. I'm not sure what went wrong, but it's entirely possible that during the web-GUI expansion process some bug caused something to go awry.

I also feel i shouldn't recommend FreeNAS anymore, at least not for ZFS yet, until they release a well-tested version based on FreeBSD 8.0+ and ZFS version 13 or 14. They also have to test the GUI extensively.

If you're interested in FreeBSD, you can try the install guides (with screenshots) over here:
http://submesa.com/data/bsd

Once properly setup, it probably doesn't require you to do much maintenance, if any. You would use PuTTY as a command-line interface to the NAS; it doesn't really need a keyboard + monitor attached. But, of course, no nice web-GUI with FreeBSD. You would be running the 'production' version of ZFS, though. While the version in FreeNAS is still the older 'experimental' version; bugs still exist there that are ugly; though i'm unsure if your bug/issue originates from ZFS or FreeNAS' scripts, or perhaps both. :)
 
Yes, this is definitely a serious bug/issue on FreeNAS' part. I'm not sure what went wrong, but it's entirely possible that during the web-GUI expansion process some bug caused something to go awry.

I also feel i shouldn't recommend FreeNAS anymore, at least not for ZFS yet, until they release a well-tested version based on FreeBSD 8.0+ and ZFS version 13 or 14. They also have to test the GUI extensively.

If you're interested in FreeBSD, you can try the install guides (with screenshots) over here:
http://submesa.com/data/bsd

Once properly setup, it probably doesn't require you to do much maintenance, if any. You would use PuTTY as a command-line interface to the NAS; it doesn't really need a keyboard + monitor attached. But, of course, no nice web-GUI with FreeBSD. You would be running the 'production' version of ZFS, though. While the version in FreeNAS is still the older 'experimental' version; bugs still exist there that are ugly; though i'm unsure if your bug/issue originates from ZFS or FreeNAS' scripts, or perhaps both. :)

Thanks for the website info, I would advise anyone reading this thread to think very hard about their data integrity if they value it at all, if you are running FreeNAS re-think again.

I have just lost under 4tb of data which took 7 months of hard work to accumulate my data into one pool, yes that's right the FreeNAS ZFS pool.

Well if you think your data is safe, think again, I have managed to loose it all with FreeNAS and ZFS even without any hard drive failure.

So back to the drawing board for me, what is it to be FreeBSD (humm...reminder of FreeNAS), although FreeBSD 8 is running ZFS 13 if I am right, OpenSolaris? maybe, WHS, no way (no parity at all), Nexensta, eon?????

well who knows, next time before going for a NAS OS I will have to think twice about my data and the hard work that it took to put it together.

Appreciate any suggestion for a good NAS OS, preferably one that you can access via web gui as my NAS is kept in the loft, an OS that is not too slow, offer good software Raid or ZFS, that can be used with WOL and be able to be switched off remotly.

Many thanks for all your help.
 
Yes, this is definitely a serious bug/issue on FreeNAS' part. I'm not sure what went wrong, but it's entirely possible that during the web-GUI expansion process some bug caused something to go awry.

I also feel i shouldn't recommend FreeNAS anymore, at least not for ZFS yet, until they release a well-tested version based on FreeBSD 8.0+ and ZFS version 13 or 14. They also have to test the GUI extensively.

If you're interested in FreeBSD, you can try the install guides (with screenshots) over here:
http://submesa.com/data/bsd

Once properly setup, it probably doesn't require you to do much maintenance, if any. You would use PuTTY as a command-line interface to the NAS; it doesn't really need a keyboard + monitor attached. But, of course, no nice web-GUI with FreeBSD. You would be running the 'production' version of ZFS, though. While the version in FreeNAS is still the older 'experimental' version; bugs still exist there that are ugly; though i'm unsure if your bug/issue originates from ZFS or FreeNAS' scripts, or perhaps both. :)

I think your post is very well insightful

I am sure nbat58 could recover data (since he hasn't over written anything) but that would require perhaps some extra knowledge, nbat58 you can try the opensolaris forums / mailing lists

Obviously you already have a strong suggestion to go freebsd with those screenshots
I will always suggest opensolaris straight from genunix.org, get /dev right away -- works excellent headless as sub.mesa has said. Plus this way you're eating straight from the farm
 
I think your post is very well insightful

I am sure nbat58 could recover data (since he hasn't over written anything) but that would require perhaps some extra knowledge, nbat58 you can try the opensolaris forums / mailing lists

Obviously you already have a strong suggestion to go freebsd with those screenshots
I will always suggest opensolaris straight from genunix.org, get /dev right away -- works excellent headless as sub.mesa has said. Plus this way you're eating straight from the farm

I am not sure about getting any data back would be feasable seeing I don't get anything back with the zpool import -D command.

I will definitely look into open OpenSolaris, thank you for the suggestion, are there any different version specifically for NAS usage that I need to consider or not?
 
I have a similar problem, last night my FreeNAS server began to beep constantly so I rebooted it through the webgui. When the machine finished booting I logged back in to the webgui and I noted that my zpool (Raidz) was faulted. Here is a little more info,
my setup:

--------------------------------------------
FreeNAS 0.7.1 Shere (revision 4997)
intel pentium 4
tyan s5161
2Gb ram
adaptec aar-21610sa
1x 250Gb maxtor boot drive/storage
6x 1tb WD drives in raidz (pool name Raidz)
--------------------------------------------
Some commands I check per this thread (zdb command was not found)


freenas:~# kldstat
Id Refs Address Size Name
1 5 0x80400000 ad6c78 kernel
2 1 0x80ed8000 6a44c acpi.ko
3 1 0x85a47000 a6000 zfs.ko
4 1 0x85a30000 2000 opensolaris.ko

freenas:~# ls -l /dev/sd*
ls: No match.

freenas:~# ls -l /dev/ad*
crw-r----- 1 root operator 0, 72 Jul 17 10:53 /dev/ad2
crw-r----- 1 root operator 0, 73 Jul 17 10:53 /dev/ad2s1
crw-r----- 1 root operator 0, 85 Jul 17 10:53 /dev/ad2s1a
crw-r----- 1 root operator 0, 86 Jul 17 10:53 /dev/ad2s1c
crw-r----- 1 root operator 0, 74 Jul 17 10:53 /dev/ad2s2
crw-r----- 1 root operator 0, 89 Jul 17 10:53 /dev/ad2s2a
crw-r----- 1 root operator 0, 90 Jul 17 10:53 /dev/ad2s2c
crw-r----- 1 root operator 0, 75 Jul 17 10:53 /dev/ad2s3
crw-r----- 1 root operator 0, 91 Jul 17 10:53 /dev/ad2s3a
crw-r----- 1 root operator 0, 92 Jul 17 10:53 /dev/ad2s3c
freenas:~# ls -l /dev/da*
ls: No match.

freenas:~# dmesg | grep -i ZFS
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6

freenas:~# zpool import
no pools available to import

freenas:~# zpool status -v
pool: Raidz
state: FAULTED
status: The pool metadata is corrupted and the pool cannot be opened.
action: Destroy and re-create the pool from a backup source.
see: http://www.sun.com/msg/ZFS-8000-CS
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Raidz FAULTED 0 0 6 corrupted data
raidz1 FAULTED 0 0 6 corrupted data
aacdu0 ONLINE 0 0 0
aacdu1 ONLINE 0 0 0
aacdu2 ONLINE 0 0 0
aacdu3 ONLINE 0 0 0
aacdu4 ONLINE 0 0 0
aacdu5 ONLINE 0 0 0

freenas:~# zpool import -f
no pools available to import

freenas:~# zpool import -f Raidz
cannot import 'Raidz': no such pool available

freenas:~# mount
/dev/ad2s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local)
====================================================

Most of the data on this pool is replaceable but I had some pictures on this pool that were not backed up that I would really like to recover. At the time that the machine started to beep I was verifying a torrent and streaming a movie. Any help would be really appreciated.
 
Looks like one of the early bugs hit you; the old and 'experimental' ZFS code is long obsolete and no one should use it now that ZFS has stable code in FreeBSD 8.0/8.1.

However, you could simply try booting a ZFS v14 and see if it can access/fix your pool. For example, you could try my Mesa preinstalled image, if just to quickly have a FreeBSD 8.1 / ZFS v14 environment to test your drives with. zpool import should display it; this can also be done through the web-GUI.

Instructions on the pre-installed binary Mesa image containing FreeBSD 8.1RC1:
http://submesa.com/mesa

Also, did you run memtest86+ on that system? Can you rule out a hardware failure?
 
As for a hardware failure it may be possible but I am unsure. I have been having problems with my setup since I switched to ZFSv6 which I attributed to only having 2GB of ram.

I booted your custom image on a different motherboard as freebsd would ask to mountroot manually on my old tyan motherboard. This is the output I get from clicking "search from hidden pools".

===============================================
pool: Raidz
id: 14119036174566039103
state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:

Raidz ONLINE
raidz1 ONLINE
ada3 ONLINE
ada5 ONLINE
ada4 ONLINE
ada1 ONLINE
ada2 ONLINE
ada0 ONLINE
=================================================
(BTW your custom image is awesome). I look forward to 0.20 release.

I think this is good news (I hope). How would I go about importing the zpool? I couldn't find this option of the gui. Is this where the cmdline comes in? Would I just use "zpool import -f Raidz"? After its imported then how would I go about accessing the files?

Thank you so much.

edit: thanks for the quick reply, I had been up all night trying to figure this out and decided to post before getting some rest.
 
Go to the root command line (enable in preferences first), and enter:
zpool import Raidz

The Raidz is case-sensitive. Note that raidz is a reserved word, you generally should chose a different name than "mirror" "raidz" or "raidz2" etc. Refer to your drives instead, like "samsung" if it consists of Samsung disks. The disks themselves i label as "samsung2T-disk1" for example, 2T meaning 2TB. But you can't/shouldn't apply labels on an already existing setup; you have to do that before you setup the array.

That's just background info though. After using the zpool import <poolname> command manually, you should be seeing the pool in the Pools page, and your filesystems and all. Try it! If it says online then it means its not corrupted at all. Rather a bug with old ZFS contained in FreeNAS.

So even though the version of your pool is still version 6, the fact that you use a newer/modern implementation of ZFS (v14) that is labeled 'stable' would resolve your bug; as the bug is likely not in the stored data on the physical disks, but rather how ZFS deals with them. Thus, your data should be fine. If i'm right, that is. You will see soon enough after the import, check the status of your array with zpool status or look at the web-interface.

Good luck!
 
I tried root access "zpool import Raidz" with "WARNING: got return value 1" and with output "mesa.bsd$ zpool import Raidz" only. The pool is not imported. I then tried "zpool import -f Raidz" and it caused a fatal trap 12 and the system became unresponsive. I tried the same commands through ssh as root and I got this below:
----------------------------------------------------------------------------
[root@mesa ~]# zpool import Raidz
cannot import 'Raidz': pool may be in use from other system
use '-f' to import anyway
----------------------------------------------------------------------------

Then I tried this command and I got this:
+++++++++++++++++++++++++++++++++++++++++++++

[root@mesa ~]# zdb -l /dev/ada0
--------------------------------------------
LABEL 0
--------------------------------------------
version=6
name='Raidz'
state=0
txg=11730350
pool_guid=14119036174566039103
hostid=0
hostname='freenas.local'
top_guid=16879648846521942561
guid=6282477769796963197
vdev_tree
type='raidz'
id=0
guid=16879648846521942561
nparity=1
metaslab_array=14
metaslab_shift=32
ashift=9
asize=6000992059392
children[0]
type='disk'
id=0
guid=6543046729241888600
path='/dev/aacdu0'
whole_disk=0
children[1]
type='disk'
id=1
guid=14313209149820231630
path='/dev/aacdu1'
whole_disk=0
children[2]
type='disk'
id=2
guid=5383435113781649515
path='/dev/aacdu2'
whole_disk=0
children[3]
type='disk'
id=3
guid=9586044621389086913
path='/dev/aacdu3'
whole_disk=0
DTL=1372
children[4]
type='disk'
id=4
guid=10401318729908601665
path='/dev/aacdu4'
whole_disk=0
children[5]
type='disk'
id=5
guid=6282477769796963197
path='/dev/aacdu5'
whole_disk=0
--------------------------------------------
LABEL 1
--------------------------------------------
version=6
name='Raidz'
state=0
txg=11730350
pool_guid=14119036174566039103
hostid=0
hostname='freenas.local'
top_guid=16879648846521942561
guid=6282477769796963197
vdev_tree
type='raidz'
id=0
guid=16879648846521942561
nparity=1
metaslab_array=14
metaslab_shift=32
ashift=9
asize=6000992059392
children[0]
type='disk'
id=0
guid=6543046729241888600
path='/dev/aacdu0'
whole_disk=0
children[1]
type='disk'
id=1
guid=14313209149820231630
path='/dev/aacdu1'
whole_disk=0
children[2]
type='disk'
id=2
guid=5383435113781649515
path='/dev/aacdu2'
whole_disk=0
children[3]
type='disk'
id=3
guid=9586044621389086913
path='/dev/aacdu3'
whole_disk=0
DTL=1372
children[4]
type='disk'
id=4
guid=10401318729908601665
path='/dev/aacdu4'
whole_disk=0
children[5]
type='disk'
id=5
guid=6282477769796963197
path='/dev/aacdu5'
whole_disk=0
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3
+++++++++++++++++++++++++++++++++++++++++

Seems like hostname is retained from the freenas install and the path= is the old drive names. Any ideas?
 
That shouldn't be the problem; you say the system locks up when using the import -f command? Then perhaps there is some damage ZFS really can not fix, and even crashes upon. You would do well to post this to freebsd-fs mailinglist, or OpenSolaris ZFS strongholds to get help from some (real) experts.

I'm a little surprised it crashes on the import though; that must be a bug regardless of your on-disk configuration. Can you capture any output when that happens?

On a related note, my image does run 8.1RC1; the current release is RC2 and the final has not yet been released. There's a chance the import fails due to this.

You can also try an OpenSolaris image with recent/development ZFS version (v24 is latest available dev i believe). Try to import your pool the same way (zpool import and then zpool import -f Raidz).
 
He said that he rebooted because it was "beeping constantly". In my experience computers don't randomly beep constantly without being programmed to do so. It sounds like a hardware problem. Could be as simple as keyboard failure but something effed up to make it beep.

Dustin
 
Yea, it was beeping but no monitor was connected to the system and I am not sure where the error logs would be stored to see what messed up. I do know that I connected the drives back to the controller and the led's for one of the ports did not light up but the drive was recognized by the OS. I then booted up opensolaris (2009.06) and 4 out of the 6 disks showed corrupt data when I did "zpool import." When I tried to import the pool it couldn't but I can't remember the output. I have never used opensolaris and could not ssh into it to record the outputs. With "zpool import -f Raidz" opensolaris did not crash but wouldn't import either.

I am currently looking for the dev opensolaris with the latest ZFS version but can't find a the download. If this doesn't work I think I may have to hire a professional as I feel very guilty for losing my family's pictures.
 
Back
Top