OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

What about Mac extended attributes on files? At first glance it appears they're not coming through. I say this because an example JPEG which, when stored locally on the Mac, shows lots of info in Finder Get Info, such as camera model, dimensions, etc, but when copied to the server (OmniOS/kernel SMB), does not display any of that info.

OSX and its file system capabilities are based on HFS.
Most but not all features are kept with extra .files (same filename) when you use "foreign" filesystems like fat or any other example via NFS/SMB shares.

OSX hides the extra files. If you connect such a share via Windows or Unix/Linux you see the extra files
 
Another question. What about Spotlight support for netatalk on Illumos-based OSs like OmniOS?

I know this requires Gnome Tracker being installed and netatalk being compiled with support for it. This seems possible on FreeBSD although my half-hearted efforts at making it work weren't fruitful.

Would it work on Illumos?

This depends not on Illumos but the question if you (can) compile all netatalk features.
I use the ips repo at University Ulm for netatalk on Illumos, OmniOS Package Repository: uulm.mawi
The netatalk there is without Spotlight support.

Beside that (my personal opinion)
AFP is EoL, Apple moved to SMB as default protocol
Netatalk is EoL for the same reason, Netatalk - SerNet
Beside that, netatalk was a pain because of Applle and all the incompatibilities with non HFS filesystems
 
OSX and its file system capabilities are based on HFS.
Most but not all features are kept with extra .files (same filename) when you use "foreign" filesystems like fat or any other example via NFS/SMB shares.

OSX hides the extra files. If you connect such a share via Windows or Unix/Linux you see the extra files

So I played around with this further. I think the data the Finder is showing about the pics is NOT actually extended attributes since running xattr -l file.jpg returns no info. But still something odd is happening. Maybe it's just standard Finder behavior. Consider this sequence of events:

1. File is on local HFS volume. Finder Get Info displays camera data.

2. Copy file to OmniOS server. Finder displays no camera info. There is also no hidden ._file.jpg.

3. The odd part. Copy the file back to the HFS volume and it AGAIN displays camera info, nothing was lost.

So it seems that the camera info isn't extended attributes at all, it's just read from the metadata of the picture. But Finder only displays the info if the file is on an HFS volume (well actually local OpenZFS volumes also display it).

BUT returning to actual extended attributes, by typing a comment into the Finder Get Info window or setting a color label on the file, you DO generate actual extended attributes. xattr -l file.jpg shows these. Copying this file to OmniOS, the comments and labels stay. And, this is the important part, there is still NO ._file.jpg. So I can only assume the attributes are being stored natively by ZFS.
 
3. The odd part. Copy the file back to the HFS volume and it AGAIN displays camera info, nothing was lost.

So it seems that the camera info isn't extended attributes at all, it's just read from the metadata of the picture. But Finder only displays the info if the

There are no extended Mac attributes on ZFS/SMB beside the ._ files
I suppose the ._ file is hidden.
Do you have a Windows machine around. Windows shows the . files
 
There are no extended Mac attributes on ZFS/SMB beside the ._ files
I suppose the ._ file is hidden.
Do you have a Windows machine around. Windows shows the . files

I looked via a Windows VM. There are no ._ files visible. And are you saying that somehow, logged onto the OmniOS machine and doing an ls -a would not show the files?

What there is, is a .DS_Store file created by Finder when it visits any folder. Via the Windows VM, I deleted that, on the theory that the extended attributes were coming from it. Then, without visiting the folder from Finder, I used Terminal to go there (cd /volumes/myshare/myfolder). Then I did xattr -l file.jpg, which STILL shows the extended attributes (a comment and color label that I added). Also ls -a showed no files beginning with a .

So, unless OmniOS/ZFS/SMB is using some kind of crazy file hidden from ls -a and Windows, I can only conclude that the extended attributes are stored in the file itself. Am I missing something? Is there any way, on Illumos, to issue some command that would show the extended attributes of a file, or at least confirm their presence? That would verify that somehow they really are getting into the file itself.

On the netatalk side of the fence, am I correct that an ea = sys statement in the config file WILL actually, for sure, result in the extended attributes being stored directly inside the files by ZFS?
 
I am not aware of an option to safe extended Mac attributes over SMB to ZFS.
This is different to netatalk that can safe extended Mac attributes (ex via the ea setting)
 
I am not aware of an option to safe extended Mac attributes over SMB to ZFS.
This is different to netatalk that can safe extended Mac attributes (ex via the ea setting)

Well, it sure LOOKS like it's saving them over SMB to ZFS. If anyone cares to know for sure, I'm happy to test further, but until someone suggests how to test or pokes a hole in what I've done, I'll have to conclude that it's working.
 
Well, it sure LOOKS like it's saving them over SMB to ZFS. If anyone cares to know for sure, I'm happy to test further, but until someone suggests how to test or pokes a hole in what I've done, I'll have to conclude that it's working.

Sorry to keep going with this if nobody else is interested.

I found that you can run ls -@ to confirm the presence of extended attributes in a file. If the file has them, instead of a + at the end of the permissions to indicate an ACL, it will print an @ only if extended attributes are present.

I ran this test:

1. Make a file on OmniOS server by vi test.txt.

2. Run ls -@ file.txt on Omni, only + displayed.

3. Do Get Info in Finder, or open the file in TextEdit, but make no changes.

4. Run ls -@ file.txt on Omni, only + displayed.

5. Set a Tag (color) in Finder, or put a comment in Get Info. I know tags and comments are stored as extended attributes on Macs.

6. Run ls -@ file.txt on Omni, now @ is displayed indicating presence of extended attributes in the file.

For step 5, I could also have edited the file on the Mac using TextEdit and the @ would have appeared. But, editing on the Mac using vi does not cause the @ to appear.

Summary: ls -@ confirms presence of extended attributes on files only after they have metadata set in Finder or are edited with GUI tools.
 
Is it possible to turn off ACLs and just use linux style file permissions? ACLs are giving me loads of problems.
 
Is it possible to turn off ACLs and just use linux style file permissions? ACLs are giving me loads of problems.

I tried that. Didn't work for me because if a file doesn't have ACLs, my Mac would invent them, and they were screwy. If the file already has legitimate ACLs the Mac respects them and all is good.

What kinds of problems are you having? Once I got mine worked out, they work great.
 
I tried that. Didn't work for me because if a file doesn't have ACLs, my Mac would invent them, and they were screwy. If the file already has legitimate ACLs the Mac respects them and all is good.

What kinds of problems are you having? Once I got mine worked out, they work great.

I use my NAS as an NFS mount for various applications. Some of those applications (coincidentally all linux based) apparently want to chown and/or chmod files. The NAS is then interpreting those changes (I assume some interpretation) in to ACL changes. The application still works fine, however I then do a pool to pool rsync of those folders to a backup, and then finally have a different application (windows based using samba) read from those backups for an off-site cloud backup. That backup application looses read access because the permission tom-foolery that the application does trickles through to the backup copies. This is even though my samba share of the backup pool permits guest access; apparently ACLs take precedence.

I used napp-it in the past in the same manner without any issues but back then it was just simple unix file permissions. I then dabbled with btrfs for a while and eventually came back to zfs but have discovered this issue that I didn't have before.

I see a couple avenues to fix it, but haven't had luck on any of them.

1. Assuming I'm correct about the ACLs taking precedence over samba guest access - just turn off ACLs.
2. Configure the NFS clients to connect to NFS in a way that doesn't permit them to attempt permission changes on files
3. Configure the NFS server in a way that doesn't permit clients to attempt permission changes on files.

I don't know if any of these are possible, I've tried all 3 approaches without success. I'm all ears to other options too! Thanks.

EDIT: I found a hacky solution that might work; adding the --chmod option to my pool-to-pool rsync. I'll test it.
 
Last edited:
Sorry to keep going with this if nobody else is interested.

I found that you can run ls -@ to confirm the presence of extended attributes in a file. If the file has them, instead of a + at the end of the permissions to indicate an ACL, it will print an @ only if extended attributes are present.

I ran this test:

1. Make a file on OmniOS server by vi test.txt.

2. Run ls -@ file.txt on Omni, only + displayed.

3. Do Get Info in Finder, or open the file in TextEdit, but make no changes.

4. Run ls -@ file.txt on Omni, only + displayed.

5. Set a Tag (color) in Finder, or put a comment in Get Info. I know tags and comments are stored as extended attributes on Macs.

6. Run ls -@ file.txt on Omni, now @ is displayed indicating presence of extended attributes in the file.

For step 5, I could also have edited the file on the Mac using TextEdit and the @ would have appeared. But, editing on the Mac using vi does not cause the @ to appear.

Summary: ls -@ confirms presence of extended attributes on files only after they have metadata set in Finder or are edited with GUI tools.

Good news as this gives Mac users a better user experience.

btw
In current napp-it 16.04 you can just enable Bonjour/ multicast dns in Menu Services > Bonjour to connect SMB shares directly from a Mac with the nice Xserve icon instead the oldstyle SMB monitor icon with a Windows bluescreen.
 
I tried that. Didn't work for me because if a file doesn't have ACLs, my Mac would invent them, and they were screwy. If the file already has legitimate ACLs the Mac respects them and all is good.

What kinds of problems are you having? Once I got mine worked out, they work great.

You cannot disable ACL as the Solaris SMB server works like Windows: NTFS alike permissions only.
What is your problem. ACLs are superior.
 
You cannot disable ACL as the Solaris SMB server works like Windows: NTFS alike permissions only.
What is your problem. ACLs are superior.

Multiple linux applications mount various directories via NFS and make chmod changes. A single offsite backup application needs to read everything.

The linux applications setting chmod=xx0 turn in to ACLs that prevents the backup application connecting via samba from reading the directories. Whatever ACLs I set natively are squashed by the linux applications.

ACLs might be superior in some circumstances, but in this circumstance they're just causing problems and I haven't found any way to reign them in and get them under control.
 
EDIT: I found a hacky solution that might work; adding the --chmod option to my pool-to-pool rsync. I'll test it.

This might work, but unfortunately in napp-it when a job has an = sign, that seems to cut off the display of the full command in the UI. I checked and the full command is still there in the job's .par file but I don't have the best memory so 6 months down the line I'd forget what the command is if I can't just read in the gui and I certainly don't want to have to re-discover where those job files are located any time I want to make a change or I just want to refresh my memory.
 
Multiple linux applications mount various directories via NFS and make chmod changes. A single offsite backup application needs to read everything.

The linux applications setting chmod=xx0 turn in to ACLs that prevents the backup application connecting via samba from reading the directories. Whatever ACLs I set natively are squashed by the linux applications.

ACLs might be superior in some circumstances, but in this circumstance they're just causing problems and I haven't found any way to reign them in and get them under control.

There is indeed an incompatibility as
- Solaris CIFS is Windows alike
- NFS3 is Unix alike but without Authentication or Authorisation
- rsync does not support Solaris NFS4 ACL at all

What you can try
- Select an everyone@=allow ACL with inheritance enabled
- hinder NFS to modify permissions with aclmode=restricted
 
There is indeed an incompatibility as
- Solaris CIFS is Windows alike
- NFS3 is Unix alike but without Authentication or Authorisation
- rsync does not support Solaris NFS4 ACL at all

What you can try
- Select an everyone@=allow ACL with inheritance enabled
- hinder NFS to modify permissions with aclmode=restricted

Thank you. Is aclmode=restricted set on the server or client side?

EDIT: Oh I see, I think you're talking about in the filesystem. That's not an NFS option right?
 
I have some limited experience with Storage Spaces and ReFS. First, if you're using their equivalent of RAIDZ/2, which they call "parity spaces" I think, the performance is awful. This is well-known. Their equivalent of a mirror is fine. Also, dealing with Storage Spaces is very convoluted compared to ZFS. I tried to implement it for our server at work but I just couldn't bring myself to. It was just so clunky compared to ZFS. I ended up with a standard hardware RAID controller (gasp!). If I were doing it over now I'd do either OmniOS running an iSCSI target or have a go at directly sharing via the built-in SMB 2.1 server, assuming that can be properly and seamlessly integrated into an AD domain, which I have no experience or information about.

Thanks. I've done my own reading and most reports do say it's not great. Especially when compared to ZFS. So i'll stick with ZFS (and no Windows at all).

Can someone explain the read/write cache options with SSDs in ZFS. I'm going to get some (of the better ones) 128GB SSDs but I'm not sure on configuring them or how many to use for best results. I'll lose how ever many spinning disks as SSDs I use, which is fine, because I don't have port capacity.

Is it one (or more) SSD per raid group, or per pool.
 
There are different caches and options with SSDs and ZFS.

Write cache:
This is always RAM based (a few seconds of write) and transforms small slow random writes to a single fast sequential write.

Read cache
The first readcache is rambased and called ARC. It works blockbased in a combination of last and most accessed blocks. You can extend the ARC with an SSD. This is called L2ARC. Such an L2ARC works on a per pool base.

Slog device
This is not a cache but a logdevice that logs all writes to a dedicated logdevice to allow fast writes over the regular rambased writecache to your pool. In case of a power outage, last writes can be restored from this logdevice. You need powerloss protection on the device ex ZeusRAM, Intel S3700 or Intel P750. Think of the Slog like a BBU + cache on a hardware raid. Similar - but faster.

A logdevice can be quite small (10GB) but must must offer a very low latency and very high write iops. You must enable or allow sync write if you want to use the Slog. This is not a performance setting but a security option. Even with the fastest slog, writes are slower than with sync=disabled as you must write all data to your pool and additionally to the logdevice. A logdevice is also per pool but you can set sync on a per filesystem base.
 
Last edited:
I've been running new OmniOS r151018 with kernel SMB2 since a day or two after it appeared. So far I'm pretty impressed. I'd still like Spotlight and finally did find that OpenCSW has a Netatalk package that installs Tracker too and is pre-configured for Spotlight. I'm testing it in a virtual machine. But SMB2 is so good and stable and everything that I'm not sure I'll ever use it. We'll see.

This post is not about that however. The one issue I seem to have, using the Mac Finder, is that some operations are slower than I would expect. I'm running over 10GbE and find copying to and from the server to be quick. However, three things seem slow so far.

1. Renaming a folder with a lot of stuff in it.
2. Moving a folder with a lot of stuff in it.
3. Deleting a folder with a lot of stuff in it.

Folders with little content do all three operations quickly via Finder.

Number 3 I can kind of understand, after all, there's more to delete. But it's still way slower than if it were happening locally, and way slower than I would expect, given the general speed of the server otherwise.

Numbers 1 and 2 I don't understand at all. I can't see how the contents of the folder should effect renaming it, or even moving it. I can log onto the server by ssh and rename a folder and it happens instantaneously. Why slow via Finder? Does it maybe have to do with ACL inheritance (which I do have)?
 
1. Renaming a folder with a lot of stuff in it.
2. Moving a folder with a lot of stuff in it.
3. Deleting a folder with a lot of stuff in it.

Folders with little content do all three operations quickly via Finder.
Number 3 I can kind of understand, after all, there's more to delete. But it's still way slower than if it were happening locally, and way slower than I would expect, given the general speed of the server otherwise.
Numbers 1 and 2 I don't understand at all. I can't see how the contents of the folder should effect renaming it, or even moving it. I can log onto the server by ssh and rename a folder and it happens instantaneously. Why slow via Finder? Does it maybe have to do with ACL inheritance (which I do have)?

Well, never mind I guess. 1 and 2 are immediate now (didn't test 3). Not sure what changed... Something on the Mac side I guess.
 
Quick question on which drives are bad on this system.

The "degraded" is next to the spare-1, spare-2, and spare-3. Are the spares degraded or the drives that say "unvailable"?

And why does the pool still show degraded if the spares resilvered?


Output of zpool status:

NAME STATE READ WRITE CKSUM
backup DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c0t50014EE20B220BA4d0 ONLINE 0 0 0
c0t50014EE20B237D95d0 ONLINE 0 0 0
spare-2 DEGRADED 0 0 0
c0t50014EE20B2550DAd0 UNAVAIL 46 61 0
c0t50014EE2B5D2F865d0 ONLINE 0 0 0
c0t50014EE20B271A01d0 ONLINE 0 0 0
c0t50014EE20B277DD8d0 ONLINE 0 0 0
c0t50014EE20B27A6C2d0 ONLINE 0 0 0
c0t50014EE20B27ABE2d0 ONLINE 0 0 0
c0t50014EE20B27C6C4d0 ONLINE 0 0 0
c0t50014EE20B282A71d0 ONLINE 0 0 0
c0t50014EE20B28397Ad0 ONLINE 0 0 1
raidz2-1 ONLINE 0 0 0
c0t50014EE20B28456Dd0 ONLINE 0 0 0
c0t50014EE20B286057d0 ONLINE 0 0 0
c0t50014EE20B28607Fd0 ONLINE 0 0 0
c0t50014EE20B286F21d0 ONLINE 0 0 0
c0t50014EE20B287593d0 ONLINE 0 0 0
c0t50014EE260791674d0 ONLINE 0 0 0
c0t50014EE260794440d0 ONLINE 0 0 0
c0t50014EE2607B5CE5d0 ONLINE 0 0 0
c0t50014EE2607C7F95d0 ONLINE 0 0 0
c0t50014EE2607C7FD9d0 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
c0t50014EE2607CBA76d0 ONLINE 0 0 0
c0t50014EE2607CC79Cd0 ONLINE 0 0 0
c0t50014EE2607CC7BFd0 ONLINE 0 0 0
c0t50014EE2607CC7D5d0 ONLINE 0 0 0
c0t50014EE2607CE1F6d0 ONLINE 0 0 0
c0t50014EE2607D477Fd0 ONLINE 0 0 0
c0t50014EE2607D4D92d0 ONLINE 0 0 0
c0t50014EE2607D4DCCd0 ONLINE 0 0 0
c0t50014EE2607D6630d0 ONLINE 0 0 0
c0t50014EE2607DC913d0 ONLINE 0 0 0
raidz2-3 DEGRADED 0 0 0
c0t50014EE2607DDC11d0 ONLINE 0 0 0
spare-1 ONLINE 0 0 1
c0t50014EE2607DE23Cd0 ONLINE 0 0 0
c0t50014EE2B5D28DC2d0 ONLINE 0 0 0
c0t50014EE2B5CFC8FDd0 ONLINE 0 0 0
spare-3 DEGRADED 0 0 0
c0t50014EE2B5D198A3d0 UNAVAIL 133 170 0
c0t50014EE2B5D2A589d0 ONLINE 0 0 0
c0t50014EE2B5D21984d0 ONLINE 0 0 0
c0t50014EE2B5D237ECd0 ONLINE 0 0 0
c0t50014EE2B5D2507Dd0 ONLINE 0 0 0
c0t50014EE2B5D272E9d0 ONLINE 0 0 0
c0t50014EE2B5D27B4Ed0 ONLINE 0 0 0
c0t50014EE2B5D28A77d0 ONLINE 0 0 0
spares
c0t50014EE2B5D28DC2d0 INUSE
c0t50014EE2B5D2A589d0 INUSE
c0t50014EE2B5D2F865d0 INUSE
c0t50014EE2B5D358CFd0 AVAIL
c0t50014EE2B5D3AF6Ed0 AVAIL
 
Only the disks marked as UNAVAIL are bad.
The other messages may occur for example if if you have several short power outages where disks (pooldisks and spares) fail for a short time.

What you should do now
optionally: power off and check all cables and disk bays if they are inserted correctly, then

- remove one AVAIL disks from the spares
- replace then one UNAVAIL disk with this disk

redo the same for the second UNAVAIL disk
the active spares should then be AVAIL again

- remove the bad UNAVAIL disk and check them (ex with a low level tool of the manufacturer)

in raid-z2-3 you have two active spares but only one unavail disk
If one active spare keeps online, try to remove, optionally try an export/import after a clear error

- clear errors

You should then be online again without bad disks or suspicious messages
 
napp-it as a 1/10G network switch

I am just playing with the software bridge feature in OmniOS that allows to use Solaris/ OmniOS like a network switch.

Example: If you have 4 nics in OmniOS, you can use one as usual as uplink (assign an ip) then bridge this nic with the others. When you connect clients to the other nics, they can see the server, each other and everything that is on the uplink port.

Use cases now:
- Some 1 or 10G nics on the server with some 1/10G clients (no extra 1/10G switch is needed).
This will allow a silent and cheap high performance office storage server setup especially when paired with NVMe and SSDs.

Near future!
There are Illumos drivers on the way for the Intel X710 with 1 or 2 x QSFP+/ 8 x 10G per nic. With one of these cards (they are quite affordable) and a QSFP+ to QSFP+ or 2 QSFP+ to 8 x SFP+ Copper QSFP+ cables you should be able to create high performance 10G or 40G lab or video editing installations without a 10G/40G switch - with up to 8 x 10G or 2 x 40G clients per nic.


You can enable bridging/ switching manually, see
Bridging Overview - Oracle Solaris Administration: Network Interfaces and Network Virtualization

I have added the bridge feature in napp-it Pro/2016.05 dev in menu System > Network Eth
Comments especially about stability are welcome

see http://www.napp-it.org/doc/downloads/napp-it.pdf
page 43
 
Last edited:
Hey Gea, I'm unable to detach a disk from a pool and don't understand why. I'm trying to detach the first unavailable drive in this VDEV, which seems to have been replaced by spare-1. But when I try to detach it I get the following error:


Code:
zpool detach "backup" c0t50014EE2607DDC11d0

oops...
Could not proceed due to an error. Please try again later or ask your sysadmin.
Maybe a reboot after power-off may help.

cannot detach c0t50014EE2607DDC11d0: only applicable to mirror and replacing vdevs



back/ return to former menu

Code:
raidz2-3                   DEGRADED     0     0     0
            c0t50014EE2607DDC11d0    UNAVAIL    131   129     0
            spare-1                  ONLINE       0     0     0
              c0t50014EE2607DE23Cd0  ONLINE       0     0     0
              c0t50014EE2B5D28DC2d0  ONLINE       0     0     0
            c0t50014EE2B5CFC8FDd0    ONLINE       0     0     0
            c0t50014EE20D474BECd0    ONLINE       0     0     0
            spare-4                  DEGRADED     0     0     0
              c0t50014EE2B5D21984d0  UNAVAIL    133   131     0
              c0t50014EE2B5D358CFd0  ONLINE       0     0     0
            c0t50014EE2B5D237ECd0    ONLINE       0     0     0
            c0t50014EE2B5D2507Dd0    ONLINE       0     0     0
            c0t50014EE2B5D272E9d0    ONLINE       0     0     0
            c0t50014EE2B5D27B4Ed0    ONLINE       0     0     0
            c0t50014EE2B5D28A77d0    ONLINE       0     0     0


EDIT: Ok, realized I need to do zpool offline, not zpool detach. But after doing zpool offline on that disk, it still shows up in zpool status. What gives?
 
Last edited:
Try the napp-it menu
Disks >> Remove on the unavail disk and one of the disks under spare-1
 
Today I got some new toys
- A mainboard SuperMicro X10SDV-2C-7TP4F

This board includes a 25W CPU (2C/4HT), 2 x Intel 10G SFP+, 2 x 1G and 16 x SAS/Sata (LSI 2116).
You can add up to 128 GB RAM. I paid around 500 Euro for the board.
The boards is also available with 8 or 16 cores.

Then I received an Intel X710 QSFP+ (40G QSFP+ or 4 x SFP+) for around 400 Euro

First impression:
A perfect board for a ZFS storage and All-In-One (beside driver support for 10G)
Currently I use the 1G nics

The size of the Board is Flex ATX, a little smaller than uATX but it fits perfectly in a regular uATX/ATX case


I have tried the following:
Use the napp-it systemimage with a Intel S3510-80: ok
Installation of OmniOS from USB stick: ok
HBA 2116: ok, comes with IT mode firmware

Intel X550: not yet supported
Intel X710: not yet supported

For both drivers are under development, see Illumos mailling list


ESXi 6.0U2
Installation: ok
Pass-through of the LSI 2116: ok
X550: not supported
X710: not supported

Hope that the drivers included soon or they can be installed seperately


Napp-in-One
Deploying the napp-it ZFS storage server template with pass-through: ok


A short test with Solaris 11.3

Setup from USB stick: fails
Setup from Sata DVD: fails
I have tried different bios settings regarding USB2/3 settings
 
Last edited:
I'm having trouble understanding what is happening with a couple of my VDEVs. I don't fully understand which devices are resilvering, if the spares are actually active, and which devices are the spares.



Code:
raidz2-2                     DEGRADED     0     0 76.6M
            c0t50014EE2607CBA76d0      DEGRADED     0     0 76.6M
            spare-1                    DEGRADED     0     0     0
              replacing-0              DEGRADED     0     0     0
                c0t50014EE2607CC79Cd0  UNAVAIL      0     0     0
                c0t50014EE2B7F1C6EAd0  DEGRADED     0     0     0  (resilvering)
              c0t50014EE2B5D2A589d0    DEGRADED     0     0     0  (resilvering)
            c0t50014EE2607CC7BFd0      DEGRADED     0     0 76.6M
            c0t50014EE2607CC7D5d0      DEGRADED     0     0 76.5M
            c0t50014EE2607CE1F6d0      DEGRADED     0     0 76.6M
            c0t50014EE2607D477Fd0      UNAVAIL    131   175     0
            c0t50014EE2607D4D92d0      DEGRADED     0     0 76.6M
            c0t50014EE2607D4DCCd0      DEGRADED     0     0 76.5M
            c0t50014EE2607D6630d0      DEGRADED     0     0 76.6M
            spare-9                    DEGRADED     0     0     0
              c0t50014EE2607DC913d0    UNAVAIL      0     0     0
              c0t50014EE2B5D2F865d0    DEGRADED     0     0     0  (resilvering)
          raidz2-3                     DEGRADED     0     0     0
            replacing-0                DEGRADED     0     0     0
              c0t50014EE2607DDC11d0    UNAVAIL      0     0     0
              c0t50014EE20D479E32d0    DEGRADED     0     0     0  (resilvering)
            spare-1                    ONLINE       0     0     0
              c0t50014EE2607DE23Cd0    ONLINE       0     0     0
              c0t50014EE2B5D28DC2d0    ONLINE       0     0     0
            c0t50014EE2B5CFC8FDd0      ONLINE       0     0     0
            c0t50014EE20D474BECd0      ONLINE       0     0     0
            c0t50014EE2B5D358CFd0      ONLINE       0     0     0
            c0t50014EE2B5D237ECd0      ONLINE       0     0     0
            c0t50014EE2B5D2507Dd0      ONLINE       0     0     0
            c0t50014EE2B5D272E9d0      ONLINE       0     0     0
            c0t50014EE2B5D27B4Ed0      ONLINE       0     0     0
            c0t50014EE2B5D28A77d0      ONLINE       0     0     0
        spares
          c0t50014EE2B5D28DC2d0        INUSE
          c0t50014EE2B5D2A589d0        INUSE
          c0t50014EE2B5D2F865d0        INUSE

What is spare-1? And which drive is it replacing?
 
You must check the indentation starting from a vdev like raid z2-3

the following is a replacing with two disks
Code:
-> raid z2-3
    ->replacing 0 (ignore naming, check only following replacements for this "disk position")
       c0.. unavail
       c0.. resilvering
   ->spare-1 (this is also a "disk position" with two following replacements)
       c0.. unavail
       c00  resilvering
   -> c0.. (disk ok)

replacing -0 and spare-1 means that there are two events where a failed disk is beeing replaced by another one. Ignore the naming. See it as at this position of the vdev a disk fails and another tries to take the place.

Such a confusing situation also regarding namings can happen if you have a short power failure or a cable disconnect where a disk fails. In this case a spare jumps in. On next occurance the next spare jump in and so on. This interferes with the former faulted disks come back.

This is a reason why you may not activate all disks as hot spare. One is enough. Activate the others or replace manually on problems.

What you can do now
- wait until all replaces are finished then manually remove a disk from such a pair where both are online or remove the unavail pair member
- or power down and check cabing and backplanes first. After power on a rebuild will start again
 
Last edited:
I ran into the same issue. the solution here was:


cd /usr/bin

mv perl perl-

ln -s ../perl5/5.16.1/bin/i386/perl

mv /var/web-gui/data/napp-it/CGI/auto/IO /var/web-gui/data/napp-it/CGI/auto/IO-
 
Back
Top