OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Does not really matter
- you can update OmniOS + napp-it independently

- you can use a newer VM template (with a newer OmniOS and napp-it)
You may need to redo user settings or napp-it settings

- Export > Import a pool is the proper way
But you can import a pool without a prior export - no problem
 
Thx Gea,
Cant believe how easy this was...
Esxi update and brand new Napp-it WM within minutes. Import pool and you are ready to go!
I can see why there is really no need for mirrored bootdevice for esxi+napp-it
 
Hi,

Its been a while since I posted here since my ZFS All-in-One had been going steady for years.

Yesterday I started to replaced one of my drives with a spare and the resilvering is taking a very long time and my VM's are barely usable.

I built my All-in-One esxi/OI:napp-it server about 3 years ago and it has been running very smoothly, until now. I setup the server with 8x 3TB drives in Raid10 (4x Pairs).

I expected the replacement of the mirrored disk to simply copy over the data from the good mirror, but it says it has 143hours to go!

It would be ok if it resilvered in the background allowing us to carry on working, but we can't. There are currently only 2 VM's running, 'Open Indiana / Napp-it' and 'Windows Small Business Server 2011 Standard'. Our files are on SBS2011 and I have copied the files we need to work on to our desktops and leaving the server to work away on its own. The exchange server, which is part of SBS2011 is struggling to receive emails and there are hours where I am not able to even remote desktop into SBS2011. CPU/memory/Disk usage is low for both OI/nappit and SBS2011.

Is this normal?
Is there a way I can get SBS2011 into a usable state while the disk resilvers?
Is there a way of speeding up the resilvering?
...I'm also concerned the resilvering is going to break my other disks, several of which are the same age as the disk I am replacing.

The server is:
Supermicro X8SIL-F-O, Xeon 3450, 32gb ECC
1x 16Gb usb (esxi)
1x 60gb SSD (OI/napp-it)
1x 60gb SSD (spare)
2x IBM M1015
8x 3TB WD Red EFRX (tank)
1x 3TB WD Red EFRX (spare, replacing 1 of above)
1x 3TB WD Red EFRX (new, not in server)
1x 128gb SSD (tank2)
1x 128gb SSD (cache)
1x Intel 313 24gb (logs)

VMware vSphere 5.5.0

Storage:
datastore1 - 60gb SSD OCZ 2d Vertex 3 (motherboard sata controller)
- OISAN (16GB mem, 2xCPU)
datastore2 - 60gb SSD Kingston sv300s37a60g (motherboard sata controller)
- Empty
/tank/ZFSdata (10.43TB, used:4.9TB)
- SBS2011 (12GB mem, 2xCPU)
- other VM's off
/tank2/VMdata (105.44GB, used:24.56GB)
- empty​


About:

pool: tank
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Jul 21 21:05:08 2016
516G scanned out of 4.90T at 8.92M/s, 143h44m to go
162G resilvered, 10.27% done
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c4t50014EE0035E4708d0 ONLINE 0 0 0
c4t50014EE0035E6499d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c4t50014EE0035E6F58d0 ONLINE 0 0 0
replacing-1 ONLINE 0 0 0
c4t50014EE058B391C7d0 ONLINE 0 0 0
c4t50014EE0AE092086d0 ONLINE 0 0 0 (resilvering)
mirror-2 ONLINE 0 0 0
c4t50014EE0AE090EC9d0 ONLINE 0 0 0
c4t50014EE603E4F695d0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c4t50014EE2B33F3626d0 ONLINE 0 0 0
c4t50014EE2B33F3F60d0 ONLINE 0 0 0
logs
c4t5001517BB287C992d0 ONLINE 0 0 0
cache
c4t5E83A97C9E1F7B12d0 ONLINE 0 0 0

errors: No known data errors​

Disk Info:

Smartinfos (buffered info max 40 s old. Health-check can increase the counter for Soft-Errors!! Click on smart_sn to display Smart details)
id diskcap pool vdev state error smart_model smart_type smart_health temp smart_sn smart_selftest smart_check
c4t50014EE0035E4708d0 tank mirror ONLINE S:47 H:0 T:0 WDC WD30EFRX-68AX9N0 sat,12 PASSED 35 °C WDWMC1T0088027 -- short long abort log
c4t50014EE0035E6499d0 tank mirror ONLINE S:47 H:0 T:0 WDC WD30EFRX-68AX9N0 sat,12 PASSED 36 °C WDWMC1T0090197 -- short long abort log
c4t50014EE0035E6F58d0 tank mirror ONLINE S:49 H:0 T:0 WDC WD30EFRX-68AX9N0 sat,12 PASSED 36 °C WDWMC1T0083774 -- short long abort log
c4t50014EE058B391C7d0 tank basic ONLINE S:50 H:1285 T:1920 WDC WD30EFRX-68AX9N0 sat,12 PASSED 36 °C WDWMC1T0088665 -- short long abort log
c4t50014EE0AE090EC9d0 tank mirror ONLINE S:47 H:10 T:33 WDC WD30EFRX-68AX9N0 sat,12 PASSED 37 °C WDWMC1T0090101 -- short long abort log
c4t50014EE0AE092086d0 tank basic ONLINE S:48 H:0 T:0 WDC WD30EFRX-68AX9N0 sat,12 PASSED 39 °C WDWMC1T0088245 -- short long abort log
c4t50014EE2B33F3626d0 tank mirror ONLINE S:47 H:10 T:18 WDC WD30EFRX-68AX9N0 sat,12 PASSED 36 °C WDWCC1T1071834 -- short long abort log
c4t50014EE2B33F3F60d0 tank mirror ONLINE S:47 H:0 T:0 WDC WD30EFRX-68AX9N0 sat,12 PASSED 37 °C WDWCC1T1071728 -- short long abort log
c4t50014EE603E4F695d0 tank mirror ONLINE S:47 H:10 T:0 WDC WD30EFRX-68EUZN0 sat,12 PASSED 36 °C WDWMC4N0770500 -- short long abort log
c4t5001517BB287C992d0 tank logs ONLINE S:47 H:0 T:0 INTEL SSDSA2VP024G3 sat,12 PASSED - CVHA2106004N024BGN -- short long abort log
c4t5E83A97C9E1F7B12d0 tank cache ONLINE S:48 H:189 T:0 OCZ-VERTEX4 sat,12 PASSED - OCZM749783502Y6WN23 -- short long abort log
c4t5E83A97D7DAA481Bd0 tank2 basic ONLINE S:48 H:192 T:0 OCZ-VERTEX4 sat,12 PASSED - OCZFBQ5NGJC21YD81QP -- short long abort log
c6t0d0 rpool basic ONLINE S:0 H:0 T:0 - - - - n.a. - short long abort log​


The disk I am replacing had errors, napp-it said it had finished repairing the disk and had 0 errors. repaired disk had a Raw_Read_Error_Rate: 2012, Reallocated_Event_Count: 24, Current_Pending_Sector: 5. I'm not sure if these are considered high, but the other disks are 0 for each of these. I had a spare disk in there so, rather than wait for it to fail I decided to replaced it.

'Repaired' Disk, currently being replaced.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 198 051 Pre-fail Always - 2012
3 Spin_Up_Time 0x0027 176 175 021 Pre-fail Always - 6200
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 30
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 27
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 066 066 000 Old_age Always - 25203
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 30
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 29
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 111 108 000 Old_age Always - 39
196 Reallocated_Event_Count 0x0032 176 176 000 Old_age Always - 24
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 5
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0


c4t50014EE058B391C7d0_error S:50 H:1285 T:1920
c4t50014EE058B391C7d0_ill_request 255
c4t50014EE058B391C7d0_info
c4t50014EE058B391C7d0_part (!parted)
c4t50014EE058B391C7d0_partcap 3 TB
c4t50014EE058B391C7d0_partcap2 2.7 TiB
c4t50014EE058B391C7d0_pool tank
c4t50014EE058B391C7d0_pred_fail_anal 0
c4t50014EE058B391C7d0_product WDC WD30EFRX-68A
c4t50014EE058B391C7d0_recoverable 50
c4t50014EE058B391C7d0_revision 0A80
c4t50014EE058B391C7d0_rwc 0
c4t50014EE058B391C7d0_sn WDWMC1T0088665
c4t50014EE058B391C7d0_stat ok
c4t50014EE058B391C7d0_state ONLINE
c4t50014EE058B391C7d0_vdev basic
c4t50014EE058B391C7d0_vendor ATA​
 
- A resilver must process all metadata on all vdevs - not only the mirror.
If data is found that is on the old disk, this data is copied to the new disks,
either from old disk or raid redundancy.

- Resilver is a low level background process so it allows to continue to work

- Resilver time depends on pool iops, number of files, fillgrade and load

In your case, the problem seems the old disk or new disk of the replace.
If the old disk is not completely dead or removed, it is accessed for a resilver.
The same with the new disk. On read/write problems ZFS waits up to 60s per error
You can set timout to a lower value, see napp-it tuning.

The key info is in (old disk to replace)
c4t50014EE058B391C7d0 tank basic ONLINE S:50 H:1285 T:1920 WDC WD30EFRX-68AX9N0 sat,12 PASSED
These iostat hard ot transfer error warnings are always something to care about. Even when they are not real errors but driver warnings, they indicate problems with lots of hard or transfer errors as they cause ZFS to wait

I would offline or remove c4t50014EE058B391C7d0
The resilver process will then use only the other disks to resilver.

You can also check System > log and System > faults for additional infos
 
Last edited:
Thank you Gea!

It was the hottest day of the year this Tuesday and unfortunately the ventilation holes to the server cabinet were blocked. The server made a high pitched noise started to shut itself down. This lines up with the time of the faults listed below.

After this, the 'old disk to replace' you noted, started with maybe H:200 and T:700, so after a bit of googling, I discovered this is not good and I ordered a new disk while the machine was 'repairing'. I forgot I already had a spare disk in the server, so I decided to use that to replace the old disk instead.

Before I remove the disk, here are the logs and faults:
I'll look into reducing timeout value now.

System Fault:

--------------- ------------------------------------ -------------- ---------
TIME EVENT-ID MSG-ID SEVERITY
--------------- ------------------------------------ -------------- ---------
Jul 19 16:09:42 eafeadeb-4433-6219-84f4-be7b01f7a38a PCIEX-8000-0A Critical

Host : OISAN
Platform : VMware-Virtual-Platform Chassis_id : VMware-56-4d-b8-11-bf-0c-1b-a4-85-a0-da-c3-67-e6-5f-04
Product_sn :

Fault class : fault.io.pciex.device-interr
Affects : dev:////pci@0,0/pci15ad,7a0@17/pci1000,3020@0
faulted but still in service
FRU : "MB" (hc://:product-id=VMware-Virtual-Platform:server-id=OISAN:chassis-id=VMware-56-4d-b8-11-bf-0c-1b-a4-85-a0-da-c3-67-e6-5f-04/motherboard=0)
faulty

Description : A problem was detected for a PCIEX device.
Refer to PCIEX-8000-0A for more
information.

Response : One or more device instances may be disabled

Impact : Loss of services provided by the device instances associated with
this fault

Action : Schedule a repair procedure to replace the affected device. Use
fmadm faulty to identify the device or contact Sun for support.​

System Log (I have attached, but these stand out to me):

Jul 16 04:17:19 OISAN smbd[749]: [ID 807464 daemon.error] ndr_rpc_bind: smbrdr_ctx_new(S=, D=SCOTTARCHITECTS, U=OISAN$), err=61
Jul 16 04:17:19 OISAN smbd[749]: [ID 807464 daemon.error] ndr_rpc_bind: smbrdr_ctx_new(S=, D=SCOTTARCHITECTS, U=IPC$), err=61
Jul 16 04:17:22 OISAN smbd[749]: [ID 702911 daemon.notice] smbd_dc_monitor: domain service not responding
Jul 16 04:17:22 OISAN smbd[749]: [ID 663266 daemon.notice] DNS query for _ldap._tcp.dc._msdcs failed: Host name lookup failure
Jul 16 04:17:22 OISAN smbd[749]: [ID 702911 daemon.notice] smbd_dc_update: ad.scottarchitects.co.uk: located

This repeats until Tuesday this week, when this happens:
Jul 19 16:09:06 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:06 OISAN Disconnected command timeout for Target 12
Jul 19 16:09:10 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:10 OISAN Log info 0x31130000 received for target 12.
Jul 19 16:09:10 OISAN scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
...
Jul 19 16:09:10 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:10 OISAN mptsas_check_task_mgt: Task 0x3 failed. IOCStatus=0x4a IOCLogInfo=0x0 target=12
Jul 19 16:09:10 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:10 OISAN mptsas_ioc_task_management failed try to reset ioc to recovery!
Jul 19 16:09:12 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:12 OISAN mpt1 Firmware version v14.0.0.0 (?)
Jul 19 16:09:12 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:12 OISAN mptsas_restart_ioc failed
Jul 19 16:09:12 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:12 OISAN Target 12 reset for command timeout recovery failed!
Jul 19 16:09:22 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:22 OISAN MPT Firmware Fault, code: 1500
Jul 19 16:09:23 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:23 OISAN mpt1 Firmware version v14.0.0.0 (?)
Jul 19 16:09:23 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1):
Jul 19 16:09:23 OISAN mpt1: IOC Operational.
...
Jul 19 16:09:42 OISAN genunix: [ID 408114 kern.info] /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas1) down
Jul 19 16:09:42 OISAN pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Jul 19 16:09:42 OISAN genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Jul 19 16:09:42 OISAN fmd: [ID 377184 daemon.error] SUNW-MSG-ID: PCIEX-8000-0A, TYPE: Fault, VER: 1, SEVERITY: Critical
Jul 19 16:09:42 OISAN EVENT-TIME: Tue Jul 19 16:09:42 BST 2016
Jul 19 16:09:42 OISAN PLATFORM: VMware-Virtual-Platform, CSN: VMware-56-4d-b8-11-bf-0c-1b-a4-85-a0-da-c3-67-e6-5f-04, HOSTNAME: OISAN
Jul 19 16:09:42 OISAN SOURCE: eft, REV: 1.16
Jul 19 16:09:42 OISAN EVENT-ID: eafeadeb-4433-6219-84f4-be7b01f7a38a
Jul 19 16:09:42 OISAN DESC: A problem was detected for a PCIEX device.
Jul 19 16:09:42 OISAN Refer to PCIEX-8000-0A for more information.
Jul 19 16:09:42 OISAN AUTO-RESPONSE: One or more device instances may be disabled
Jul 19 16:09:42 OISAN IMPACT: Loss of services provided by the device instances associated with this fault
Jul 19 16:09:42 OISAN REC-ACTION: Schedule a repair procedure to replace the affected device. Use fmadm faulty to identify the device or contact Sun for support.
Jul 19 16:09:42 OISAN genunix: [ID 846333 kern.warning] WARNING: constraints forbid retire: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0
Jul 19 16:10:28 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:28 OISAN Disconnected command timeout for Target 9
Jul 19 16:10:32 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN Log info 0x31130000 received for target 9.
Jul 19 16:10:32 OISAN scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
Jul 19 16:10:32 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN Log info 0x31130000 received for target 9.
Jul 19 16:10:32 OISAN scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
Jul 19 16:10:32 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN Log info 0x31130000 received for target 9.
Jul 19 16:10:32 OISAN scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
Jul 19 16:10:32 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN Log info 0x31130000 received for target 9.
Jul 19 16:10:32 OISAN scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
Jul 19 16:10:32 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN Log info 0x31140000 received for target 9.
Jul 19 16:10:32 OISAN scsi_status=0x0, ioc_status=0x8048, scsi_state=0xc
Jul 19 16:10:32 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN mptsas_check_task_mgt: Task 0x3 failed. IOCStatus=0x4a IOCLogInfo=0x0 target=9
Jul 19 16:10:32 OISAN scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:32 OISAN mptsas_ioc_task_management failed try to reset ioc to recovery!
Jul 19 16:10:34 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:34 OISAN mpt0 Firmware version v14.0.0.0 (?)
Jul 19 16:10:34 OISAN scsi: [ID 365881 kern.info] /pci@0,0/pci15ad,7a0@16/pci1000,3020@0 (mpt_sas0):
Jul 19 16:10:34 OISAN mpt0: IOC Operational.
...
Jul 19 16:15:24 OISAN fmd: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-FD, TYPE: Fault, VER: 1, SEVERITY: Major
Jul 19 16:15:24 OISAN EVENT-TIME: Tue Jul 19 16:15:24 BST 2016
Jul 19 16:15:24 OISAN PLATFORM: VMware-Virtual-Platform, CSN: VMware-56-4d-b8-11-bf-0c-1b-a4-85-a0-da-c3-67-e6-5f-04, HOSTNAME: OISAN
Jul 19 16:15:24 OISAN SOURCE: zfs-diagnosis, REV: 1.0
Jul 19 16:15:24 OISAN EVENT-ID: 726fea8d-7552-4f03-98c8-f72eaba30b8a
Jul 19 16:15:24 OISAN DESC: The number of I/O errors associated with a ZFS device exceeded
Jul 19 16:15:24 OISAN acceptable levels. Refer to ZFS-8000-FD for more information.
Jul 19 16:15:24 OISAN AUTO-RESPONSE: The device has been offlined and marked as faulted. An attempt
Jul 19 16:15:24 OISAN will be made to activate a hot spare if available.
Jul 19 16:15:24 OISAN IMPACT: Fault tolerance of the pool may be compromised.
Jul 19 16:15:24 OISAN REC-ACTION: Run 'zpool status -x' and replace the bad device.
...


​
 

Attachments

  • log220716_2000.zip
    313.5 KB · Views: 47
The initial problem seems then overheating.
The logs now indicate problems at least with two disks
target 9 and target 12.

If you have a newer napp-it Pro monitor, you will find a translation
of driver target numbers to WWN disks in menu Disks >Detection
or Disks > SAS extension

in System logs there are more timeout infos with disks.

If the overheating problem is fixed and the problem disks have intact
mirrors you can remove/offline them and replace -
otherwise try a resilver and wait and hope that the disks are good enough
to resilver the pool. Mostly a short overheating does not finally damage disks
but produce temporary read/write errors only.
 
Thanks Gea,

Yes, the overheating problem is fixed for now. I assume the mirrors are intact, none are reporting as degraded or unavailable. yet.
Hopefully I can just replace the drive with the really high errors. I'll keep an eye on the other ones (H:10/T:33, H:10/T:18, H:10/T:0), to see if they worsen.

I don't have napp-it pro or any extensions yet as I wanted to keep things simple.
Unless I come up with a better idea, I'll have to read the serial numbers on the physical disks to know which one is the one I'm replacing.
I've not updated OI or Napp-it as the system seemed stable (OI v151a8 / Napp-it v.0.9d2 nightly Nov.03.2013) Is it important to update either of these?

Setting the timeout to a lower value looks quite involved, perhaps I try to remove/offline the old disk first...

The Disks > Remove, doesn't include on the list the 2 disks currently involved in replacing.
but Disks > Hotswap > set offline shows all disks, should I just click offline - and the system will automatically start resilvering from the other disks?


It now notes both drives as (resilvering) with cksum on the new disk:
pool: tank
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Jul 21 21:05:08 2016
922G scanned out of 4.90T at 10.2M/s, 114h21m to go
283G resilvered, 18.36% done
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c4t50014EE0035E4708d0 ONLINE 0 0 0
c4t50014EE0035E6499d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c4t50014EE0035E6F58d0 ONLINE 0 0 0
replacing-1 ONLINE 0 0 0
c4t50014EE058B391C7d0 ONLINE 0 0 0 (resilvering)
c4t50014EE0AE092086d0 ONLINE 0 0 78 (resilvering)
mirror-2 ONLINE 0 0 0
c4t50014EE0AE090EC9d0 ONLINE 0 0 0
c4t50014EE603E4F695d0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c4t50014EE2B33F3626d0 ONLINE 0 0 0
c4t50014EE2B33F3F60d0 ONLINE 0 0 0
logs
c4t5001517BB287C992d0 ONLINE 0 0 0
cache
c4t5E83A97C9E1F7B12d0 ONLINE 0 0 0

errors: No known data errors​
 
oh no ...its all gone wrong.

I tried to update Napp-it, but it failed. I suspected DNS issue, so went into OI desktop and tried firefox and it couldn't reach external websites.
I added nameserver addresses to our SBS2011 server and 8.8.8.8 and Firefox could then reach external websites. Napp-it found the latest update 16.0_f.
I also downloaded the latest updates for OI. To set the timeouts, I followed the everycity.co.uk guide refered to on the napp-it tuning webpage,
I added to /etc/system
set sd:sd_io_time=0x0A
set ssd:ssd_io_time=0x0A​
and to /kernel/drv/sd.conf
sd-config-list = "ATA WDC WD30EFRX-68A", "retries-timeout:3";
"ATA WDC WD30EFRX-68E", "retries-timeout:3";
"ATA OCZ-VERTEX4 ", "retries-timeout:3";​

I restarted and now I can't get back to the OI desktop login (screen grab attached).

I have no recent snap shots of the OI/napp-it VM. When I shut down the server, the disk were still resilvering as before, but I had not yet set any to be offline or removed (I had planned to do this after the restart).

I tried other previous versions on the initial boot list, but these also either could not load to the desktop login, or got to the login page but didn't accept my user/pass.

Can I still salvage OI/Napp-it?
 

Attachments

  • boot.jpg
    boot.jpg
    168.7 KB · Views: 84
some remarks

- OI 151a is a dead end as it is several years old without any fixes or updates
The successor is OpenIndiana Hipster that require more or less a complete reinstall

- There are some serious OI 151a OpenSSL issues and security problems fixed in newer releases.
OI 151a can give problems with newer napp-it as this require a newer OpenSSL
a workaround see napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux : OpenIndiana

- Your former system states are available not as snaps but as bootable bootenvironments (BE).
You can select a default BE at menu snaps or select any during reboot

- Current free napp-it is 1604f
Update from very old versions is a two step update over 16.01f

- newer OS releases like OI Hipster or OmniOS have a newer ZFS with many improvements
and SMB 2.1 that is faster, http://napp-it.de/doc/downloads/performance_smb2.pdf

- OI Hipster (dev only) is the option if you need the OS GUI.
Without the GUI, OmniOS is a better option as this comes with regular stable editions and bugfixes.

- A reinstall is quite trouble free.
Install the OS and import the pool, update the pool to the recent ZFS v5000.
http://www.napp-it.org/doc/downloads/napp-it.pdf
 
Last edited:
Ok, looks like I need to move to OmniOs. I choose the OI desktop version as I'm not familiar with CLI's. I only access the desktop for network settings and tweaking setting files

But it seems like it might be best to first get the existing system back online so I can tidy things up and prepare myself. Mainly, I hope this would take less time than reading the guides (thoroughly), installing the OS, setting it up right for my VM's, but if I hit a brick wall, then it might force me to reinstall today.

With a reinstall, could I re-set up the NFS share exactly as it is now, so that esxi is 'none the wiser' and my esxi VM's just work?

I'm look into bootable bootenvironments (BE) now.
 
Disappointingly I can't seem to get into any BE or snapshots since napp-it was installed. I tried cloning an old snapshots to new BE as described here illumos: manual page: beadm.1m, but I might not be doing it right (attached).

I can only access OI desktop with a snapshot taken just before napp-it was installed (151a7 2013), but I'm not sure if this will cause issues with my zfs pool which was running on 151a8 or if using the old snapshot will take less time to setup than taking the plunge and reinstalling with OmniOS.

When booting it appears to be having problems with svc.startd[10] timing out, a readme takes me to SMF-8000-QD but I cant see what to try from there.
I tried using vi to check the 2 files I changed mentioned in my post #7608 (system & sd.conf) and they don't appear to include my changes and look unaltered, however the file resolv.conf does have my changes in it (added nameservers 8.8.8.8 & 8.8.4.4)

This has turned into a bit of a nightmare.
 

Attachments

  • 99.JPG
    99.JPG
    60 KB · Views: 67
I suppose you expect a different behaviour of boot environments.
If you boot into a BE and modify anything this is the state when you reenter this BE
so a BE must not be the state on creation time but state of last modifications.

If you only create a BE without booting into, this is like a freeze of this state
Any napp-it BE`s are of this kind

If you do an OS update it creates a BE and you are asked to reboot.
You will then boot into the new BE as default BE, the former state keeps available in the former active BE

If you have not done an OS update, I suppose your former state is a very old
BE like the OpenIndiana that is created during install and where you had done
everything in the last years.

And
You do not need to clone anything
Just select a BE during boot and if its the desired one, set it as default for next boots
in menu Snaps > bootenvironment
 
Last edited:
There are 7 states saved, I've tried the last few and I have one for each OI version: 151a9(this week),151a8(2013), 151a7(2013) they all seem to have the same issue:

Jul 23 18:44:40 svc.startd[10]: svc:/system/boot-archive:default: Method or service exit timed out. Killing contract 17.
Jul 23 18:44:40 svc.startd[10]: svc:/system/boot-archive:default: Method "/lib/svc/method/boot-archive" failed due to signal KILL.
...Killing contract 23.
...Killing contract 25.
Jul 23 18:44:41 svc.startd[10]: system/boot-archive:default: failed: transitioned to maintenance (see 'svcs -xv' for details) "/lib/svc/method/boot-archive" failed due to signal KILL.

I can't find anything on fixing this :-/
 
Seems your boot disk is messed up.
Does not matter if by chance or overheating was the reason or one of your actions.

Now there are 20 minutes between your current state and a working NFS if you
- have decided the OS, either old OI 151a9, newer OI Hipster or OmniOS

- have already download the installer (iso or USB image), otherwise count additional 30 minutes

If your current disk was working, you would had the option to use a new disk, install, and import
the pool - without a pool update. This would allow a go back.

Now I would simply re- install the OS (I would use OmniOS 151018) and import the pool.
The NFS share should be set to on after the import as shares on Solarish are a ZFS property.

In ESXi the NFS datastore should be available after it come back, if not
- reboot ESXi and retry
- otherwise delete the NFS datastore, re-add and re-add the VM s to the inventory as worst case.
(ESXi databrowser, mouse right click on the .vms file of a VM > add to inventory)

If you use OmniOS, you need 3 CLI commands to setup network, see my manual or read
napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux : OmniOS

You can do the other settings in napp-it.
I would update the pool from v28 to v5000 when everything is ok.
 
Thanks _Gea

I downloaded: OmniOS_Text_r151018.iso & the Napp-it ToGo VM: napp-it_2016.4_ova_for_ESXi_5.5u2-6.0u2. (Apr 20,2016)
...also upgraded esxi to 5.5U3 as my version is 5.5, will go upto 6.0U2 another day

I restarted the server a couple of times (and went into the disk controller info to note down where each disk is connected).
Rebooted and the whole thing has come back to life!!! It has the previous version OI151a8, but no napp-it. I set it to download and it now has v16.03f2 2016.
I made a BE snapshot.

I have a spare 60Gb SSD disk attached to the controller I could install OmniOS to. Hopefully OI will stay alive a little longer so I can try to import the pool. maybe not.
 
I installed the Napp-it ToGo VM to a different drive (old OI boot drive not working again).

I'm trying to import my the pool but it is hanging:
I didn't select any of the tick boxes on import, just selected the pool and imported.

napp-it-san config header request timeout
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san NULL command for address reply in slot 1
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san config header request timeout
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san config header request timeout
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san config header request timeout
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san NULL command for address reply in slot 3
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san NULL command for address reply in slot 4
napp-it-san scsi: WARNING: /pci@0,0/pci15ad,7a0@17/pci1000,3020@0 (mpt_sas16):

napp-it-san NULL command for address reply in slot 5​

Not sure what to do, I don't want to lose the pool.

When I last checked on the previous OI/nappit VM SAN (before it died) the 'new' drive replacing the old drive had a Raw Read Error Rate of over 5000, I had offlined the old disk, so it was resilvering from the good mirror, the high count over a short time might have been to do with the 'set sd:sd_io_time=0xa' 10sec (WD Reds, I understood to be 7sec). The new disk is also the same age as the old disk, but was in server only as a spare for 3 years. I have not yet reduced the timeout for the new OmniOS install, so I assume its still 60sec. Is it likely there is a fault with the new disk, or is related to the partial resilvering and offline old disk during a replace? Should I force reboot, and try differnet options for import?Should I replace the new disk with another disk (bought last week)?
 
ARNiTECT said:
Not sure what to do, I don't want to lose the pool.
Click to expand...

You are near a real disaster and for such a case, you need backups even with ZFS

Your possible problem:
- more than one disk fails.
If this happens in a mirror, the pool is lost. It is uncritical if it happens in different mirrors

- one disk fails and is blocking the controller

- your pool structure is damaged. Can happen during last writes when hardware was too hot

One word about the behaviour of ZFS pools:
If a disk fails, the pool goes to degraded but remains accessable
If more disks fail than the allowed redundancy level, the pool goes to offline/ unavailable
If enough disks come back, the pool goes back to degraded or online. So the basic behaviour is very fault tolerant.

What I would do.

1. check for possible bad disks
- Remove all pooldisks and boot.
Insert disk by disk, wait a short time and call format (followed by ctrl-c) to check if its available.
Look for a delay when disks are listed

optionally: do a short smart check of the disks (napp-it)
If you can identify bad disks and they are limited to a single mirror part, keep them removed

2. try to import the pool with option readonly
After a success, you can try a backup

3. try to import with option -F
(import former state of a damaged pool). This can skip last writes (last writes are lost)
 
Hi _Gea,

Thank you for your time on this, I really appreciate it.

My pool is currently resilvering at 200-300M/s with 3h to go!

Last night I did a smart check of all drives and noted down the serial numbers. Together with drive info from the controller, I opened up my server and fully catalogued every drive and where its connected. I removed the 1st damaged mirror disk and also the spare disk, which I had tried to use to replace the damaged disk; as both were not looking too healthy. I put in a brand new disk and booted up.

I restarted a couple of times, but in the end I imported the pool and appears to be intact. The physically removed disk is now listed as 'UNAVAIL', so I used 'replace' to replace the unavailable disk with the new one.

mirror-1 DEGRADED 0 0 0
c3t50014EE0035E6F58d0 ONLINE 0 0 0 3 TB WDC WD30EFRX-68A S:0 H:0 T:0
replacing-1 UNAVAIL 0 0 0
1351237688965085373 UNAVAIL 0 0 0 was /dev/dsk/c3t50014EE0AE092086d0s0
c3t50014EE2B7DD75CAd0 ONLINE 0 0 0 (resilvering) 3 TB WDC WD30EFRX-68E S:0 H:0 T:0

I assume when this is done, either the unavailable disk will disappear, or I'll have to 'Remove' it?

The pool has automatically appeared in esxi storage and my other VM's stored on this are now showing again. I have not started them yet, as the resilvering is going quite quickly and it seams sensible to let it focus on the one task.

Only one of my 2 pools is showing in esxi, fortunately its the main raid10 pool, so I can browse the content. The other small pool is listed as '(inactive)', perhaps napp-it or esxi needs a restart as you explained earlier, which I'll do after the resilver.

Assuming it all completes, after backing up I'll look into updating the pool to v5000, should I look into lowering the error timeout times as before or leave as 60s?

I have a backup of my work stuff, I back up using SBS2011 over iscsi to usb drives connected to my workstation (using the excellent Starwind) I've only used these back ups to retrieve individual files, but not had to try a full restore. Not backed up: the VMs, 20-30 recent cd rips, and some other consolidated data from older back ups, replacing it all would be a lot of work. The first thing I'll do after the resilver, is back all this up.

In preparation for the next failures, I have ordered 2 new WD Red 3TB disks, as 4 of the remaining disks were from the same order. To prevent damage from heat, I have fitted 2 constant fans to the top vents of the cabinet, vacuumed dust from the server and explained to all about the perils of blocking up the bottom vents to the cabinet.

I also have a spare AiO server on a shelf built at the same time as my current server with very similar spec. I will set this up for back ups, or maybe something more advanced like syncing between the two.
 
Good to hear that you have sorted out you disks.

regarding timeout:
With a lower timeout a pool faults earlier but ESXi must not wait so long.
TLER disls have a timeout of 7s. This may be too short for a raid over many disks but I would reduce the timeout from 60s to 10-20s.

If you have a second AiO, you should replicate the filesystems to this system. This allows to keep them in sync down to a minute.
In case of problems you can start the VMs from there.
 
I'll reduce the time out to 20s.

I'm having trouble accessing napp-it from one PC in the office, literally every other PC or device can access the napp-it webpage, but mine gets stuck at "initialize Web-UI, please wait..." strange, maybe a network/dns issue I havent found

Also, I noticed ESXi has imported the Datastores in with the e1000 nic, instead of the vmxnet3, I assume I should swap over the nic's IP addresses in napp-it, rather than delete and re-import the datastores in ESXi.

I'll probably use RaidZ2 on my secondary AiO (8x 2TB) as its only backup for my primary AiO (8x3TB RAID10) ...also have an 8Gb HP N40L with iLo, modified to fit 7x 3.5" disks, no use for it though :-/
 
Is there any suggested L2Arc guide by memory. Wondering what is a reasonable L2Arc size for 64GB. Has persistent L2Arc been released yet?
 
There is no global rule - depends mostly on workload, check arcstat logs first
You loose also some fast RAM for the slower L2Arc, calculate about 5% so with 64GB I would not use more than around 200GB

While performance improvement of an L2Arc is mostly iops related and optimized for small files/ metadata, it should not be too slow as reads are as fast/slow as the cache . If you enable caching of sequential data, it should be quite fast.

Even a low write performance can slow down overall write performance
as new data must be written to the pool and the cache.

Ex: On a very fast pool with many fast disks and sequential cache enabled you need a high performance NVMe up to an Intel P3608 to be in par with the pool regarding sequential performance. For a low performance pool, even an average desktop SSD may help due much higher iops.

persistent L2Arc
Feature #3525: Persistent L2ARC - illumos gate - illumos.org
 
Last edited:
So for 64GB, would a 128GB SSD be good enough? Don't bother with 256GB? Thinking either Samsung 850 Pro or their 950. Obviously the former is a lot cheaper. I already have a S3710 as my SLOG.
 
Put in any SSD and check arcstat/l2arcstat for L2arc usage
Then select performance in relation to your pool.
 
My new server is getting closer....:)
Just a quick question - should LSI firmware P20 still be avoided?
TIA
 
Last edited:
A question on linux user mapping with solaris:

I have a linux VM running an app under "jvm1" user. I have an nfs mount on my solaris zfs (running napp-it) via /etc/fstab. I keep getting permissions error whenever my app runs, and when I try to do a recursive chown within that nfs mount, it obviously fails. Is there any way for that jvm1 user in linux to have elevated rights within the mounted zfs share? Does Napp-it have that via the gui?
 
What command should be used to partition a ssd to be a desired l2arc size.

For my zil I over provisioned through some fancy utility in Linux and I'd prefer not to go that route if it isn't necessary.
 
jad0083 said:
A question on linux user mapping with solaris:

I have a linux VM running an app under "jvm1" user. I have an nfs mount on my solaris zfs (running napp-it) via /etc/fstab. I keep getting permissions error whenever my app runs, and when I try to do a recursive chown within that nfs mount, it obviously fails. Is there any way for that jvm1 user in linux to have elevated rights within the mounted zfs share? Does Napp-it have that via the gui?
Click to expand...

User mapping in Solaris is mainly to map SMB groups to Unixgroups or AD user to Unix user.
With NFS 3 I would completely forget user authentication as there is no real authentication or authorisation with NFS3.

While you can try different share options or create a user with same uid/gid on Solaris like you have on Linux, I always use everyone=allow setting on NFS shares to avoid permission problems on NFS - paired with aclmode=restricted to hinder NFS to modify permissions.
 
davewolfs said:
What command should be used to partition a ssd to be a desired l2arc size.

For my zil I over provisioned through some fancy utility in Linux and I'd prefer not to go that route if it isn't necessary.
Click to expand...

I would use HPA/ host protected areas to overprovision SSDs as this is hidden to the OS
You can use Linux/hdparm or DOS/hdat2 or tools from the SSD manufacturer ex Samsung.

Another option is partitioning the SSD (you can use parted or napp-it)
In any case, use new or secure erased SSDs
 
Is ProFTPD no longer working with Napp-it?
I installed it but FTP does not appear under Services, so I cant enable it.
I am running ver 16-07f.
 
napp-it includes menu sets for OEMs, languages or a lot of extra services and unsupported free add-ons. Per default, you see the menu set sol (Solarish) with all default menus for Solarish and ZFS functions,

You can switch to "en", the default English menu set with all menus, either in the top menu nearby Logout or in About > settings to modify defaults.
 
Looking at creating a backup server and want to keep costs down.

If running on bare metal is there anything wrong with using the built in sata ports or do I really need to get a LSI controller?
 
Sata/AHCI is fine

You can try if hotplug works when you enable hotplug support in /etc/system
via set sata:sata_auto_online=1 or with napp-it menu System > Appliance Tuning
 
Hi Gea,

I'm moving over all of my data from an OI+Napp build to OmniOS. I was a big fan of the Timeslider app and how easy it worked (set it and forget it), the defaults pretty much suited my needs.
I am trying to mimic Timeslider using Napp-IT autosnap jobs, but I don't understand the Keep VS Hold settings as well as the "Retention policy, keep and optional hold must be both true to delete".

Does the following setup make any sense? (This is similar to a post I saw on STH)

5 min for two days
hourly for 1 week
daily for two week
weekly for two months
montly for 1 year

Code: 
Joblog      Text1      Opt1/ from      Text2      Opt2/ to      Opt3      Month      Day      Hour      Min      ID/ edit      Status      Last           Jobstate        Execute        Job
                                                                                                           
  snap      omnizfs/testsmb      keep 12      monthly      del zero, hld 8      -       every       1       23       0       1470686929       active      new      -       run now       delete 
  snap      omnizfs/testsmb      keep 8      weekly      del zero, hld 8      -       every       sun       23       0       1470687058       active      new      -       run now       delete 
  snap      omnizfs/testsmb      keep 14      daily      del zero, hld 8      -       every       every       23       0       1470691217       active      new      -       run now       delete 
  snap      omnizfs/testsmb      keep 2      hourly      del zero, hld 7      -       every       every       every       0       1470691334       active      new      -       run now       delete 
  snap      omnizfs/testsmb      keep 2      minute      del zero, hld 2      -       every       every       every       every-5       1470691420       active      new      -       run now       delete

Thanks for the help :)
 
You can add two retention settings to a snap job
hold n days
keep n snaps (number of snaps)

Snaps are only deleted when allowed by both rules
ex If you create a snap job with hold 2 days any keep 10 snaps with a snap every hour:
You will find 48 snaps as the two days rule must be respected

about your snap jobs
minute: takes a snap every 5 minutes
keep2, hold 8: result=hold 8 days with number of snaps: 12 x 24 x 8


hourly takes additional snaps with its own poliocy
it takes a snap every full hour with keep=2 and hold=7: 24 x 7

daily takes a snap at 11pm, keep=14, hold=8:
In this case keep is relevant: snaps=14

In general, each snap job creates snaps with its own job id
A job cares only about his snaps and will not touch snaps from other jobs.
You can use mass-delete to delete snaps based on other rules

In most cases, you add only one rule that should be relevant like
minutely: a snap every 5 minutes, keep 12 (allow to go back to any 5minute for last hour)
hourly: a snap every hour, keep=24 (allow to go back to any hour for last 24hours)
daily: a snap every 11pm, hold=7 (allow to go back to any day for week month)
weekly: a snap every sunday 11pm, hold=4 (allow to go back to any sunday for last month)
monthly: a snap every first sunday 11pm, keep=12 (allow to go back to any month for last year)

This follows the rule: more recent autosnaps, less older autosnaps
 
Thanks Gea! I think I understand now!

Did I catch a typo?
You said:
"weekly: a snap every sunday 11pm, hold=4 (allow to go back to any sunday for last month)"
Shouldn't it be:
"weekly: a snap every sunday 11pm, KEEP=4 (allow to go back to any sunday for last month)"
 
I decided to upgrade my old All-in-one ESXi machine from Openindiana 151a to the latest OmniOS (151018 I think). However, I've run into a problem...when I import my pool (RAIDZ2 10x 2TB + RAIDZ2 10x 3TB hdds), it reports 0 bytes free! While in OI151a, it reports 262 GB. Deleting files doesn't seem to work in OmniOS - still reports 0 bytes available even after I deleted a 13 GB file.

Any ideas?
 
There is no known problem with OmniOS regarding this.
But the free message is always an estimation with physical disk blocksize and ZFS stripesize as variables. If you write data its space consumption also depends on the difference to the next possible blocksize and the raid stripes.

In your case, it may be that OmniOS with a much newer ZFS with more features either needs some Bytes more or calculate a little different, but the real problem is

Your pool has a usable capacity of 40TB. 260 GB is less than 1%. Your pool is full at around 99,95%
In this situation, your pool becomes very slow with a huge fragmentation. You should try to go below 90%
what means you must delete not 13GB but 4TB.

Do you have snaps that you can delete?
 
You must log in or register to reply here.
Back
Top