OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Ok, menu disks is using format to detect disks.
You can now
- mirror manually with the disk-ids

What happens if you enable partition support in napp-it
where parted is used instead of format.
>> Menu disks >> Partitions >> enable Partition support
Gea

You picked it. That works, both USB's are displayed and have a status of removed.
That said enabling partition support does not appear to be 'sticky', clicking enable partition support results in the USB's being displayed, but then clicking on 'partitions' results in just the main disks being displayed.
 
Running SunOS napp-it-15a 5.11 omnios-10b9c79 on esxi 5.5. I have an existing pool that I tried to destroy via the napp-it gui.

Now the napp-it gui is not responding and zfs commands are not responding. Other commands seems to be fine. The zpool destroy has been running for over 9 hours.


root@napp-it-15a:/root# ps -ef | grep zpool
root 6 0 0 Jan 03 ? 0:00 zpool-rpool
root 8526 24018 0 12:04:16 pts/2 0:00 zpool get freeing
root 8611 0 0 21:20:29 ? 0:00 zpool-tank2
root 23882 23881 0 04:35:16 ? 0:12 /usr/sbin/zpool destroy -f tank2
root 24137 1 0 04:40:30 ? 0:00 sudo /usr/sbin/zpool list -H -o name
root 23881 1 0 04:35:16 ? 0:00 sudo /usr/sbin/zpool destroy -f tank2
root 24138 24137 0 04:40:30 ? 0:00 /usr/sbin/zpool list -H -o name
root 7810 1 0 11:44:35 ? 0:00 /usr/sbin/zpool list -H -o name
root 9734 1 0 12:38:31 ? 0:00 sudo /usr/sbin/zpool list -H -o name
root 9735 9734 0 12:38:31 ? 0:00 /usr/sbin/zpool list -H -o name
root@napp-it-15a:/root#
 
Is this on ESXi 5.5 or 5.5U2?

5.5 is buggy especially with e1000
5.5 U1 is buggy with NFS
5.5 U2 seems ok
 
napp-it autosnap: include ESXi hot snaps in ZFS snaps

New function in napp-it dev (0.9f4=unstable 0.9, free download if you want to try)
autosnap: include ESXi hot snaps in ZFS snaps on NFS datastores

The idea:
- create a hot ESXi snap with memory state (remotely via SSH, datastore on NFS/ZFS)
- create a ZFS snap
- delete ESXi snap

in case of problems
- restore ZFS snap
- restore hot ESXi snap with Vsphere

see menu Services >> SSH >> SSH keys and menu Jobs >> ESXi hot-snaps [Help] and
http://napp-it.org/downloads/changelog_en.html


Are there any problems to be expected? like the comment I already got at STH:
- with Vcenter you may need to re-register the VM after restore as
Vcenter was aware that you have deleted the snap.
 
Last edited:
EDIT: nvm on that question, but iv'e got another hehe. I've never setup iSCSI before and well i think i did everything right, ESXi was able to pull up the iSCSI name just fine after punching in the IP, but the LUN isn't showing. I've included screenshot of the comstar main page in hopes someone will see something i've goofed thats keeping ESXi from seeing it?

iscsi1.png
 
Last edited:
You need 4 steps with Comstar

- create LUs
- create targets
- create target groups with targets as a member
- create views LU -> target groups to make them visible to all member-targets

In your case, you set a view to an empty target group so no visibility to any target
 
yeah i realized i forgot to add member to the target group and added one but still see nothing. I updated the original pic
 
Setting seems ok now.
Can you try to connect from Windows to check if the problem is on ESXi.
 
I see no reason why the LUN should not be visible.
Can you compare a simple setup (menu ZFS filesystems, enable iSCSI for a filesystem)
 
I would suppose it will work as well if you redo all 4 steps manually.
 
New function in napp-it dev (0.9f4=unstable 0.9, free download if you want to try)

Is there a "Pro" requirement to run Gea?
I upgraded my "Free" version to:
"running version : 0.9f4_dev Jan.09.2015", yet the tree is:
"Services >> SSH >> SSH keys" -period- no "and menu Jobs"
Did "download 0.9unstable& activate 0.9unstable" but obviously, some other version.
 
You can currently update a free version to 0.9 unstable to try the ESXi + ZFS autosnap feature.
This gives you the newest 0.9f4_dev from Jan. 09
 
How?
Pls give detailed instructions, as I did everything I think reasonable towards this direction.
 
read
- menu Services >> SSH >> SSH keys (howto create and exchange keys)
- menu Jobs >> ESXi hot snaps >> help
 
Ok, I got it now. I was looking for a "SSH keys and menu Jobs" entry.
Thanks, I will try it.
 
Now that I have read the instructions I have a policy query and need your recommendation:
I have a limited SSD mirror pool for the VM files and an amply sized HDD raidz2 for general storage.
When the ZFS snap gets created it will include all VMs regardless of pre-selection or not.
The difference between the two is that those pre-selected will have a freshly taken ESXi snap, so they can be consistently reinstated afterwards.
With the non selected VMs, lacking this ESXi snap, this may work or not depending on the state that the VM was at the time the ZFS snap was taken.

If the aforementioned are correct and taking into consideration that I don’t use dedup, should I set the SSD ZFS snap to be written onto the HDD datastore, as the former one will get quickly full after a series of snaps?

PS The above is based on my assumption that within ZFS one can take snaps only of datastores and not folders or individual files. I may be wrong though.
 
Last edited:
If you include ESXi hot state snaps or not has no influence to the size of the ZFS snap.
What I would do: Keep only one or a few snaps on the SSD and use replication to backup
them to the larger pool where you can keep more snaps

ps
You can take snaps only on ZFS filesystems not on "regular folders"
What you can do, create as many filesystems as you need.
 
Last edited:
If you include ESXi hot state snaps or not has no influence to the size of the ZFS snap.
How come this?
When ESXi takes a snap, doesn't it create another file?
What I would do: Keep only one or a few snaps on the SSD and use replication to backup
them to the larger pool where you can keep more snaps
Solaris "Time Slider" has built in this option "Replicate backups to an external drive".
I think napp-it does it too, but I use the Time Slider.

You can take snaps only on ZFS filesystems not on "regular folders"
What you can do, create as many filesystems as you need.

Yes, I already thought about assigning one file system per VM.

Thanks for your input and endeavours with napp-it
 
Last edited:
1.
When ESXi creates a snap, it writes any modifications after the snap to a new file.This is not as sophisticated as ZFS snaps with the Copy On Write filesystem where you only freeze data pointers.

But as you create the ZFS snap immediatly after the ESXi snap the difference is minimal as the ESXi delta-file is minimal..

2.
You can use any zfs send script, use a manual zfs send over SSH or use the napp-it Pro replication

3.
I do a nightly snap on any VM and replicate the whole NFS to my backup systems.
Keep it simple.

This may be different if you need a hot snap more often.
But be aware that ESXi snaps can halt a VM for some seconds or minutes depending on storage performance.
 
Last edited:
1. Reasonable argument
2. Will see if I can sync it with Time Slider
3. I take backups every Saturday, so no need for anything exotic here.

I have a second ESXi (free) server and use is as "poor man's" vMotion (manually).
I will experiment on this with a single VM.

Thanks and Good Night
 
EDIT: I found my error Gea. Copy-paste problem (first character from ssh-rsa .... missing). Silly me.

I leave the following as it may serve as an example of what you will face with if you don't copy correctly the public key.

1) Instruction: "open file id_dsa.pub and copy the key (ssh-dss AAAAB3NzaC1kc3MA...) so you can paste in step 5"
The id_dsa.pub file ends with "....== root@servername"
Must this aswell (i.e. everything) be included into the "authorized-keys" file of ESXi?
Yes, copy everything

2) Instruction: "On OmniOS/OI/Solaris, open a Console, login as root and enter
ssh -l root ip_of_ESXi uname -a"

I get asked this question:
"The authenticity of host '192.168.0.xy (192.168.0.xy)' can't be established.
RSA key fingerprint is .......................................................
Are you sure you want to continue connecting (yes/no)?"

Looks like there is some key problem here.
Answering "yes", this is printed:
"Warning: Permanently adding '192.168.0.xy' (RSA) to the list of known hosts.
Password:"

Entering the root password, ESXi returns the OS data, the VMs list, as per "Step 6: Test".
With correct public key, no question/password asked.

3) On napp-it "Jobs>ESXi hot-swaps>Request ESXi VM list" I get:
"try to connect to ESXi server 192.168.0.xy:
Permission denied (publickey,keyboard-interactive)"

With correct public key, you get the VMs list as expected.
 
Last edited:
Finally, I took my first hot ESXi/ZFS snapshot.
Some feedback for Gea:
In the "Jobs > snap > Hot ESXi snaps" menu I see all and every snapshot (the same content with the top "Snapshots" menu).
It would be nice to have a way of discriminating between ordinary and hot ESXi snapshots.
What I tried:
In the "Jobs > create autosnap job" I substituted the default entry "daily" with "esxivms" and as such these snaps sort differently in the full list.
Now Gea, if you can standardise some relative name and only show these in the "Jobs > snap > Hot ESXi snaps" menu, this would do the trick.
 
thanks for the feedback.

You can check the opt3 field of a job.
When you enable ESXi hotsnaps with a SSH prejob, you will see there a "pre" note.
 
You are welcome.

One other very secondary note:
When napp-it executes the ESXi snap command, this is registred within the ESXi snapshot manager under the Name "ZFS_autosnap" bearing as Description ",job=xxxxxxxxxx"
Normally, this will not be displayed after the auto snap deletion, but if someone (I for example tried it for testing purposes) elects not to erase the ESXi snapshot, the Name looks fine, but the Description starting with a comma "," doesn't look very cricket.
Personally, appart from the job's number, I would have liked to see the date and time this snapshot had been taken.

But as I said, this is an unimportant thing.


capture screen
 
Last edited:
Gea I may have found a bug.

The Post script was not executing, so I compared it's content with the Pre and found that there was a "40" missing at the end.
So, I patched the exact line from the Pre script and now it gets executed.
See below (first line not working, added the second and it works):

ssh -l root 192.168.0.44 vim-cmd vmsvc/snapshot.removeall; # [SILO2_VMs] WHS/WHS.vmx
ssh -l root 192.168.0.44 vim-cmd vmsvc/snapshot.removeall 40;

What is the significance of this "40"?
 
edit: fixed

- The 40 is the id of the VM, this is now preset when you import VMs
- the ,job should be date, job
 
Last edited:
quick question, I got 2 240gb crucial m500's I was wanting to use as a napp-it mirror and use the rest for datastores crucial tells me I can just leave unpartitioned space for over provisioning without having to create a HPA... I was planning on overprovisioning 20% or so that way. My one concern is that if I use the ssds for datastores and napp-it will the drives actually have time to idle to do garbage collection? I will probably have a pfsense vm with couple windows/linux vm's always running....
I have found a article with some ssd esxi info stating that ssd + vaai = trim over here http://www.v-front.de/2013/10/faq-using-ssds-with-esxi.html any suggestions and info are most welcome. Thanks
 
It does not matter if you partition a disk or create a HPA
Advantage of HPA is that it is a set and forget setting
 
It does not matter if you partition a disk or create a HPA
Advantage of HPA is that it is a set and forget setting

Thanks _Gea
Should I be concerned with napp-it and datastores not allowing enough idle time for garbage collection to happen?
 
Manual overprovisioning is nothing that affects data security.
It helps a little to keep write performance of desktop SSDs high under load.

As ESXi or OmniOS/napp-it on a local datastore produce only a minimal write load
(mainly logs), this is not really relevant. Even if you use a regular disk that is
10-100 times slower regarding write iops, you will see no real difference in storage performance,
boot is not as fast and the web-ui may react slower.
 
Manual overprovisioning is nothing that affects data security.
It helps a little to keep write performance of desktop SSDs high under load.

As ESXi or OmniOS/napp-it on a local datastore produce only a minimal write load
(mainly logs), this is not really relevant. Even if you use a regular disk that is
10-100 times slower regarding write iops, you will see no real difference in storage performance,
boot is not as fast and the web-ui may react slower.

Thanks _Gea for the explanation, it helps put me at ease going forward with upgrading my datastores to ssd!
 
I might have encountered a slight problem with my ZFS pool. The pool consists of 20 drives and the pool is logically made of 2x 10 drive raidz2's.

I firsts noticed the problems after my network drives went suddenly down. After logging to the Solaris I found that all the disks in my pool had pretty much the same amount of write errors. They also were all in UNAVAIL status. I obviously assumed there would be some problems with the SAS expander or with the disk controller. I've now got a new disk controller and I found out some additional problems when debugging: apparently three of the disks are completely 'dead' (i.e. not detected). I also have two other disks that were offlined by Solaris (I'm hoping due to transient errors caused by the actually broken disks...). Now the pool is completely UNAVAIL but the SMART on these two disks seems to be fine, however, so I'm cautiously optimistic on recovering the pool, or atleast the actually important data. Also, I'm planning on pulling the trigger for Supermicro 846BE16-R920B as the Norco is apparently a piece of junk..

How would you suggest me on recovering the pool? I'm thinking I should mount the pool as read only and backup the data to a separate NAS as soon as possible before doing any resilvering? Can I disable something in order to prevent Solaris from offlining additional disks because of read/checksum errors and thus setting the pool into UNAVAIL state?
 
There are two possibilities:
- The disks (more than your redundancy level allows) are really damaged.
The pool is lost then and you need a backup

The more propable reason if you have sudden problems with many disks
Your psu, cabling, backplane, disk controller or expander is bad.
In this case the pool come back to online or degraded automatically
whenever enough disks come back. Unlike hardware raid, a sudden
remove of some disks from a pool does not matter as it only offlines the pool.
 
There are two possibilities:
- The disks (more than your redundancy level allows) are really damaged.
The pool is lost then and you need a backup

The more propable reason if you have sudden problems with many disks
Your psu, cabling, backplane, disk controller or expander is bad.
In this case the pool come back to online or degraded automatically
whenever enough disks come back. Unlike hardware raid, a sudden
remove of some disks from a pool does not matter as it only offlines the pool.

Thanks for the reply. I can confirm that I've got indeed two dead drives in my raidz2. I've also got a third one which has been set to UNAVAIL for some reason by Solaris.The SMART is OK on this drive.

Do I have any options in preventing the Solaris from setting this third disk to UNAVAIL status again even though there might be some errors, just so I could do my backup?
 
You cannot prevent ZFS to set a damaged OR MISSING disk to unavail.

If the disk is missing because it gots a new id, you can fix this with a pool export/import
where all disk-ids are reorganized optionally with a "ignore last faults"
- with napp-it you can select on import.

Other options.
- reboot after a power off
- switch missing disk to another port, reboot or export/import
- menu pool - clear errors
 
Last edited:
I was setting up a hp microserver last week for a friend with nappit /omni os directly installed.. nice machine not to expensive. While testing it maxed out my 1gb LAN immediately.
For myself i have a esxi setup with omnios and couiple of vm,'s. spend couple of thousand euro's and not getting any further then 600mbit.. so frustrating. in omni i use the VMXNET3 adapter. tested with bonnie: 480 MB/s Seq-Write . 546 MB/s Seq Read

where to begin?
 
Back
Top