OpenZFS NAS (BSD, Illumos, Linux, OSX, Solaris, Windows + Storage Spaces) with napp-it web-gui

Hey, i'm redoing my pool and decided to give esxi a shot and set my drives as passthrough and allocated 7gb and 2 cores to my openindiana virtual machine installed napp-it as i would on a normal system left the default virtual network adapter created a pool in raidz with 3 drives at first to try it out (that's one of the optimal setups for zfs raidz1 right? 3 drives 2 data 1 parity)

Then i shared it through smb and copied over a file with 8gb and my transfer speeds start at around 90mb/s and then slowly drop like it goes up 2 then down 3mb/s etc but other than some weird result first time the lowest i saw it was around 63 to 70mb/s, is that normal? should i add zil if i want to max the connection? The drives are wd red 3tb, anything extra i need to do to get good network performance?

I also installed the vmware guest tools, and that's it, install esxi pick the ports for passthrough upload openindiana image install openindiana install napp-it create pool and filesystem and bring it online.

Edit: tried the vxnet3 network adapter and same thing, also added 2 more drives in basic mode and same speeds. Is it just the expected speed raidz1 with just 3 drives before i mirror vdevs etc or is it something with vmware?

Thanks
 
Last edited:
gasu said:
Hey, i'm redoing my pool and decided to give esxi a shot and set my drives as passthrough and allocated 7gb and 2 cores to my openindiana virtual machine installed napp-it as i would on a normal system left the default virtual network adapter created a pool in raidz with 3 drives at first to try it out (that's one of the optimal setups for zfs raidz1 right? 3 drives 2 data 1 parity)

Then i shared it through smb and copied over a file with 8gb and my transfer speeds start at around 90mb/s and then slowly drop like it goes up 2 then down 3mb/s etc but other than some weird result first time the lowest i saw it was around 63 to 70mb/s, is that normal? should i add zil if i want to max the connection? The drives are wd red 3tb, anything extra i need to do to get good network performance?

I also installed the vmware guest tools, and that's it, install esxi pick the ports for passthrough upload openindiana image install openindiana install napp-it create pool and filesystem and bring it online.

Edit: tried the vxnet3 network adapter and same thing, also added 2 more drives in basic mode and same speeds. Is it just the expected speed raidz1 with just 3 drives before i mirror vdevs etc or is it something with vmware?

Thanks
Click to expand...

Is it droping to 2-3 MB/s or dropping below 90mb/s by 2-3MB/s? Also I am assuming you are talking MegaBYTES/s right?

If you are getting around ~90/MB/s on writes that is about as good as you will get with a single RAIDZ array since you are pretty much limited to single drive on writes.
 
Hey, i meant it will start at 100ish MB/s and then if it's a big big file or multiple folders with big files by the end it will be at around 65 to 70MB/S, i tried with different hard drives, may be a windows or my macbook pro network card.

Guess first thing i should do is try this from macos and then some other computer with an intel nic?

Cheers
 
gasu said:
Hey, i meant it will start at 100ish MB/s and then if it's a big big file or multiple folders with big files by the end it will be at around 65 to 70MB/S, i tried with different hard drives, may be a windows or my macbook pro network card.

Guess first thing i should do is try this from macos and then some other computer with an intel nic?

Cheers
Click to expand...

Also the speed of the source drive will effect it as well. But the fact is with a single RAIDZ you will not get that much faster of writes.
 
m1abram said:
Also the speed of the source drive will effect it as well. But the fact is with a single RAIDZ you will not get that much faster of writes.
Click to expand...

Thanks i won't stress about it then for now and will set up my other drives and start mirroring vdevs and see if it changes.

Appreciate it
 
gasu said:
Thanks i won't stress about it then for now and will set up my other drives and start mirroring vdevs and see if it changes.

Appreciate it
Click to expand...

I do not believe you can mirror vdevs, you can only "stripe" them.
 
_Gea said:
I want to add my own config.

I use about ten All-In-One's (university use) with 16-64 GB RAM (free and payed). On each of them we run only a few VMs,
mostly Windows server (AD 2012, Mailserver, Terminalserver, Webserver), OSX server for ical or OmniOS
based webserver/ webmailer.

I prefer many smaller systems over some large systems to have a better load balancing and to have not too many
VM's down on problems. I also use SSD only pools for VMs with two backups to disk pools (plus one external backup on different location)

All servers are connected via an CX4 10 GbE vlan based network for fast backup/ move/ clone/ vmotion + storage motion

I would always prefer such a multiple cheap systems config over one or two large systems that are not allowed to fail.
It is easier to update and test a small system out of many than to update the one system that is not allowed to fail-
even with ZFS where you can use BE system snaps to undo an update or All-In-Ones, where you can add a second
updated local SAN system where you only need to switch the HBA due to pass-through.

I also use external hardware raid-1 enclosures for 2 x 2,5" disks for the reason to clone system disks and to have the option
to boot different systems on the fly. You can also use my preconfigured ESXi 5.5 appliance image to update the whole system.

On a real hardware crash, we unplug the disks, put them into the next All-In-One (we have always enough free bays),
import the pool and the VMs and we are back on service after 30min (to avoid using backups that are not as up to date)
an to avoid real HA setups with its complexity.
Click to expand...

Thanks GEA! Really interesting how you use AIO, ZFS, and ESXi to handle high availability.
 
My SMB users are getting temporary hiccups. Windows complains that the mapped network drive is no longer available and then if we wait a minute or two and don't touch anything, it comes back.
Is there a common cause for this? I'm not having any luck with google. I can see a CPU spike in vsphere client for the OmniOS VM that might be at the same time as the last hiccup. I can't tell if it's something happening in OmniOS, or another VM. Most of the time, everything is pretty much idle.
 
Have you similar problems within other VMs, then it is a ptoblrm within this box.
Can you isolate to external guests, then its a network problem
 
Just caught OmniOS having a panic and rebooting itself?
Now I'm noticing 41 dumps in /var/crash/unknown in about 15 days.
eep
 
Hey just an update on my performance, i installed windows 2012 server on my esxi box and uploaded a file to there (still the slow speeds) but then transfered a file from the windows server to the omnios through smb (both on the esxi) and my transfer speeds were 130mb/s dropping sometimes to 127 but was quite stable at 130 and ended at 132mb/s or so.

That should sort of indicate an issue on my macbook network than anything right? if when i transfer through samba from my macbook pro to the esxi it's slow and unstableish and if within the server itself i can achieve good speeds through smb?

Cheers, happier now at least :)
 
gasu said:
That should sort of indicate an issue on my macbook network than anything right? if when i transfer through samba from my macbook pro to the esxi it's slow and unstableish and if within the server itself i can achieve good speeds through smb?

Cheers, happier now at least :)
Click to expand...

The current SMB implemenation on OSX is quite slow, sometimes only half as fast than on Windows.
On OSX AFP and NFS are much faster. - hope for 10.9 -
 
tladuke said:
Just caught OmniOS having a panic and rebooting itself?
Now I'm noticing 41 dumps in /var/crash/unknown in about 15 days.
eep
Click to expand...

Kernel panic = hardware problem, mostly RAM or power.
Do extensive RAM tests first (or remove half the RAM, test, then remove the other half)
 
Ah i was trying from windows with bootcamp, but at least it seems to hit stable at 70MB/s now from windows bootcamp to esxi, so i'll upload everything i need and then it won't matter anymore :), since it will be all inside the esxi box.

But yes i have noticed before that OSX smb performance is poor :(.
 
_Gea said:
Kernel panic = hardware problem, mostly RAM or power.
Do extensive RAM tests first (or remove half the RAM, test, then remove the other half)
Click to expand...

:(
I even bought the supermicro approved samsung ram.
Actually I hope it's the RAM and not some problem with running only like 4 VMs
 
Untitled.jpg


Can someone help me with my throughput problem? notice how uneven the graph is and how slow it is at the start? I have a raidz1 array with 5 drives. The image is posted from a Win7 VM on the server itself.

When I run dd bench, I get the following:
Memory size: 6144 Megabytes

write 12.8 GB via dd, please wait...
time dd if=/dev/zero of=/pool/dd.tst bs=2048000 count=6250

6250+0 records in
6250+0 records out

real 50.0
user 0.0
sys 3.0

12.8 GB in 50s = 256.00 MB/s Write

wait 40 s
read 12.8 GB via dd, please wait...
time dd if=/pool/dd.tst of=/dev/null bs=2048000

6250+0 records in
6250+0 records out

real 42.5
user 0.0
sys 2.6

12.8 GB in 42.5s = 301.18 MB/s Read
 
Also on ESXI and the all in one or just a manual omnios install on esxi at boot time it says 2 times failed to update microcode 0xsomething to 0x11, is it normal? Doesn't happen on openindiana.

Cheers
 
gasu said:
Also on ESXI and the all in one or just a manual omnios install on esxi at boot time it says 2 times failed to update microcode 0xsomething to 0x11, is it normal? Doesn't happen on openindiana.

Cheers
Click to expand...

From a kernel and driver view, OmniOS and OI are quite the same
but OmniOS is mostly based on newer Illumos bits.

On my current OmniSetups, I do not get this message during boot on ESXi,
so it seems more a problem on a special hardware.

If the message indicates further problems, you may either try a newer OmniOS bloody
or stay with OI 151.a8
 
Running OmniOS v11 r151006 and Napp-It 0.9b3. I'm trying to figure out how to set a monthly scrub job. I just want a scrub to run once a month on the first Sunday night. Any guidance? Thanks!
 
joltman said:
Running OmniOS v11 r151006 and Napp-It 0.9b3. I'm trying to figure out how to set a monthly scrub job. I just want a scrub to run once a month on the first Sunday night. Any guidance? Thanks!
Click to expand...

you cannot set this with autojob - scrub
you can only set to every sunday or every month at a special day like the 1st.

you can use other jobs with your own script or you need to modify the autoscrub script.
 
_Gea said:
From a kernel and driver view, OmniOS and OI are quite the same
but OmniOS is mostly based on newer Illumos bits.

On my current OmniSetups, I do not get this message during boot on ESXi,
so it seems more a problem on a special hardware.

If the message indicates further problems, you may either try a newer OmniOS bloody
or stay with OI 151.a8
Click to expand...

No evil has come out of it so i'll let it stay just found it odd as it doesn't display on Openindiana, still does on bloody, and doesn't happen if i just install the os without virtualization, anyway i'm sure it's no big deal but if i notice poor performance down the line i'll switch to OpenIndiana, maybe it's due to having kvm support out of the box as besides saying failed to update microcode from 0x10 to 0x11 also says no kvm not supported, speaking of that maybe i'll give omnios + kvm a try.

Edit: Fixed, applied the latest microcode from the intel downloads and no more message :), since it's a old 920 and latest bios is from 2011 or so guess it was outdated and omnios was trying to update it itself and esxi blocks it : ), so it should be done from the host itself, had to add a entry to do it every time it boots or it will be lost in between reboots, it already does so by default but can't replace the .dat or it will be overwritten on boot up, so best to call it in rclocal, anyway just info if anyone runs into it in the future.
 
Last edited:
Can anyone offer me a bit advice regarding the following problem?

I have four basic vdevs in a pool:
3tb
3tb
3tb
4tb

The ressource consumption is about 5 tb.

I have just received two new 4tb HDD's.

What I what:

1 raidz vdev of 3tb+3tb+3tb

1 raidz vdev of 4tb+4tb+4tb

The first thing I have to do is empty the 4tb HDD in use, create a new raidz1 vdev of 4tb+4tb+4tb and then move all the data from the three 3tb basic vdevs to the new raidz vdev.

Any advice is much appreciated.
 
This sounds like the problem with the corn, the chicken, and the fox crossing the river.
Gnavox said:
The first thing I have to do is empty the 4tb HDD in use, create a new raidz1 vdev of 4tb+4tb+4tb and then move all the data from the three 3tb basic vdevs to the new raidz vdev.
Click to expand...
You can't evacuate data from a device like you want. This requires a feature called "bp-rewrite" which has been requested for years, but it's a hard problem and there's no sign it will be implemented any time soon.
Gnavox said:
Any advice is much appreciated.
Click to expand...
If you have a little more extra space (say, some 1TB hard drives sitting in a closet), here's what I'd do.
  • Step 1: Create pool2, composed of 2*4TB + 2*1TB mirrors. Move all data to pool2 with zfs send.
  • Step 2: Destroy pool1, and create pool3 from 3*3TB raidz1. Move all data to pool3 with zfs send.
  • Step 3: Destroy pool2, and add another vdev to pool3 composed of 3*4TB raidz1. Copy and delete data until balanced enough for your use case.
 
I have an odd problem with nappit.

Before I describe the issue, here's what I'm working with.

MB: Supermicro H8SGL-F (AMD G34)
CPU: Opteron 6136 (2.4ghz 8 core)
Ram: 64gb DDR3 1066mhz
Raid Card: LSI 2008 (flashed to IT)
ESXi Version 5.1.0 799733
Supermicro Hotswap Bay
Drives: 3x 4tb Seagate Barracuda XT
2x 4tb Hitachi 7200rpm

OS: open indiana 151a8 w/ Napp-it v0.93 nightly in a VM.

I can't build a pool. If I try to build one, or get info on drives, the appliance just sits saying "processing, please wait..."

HW passthrough works. OI sees all 5 4tb drives on the controller. ESXi sees the controller as well as the bios & boot phase with it.

Any ideas? I've been fighting with this for the past few days.

Edit: I've tried Chrome/IE10/FF and the results are the same.
 
By the way, OmniOs unstable spells trouble for me in Esxi, if you update the system before installing napp-it i get ramdisk initialization error, if i install napp-it then update the system (because no ldd by default which is required for vmware tools) and install vmware tools then same ramdisk error, did anyone else try it?

If i just install napp-it update and do nothing else it works fine, but installing vmware tools = disaste, normal omnios works fine tho.
 
Last edited:
Synthetickiller said:
I
I can't build a pool. If I try to build one, or get info on drives, the appliance just sits saying "processing, please wait..."

.
Click to expand...

Check the minilog if it hangs at the parted command (or call parted at CLI).
This happens mostly with new unpartitioned disks.

To solve, you can either
- initialize disks (menu disks - initialize) or
- disable partition support ( napp-it 0.9d1)
 
It seems that my rpool size went up a little bit and now triggers napp-it alerts for 15% space left. I am on OI_151a8 (with UI). Here's the alert I get:
Code: 
------------------------------
zpool list (Sum of RAW disk capacity without redundancy counted)
------------------------------
NAME        SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
VMstorage   278G   200G  77.9G         -    71%  1.00x  ONLINE  -
rpool      19.9G  13.5G  6.33G         -    68%  1.00x  ONLINE  -

Real Pool capacity from zfs list
------------------------------
NAME		 USED		 AVAIL		 MOUNTPOINT		 %
VMstorage		 214G		 73.5G		 /VMstorage		 26%
rpool		 16.6G		 2.97G		 /rpool		 15%!

Questions:
1. Why the difference betwee ALLOC and USED (13.5G vs. 16.6G) since there is no dedub active?
2. My other all in one systems have the rpool at around 10G. Is there a way to free up some space (e.g. is there any temp data from the OI151a7 to a8 upgrade that can be deleted safely)?

Thanks!
-TLB
 
joltman said:
Running OmniOS v11 r151006 and Napp-It 0.9b3. I'm trying to figure out how to set a monthly scrub job. I just want a scrub to run once a month on the first Sunday night. Any guidance? Thanks!
Click to expand...



Code: 
su root
export EDITOR=nano
crontab -e

#*     *     *   *    *        command to be executed
#-     -     -   -    -
#|     |     |   |    |
#|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
#|     |     |   +------- month (1 - 12)
#|     |     +--------- day of        month (1 - 31)
#|     +----------- hour (0 - 23)
#+------------- min (0 - 59)#
# 13th day every month, at 3:00 am
0 3 13 * * zpool scrub tank
 
TheLastBoyscout said:
2. My other all in one systems have the rpool at around 10G. Is there a way to free up some space (e.g. is there any temp data from the OI151a7 to a8 upgrade that can be deleted safely)?
Click to expand...

151 a7 is yet there as a boot environment
check beadm list and use beadm destroy to delete unneeded bootsnaps.
 
_Gea said:
Check the minilog if it hangs at the parted command (or call parted at CLI).
This happens mostly with new unpartitioned disks.

To solve, you can either
- initialize disks (menu disks - initialize) or
- disable partition support ( napp-it 0.9d1)
Click to expand...

Thanks! The first option worked perfectly. :D

You saved me a lot of time!;)
 
Here's the error on OmniOS bloody on Esxi 5.1 update 1 after installing napp-it > system update > install vmware tools with default options and reboot

http://i.imgur.com/eUyadVw.png

Is it just broken for vmware tools right now or?

Stable works fine.
 
Last edited:
Hello, I am using OmniOS and I installed napp-it some months ago, then I removed it because I didn't really need it.

Now I found something strange and I need confirmation before proceeding, or I need an older install script to check by myself.

I found two copies of the "smartmontools" binaries (smartd, smartctl, …), one in /opt/omni/... and one in /usr/...
The one in /opt/omni/... was installed by me, and I have the proof: by uninstalling the pkg, it disappeared.
The other one has an install date I cannot find anywhere else in the repositories: 31st of March. I just remember I had installed OmniOS some days earlier and I was setting it up.

I gave another look at /usr/sbin and I found many other files/links dated 31st of March, most of them are links to the same binaries, located in a more "regular" location, such as /sbin. I checked the linked files and they are dated according to the latest OmniOS release.

All this makes me think those links were placed by napp-it.
Could someone (the developer hopefully) confirm this? I put some of the files as example:

Code: 
accept -> ./print-service
automount -> ../lib/fs/autofs/automount
autopush -> ../../sbin/autopush
beadm -> ../../sbin/beadm
bootadm -> ../../sbin/bootadm
cachefslog -> ../lib/fs/cachefs/cachefslog
cachefswssize -> ../lib/fs/cachefs/cachefswssize
cfsadmin -> ../lib/fs/cachefs/cfsadmin
cryptoadm -> ../../sbin/cryptoadm
dcopy -> ./clri
devnm -> ./df
dladm -> ../../sbin/dladm
dlstat -> ../../sbin/dlstat
edquota -> ../lib/fs/ufs/edquota
fdisk -> ../../sbin/fdisk
fiocompress -> ../../sbin/fiocompress
flowadm -> ../../sbin/flowadm
flowstat -> ../../sbin/flowstat
fsdb -> ./clri
fsirand -> ../lib/fs/ufs/fsirand
fssnap -> ./clri
hostconfig -> ../../sbin/hostconfig
ifconfig -> ../../sbin/ifconfig
inetd -> ../lib/inet/inetd
init -> ../../sbin/init
installgrub -> ../../sbin/installgrub
ipadm -> ../../sbin/ipadm
ipmpstat -> ../../sbin/ipmpstat
labelit -> ./clri
lockfs -> ../lib/fs/ufs/lockfs
lpadmin -> ./print-service
lpmove -> ./print-service
mbuffer
metadb -> ../../sbin/metadb
metadevadm -> ../../sbin/metadevadm
metainit -> ../../sbin/metainit
metarecover -> ../../sbin/metarecover
metastat -> ../../sbin/metastat
mkfs -> ./clri
mount -> ../../sbin/mount
ncheck -> ./ff
newfs -> ../lib/fs/ufs/newfs
quot -> ../lib/fs/ufs/quot
quota -> ../lib/fs/ufs/quota
quotacheck -> ../lib/fs/ufs/quotacheck
quotaoff -> ../lib/fs/ufs/quotaoff
quotaon -> ../lib/fs/ufs/quotaon
reject -> ./print-service
repquota -> ../lib/fs/ufs/repquota
route -> ../../sbin/route
routeadm -> ../../sbin/routeadm
showmount -> ../lib/fs/nfs/showmount
smartctl
smartd
sync -> ../../sbin/sync
tunefs -> ../lib/fs/ufs/tunefs
tzreload -> ../../sbin/tzreload
uadmin -> ../../sbin/uadmin
ufsdump -> ../lib/fs/ufs/ufsdump
ufsrestore -> ../lib/fs/ufs/ufsrestore
umount -> ../../sbin/umount
update-smart-drivedb
wusbadm -> ../../sbin/wusbadm
xntpdc -> ntpdc
zfs -> ../../sbin/zfs
zpool -> ../../sbin/zpool
 
push3r said:
Test re-done with OmniOS and Napp-IT

Intel DC S3700 100GB as ZIL device partitioned to 3GB
...
Click to expand...
This is maybe a stupid question, but why are you reducing the size of the ZIL device? Sure, the system doesn't benefit from more space, but is there a benefit of having a smaller partition?

Thanks!
TLB
 
zuhizam said:
confirmation before proceeding, or I need an older install script to check by myself.

I found two copies of the "smartmontools" binaries (smartd, smartctl, …), one in /opt/omni/... and one in /usr/...
Click to expand...

Smartmontools (compiled from sources) is part of the napp-it installer.

You have thre options for installation:
- compiling from sources
- use one or another IPS repository
- use a pkgin repository

they do not care about each others setup and can install to different locations
resulting in double installations.

Napp-it itself is all in the folder /var/web-gui. The installer configures your
storage server ready to use with some tools (smartmontools, mc, nc etc)

If you like a clean setup without napp-it and its modifications, you can go back to the prior napp-it BE
 
Konowl said:
Untitled.jpg


Can someone help me with my throughput problem? notice how uneven the graph is and how slow it is at the start? I have a raidz1 array with 5 drives. The image is posted from a Win7 VM on the server itself.

When I run dd bench, I get the following:
Memory size: 6144 Megabytes

write 12.8 GB via dd, please wait...
time dd if=/dev/zero of=/pool/dd.tst bs=2048000 count=6250

6250+0 records in
6250+0 records out

real 50.0
user 0.0
sys 3.0

12.8 GB in 50s = 256.00 MB/s Write

wait 40 s
read 12.8 GB via dd, please wait...
time dd if=/pool/dd.tst of=/dev/null bs=2048000

6250+0 records in
6250+0 records out

real 42.5
user 0.0
sys 2.6

12.8 GB in 42.5s = 301.18 MB/s Read
Click to expand...

Anyone?
 
You must log in or register to reply here.
Back
Top