OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Hi,

I have an all-in-one system with OI 151a and run into a problem trying to copy the VMs from the OI to a different system on the network. Whenever the copy process gets to the (large) .vmdk file, I get an error that "Permissions are denied".
I already removed the VM from vSphere's inventory, but the error persists.
Looking at the permissions, I see the following:
Code: 
-rwxrwxrwx+  1 nobody   nobody    270622 Apr 15 11:56 vmware.log
-rwxrwxrwx+  1 nobody   nobody    141975 Feb 29 14:14 vmware-8.log
-rwxrwxrwx+  1 nobody   nobody    100167 Feb 17 09:53 vmware-7.log
-rwxrwxrwx+  1 nobody   nobody     99692 Feb  3 11:45 vmware-6.log
-rwxrwxrwx+  1 nobody   nobody     95830 Feb  2 11:13 vmware-5.log
-rwxrwxrwx+  1 nobody   nobody    164366 Dec  9 08:29 vmware-4.log
-rwxrwxrwx+  1 nobody   nobody    309607 Sep 26  2011 vmware-3.log
-rw-rw-rw-   1 nobody   nobody      2847 Sep 21  2011 testvm.vmxf
-rwxrwxrwx+  1 nobody   nobody      2852 Apr 15 11:56 testvm.vmx
-rw-rw-rw-   1 nobody   nobody       380 Apr 15 11:56 testvm.vmsd
-rw-------   1 nobody   nobody       520 Feb  3 14:00 testvm.vmdk
-rw-rw-rw-   1 nobody   nobody      8684 Apr 15 11:56 testvm.nvram
-rw-rw-rw-   1 nobody   nobody     30926 Feb 17 09:54 testvm-Snapshot2.vmsn
-rw-------   1 nobody   nobody   53687091200 Feb 17 09:53 testvm-flat.vmdk
-rw-rw-rw-   1 nobody   nobody       326 Feb 29 14:21 testvm-000001.vmdk
-rw-rw-rw-   1 nobody   nobody   10854965248 Apr 15 11:56 testvm-000001-delta.vmdk

Any ideas what I can try?

Thanks,
-TLB
 
I am a "Windows child" of sorts :eek: So yeah, looking at the permissions of the .vmdk files (only the owner can rw) it does make sense (?).
However, why are all the other files not set up in the same way (ESXi 5.0 behavior)?

Is there any danger in changing the file permissions (permanently), so that I can copy the entire VM automatically via a script?

Thanks,
-TLB
 
TheLastBoyscout said:
I am a "Windows child" of sorts :eek: So yeah, looking at the permissions of the .vmdk files (only the owner can rw) it does make sense (?).
However, why are all the other files not set up in the same way (ESXi 5.0 behavior)?

Is there any danger in changing the file permissions (permanently), so that I can copy the entire VM automatically via a script?

Thanks,
-TLB
Click to expand...

no problem, you can change all file permissions to 666 or 777.
If you want to copy running VM's, you can use (cold) ZFS snaps

If you need hot snaps with memory state, do a ESXi snap, then a ZFS snap
and copy the snap. You can delete the ESXi snap after doing a ZFS snap.
 
Under the new 0.8e nightly version of napp-it I noticed that under home->ZFS Folder->create, the "share settings" no longer has NFS. Assuming this is a bug, unless that was removed intentionally.

Thanks,
--peter
 
Last edited:
siamsquare said:
Yo,

Sorry for repeating my question but is there any1 here who could help me install Squeezebox Server and SABnzb on my OI+Napp-It server. I allready found this :

http://forums.slimdevices.com/showthread.php?93300-Solaris-11-(NOT-Express)-Problem-with-UPnP-Plugin-and-Image-Browser

But as I'm a Linux noob and used working in windows it is a bit hard!
Squeezebox server and Sabnzb are running on my QNAP NAS but I would like to use my OI server only and sell the NAS.

Thanks
Click to expand...

siamsquare -- what version of Linux are you using for squeezeboxserver? If it is RHEL or one of its derivatives, there is a yum repository for it (now called logitechmediaserver):

http://wiki.slimdevices.com/index.php/Logitech_Media_Server_RPM

--peter
 
siamsquare said:
Yo,

Sorry for repeating my question but is there any1 here who could help me install Squeezebox Server and SABnzb on my OI+Napp-It server. I allready found this :

http://forums.slimdevices.com/showthread.php?93300-Solaris-11-(NOT-Express)-Problem-with-UPnP-Plugin-and-Image-Browser

But as I'm a Linux noob and used working in windows it is a bit hard!
Squeezebox server and Sabnzb are running on my QNAP NAS but I would like to use my OI server only and sell the NAS.

Thanks
Click to expand...

You dont have virtualization going?? Put logitecmediaserver on a windows machine---theres a lot less BS you have to put up with
 
siamsquare said:
Yo,

Sorry for repeating my question but is there any1 here who could help me install Squeezebox Server and SABnzb on my OI+Napp-It server. I allready found this :

http://forums.slimdevices.com/showthread.php?93300-Solaris-11-(NOT-Express)-Problem-with-UPnP-Plugin-and-Image-Browser

But as I'm a Linux noob and used working in windows it is a bit hard!
Squeezebox server and Sabnzb are running on my QNAP NAS but I would like to use my OI server only and sell the NAS.

Thanks
Click to expand...

SAB is quite easy, its basically just a python script. You can just use git to clone the repo from github, install a few dependencies and I can send you an SMF manifest to run it as a service. I run sabnzbd+sickbeard+couchpotato on my solaris box with no issues.

pm me if you need help with SABnzbd. I'm on vacation for a few days so responses might be slow.

I don't have experience with Squeeze box but my quick google search turns up the logitech media server and it appears to be a perl program so that should be relatively easy to get up and running on OI.

see: http://wiki.slimdevices.com/index.php/Solaris_Installation_Guide
 
So after building all my ESXi i've now spent some time with openindiana and nappit and I must say, Gea you did an amazing job.

I have two questions reguarding folder/smb

If i don't enable the guest (777) i don't know what user/pass to use to get on the share. How do I add a user for this? (local user seem to require the paid plugin)

Is there a way that I leave the file wild open to the host, but add permission for smb? (So if something goes wrong with my smb permission the host can still access the files)?

other than that the dd disk benchmark dosen't seem to ouput any result even when refreshing.

Thanks you
 
x-cimo said:
So after building all my ESXi i've now spent some time with openindiana and nappit and I must say, Gea you did an amazing job.

I have two questions reguarding folder/smb

If i don't enable the guest (777) i don't know what user/pass to use to get on the share. How do I add a user for this? (local user seem to require the paid plugin)

Is there a way that I leave the file wild open to the host, but add permission for smb? (So if something goes wrong with my smb permission the host can still access the files)?

other than that the dd disk benchmark dosen't seem to ouput any result even when refreshing.

Thanks you
Click to expand...

Add users, this is easy and free
Go to menu users and add some users
Go to menu ZFS folder - acl extension add add permissions for these users

If anything goes wrong or if you accidentally remove all permissions, you must
know that root and the owner have always full access. You cannot lock them out.

Paid extensions
The paid extension do not block or hinder any OI functionality. You can all do
like you would do without knowing the extensions via CLI and shell-scripts.

They are comfort-functions. And they offer a lot of functionality for free.
The ACL extension (use newest) allows setting trivial ACL like everyone@ and user ACL
for free as well as resetting all ACL recursive to a modify for everyone.

The paid ACL extras are for Active Directory, Groups, complex or deny rules or a more detailed
view to file ACL, usually needed by larger organisations. I need these earnings to proceed development.
(There is no enterprise sponsor behind napp-it, all my own effort in my free time)

Thats the same with other extensions like async replication between appliances or the
SAS2 slot and monitoring tool or the pool encryption for OpenIndiana and ZFS V.28 pools with ZFS-2 safe
backup options for encrypted pools to a cloud provider or any external disk.


if you miss bonnie benchmarks on OI:
There was a bug with OI < 151.a3, where multiple package installs failed.
Rerun the napp-it wget installer or do su and then a
pkg install bonnieplus
 
s0rce said:
SAB is quite easy, its basically just a python script. You can just use git to clone the repo from github, install a few dependencies and I can send you an SMF manifest to run it as a service. I run sabnzbd+sickbeard+couchpotato on my solaris box with no issues.

pm me if you need help with SABnzbd. I'm on vacation for a few days so responses might be slow.

I don't have experience with Squeeze box but my quick google search turns up the logitech media server and it appears to be a perl program so that should be relatively easy to get up and running on OI.

see: http://wiki.slimdevices.com/index.php/Solaris_Installation_Guide
Click to expand...

Thanks for all the replies!
I already looked up the internet for a few tutorials, problem is I don't know what's already installed on my OI + Napp-it because those tuts talk about prerequisitories like Perl and such which I guess are already on my machine installed by Napp-It.
I would prefer to run SABnzb and squeezebox natively on my OI machine and try to avoid VTD because then I would need to buy more hardware and all my data is already backed up on 2 pools on my box.

thanks
 
_Gea said:
no problem, you can change all file permissions to 666 or 777.
If you want to copy running VM's, you can use (cold) ZFS snaps
Click to expand...
Changing the permissions on all files for a given VM (in the VMs folder do a chmod 666 *.*) fixed this problem. :) Thanks!

Now I have another problem when trying to bring the backed up VMs back to life: After adding a VM to the inventory on the same, original ESXi host, I get an "unable to start VM, permission denied" error, when trying to start the VM.
I noticed that the owner of the files had changed and changed it back to nobody (in the VMs folder do a chown nobody:nobody *.*) and changed the permissions back to to what they used to be, but no luck.

Looking at the VMware.log file, I see the following near the end of the logfile:
Code: 
Apr 18 14:40:48.063: vmx| CreateVM: Swap: generating normal swap file name.
Apr 18 14:40:48.065: vmx| Swap file path: '/vmfs/volumes/de21732a-7dc18348/WinXPVM01/WinXPVM01-bba7bc2c.vswp'
Apr 18 14:40:48.066: vmx| Msg_Post: Error
Apr 18 14:40:48.066: vmx| [msg.vmmonVMK.creatVMFailed] Could not power on VM : Permission denied.
Apr 18 14:40:48.066: vmx| [msg.monitorLoop.createVMFailed] Failed to power on VM----------------------------------------
It sound like the swap file cannot be created, but why is that?

Thanks!
-TLB

While typing this I realized that I updated the owners and permissions of all files IN the VM's folder, but not the owner and permissions for the folder itself...
After doing that, the VM starts fine :cool:
 
I have built an all-in-one on some pretty good hardware (HP ProLiant DL380 G6 with 16gb of mem soon to be 32gb, Xeon E5530 cpu, 2 mirrored 72gb boot drives on built-in raid controller, separate LSI-9211-8i flashed to IT mode, 6 1TB WDC hard drives in 3 mirrored vdevs). Running ESXi 5.0 with three virtual hosts: OI for ZFS SAN use, Centos 6.2 host to be a firewall, DNS and mail server, and an Oracle Linux 5.8 database machine to be a standy Oracle system.

I created a new linux virtual host under ESXi 5.0 using a zfs folder (NFS-shared). Created it to be 64gb. Then I created a 2nd virtual disk for that machine of 2TB (another ZFS nfs vol). Then under linux (Oracle Linux 5.8) I ran fdisk on the 2tb virtual drive, creating a single LVM partition. Then I used LVM to create one volume group for the whole disk. Then I started making logical volumes. The first one was only 15GB -- no problem. The second was 300GB and when I did the lvcreate, all my ESXi virtual machines stopped responding (even pings did not work). Eventually (maybe 15-20 minutes later) everything came back. I noticed the load avg on my OI zfs host was 22! There must be some weird interaction of using LVM inside an ESXi virtual disk (that in turn is NFS-mounted from OI). I am now going to blow away the LVM stuff and the 2TB virtual disk and instead just NFS-mount file systems directly from OI and ZFS.

Has anyone here experienced similar problems with logical volume creation with large ESXi virtual disks that are NFS mounted from OI? Any other thoughts?

Thanks,
--peter
 
_Gea said:
Add users, this is easy and free
Go to menu users and add some users
Go to menu ZFS folder - acl extension add add permissions for these users

If anything goes wrong or if you accidentally remove all permissions, you must
know that root and the owner have always full access. You cannot lock them out.

Paid extensions
The paid extension do not block or hinder any OI functionality. You can all do
like you would do without knowing the extensions via CLI and shell-scripts.

They are comfort-functions. And they offer a lot of functionality for free.
The ACL extension (use newest) allows setting trivial ACL like everyone@ and user ACL
for free as well as resetting all ACL recursive to a modify for everyone.

The paid ACL extras are for Active Directory, Groups, complex or deny rules or a more detailed
view to file ACL, usually needed by larger organisations. I need these earnings to proceed development.
(There is no enterprise sponsor behind napp-it, all my own effort in my free time)

Thats the same with other extensions like async replication between appliances or the
SAS2 slot and monitoring tool or the pool encryption for OpenIndiana and ZFS V.28 pools with ZFS-2 safe
backup options for encrypted pools to a cloud provider or any external disk.


if you miss bonnie benchmarks on OI:
There was a bug with OI < 151.a3, where multiple package installs failed.
Rerun the napp-it wget installer or do su and then a
pkg install bonnieplus
Click to expand...


So the way I understand this is that you want to set the ACL at the ZFS folder level and NOT the SMB shares level? You leave the SMB level at 777 correct? Am I correct in assuming this is how it works best?
 
I though the opposite since it's easier to manage the permission from windows..

ALSO: I am trying to add OI to a domain however its failing with:
failed to find any domain controllers for XXXXXXXXXXXXXX

if I click on the check krb5.conf I get:
Could not proceed due to an error. Please try again later or ask your sysadmin.
Maybee a reboot after power-off may help.

(367 autocreate default action) write-error: /var/web-gui/data/napp-it/_my/zfsos/02_services/08_SMB/03_check krb5conf/action.pl
 
More info: Joining a 2003 AD works but not 2008R2..

Also when playing with local user and group I get Oops! when adding and removing a user, however the user seem to get added and removed properly:

Error on remove:

oops...
smbadm remove-member -m 'username' user
 
x-cimo said:
More info: Joining a 2003 AD works but not 2008R2..

Also when playing with local user and group I get Oops! when adding and removing a user, however the user seem to get added and removed properly:

Error on remove:

oops...
smbadm remove-member -m 'username' user
Click to expand...

what OS/OS release/ napp-it release are you using?

the problem with editing krb5.conf will be fixed in next release (problem with "." from krb5.conf in menu)
edit /etc/krb5/krb5.conf manually.

If you try to join a domain and get failed to find,
its mostly a problem with lmauth. Try lmauth=3 with W2008
 
Last edited:
mmmmmdonuts said:
So the way I understand this is that you want to set the ACL at the ZFS folder level and NOT the SMB shares level? You leave the SMB level at 777 correct? Am I correct in assuming this is how it works best?
Click to expand...

You can do either.
I would start with folder permission 777 on a shared folder and everyone@=full on a share itself (default value) and set ACL. Look only at ACL with SMB

look at "What you should know about ACL and SMB" in
http://napp-it.org/extensions/acl_en.html
 
Overall its been going pretty well. I am getting 100MB/s R/W off the SMB share, even for multiple small files.

One thing I noticed however, is if I start two or more transfer (WRITE TO ZFS STORE) the bandwidth goes away. For example, I start a single file transfer of many small and big files, if I start three other one they all go to around 2.5MB/s which is around 10-15MB/s total

Anyone have an idea how to sort that one out?

Im writing to a Raidz1 of three WD 2TB black 7200 rpm.

Edit: if i start one file write i get about 95MBps but if i juste start one other write the first one slow down fast and the second is stuck at 2.5MBps...
 
Last edited:
@ x-cimo

Try adding a ZIL write cache to your pool like a SSD. Did improve my upload speeds a lot to my server!
 
Ive tried to disable sync and it didint improve, something else i am seeing is if I start one large file transfer i get 90MB/s if I start another one it stated around 30MB/s and a third one it goes at 2.5MB/s over time they both get around 6MB/s even if the first transfer has finished, I've let both run for about 2 min, then killed one, and the last one remaining started to speed up really fast

System is a OI with ESXi5 and 4 vcore and 10GB of ram allocated to the vm (32 total)
 
Last edited:
x-cimo said:
Ive tried to disable sync and it didint improve, something else i am seeing is if I start one large file transfer i get 90MB/s if I start another one it stated around 30MB/s and a third one it goes at 2.5MB/s over time they both get around 6MB/s even if the first transfer has finished, I've let both run for about 2 min, then killed one, and the last one remaining started to speed up really fast

System is a OI with ESXi5 and 4 vcore and 10GB of ram allocated to the vm (32 total)
Click to expand...

I dont seem to have that issue. I have almost same setup as you mentioned (i have raidz2 and sync disabled) . image below is going from my win7 workstation (not a vm) to OI CIFS Share

http://i.imgur.com/BaKHC.png

Is yours also from a non-vm -> OI CIFS share? Might try upgrading/installing nic card drivers, I had lots of issues if i used the windows stock realtek drivers for my workstation machine
 
Last edited:
A round of Beers for Gea and OINappIt. Very nice stuff.

Now for a noob question.
nappIt docs, section 10.2, is this to be performd ONLY when the OS creates a new boot environment, or as part of the initial setup?

What harm comes if I re-run the installer instead of all the patching?
 
pitne said:
I dont seem to have that issue. I have almost same setup as you mentioned (i have raidz2 and sync disabled) . image below is going from my win7 workstation (not a vm) to OI CIFS Share

http://i.imgur.com/BaKHC.png

Is yours also from a non-vm -> OI CIFS share? Might try upgrading/installing nic card drivers, I had lots of issues if i used the windows stock realtek drivers for my workstation machine
Click to expand...

This is correct, I am testing from a bare metal win 2k8 server. not sure why its doing that yet..
 
Just wanted to say thanks to Gea and everyone's efforts. After much trial and error, with zero linux experience, I was able to get an all-in-one setup working on the following hardware:

Gigabyte GA-970A-UD3 AM3+ AMD 970
AMD FX-8120 Zambezi 3.1GHz
60GB SSD
3 Seagate 3TB in zfs1.
IBM M1015 HBA for Passthrough to the OI VM which is installed on the SSD of course. Already flashed to IT mode.
Storage will mostly be for movie files.

I had to update the motherboard's bios and put the M1015 in the pci-express slot closest to the CPU. I do not recommend this motherboard personally with all the troubleshooting that I ended up having to do originally I only wanted a system for ESXi..

Much to learn.
 
Yo,

Is it possible to shut down the server using only the servers' power button or do I need to login everytime into Napp-It to shut down?
Would be easier if one could use the power button!
 
siamsquare said:
Yo,

Sorry for repeating my question but is there any1 here who could help me install Squeezebox Server and SABnzb on my OI+Napp-It server. I allready found this :

http://forums.slimdevices.com/showthread.php?93300-Solaris-11-(NOT-Express)-Problem-with-UPnP-Plugin-and-Image-Browser

But as I'm a Linux noob and used working in windows it is a bit hard!
Squeezebox server and Sabnzb are running on my QNAP NAS but I would like to use my OI server only and sell the NAS.

Thanks
Click to expand...

This post should help quite a bit for the SABNZBD install...you're going to want to pull the latest version from sourceforge though of course. Github as S0urce mentioned is another option which keeps the version up to date easier.

http://hardforum.com/showpost.php?p=1037508313&postcount=1213
 
x-cimo said:
Well I still haven't figured out my multi smb connection slowing down.. Ive reinstalled OI update 3 still the same..

I wonder if 151u3 could be broken..

Here is an example:

http://imageshack.us/photo/my-images/577/netkw.png/


It show 2 transfer then a single transfer, then 2 transfer then back to one transfer...
Click to expand...

are you using vmxnet3 or e1000 on the openindiana VM? i dont have any issues with update3, using e1000 on the openindiana vm and vmxnet3 on the windows 2008r2 vm. havent tried the vmxnet3 on the openindiana vm though because ive heard people report issues with it and solaris
 
So i've installed the older IO 151a and the same happen..

I've deleted the pool and tried Mirror, raidz1/z2 and same happen..

Here's how I install, on ESXi5 update 1,

Create a VM on a Intel SSD X25-M 80Gb, give it 4 vcore, 12gb ram, add pci passthrough, add two E1000 nics, install OI, install vmware tool, install napp-it, create share (integrated smb), bench from a barebone Win7 host.

humm i'm running out of idea.

Wireshark idea is good however it end up crashing quickly due to the amount of data that i need to transfer for testing..

Edit: 1vcpu, no vmware tool, 1 e1000 nic, same results..
Edit2: Solaris 11 express does the same...
 
Last edited:
unclerunkle said:
This post should help quite a bit for the SABNZBD install...you're going to want to pull the latest version from sourceforge though of course. Github as S0urce mentioned is another option which keeps the version up to date easier.

http://hardforum.com/showpost.php?p=1037508313&postcount=1213
Click to expand...

Those instructions are for nexenta which has the debian build tools on OI/solaris its a bit different to install the dependencies. For example, AFAIK python-yenc is not available in any of the repos but it can be easily installed with pip or easy install, this goes for python-cheetah too.
 
i'm creating the share in ZFS folder create then enable smb share and guest (for testing) ?

I assume this is CIFS?
 
Anyone else are experiancing the same as I do ? (Major drop in write performance when writing two or more simultaneous file over SMB)

Here is my exact setup:

1X Supermicro AOC-USAS2-L8E
1X Supermicro SC743T-665B
1X Asus P9X79 (standard)
1X Intel i7 3930K
3X WD Black 2TB 7200RPM 64MB 3.5'' SATA6 HDD
1X CMP32GX3M4X1600C10
1X Intel PCIe Dual GigE port PT
1X Intel SSD X25-M 80Gb

Disabled Powersaving in bios
Install ESXi5 Update 1
Installed Solaris or OI with PCI passthrough, 4vcore, 12GB ram, E1000
Create raidz1 Pool, create Folder with SMB

Write 2 files from a Win7 Host

That's it.

Any idea what I could try?
 
x-cimo said:
Anyone else are experiancing the same as I do ? (Major drop in write performance when writing two or more simultaneous file over SMB)

Any idea what I could try?
Click to expand...

A speed drop with multiple copies is not typical for Solaris.
The problem may also be at the Win7 side (Network card, copy tools etc)
 
You must log in or register to reply here.
Back
Top