![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)
- Thread starter _Gea
- Start date
Hey Gea,
New question about Solaris 11.1 / napp-it
I just added a new ZFS folder. I set NFS sharing to true, the folder was not actually exported. I tried to change NFS to "off" in the ZFS Folder menus of Napp-it, but nothing changed.
If I share it manually using the shell "share" command it gets shared perfectly.
Did Oracle to something that breaks your scripts for NFS?
Thanks in advance for any help. I know you are not a real fan of Oracle's version of Solaris and I do appreciate any time you spend on it.
New question about Solaris 11.1 / napp-it
I just added a new ZFS folder. I set NFS sharing to true, the folder was not actually exported. I tried to change NFS to "off" in the ZFS Folder menus of Napp-it, but nothing changed.
If I share it manually using the shell "share" command it gets shared perfectly.
Did Oracle to something that breaks your scripts for NFS?
Thanks in advance for any help. I know you are not a real fan of Oracle's version of Solaris and I do appreciate any time you spend on it.
There are four problems with hardware raid and ZFS
when using single disks (as sort of a raid-0 from one disk)
1. extra layer of unneded raid software
2. special on disk format that may hinder a disk move to another controller
in general
3. not or badly supported driver (there are only a few LSI HBA that are perfect)
when using raid-functionality
4. in case of disk failure, your hardware raid is fine, in any other case (bit rot error, sector error, cabling or driver problem) ZFS detects the failure due to checksums but it cannot repair the error because the whole is a failed single disk for ZFS (no data self-healing an access or scrub)
Knowing that a perfect HBA like a IBM 1015 when flashed to LSI 9211-IT mode is at about 100 Euro/$, I would try to sell the 9750.
you may use the recursiv reset acl/permission with napp-it
menu zfs folder - acl extension - acl on folders -> reset acl (basic settings are free)
There are always minor changes that breaks compatibility like
OpenSolaris -> Solaris Express: new way of network setting
Solaris Express -> Solaris 11.11: new way to share, different naming of ipadm-ip vs former ipadm-if
Solaris 11.11 -> 11.1: different location and structure of pam.conf
So on every update you must check for broken things.
Currently i am working on a OmniOS port with napp-it 0.9 so I will check this next year.
If you like, you can look at the zfslib.pl (functions ask property and modify property) for the needed action.
redskilldough
n00b
- Joined
- Dec 10, 2012
- Messages
- 1
Hi, my first post here.
Just want to thank _Gea for saving me many hours, with with his napp it application.
Also, recently, I installed solaris 11.1 on an Asus P8B-X board, with the initial bios. The installation went fine. Then, I upgraded the bios, after that, it just wouldn't boot. I tried to reinstall with the usb stick (that's who I installed it in the first place), and it would just hang right after the initial Message (^MSunOS Release 5.11 Version 11.1 64-bit).
I googled it, there isn't much information on this, you just read here and there that some people are having problems like these with Asus boards. The closes thing I could find on the problem some people having success after disabling USB legacy support. Unfortunately, didn't work for me.
So, after hours of testing the bios setting, I found out that by disabling vt-d, everything worked fine after that.
So, to summarize,
Asus P8B-X, if it hangs when trying to install, disable vt-d.
I hope this helps someone.
Just want to thank _Gea for saving me many hours, with with his napp it application.
Also, recently, I installed solaris 11.1 on an Asus P8B-X board, with the initial bios. The installation went fine. Then, I upgraded the bios, after that, it just wouldn't boot. I tried to reinstall with the usb stick (that's who I installed it in the first place), and it would just hang right after the initial Message (^MSunOS Release 5.11 Version 11.1 64-bit).
I googled it, there isn't much information on this, you just read here and there that some people are having problems like these with Asus boards. The closes thing I could find on the problem some people having success after disabling USB legacy support. Unfortunately, didn't work for me.
So, after hours of testing the bios setting, I found out that by disabling vt-d, everything worked fine after that.
So, to summarize,
Asus P8B-X, if it hangs when trying to install, disable vt-d.
I hope this helps someone.
That doesn't sound right. It also seems like it will take away a lot of Solaris' built-in virtualization capabilities.
Not just VBox, but the Virtualized file system that is ZFS, Crossbow and NWAM, Zones, OVM templates, and quite a few other things that we never see in userland.
Is it possible to boot the board with a Solaris 11.1 Live DVD or USB? Do you have EHCI or "Plug and Play OS" enabled? Can you change any boot agent settings?
Not just VBox, but the Virtualized file system that is ZFS, Crossbow and NWAM, Zones, OVM templates, and quite a few other things that we never see in userland.
Is it possible to boot the board with a Solaris 11.1 Live DVD or USB? Do you have EHCI or "Plug and Play OS" enabled? Can you change any boot agent settings?
napp-it 0.9 developer preview1
A first developer preview of napp-it 0.9 is available
There should not be serious problems in it but I hope for infos what's not working
(its a complete redo of a lot of programmparts so I expect some problems especially with disk-detection and smart on some configs)
Install:
wget -O - www.napp-it.org/nappit09 | perl
followed by a reboot and a resetting of root password
You can go back to 0.8 with system snaps (boot environments)
on problems, send a short buginfo to
community@napp-it.org
A first developer preview of napp-it 0.9 is available
There should not be serious problems in it but I hope for infos what's not working
(its a complete redo of a lot of programmparts so I expect some problems especially with disk-detection and smart on some configs)
Install:
wget -O - www.napp-it.org/nappit09 | perl
followed by a reboot and a resetting of root password
You can go back to 0.8 with system snaps (boot environments)
on problems, send a short buginfo to
community@napp-it.org
Code:
changelog:
0.9 pre1
improved disk detection (hot plug)
improved performance for disk details and smart infos
performance improved smartdetection
smart: start short/long checks
support for disk partitions
disk detection with disks like c3d1 fixed
UNAVAIL disk replace with GUID disk not working
jobmanagement: edit timetable and parameter, code and logs optimized, all actions in separate files
monitor extension:
topmenu edit = display cli commands, return values and hashvalues without reload
log actions, realtime mini log, realtime page update
disable monitor: topmenu "Mon" for best performance
replication extension: code and logs optimized, speed improvements with buffered transfer
acl extension: new feature: reset ACL recursively to current folder settings, bugfix with resetting files
manual network settings: bugfix
Solaris 11.1: NFS sharing, and pam settings fixed
OmniOS: supported
NexentaCore: no longer supportedHaving some lockup issues with my oi151a5 box with napp-it (other OSes installed on the same hardware show no issue - see thread http://hardforum.com/showthread.php?t=1733419). Someone there suggested that they have had similar issue but they went away after updating to the latest.
What do you guys think? Is the latest development version (oi151a7, I think...) more or less stable than oi151a5?
What do you guys think? Is the latest development version (oi151a7, I think...) more or less stable than oi151a5?
One possible suggestion I have after rebuilding my virtualbox test environment:
I tried to add 2 drives for l2arc cache and I expected the UI to stripe those behind the scenes. I got an error that only 1 disk can be used which was a little confusing. I would either change the message to state you must add one disk at a time or (ideally) determine if multiple disks were selected and then stripe them in the background process. I think option 2 would make a little more sense and be a bit more user friendly.
The UI seems to be much more responsive but that could be due to seeing the command window at the bottom. That's an awesome feature by the way because it gives me the chance to see a lot more of the background command set so I can get a little bit more ZFS knowledge here and there.
The disk partitioning is working well. I re-sliced my simulated ZIL and L2ARC disks and the process was intuitive and easy to follow. I really like it and I was able to do 100% of my test config in the UI now where before I felt like half my time was in the CLI for things.
I tried to add 2 drives for l2arc cache and I expected the UI to stripe those behind the scenes. I got an error that only 1 disk can be used which was a little confusing. I would either change the message to state you must add one disk at a time or (ideally) determine if multiple disks were selected and then stripe them in the background process. I think option 2 would make a little more sense and be a bit more user friendly.
The UI seems to be much more responsive but that could be due to seeing the command window at the bottom. That's an awesome feature by the way because it gives me the chance to see a lot more of the background command set so I can get a little bit more ZFS knowledge here and there.
The disk partitioning is working well. I re-sliced my simulated ZIL and L2ARC disks and the process was intuitive and easy to follow. I really like it and I was able to do 100% of my test config in the UI now where before I felt like half my time was in the CLI for things.
You have two options:
1. create a primary Solaris partition and slice this partition
where slice 2 (s2) is a synonym of the whole disk
2. and/or create up to four partitions where partition 0 (p0) is a synonym of the whole disk
napp-it uses the second way because its more flexible and easier to handle
You can partition your SSD with menu disk-partition (1.. 4 partitions) and use them
(p1..p4) like regular disks.
Some systems (have it myself with Solaris 11.1) have problems with discovering partitions with the comand line tool parted that is used by napp-it. In this case you currently cannot create partitions with napp-it. (may create and check manually)
I believe Gea in the past has recommended the E1000 adapter.
I did ask some questions about multiple adapters in the past and it seems that you can go with 1 adapter unless you are running across different networks. My case was setting storage on an isolated network. If you go with the all-in-one solution you do get some advantages to networking by setting VMWare to use multiple ports in an active/failover scenario.
I did ask some questions about multiple adapters in the past and it seems that you can go with 1 adapter unless you are running across different networks. My case was setting storage on an isolated network. If you go with the all-in-one solution you do get some advantages to networking by setting VMWare to use multiple ports in an active/failover scenario.
E1000 is rock solid and stable for years.
VMXNET 3 is newer and can offer much better performance with a lower CPU load
- but can have problems on some configs
Can you provide more info on how you resolved the cabling? I am stuck with a R720xd with an H310. Dell has acknowledged that it has a JBOD performance problem that they will not be able to correct in firmware so I am looking to do like you did and use an LSI 9211-8i card in IT mode. You can see me post in their forums with more details (http://en.community.dell.com/support-forums/servers/f/906/t/19480834.aspx) on the issues I ran into.
I have not looked inside yet, but were you able to connect the add-on drives in the back directly to an LSI card? Dell made it sound like it was proprietary. Would a 9211-4i do for those?
thanks.
can you expand on this problem?
Start a dd to an SSD and iostat shows about 400MB/s. Start a second dd to a spinning disk and the SSD drops to 140MB/s just like the spinning disk.
This makes the R720xd a bad machine to do ZFS on if you plan on using ZIL and L2ARC. The PERC H310 is the only one of their internal SAS adapters that supports JBOD. With the others you need to set them up as single disk RAID 0 which won't let SMART go through.
I also had a problem with my max throughput on the spinning disks limited to about 540MB/s, i.e. about 4 disks. As I increased the number of disks I was dding to the average throughput per disk dropped such that the total throughput was about 540MB/s. However this may have been a contributed to by the CPU (single E5-2409).
To make things worse Dell just told me they won't be fixing the H310. I guess JBOD is not a common enough setup for them to care about.
You got it.
The R720xd is a dual CPU 2U machine supporting 12x3.5" or 24x2.5" drives in the front plus an additional 2x2.5" in the back so at a glance seemed like a perfect self contained ZFS box. Not so much.
.The key component is the ESXi virtual network switch.
Your VMs (storage or others) only need one virtual nic that is connected to that switch. Even if you use the e1000 ESXi virtual nic, you can have several Gbit/s because its software not a physical cabling. With the vmxnet3 ESXi driver, you can have about twice the performance. To separate networks, you can use vlans to your network switch.
Within ESXi you can setup link aggregation over your physical nics but must aware that the other side of the connection (switch or server) must have the same settings.
Other option is to use one physical nic for one server or ip connection and avoid link aggregation.
Depending on settings, a single transfer will be not faster with link aggregation, only concurent connections. Mostly you should think about 10Gb when 1 Gbit is not fast enough. This gives you the performance without tweaking and problems.
Last edited:
I found this link very interesting http://wahlnetwork.com/2012/04/19/nfs-on-vsphere-a-few-misconceptions/
I thought I could trick ESX into load balancing NFS with multiple NICs bound to a port group, but it doesn't seem that way. The whole series of posts at the above link made for some good reading on ESX network configurations and NFS.
I thought I could trick ESX into load balancing NFS with multiple NICs bound to a port group, but it doesn't seem that way. The whole series of posts at the above link made for some good reading on ESX network configurations and NFS.
I have finally instaled my zfs setup (allione oi+nappit) and was wondering, how to check if the pools on the mirror SSDs and 10x 3TB RED raid2z were created with ashift=12? I didn't use the "force" option.
edit: nvm, found it. raid2z was created with ashift=12 but the two ssd's were not.
edit: nvm, found it. raid2z was created with ashift=12 but the two ssd's were not.
Last edited:
Hi!
I need ZFS help. Currently I have 3x2TB WD green in RAID5 (integrated mobo controller). Now, I'm going the freeBSD route, with 6x2TB WD green RAIDZ2.
My mobo only has 4xSATAII so I'm looking for a freeBSD compatible SATA expander PCI card. What are my options?
How is ZFS on freeBSD?
I need ZFS help. Currently I have 3x2TB WD green in RAID5 (integrated mobo controller). Now, I'm going the freeBSD route, with 6x2TB WD green RAIDZ2.
My mobo only has 4xSATAII so I'm looking for a freeBSD compatible SATA expander PCI card. What are my options?
How is ZFS on freeBSD?
If I'm setting up two vSwitches,1 for vMotion and 1 for everything else, I want to setup LACP on the physical switch correct? 4 ports for the Server with quad nic(contains OI VM, and other vmotion'd VMs), and 2 ports for the servers with dual Nics(which just run vMotion'd VMs)
Yes, for Link aggregation, you need the same settings in ESXi and on your switch. But i skipped this completely for 10G. On problems, you may ask at http://hardforum.com/forumdisplay.php?f=122
napp-it 0.9 developer preview now updated
with massive speed improvements in the GUI and in replication, partitions, better hotplug support, realtime log and stat and many other improvements
http://www.napp-it.org/downloads/changelog_en.html
if you use OmniOS stable, you should re-run the installer
http://www.napp-it.org/downloads/omnios_en.html
especially OmniOS stable is something that is really of interest and supported from napp-it 0.9+
I expect OmniOS to be one of the most stable and up to date Illumos based ZFS storage servers in future
- either free or with commercial support
with massive speed improvements in the GUI and in replication, partitions, better hotplug support, realtime log and stat and many other improvements
http://www.napp-it.org/downloads/changelog_en.html
if you use OmniOS stable, you should re-run the installer
http://www.napp-it.org/downloads/omnios_en.html
especially OmniOS stable is something that is really of interest and supported from napp-it 0.9+
I expect OmniOS to be one of the most stable and up to date Illumos based ZFS storage servers in future
- either free or with commercial support
With an update you keep
NFS and CIFS settings (they are ZFS related)
installed services like netatalk
You must check or recreate jobs
in 0.9 jobs are managed by separate scripts while in napp-it < 0.9 actions were coded in jobs
but now you can edit job parameters and bugfixes affects older jobs
not working:
netatalk < v3
netatalk3 on Solaris 11.1 at all
you may need to update netatalk2 and reshare the folders
the good thing
you may go back to the last BE, but 0.9 ist already better than any previous version
in general
This is a major update, you can do only via wget, you can go back only via BE
You should do some test prior production use (remember its a preview - but mostly ready now)
I invite all to try the new version and to help fixing remaining bugs
napp-it 0.9 will be the default version begin of 2013 - mainly supports OmniOS and OI but mostly running on Nexenta Ilumian (base of Nexentastor4) and Oracle Solaris 11.1
Last edited:
Gea, if you were starting a new config would you go with OmniOS (not sure how close to ready it is) or stick with OI? I've just downloaded OmniOS and I am going to give it a try but I wanted to see if you had a good feel for the overall stability.
you may answer it yourself
- both rely on Ilumos
- OI is a general purpose OS incl desktop use while Omni is a minimal server OS focussed on NAS/SAN
- Omni sells commercial support and the whole company rely on a stable OS
- Omni is much closer to the Illumos development than OI
- Omni has a stable release and offers betas every few weeks
Yes I have a good feeling with OmniOS for a NAS or SAN but i hope the general
approach of OI as a general purpose OS has a future - but that may need more developers
or a major enterprise behind that sponsors to speed up development.
All of my servers are currently OI but for newer setups I will move to/try OmniOS
and thats the reason napp-it 0.9 supports Omni
I may have come across an issue with the dev version. I have partitioned a drive into 25% slices (10GB virtualbox drive) to simulate splitting out my ZIL SSDs. I have 2 of the drives mirrored for ZIL and I tried to replace one partition with a remaining partition and each attempt to replace c3t2d0p1 with any of the remaining slices from either 10GB disk results in an error that the drives are not the same size.
The actual error is:
Could not proceed due to an error. Please try again later or ask your sysadmin.
Maybee a reboot after power-off may help.
cannot replace c3t2d0p1 with c3t2d0p3: device is too small
It looks like the actual partitions are not the same size as the mini log shows the partitions as:
1:2097kB:2147MB:2145MB:::;
2:2147MB:4209MB:2062MB:::;
3:4209MB:6270MB:2062MB:::;
4:6270MB:8332MB:2062MB:::boot;
In this case I would not be able to replace my ZIL partitions unless I repartition the disk into unequal sizes such as 26/26/26/22 which would at least let me sliver 3 ZIL partitions and then keep some free space at the end for wear.
Is there a better way to handle this scenario?
The actual error is:
Could not proceed due to an error. Please try again later or ask your sysadmin.
Maybee a reboot after power-off may help.
cannot replace c3t2d0p1 with c3t2d0p3: device is too small
It looks like the actual partitions are not the same size as the mini log shows the partitions as:
1:2097kB:2147MB:2145MB:::;
2:2147MB:4209MB:2062MB:::;
3:4209MB:6270MB:2062MB:::;
4:6270MB:8332MB:2062MB:::boot;
In this case I would not be able to replace my ZIL partitions unless I repartition the disk into unequal sizes such as 26/26/26/22 which would at least let me sliver 3 ZIL partitions and then keep some free space at the end for wear.
Is there a better way to handle this scenario?
You can always replace a disk with another that is larger so replacing 3 with 1 is possible. Replacing 1 with 3 is not possible because 3 is smaller than 1.
So its only a matter of using the smaller partitions first and add mirror/ replace larger ones as it is with regular disks. The basic problem is, that you are partitioning based on percent values partition by partition. Using the exact value gives you the problem that mostly some bytes are missing for the last partition.
I may check if its possible, to set 4 x 24% where the last partion is also 24%.
Now the last one takes the remaining space.
I know its a lot to ask, but can you do a tutorial how to setup all-in-one with OmniOS & napp-it like the current one for Open Indiana? I just built a box on esxi&OI but would love to make it future proof and since i don't need fully desktop OS like OI, OmniOS sounds perfect.
I agree with what you are saying, but from within the interface there is no way to tell that one partition was/is larger than another. They all read 2.1GB in size so I had not other indication of the discrepancy.
Maybe another option on the partition screen would be to input the 4 partition sizes and then just leave the remaining space unpartitioned?