OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

All this discussion of sd.conf and ashift=12 piqued my interest. We have 6 WDC WD10JPVT-00A 1TB drives attached to an LSI 9211 (ESXi 5 passthrough to OI which is used as a ZFS SAN for other virtual hosts). The WD10JPVT-00A are advanced format 4k drives.

zfs list:
NAME USED AVAIL REFER MOUNTPOINT
rpool 13.4G 10.1G 46.5K /rpool
rpool/ROOT 5.07G 10.1G 31K legacy
rpool/ROOT/napp-it-0.8e_update_04.13 5.05G 10.1G 3.43G /
rpool/ROOT/openindiana 14.2M 10.1G 3.24G /
rpool/ROOT/openindiana-1 12.6M 10.1G 3.26G /
rpool/ROOT/openindiana-1-backup-1 133K 10.1G 3.01G /
rpool/ROOT/pre_napp-it-0.8e 100K 10.1G 3.00G /
rpool/dump 4.00G 10.1G 4.00G -
rpool/export 42.1M 10.1G 32K /export
rpool/export/home 42.1M 10.1G 32K /export/home
rpool/export/home/pag 42.0M 10.1G 42.0M /export/home/pag
rpool/swap 4.25G 14.3G 137M -
san 1.14T 1.62T 152K /san
san/datastore2 1.06T 1.53T 184K /san/datastore2
san/datastore2/dr-firewall 9.91G 1.53T 9.91G /san/datastore2/dr-firewall
san/datastore2/oscar 19.2G 1.53T 19.2G /san/datastore2/oscar
san/datastore2/oscar_disk2 1.03T 1.53T 1.03T /san/datastore2/oscar_disk2


Is there any way to convert our "san" pool to use ashift=12 in place without rebooting? Failing that, what would be the recommended procedure to do the conversion?

Thanks in advance,
--peter
 
Here is a pastebin with some of the dmesg log (http://pastebin.ca/2170797), it looks like megasas spawns off a ton of commands that trigger modprobes that just hang until they get killed, then it spawns some more.. or something similar anyway. Also here is a shot of the optionrom: http://i.imgur.com/sWtTT.png

my finding:
you have IRQ problems, check your motherboard BIOS configuration

---->megaraid_sas 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17 <---
the IRQ numner that assigned to your card.


---->[ 193.767515] scsi 6:0:0:0: megasas: RESET cmd=12 retries=0
[ 193.767519] megasas: [ 0]waiting for 1 commands to complete <---
the megasas kernel module sent commands but got timeout aka no response from your card

check you BIOS to verify that no-shared IRQ
Check you pci-ex slot to make sure that the card is not behind pci-experess multiplexer or switch ( some motherboardBIOS does not handle switch as supposed to..)

your card will happly when it has direct pci-express connection and non-shared IRQ :D.
 
OI 151a (CLI only) user here.

Can I run virtualisation straight on top of OI? Basically need a Windows machine that will act as a media server, and don't want a seperate box pointing to OI via iSCSI. Guessing I will need to carve off some space on the data pool for the VM guest(s), leaving the rest to be passed through to such guests.

This doesn't sound too promsing for an AMD user http://wiki.openindiana.org/oi/KVM!

Coming back to this - with ESXi I guess I would install this first, then virtualise OI, create pools and then pass these through to the other VMs? I'm familiar with the whole virt/passthrough stuff as I do this with Hyper-V day-to-day. In fact, could I swap ESXi for the free incarnation of Hyper-V Server 2008 R2?

If so, what are the recommendation re disks. At the moment my vanilla OI build is 2x SAS 72K disks for mirrored rpool, 14 x 2TB disks for my data pool (2x 6 disk raid-z2 with 2 hot spares).
 
All this discussion of sd.conf and ashift=12 piqued my interest. We have 6 WDC WD10JPVT-00A 1TB drives attached to an LSI 9211 (ESXi 5 passthrough to OI which is used as a ZFS SAN for other virtual hosts). The WD10JPVT-00A are advanced format 4k drives.
..

Is there any way to convert our "san" pool to use ashift=12 in place without rebooting? Failing that, what would be the recommended procedure to do the conversion?

Thanks in advance,
--peter

You can set ashift only during creation of a pool or vdev.
You must create a new ashift=12 pool and copy files over.
 
set all pool reservations to none to max out capacity for datasets
(beside the ZFS internal base reservation)

whats the internal base reservation? I thought there was that option to run only to 90% full. I don't want to get too close to being full.

But even with that I don't see why it goes from 7.88T => 7.2T => 6.4T for my folder. Sorry for being newb :(
 
Depends on if you want gui stuff like timeslider. Some do, I don't - just installed the text image...
 
whats the internal base reservation? I thought there was that option to run only to 90% full. I don't want to get too close to being full.

But even with that I don't see why it goes from 7.88T => 7.2T => 6.4T for my folder. Sorry for being newb :(

- have you read about the base reservation to avoid data-corruption on a full disk?
http://cuddletech.com/blog/pivot/entry.php?id=1013

- have you set the pool reservations to none (pool reservations avoids speed problems on nearly full pools)
to see the capacity changes for datasets?
 
Im having issues adding multiple (4, for ssd L2cache) sas interposers (LSI SS9252) to a openindiana box,
The hardware Im using is a Supermicro 847E26 chassis (LSI SAS2X28 expander chips on the backplanes) with a Supermicro X8DTH-6F motherboard (LSI 2008 onboard) and 1x LSI SAS 9211-8i.
The WWN for all interposers is g5000000000000000 and so OI thinks all 4 SSDs are 1 disk just with different paths.

here is an example of plugging 2 of the disks:
Code:
Jul 13 13:31:29 san01 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g5000000000000000 (sd28) multipath status: optimal: path 53 mpt_sas9/disk@w500605ba00406f06,0 is online: Load balancing: logical-block, region-size: 18
Jul 13 13:31:29 san01 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g5000000000000000 (sd28) multipath status: optimal: path 54 mpt_sas2/disk@w500605ba00406f05,0 is online: Load balancing: logical-block, region-size: 18

Jul 19 09:59:46 san01 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g5000000000000000 (sd28) multipath status: optimal: path 55 mpt_sas2/disk@w500605ba004083a1,0 is online: Load balancing: logical-block, region-size: 18
Jul 19 09:59:46 san01 genunix: [ID 530209 kern.info] /scsi_vhci/disk@g5000000000000000 (sd28) multipath status: optimal: path 56 mpt_sas9/disk@w500605ba004083a2,0 is online: Load balancing: logical-block, region-size: 18

Anyone ever seen this behavior or knows how to tell the scsi_vhci module that these are in fact different disks? I have already tried different SSD (currently using Plextors, but tried OCZ) and also regular 2,5" drives, they all behave the same.
 
How do I find out if I've install OI in server mode (with desktop) or workstation mode ?
When I install OI in server/text mode, I can see commands appearing at the shell on I use the Web-Gui, but I cannot see that currently...
 
How do I find out if I've install OI in server mode (with desktop) or workstation mode ?
When I install OI in server/text mode, I can see commands appearing at the shell on I use the Web-Gui, but I cannot see that currently...

In contrast to Windows, where the desktop edition Win7 has a reduced functionality
compared to the server edition Win 2008 server, the base functionality in OI text edition
(server??) is the same as in live/full/desktop edition.

The live editions adds a graphical user interface with filebrowser, systemmanagement,
firefox and thunderbird to the minimal textbased server install.

If you boot OI and you end in a console mode, you have a "server" install,
if you get the OI desktop with mouse handling etc, you have the "desktop" version.

Due to much better usability and timeslider, i always use "desktop" edition for my servers
but that is a point, you must decide yourself.
 
what/where is the correct documentation to look at for command line network interface setup for OI_151a and OI_151a4 ?
 
Can you check spin down timer of drives with nappit?

Or do you have use hd manufacturer tool?

Brought some hd204ui/vp4 have spin down issue will this effect the array

Have 4 original hd204ui and 6 hd204ui/vp4 in zraid2
 
In contrast to Windows, where the desktop edition Win7 has a reduced functionality
compared to the server edition Win 2008 server, the base functionality in OI text edition
(server??) is the same as in live/full/desktop edition.

The live editions adds a graphical user interface with filebrowser, systemmanagement,
firefox and thunderbird to the minimal textbased server install.

If you boot OI and you end in a console mode, you have a "server" install,
if you get the OI desktop with mouse handling etc, you have the "desktop" version.

Due to much better usability and timeslider, i always use "desktop" edition for my servers
but that is a point, you must decide yourself.

The challenge is that I want to have a text base for 1 reason, when I use the text base and when I use the NAPP-IT web gui, I can see the commands sent by NAPP-IT and that helps me.. so that's the MAIN reason and if I can get that back, I'll be real happy !!
 
The challenge is that I want to have a text base for 1 reason, when I use the text base and when I use the NAPP-IT web gui, I can see the commands sent by NAPP-IT and that helps me.. so that's the MAIN reason and if I can get that back, I'll be real happy !!

Hmmm...the GUI is an extra...you still can use the CLI...open a terminal or enable ssh and connect...voilá
 
The challenge is that I want to have a text base for 1 reason, when I use the text base and when I use the NAPP-IT web gui, I can see the commands sent by NAPP-IT and that helps me.. so that's the MAIN reason and if I can get that back, I'll be real happy !!

if you need console messages

using live version with gui:
- open a (remote putty) console (locally use su to get superuser permission)
- restart napp-it: /etc/init.d/napp-it restart

napp-it commands wlll be shown in this console
 
what/where is the correct documentation to look at for command line network interface setup for OI_151a and OI_151a4 ?

best is to use basically the Oracle docs likre
http://docs.oracle.com/cd/E19963-01/html/821-1458/gjhbt.html

and check where OI is different with the help of the OI wiki: please add insights! - its a community effort-
http://wiki.openindiana.org/oi/Static+IP

or check napp-it actions on different systems

or with the help of Google to discover the rest

not to forget: try and error
(OI, NexentaCore, Illumian and Solaris 11 works slightly different )
 
So I'm setting up Openindiana in esxi 5

if I let openindiana get dhcp, it can get interent access but if i use static even with the same IP, it gets local assess but no interent?

please point me into the right direction
 
OI and company are a pain in the ass to set up for static IP. It can be done but it's a drag. If possible, I'd just have your DHCP server assign him a 'static' dhcp address.
 
OI and company are a pain in the ass to set up for static IP. It can be done but it's a drag. If possible, I'd just have your DHCP server assign him a 'static' dhcp address.

that's what I end up doing, use the mac address provided by the esxi for the VM and have my router set the static IP for it

When I install nappit it creates another OI ? when I boot up my VM OI I know see two boot options?
 
I just got finished setting up static IPs (and aggregates!) on two IO machines (see my post about 3 posts back). Just follow the OI wiki. One thing that didn't seem to work like it says is the edit /etc/defaultrouter part. I had to use route -p add default. Make sure the disable nwam or it acts really weird.



Anyways, I have two almost identical NASs and if one has a bonnie bench Seq-read of 150MB/s and the other has a Seq-write of 130MB/s, shouldn't I be able to get ~100MB/s on gigabit Ethernet between them (I even aggregated two nics on each for fun, so...)? Best I can get over smb or nfs is peak 50MB/s.
 
if you need console messages

using live version with gui:
- open a (remote putty) console (locally use su to get superuser permission)
- restart napp-it: /etc/init.d/napp-it restart

napp-it commands wlll be shown in this console

THANKS !! Will you be starting a FAQ or Wiki type of FAQ so that answers don't have to be repeated. While your quickstart is great, I find that there are things that we need that are already there in your GUI but we are just not finding it.
 
OI and company are a pain in the ass to set up for static IP. It can be done but it's a drag. If possible, I'd just have your DHCP server assign him a 'static' dhcp address.

If you do it without using nwam it's difficult - if you leave nwam on and use it to set up a basic static IP it isn't too bad, e.g.: link for how-to
 
In the napp-it Web-UI (menu system-network) i do setup of a manual ip in OI this way:
(with some extra settings sometimes needed)

svcadm disable network/physical:nwam
svcadm disable network/physical:upgrade
svcadm enable network/physical:default

#down
ifconfig e1000g0 down

#delete old
ipadm delete-if e1000g0

#create new
ipadm create-if e1000g0

#use dns
cp /etc/nsswitch.dns /etc/nsswitch.conf

#create static adress
#netmask=24 on a class c or 16 on class b network
ipadm create-addr -T static -a ip/netmask e1000g0/v4


#update gateway ex gateway=192.168.1.254
echo gateway | tee /etc/defaultrouter

#update route
route flush
route -p add default gateway

#opt restart and interface up
svcadm restart network/physical:default
ifconfig e1000g0 up


#step 2: update /etc/resolf.conf with DNS entries (8.8.8.8 = open Google DNS)
domain mydomain.com
nameserver 8.8.8.8



-------------------------------------------------------------------------------------------
you can check different settings for illumian and S11 in
/var/web-gui/data/napp-it/zfsos/_lib/systemlib.pl
 
Last edited:
@ _Gea,

Still struggling with TLS not working on my system...
invalid SSL_version specified at /usr/perl5/site_perl/5.12/IO/Socket/SSL.pm line 308

For help, please send mail to this site's webmaster, giving this error message and the time and date of the error.
[Sat Jul 21 11:57:05 2012] admin.pl: invalid SSL_version specified at /usr/perl5/site_perl/5.12/IO/Socket/SSL.pm line 308

Since I installed initially the wrong version, I have the suspicion that the TLS module from cpan maybe tries to lookup the wrong version. Is it possible to uninstall the TLS module? How? I tried 'clean Net::SMTP::TLS' but it responded that there's no tarred or zipped module found and hence does nothing.

BTW this is line 308 in above script:
${*$self}{'_SSL_ctx'} = IO::Socket::SSL::SSL_Context->new($arg_hash) || return;

Thanks,

Cap'
 
If you do it without using nwam it's difficult - if you leave nwam on and use it to set up a basic static IP it isn't too bad, e.g.: link for how-to

yeah, i've done that too. one huge gotcha: you cannot use nwam with more than one NIC :( it really annoys me about OI that for something so useful for servers, there is not an out of the box config for static IP that doesn't involved jumping through hoops...
 
Hi all,

I use open Indiana with napp-it as a file server on my home network. I have configured one pool with a few samba shares and can easily access them from windows devices. Unfortunately, this is not the case with android. I can see the shares but can not access them. No matter what app I use, it complains about the rights. I suppose it has to do with the assignment of the samba and the UNIX rights.

I m trying to solve the problem via napp-it.

"member of smb group" and "idmap" are empty. This is not OK, right? How can I change it?

After I have removed some users I get error messages:
"failed to find an error occurred while retrieving data group"
"failed to find the check system log for more information"

How can I configure Users, Groups, and their rights?
Why has the napp-it web interface login only as "admin" and "operator"?
This does not correspond with those users of open indiana?
How can I modify it?

I've checked in the settings of open indiana (Administration / Users and Groups). By the properties of user "napp-it" is the following:
main group: other
shell: / bin / false
homeDirectory: / home / napp-it
user id: 102

If I select "root" instead of "other" I get an error message "can not create a home directory on an automounted file system"
As far as I know napp-it must be either admin or root. What can I set these parameters?

Plz help ...

Thanks
 
How do we do static IP in the case of aggregate lans?

In the napp-it Web-UI (menu system-network) i do setup of a manual ip in OI this way:
(with some extra settings sometimes needed)

svcadm disable network/physical:nwam
svcadm disable network/physical:upgrade
svcadm enable network/physical:default

#down
ifconfig e1000g0 down

#delete old
ipadm delete-if e1000g0

#create new
ipadm create-if e1000g0

#use dns
cp /etc/nsswitch.dns /etc/nsswitch.conf

#create static adress
#netmask=24 on a class c or 16 on class b network
ipadm create-addr -T static -a ip/netmask e1000g0/v4


#update gateway ex gateway=192.168.1.254
echo gateway | tee /etc/defaultrouter

#update route
route flush
route -p add default gateway

#opt restart and interface up
svcadm restart network/physical:default
ifconfig e1000g0 up


#step 2: update /etc/resolf.conf with DNS entries (8.8.8.8 = open Google DNS)
domain mydomain.com
nameserver 8.8.8.8



-------------------------------------------------------------------------------------------
you can check different settings for illumian and S11 in
/var/web-gui/data/napp-it/zfsos/_lib/systemlib.pl
 
can I aggregate 1 physical nic passthrough and 1 vmxnet3 from esxi? or it has to be 2 physical nics
 
can ESXI 5 report SMART Info from drives?

my situation

xeno 1260L
s1200kp (itx that does NOT support VT-D)
lian li Q25

currently I just make large VMDK and pass them to openindiana ZFS

option 1 just use the whole server as ZFS (openindiana) is it a bit overkill?
option 2 use VMDK and pass them to openindiana, performance is good just looks messy when maintaining the system
option 3 find another cpu/motherboard for ZFS, keep my s1200kp for my lab only (dont really want to spend quite a bit already) on 16 gb ECC, case ETC

hard to get itx + vt-d + ECC
 
I think you can see smart stuff if you pass drives thru using RDM, but I believe that is not supported. If you pass thru the controller it all just works.
 
Ugh, for some reason Domain\Administrator doesn't work. I finally noticed the User "NAS" that I had put in AD a long time ago and used it to join at of course it worked the first try.

Code:
Jul 22 23:44:12 solaris smbd[3605]: [ID 504979 daemon.notice] ldap_add: Operations error
Jul 22 23:44:12 solaris smbd[3605]: [ID 702911 daemon.notice] Failed to create the workstation trust account.
Jul 22 23:44:12 solaris smbd[3605]: [ID 871254 daemon.error] smbd: failed joining xxxxx.xxxxx(UNSUCCESSFUL)

This was the same hassle I had. Used a different member of the Windows 'Domain Admins' group and it joined right up.
 
Is there a problem today to download & install Napp-IT ?

Code:
stef@ZFS:~$ wget -O - www.napp-it.org | perl
--2012-07-24 15:48:01--  http://www.napp-it.org/
Résolution de www.napp-it.org (www.napp-it.org)... 80.67.28.3, 2a00:1158:0:300:9a63::1
Connexion vers www.napp-it.org (www.napp-it.org)|80.67.28.3|:80...connecté.
requête HTTP transmise, en attente de la réponse...200 OK
Longueur: 15829 (15K) [text/html]
Sauvegarde en : «STDOUT»

 0% [                                       ] 0           --.-K/s              Bareword found where operator expected at - line 11, near "<title>napp"
	(Missing operator before napp?)
String found where operator expected at - line 13, near "<meta name=""
  (Might be a runaway multi-line "" string starting on line 12)
	(Missing semicolon on previous line?)
Bareword found where operator expected at - line 13, near "<meta name="description"
	(Missing operator before description?)
String found where operator expected at - line 13, near "description" content=""
Bareword found where operator expected at - line 13, near "" content="napp"
	(Missing operator before napp?)
syntax error at - line 2, near "html>"
syntax error at - line 13, near "<meta name=""
Unrecognized character \xE4 in column 125 at - line 13.
25% [=========>                             ] 4095       --.-K/s   ds 0,05s

:rolleyes:

Cheers.

St3F
 
you missed the installer, try
Code:
wget -O - www.napp-it.org/nappit  | perl
.. and This happened :

Code:
stef@ZFS:~$ wget -O - www.napp-it.org/nappit  | perl
--2012-07-24 16:50:34--  http://www.napp-it.org/nappit
Resolution de www.napp-it.org (www.napp-it.org)... 80.67.28.3, 2a00:1158:0:300:9a63::1
Connexion vers www.napp-it.org (www.napp-it.org)|80.67.28.3|:80...connecté.
requête HTTP transmise, en attente de la réponse...200 OK
Longueur: 21919 (21K) [text/plain]
Sauvegarde en : «STDOUT»

100%[======================================>] 21919       134K/s   ds 0,2s    

2012-07-24 16:50:35 (134 KB/s) - envoi vers sortie standard [21919/21919]

stef@ZFS:~$
Then, following the installation guide (not the all in one), I reboot and after login in OI, I try to connect to http://192.168.2.119:81 from another computer.
.... and error to connect to http://192.168.2.119:81 :/

What am I missing / doing wrong ?

Cheers.

St3F
 
Last edited:
.. and This happened :

Code:
stef@ZFS:~$ wget -O - www.napp-it.org/nappit  | perl
--2012-07-24 16:50:34--  http://www.napp-it.org/nappit
Resolution de www.napp-it.org (www.napp-it.org)... 80.67.28.3, 2a00:1158:0:300:9a63::1
Connexion vers www.napp-it.org (www.napp-it.org)|80.67.28.3|:80...connecté.
requête HTTP transmise, en attente de la réponse...200 OK
Longueur: 21919 (21K) [text/plain]
Sauvegarde en : «STDOUT»

100%[======================================>] 21919       134K/s   ds 0,2s    

2012-07-24 16:50:35 (134 KB/s) - envoi vers sortie standard [21919/21919]

stef@ZFS:~$
Then, following the installation guide (not the all in one), I reboot and after login in OI, I try to connect to http://192.168.2.119:81 from another computer.
.... and error to connect to http://192.168.2.119:81 :/

What am I missing / doing wrong ?

Cheers.

St3F

-you need root permissions
(and you must start installer from home directory)

su
password ****

wget -O ...
 
-you need root permissions
(and you must start installer from home directory)

su
password ****

wget -O ...
Yep ; thank you very much Gea.

According to the "NAPP-IT ZFS SERVER: THE MISSING MANUAL 3", I was thinking to do :
Code:
su wget -O - www.napp-it.org/nappit  | perl
Watching at this video : http://www.youtube.com/watch?v=Li1Y9nI9v2g ... I understand my error.

Now I'm on Solaris 11 to test with a little box (Xeon x3440, 8 Go, 5x 2 To WD EARS, 2x STEC Mach 16, 1x OCZ Vertex 3, Intel 10 GbE Dual)

Cheers.

St3F
 
Back
Top