![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)
- Thread starter _Gea
- Start date
Hello everybody.
I have a problem with NexentaStor (Community Edition) running as a VM under ESXi 4.1.
Here is the description of the issue I am having: http://www.nexentastor.org/boards/1/topics/3352
Does anybody run this setup? So far I failed to find a solution...
I have a problem with NexentaStor (Community Edition) running as a VM under ESXi 4.1.
Here is the description of the issue I am having: http://www.nexentastor.org/boards/1/topics/3352
Does anybody run this setup? So far I failed to find a solution...
Could you please not hijack a thread totally unrelated to your problem?
I love you too...
If its a virtual CD-rom drive causing problems then why don't you just go into settings for that vm in vmware and remove the CD-rom as it's not needed unless you want to reinstall.
Thanks Latent! Since there is no option for that in the GUI I downloaded .vmx file, edited it and uploaded it back. I no longer see the device in iostat output. Thanks again!
ide1:0.present = "true" (changed to false).
I don't know why I can't find this, but if I already have an existing OI server serving out CIFS in Domain mode, iSCSI (with Auth), Time-Slider, etc do I need to do anything special to install Napp-It?
Should I at least disable Time-Slider (redundant?) or will Napp-it do it for me?
Should I at least disable Time-Slider (redundant?) or will Napp-it do it for me?
This is basic forum etiquette - I wasn't flaming you or being abusive - no need to be snide.
Captainquark
Weaksauce
- Joined
- Dec 14, 2011
- Messages
- 105
@patrickdk,
Thanks, I didn't think this through obviously... sorry for, well being a noob
Still - I found it very strange not being able to create another NFS export... so any idea anyone? I admit I don't need it right now, but may do so in the future, so I'd like to be prepared.
2nd, a reinstall did not solve my lockup problems, unfortunately. Occasionally (say once a day(!) ), I loose connectivity to all my SMB shares and the NFS export, and I cannot login to napp-it anymore. When I try to do so, it says "initializing napp-it" forever. I can logon to the console, but commands like "zpool status" do not complete anymore (also when I try to restart napp-it manually, it does not complete). I can also not shutdown and restart the server, when I do so on the console, it starts shutting down and then eventually stops on the Oracle logo (building the bar back) and does not respond anymore. I then have to cold boot the system.
Checked afterwards using zpool status, and it says that all pools are healthy.
When checking the logs after the reboot, I see that at some point, it lost contact to a target on mpt_sas0. It's not always the same target. After that, it sometimes tries to kill two or three contracts like a hundred times, until I give up and cold boot the system. It may be that it tries to kill the contracts in response to the shutdown that cannot complete somehow.
What I do not understand is it shows all disks OK.
Here is the critical snippet from the logs:
And here's an example for the contract killing part:
My system runs on pretty new hardware:
Supermicro X8SIE-F
LSI 9211-8i Controller flashed with IT Firmware
Xeon L3426 CPU
12 GB ECC RAM
7 x 1 TB Samsung Harddrives (4 of them being brand new)
I checked two of the Samsung drives several times using the Samsung (Seagate really) checking tool, and even after a long run overnight, it never found any issues with the disks.
Any help on this is much appreciated, i.e. are there some more detailed logs than this? Where can I find them? Is there a health check tool for the LSI controller?
Thanks!
Cap'
Thanks, I didn't think this through obviously... sorry for, well being a noob
Still - I found it very strange not being able to create another NFS export... so any idea anyone? I admit I don't need it right now, but may do so in the future, so I'd like to be prepared.
2nd, a reinstall did not solve my lockup problems, unfortunately. Occasionally (say once a day(!) ), I loose connectivity to all my SMB shares and the NFS export, and I cannot login to napp-it anymore. When I try to do so, it says "initializing napp-it" forever. I can logon to the console, but commands like "zpool status" do not complete anymore (also when I try to restart napp-it manually, it does not complete). I can also not shutdown and restart the server, when I do so on the console, it starts shutting down and then eventually stops on the Oracle logo (building the bar back) and does not respond anymore. I then have to cold boot the system.
Checked afterwards using zpool status, and it says that all pools are healthy.
When checking the logs after the reboot, I see that at some point, it lost contact to a target on mpt_sas0. It's not always the same target. After that, it sometimes tries to kill two or three contracts like a hundred times, until I give up and cold boot the system. It may be that it tries to kill the contracts in response to the shutdown that cannot complete somehow.
What I do not understand is it shows all disks OK.
Here is the critical snippet from the logs:
And here's an example for the contract killing part:
My system runs on pretty new hardware:
Supermicro X8SIE-F
LSI 9211-8i Controller flashed with IT Firmware
Xeon L3426 CPU
12 GB ECC RAM
7 x 1 TB Samsung Harddrives (4 of them being brand new)
I checked two of the Samsung drives several times using the Samsung (Seagate really) checking tool, and even after a long run overnight, it never found any issues with the disks.
Any help on this is much appreciated, i.e. are there some more detailed logs than this? Where can I find them? Is there a health check tool for the LSI controller?
Thanks!
Cap'
Captainquark
Weaksauce
- Joined
- Dec 14, 2011
- Messages
- 105
Sometimes, it's target 10... I can't go back any further to see if there were different disks, too.
nice: What Does One Petabyte Of Storage (And $500K) Look Like?
Meet The Petarack
We all know what a megabyte is, and gigabytes are familiar as well. Terabytes were more recently folded into our vocabulary.
But there’s a good chance that many enthusiasts still haven't wrapped their minds around the idea of a petabyte.
In short, we’re talking about one thousand terabytes, or enough space for 20 000 Blu-ray movie rips. Although we’re a long way from seeing .....
more http://www.tomshardware.com/picturestory/582-petarack-petabyte-sas.html
Meet The Petarack
We all know what a megabyte is, and gigabytes are familiar as well. Terabytes were more recently folded into our vocabulary.
But there’s a good chance that many enthusiasts still haven't wrapped their minds around the idea of a petabyte.
In short, we’re talking about one thousand terabytes, or enough space for 20 000 Blu-ray movie rips. Although we’re a long way from seeing .....
more http://www.tomshardware.com/picturestory/582-petarack-petabyte-sas.html
A thread one should read,
http://hardforum.com/showpost.php?p=1038360802&postcount=40
especially the included link with an excellent summary on practical tipps about ZFS
http://www.nex7.com/readme1st
http://hardforum.com/showpost.php?p=1038360802&postcount=40
especially the included link with an excellent summary on practical tipps about ZFS
http://www.nex7.com/readme1st
brutalizer
[H]ard|Gawd
- Joined
- Oct 23, 2010
- Messages
- 1,602
One guy added ZIL to his zpool and writes:
"I also have to mention that I use VMware ESXi for virtualization and the backend storage runs over NFS against the file server. NFS uses sync writes and this is probably where I saw the biggest difference when I added the ZIL devices. Earlier the VMs would pretty much be unresponsive whenever i copied a big file, because they couldn’t get their sync writes through to the disks. This is gone now, thankfully."
"I also have to mention that I use VMware ESXi for virtualization and the backend storage runs over NFS against the file server. NFS uses sync writes and this is probably where I saw the biggest difference when I added the ZIL devices. Earlier the VMs would pretty much be unresponsive whenever i copied a big file, because they couldn’t get their sync writes through to the disks. This is gone now, thankfully."
_Gea:
Thanks for the pointer on backing stuff up.
I want to focus on power management. This particular server will see very occasional use. I think 95-98% of the time it will be idle. What do you suggest I do for power management?
I presume there are two distinct aspects here:
-> Overall system
-> The hard-disks
When it comes to low power state, what do you recommend in terms of OI settings? I am not sure what Sleep/Suspend mean from the server point of view.When I was building my CustoMac, it was fun to see the CPU and the PSU Fan stop spinning as it went into the suspend state, only to come alive in 2-3 seconds when I pressed a key. Is there something similar I can do with OI? Suspend/Sleep is kind of shaky with OI.
The second aspect are the hard-disks. I have seen references to the spin down states for the hard-disks. Is there something specific I need to control to minimize power usage?
Thanks for the pointer on backing stuff up.
I want to focus on power management. This particular server will see very occasional use. I think 95-98% of the time it will be idle. What do you suggest I do for power management?
I presume there are two distinct aspects here:
-> Overall system
-> The hard-disks
When it comes to low power state, what do you recommend in terms of OI settings? I am not sure what Sleep/Suspend mean from the server point of view.When I was building my CustoMac, it was fun to see the CPU and the PSU Fan stop spinning as it went into the suspend state, only to come alive in 2-3 seconds when I pressed a key. Is there something similar I can do with OI? Suspend/Sleep is kind of shaky with OI.
The second aspect are the hard-disks. I have seen references to the spin down states for the hard-disks. Is there something specific I need to control to minimize power usage?
Hey Gea one quick thing to add into Napp-it when you have the time is an option to set the write cache for comstar iscsi lun's. Currently the only way I found to change it was to use the command:
stmfadm modify-lu -p wcd=true [LUN NAME]
or to trun writeback cache back on use
stmfadm modify-lu -p wcd=false [LUN NAME]
This is quite an important setting as this with the default iscsi Lun's I created using Napp-it this WCE (write cache enable) setting is set to on. This means that all iscsi writes are converted to async writes and the ZIL/Log will not be used. This means that if there is a power cut or computer crash then you have a risk of losing the writes that were cached and corrupting your data.
Having write cache enabled makes writes faster though so being able to easily set this per lun in napp-it would be a great feature. NFS has the same basic setting and this is already supported in Napp-it web GUI which is great thanks.
for now anyone wanting to toggle this feature can just find the LUN name in napp-it (its a big long number on mine under the column LU) and use the above commands to set it. You can see the current setting under the writeback cache column.
disabled is safe but slow without a high speed LOG device
enabled is fast but not safe
stmfadm modify-lu -p wcd=true [LUN NAME]
or to trun writeback cache back on use
stmfadm modify-lu -p wcd=false [LUN NAME]
This is quite an important setting as this with the default iscsi Lun's I created using Napp-it this WCE (write cache enable) setting is set to on. This means that all iscsi writes are converted to async writes and the ZIL/Log will not be used. This means that if there is a power cut or computer crash then you have a risk of losing the writes that were cached and corrupting your data.
Having write cache enabled makes writes faster though so being able to easily set this per lun in napp-it would be a great feature. NFS has the same basic setting and this is already supported in Napp-it web GUI which is great thanks.
for now anyone wanting to toggle this feature can just find the LUN name in napp-it (its a big long number on mine under the column LU) and use the above commands to set it. You can see the current setting under the writeback cache column.
disabled is safe but slow without a high speed LOG device
enabled is fast but not safe
Comstar Module needs some work but i have added this in LU overview from 0.7h
If you are going to use WOL sleep/ suspend with your machine you need to make sure that your hardware drivers are compatible with solaris and derivatives.
I remember trying to get WOL working with my machine and never managed it as the graphics drivers did not want play nice, in the end I ended up putting together a remote sensor and using a remote control to switch my machine on or off as it is located in a small attic downstairs.
_Gea,
#1 I cannot seem to set an admin/operator password for napp-it. I tried with v. 0.7g & when it failed to set the admin password I tried again after updating to v. 0.7h, but the same happened.
#2 Is there a maximum length for the admin/operator password in napp-it? I don't seem to be able to enter any long passwords.
#3 Can we now use gmail for email notifications? I remember some months ago napp-it didn't work with gmail for notification.
#4 Is there a way to enable SSL/https for napp-it web interface? So that the connection to the server is always encrypted.
Thanks.
I get the following error when trying to set admin password:
#1 I cannot seem to set an admin/operator password for napp-it. I tried with v. 0.7g & when it failed to set the admin password I tried again after updating to v. 0.7h, but the same happened.
#2 Is there a maximum length for the admin/operator password in napp-it? I don't seem to be able to enter any long passwords.
#3 Can we now use gmail for email notifications? I remember some months ago napp-it didn't work with gmail for notification.
#4 Is there a way to enable SSL/https for napp-it web interface? So that the connection to the server is always encrypted.
Thanks.
I get the following error when trying to set admin password:
Code:
oops...
admin passwort is not identical
back/ return to former menuI don't see why gmail or any other ip wouldn't work.
But if you told it to talk on the internet, directly, without correctly configuring your mail server/ip/dns/..., than that is really your fault.
The options you have is, send email to a real mail server and let it relay to gmail, or setup the tls/smtp module and have it login to gmail, to send email. There are already tons of info on how to send mail using gmail as a relay in search engines already.
But if you told it to talk on the internet, directly, without correctly configuring your mail server/ip/dns/..., than that is really your fault.
The options you have is, send email to a real mail server and let it relay to gmail, or setup the tls/smtp module and have it login to gmail, to send email. There are already tons of info on how to send mail using gmail as a relay in search engines already.
TheFriskyPen
n00b
- Joined
- Dec 10, 2010
- Messages
- 45
im getting some wierd iSCSI results..... my 2008R2 vm is getting 100MB/s read and 10mb/s write.... very odd... it took me almost 40 minutes to move a vm that was only 9gb im running 4 2tb Hitachi drives in raidz.... this is over gig networking... any ideas?
im running 4 2tb Hitachi drives in raidz.... this is over gig networking... any ideas?Hi Gea,
in the process of swapping some disks in my vdevs, I noticed that
the view of available disks from the LSI-SAS2 extension seems to sort these disks into the wrong category.
These "available" disks, although being physically attached to the LSI card, are reported as "available on non-LSI controllers".
...but these should show up in "available on LSI controllers", shouldn't they?
This is for latest napp-it stable (0.7g) applied to an All-in-One with SolEx11 and LSI 9211-8i card.
here's a screenshot:
....the "available" disk from above is actually attached to sas2_slot c0:e1:s1 and has just been swapped with the disk on c0:e1:s7.
in the process of swapping some disks in my vdevs, I noticed that
the view of available disks from the LSI-SAS2 extension seems to sort these disks into the wrong category.
These "available" disks, although being physically attached to the LSI card, are reported as "available on non-LSI controllers".
...but these should show up in "available on LSI controllers", shouldn't they?
This is for latest napp-it stable (0.7g) applied to an All-in-One with SolEx11 and LSI 9211-8i card.
here's a screenshot:
....the "available" disk from above is actually attached to sas2_slot c0:e1:s1 and has just been swapped with the disk on c0:e1:s7.
Hi.
I want to change my 5series Adaptec to 6 Series adaptec. I see that drivers are available but i can't find any info on how to get those drivers into ESXi. I can't spend allot of time trying as the server is in production, so i hope you guys can help me out.
http://ask.adaptec.com/Scripts/adap...nMuc2VhcmNoX25sJnBfcGFnZT0x&p_li=&p_topview=1
I want to change my 5series Adaptec to 6 Series adaptec. I see that drivers are available but i can't find any info on how to get those drivers into ESXi. I can't spend allot of time trying as the server is in production, so i hope you guys can help me out.
http://ask.adaptec.com/Scripts/adap...nMuc2VhcmNoX25sJnBfcGFnZT0x&p_li=&p_topview=1
As far as I can tell, neither the basic SMART info nor the LSI module handle SAS drives. I'm trying to find the problem, but not that up on Perl
The SAS2 module needs correct SMART values especially the SN.
reason:
a, The LSI tools reports physical slot and SN but not the ZFS disk id
b. The ZFS tools reports SN (sometimes differ a little to a) and ZFS disk id
To combine these values, i use the SN as key-value.
This can work only if SN is detected correctly by SMART
Next problem:
SMART on Solaris cannot auto-detect type of disk so i need to try them all to
find correct values. If it does not work, you may look at
/var/web-gui/data/napp-it/zfsos/_lib/get-disk-smart.pl
where SMART detection is done.
(With perl you can do horrible things but I hope the code is quite readable for
anyone with basic scripting skills)
Yeah, I found the code, but I have no idea what I am looking at or how to fix it
The script is calling smartctl like
$r=`sudo /usr/sbin/smartctl -a -d scsi -T permissive /dev/rdsk/$disk`;
and checks if there are usable values
You can call the same at console like
sudo /usr/sbin/smartctl -a -d scsi -T permissive /dev/rdsk/c0t0d6s0
(enter your disk id +s0)
if you get a smart-sn at console, but not in napp-it, you may modify values in that script
or report working parameter and the output of smartctl.
I figured out why it isn't working! My seagate SAS drive is NOT generating a string:
=== START OF INFORMATION SECTION ===
(no idea offhand how to fix this though...) If you want the first N lines of the output, I can certainly post that...
=== START OF INFORMATION SECTION ===
(no idea offhand how to fix this though...) If you want the first N lines of the output, I can certainly post that...
Last edited:
Wally Simmonds
Weaksauce
- Joined
- Jun 14, 2011
- Messages
- 95
Hey guys,
Looking at building my next openindiana system (running a AIO on my 990FX and opendiana with nappit) and it works great, but want another standalone system.
At this stage I'm looking at a Asus E35M1-I which has 6 sata ports and chucking that in a Lian-Li PC-Q25. I have 5x2TB WD greens and some samsung sata ssd I was looking to use for cache. I was going to give it 8 gbs of ram.
The question I have is whether or not I can/should install openindiana to USB, or whether I should use the spare pcie slot to put a sata card into (I had intended on putting in a dedicated nic). Is anyone else running openindiana/nappit off USB?
Looking at building my next openindiana system (running a AIO on my 990FX and opendiana with nappit) and it works great, but want another standalone system.
At this stage I'm looking at a Asus E35M1-I which has 6 sata ports and chucking that in a Lian-Li PC-Q25. I have 5x2TB WD greens and some samsung sata ssd I was looking to use for cache. I was going to give it 8 gbs of ram.
The question I have is whether or not I can/should install openindiana to USB, or whether I should use the spare pcie slot to put a sata card into (I had intended on putting in a dedicated nic). Is anyone else running openindiana/nappit off USB?
Rectal Prolapse
Gawd
- Joined
- Jan 19, 2007
- Messages
- 575
im getting some wierd iSCSI results..... my 2008R2 vm is getting 100MB/s read and 10mb/s write.... very odd... it took me almost 40 minutes to move a vm that was only 9gb
Maybe you don't have write caching enabled for that iSCSI "drive"?
On my Windows 7 machine, f I turn off write caching on a mounted a comstar-managed iSCSI target from my ZFS all-in-one box, I get the same result as you - 9 MB/sec writes.
for ESXi 5.0 download the driver, stick it in a datastore, download the command line stuff if you haven't already, then:
esxcli -s ip.of.your.server -u root -p PASSWORD software vib install -d [datastore]filename.zip
must be in maintenance mode, iirc, and then reboot as it tells you to.
Worked for me with our Adaptec 6405e. The commands are different for 4.1;
vihostupdate --server ip.of.your.server --install --bundle /location/of/file.zip
I don't think that this is related to what I see.
As said, it was obvious when I swapped disks...only the disks attached to a vdev/pool were listed as being "on" the controller...as soon as the disks were configured out of the pool, they showed as being "off" the controller (and without smartinfo details)...but physically, nothing has changed.
As said, it was obvious when I swapped disks...only the disks attached to a vdev/pool were listed as being "on" the controller...as soon as the disks were configured out of the pool, they showed as being "off" the controller (and without smartinfo details)...but physically, nothing has changed.
currently I check if a drive check returns the above string, otherwise I check for another disk type
if (!($r=~/=== START/s)) {
Easiest way is to add a OR check for another string from a successfull check like
if (!($r=~/=== START|another successfull string/s)) {
You may also post the string that is unique for a successfull check (contains sn, temp and health)
A successfull smartcheck is needed to identify the disk.
Without this step, napp-it does not do any further SAS2 slot checks.
But of course, i accept this as a bug in smart-detection
Last edited:
1. You cannot use blank or some special characters (see pw form)?
2. max 16 char
3. TLS is not supported out of the box, you must install the needed perl moduls manually
4. possible with minihttpd webserver but not supported
napp-it is intended to run on a secure network (LAN, SAN) or via vpn if security is important
Thanks alot. Worked like i charm. I did need to do it in maintenance mode and i also had to change software acceptance level then it worked.
It appears that a folder got deleted on my system. Of course I had the mapping done using the admin account, so there was no prompt and I didn't notice it until today. It contained all of my actual SD/DVD rips. I haven't scrubbed it but also have no snaps. Any chance of getting the folder back?
Well, I might have some old snaps:
http://hardforum.com/showthread.php?p=1038376268
Well, I might have some old snaps:
Code:
Datapool Snapshots
NAME USED AVAIL REFER MOUNTPOINT
rpool/ROOT/oi_151a@install 25.1M - 3.38G -
rpool/ROOT/oi_151a@2011-02-05-20:39:53 87.0M - 3.57G -
rpool/ROOT/oi_151a@2011-03-23-14:45:32 179M - 3.90G -
rpool/ROOT/oi_151a@2011-11-26-19:39:36 145M - 4.55G -http://hardforum.com/showthread.php?p=1038376268
Last edited:
Hi _Gea,
Can I make a suggestion? I think it would be very helpful to be able to make selections with check boxes. I've run into a few instances now where I've had to delete all my snaps and jobs. It's quite time consuming having to delete hundreds of snaps and jobs one at a time.
Thanks and continue the great work!
Riley
Can I make a suggestion? I think it would be very helpful to be able to make selections with check boxes. I've run into a few instances now where I've had to delete all my snaps and jobs. It's quite time consuming having to delete hundreds of snaps and jobs one at a time.
Thanks and continue the great work!
Riley
I have an odd SMB/Permissions issue I can't seem to solve, here goes:
In ZFS Folder = Videos:
In above folder I have another folder called MOVIES. When I make a new folder inside that one, ie NEW FOLDER, the permissions are granted only for the owner and no one else. SMB share is mounted as root. See below.
This ONLY occurs when creating that folder over SMB shares. Doing it from console yields the correct permissions.
It doesn't matter where I create a folder, result is the same. With ONLY ONE exception -> if I create a folder in the "root" of my Videos ZFS folder, the permissions is correct.
I've narrowed it down to a SMB permissions/config issue, but now I'm lost.
I have NO user mappings set, as suggested for workgroup mode.
I've tried resetting all ACL on folders+shares, see below:
Thanks for any help
/Jim
In ZFS Folder = Videos:
Code:
share/Videos Videos on off off 10.6T [51%] 10.1T none none none none standard on off n.a. o=full, e=modi full_set 777+ offIn above folder I have another folder called MOVIES. When I make a new folder inside that one, ie NEW FOLDER, the permissions are granted only for the owner and no one else. SMB share is mounted as root. See below.
Code:
drwx------+ 2 root root 3 Feb 13 21:26 New folderThis ONLY occurs when creating that folder over SMB shares. Doing it from console yields the correct permissions.
It doesn't matter where I create a folder, result is the same. With ONLY ONE exception -> if I create a folder in the "root" of my Videos ZFS folder, the permissions is correct.
I've narrowed it down to a SMB permissions/config issue, but now I'm lost.
I have NO user mappings set, as suggested for workgroup mode.
I've tried resetting all ACL on folders+shares, see below:
Code:
ACL of folder /share/Videos
drwxrwxrwx+ 12 root root 12 Feb 13 20:16 (777)
ACL User/ Group acl acl-set details inheritance type option
0 everyone@ rwxpdDaARWc--s modify_set rd(acl,att,xatt) wr(att,xatt) add(fi,sdir) del(yes,child) x, s file,dir allow delete
1 owner@ rwxpdDaARWcCos full_set rd(acl,att,xatt) wr(acl,att,xatt,own) add(fi,sdir) del(yes,child) x, s file,dir allow delete
2 user:root rwxpdDaARWcCos full_set rd(acl,att,xatt) wr(acl,att,xatt,own) add(fi,sdir) del(yes,child) x, s file,dir allow delete
Code:
ACL of SMB share /share/Videos/.zfs/shares/Videos
-rwxrwxrwx+ 1 root root 0 Feb 13 18:36 (-777)
ACL User/ Group acl acl-set details inheritance type option
0 everyone@ rwxpdDaARWcCos full_set rd(acl,att,xatt) wr(acl,att,xatt,own) del(yes,child) x, s folder only allow deleteThanks for any help
/Jim
Last edited: