OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Yo,

Still need some help in the powermanagement section...
I want my HDD's to spin down if they are not in use for 15min/900s, so I typed 900 in the disks powermanagement setting. But my disks don't seem to be spinning down after that amount of time!
So could use some help!

gr33tz
 
correct, i also add s0 to device id's within napp-it to display temp
with current napp-it i do also a check against sata, ata and scsi to display smart vaues with more disks

Thanks gea,

In that case 0.6m didn't make a difference for me with smart sorry.

Can you give me the commands it would be calling so I can test them against my system?

My disks are sata but I need to use the scsi flag to make them detect (both explicitly listed in smartdconf) and parsed with a smarts -q onecheck.

Cheers
Paul
 
Thanks gea,

In that case 0.6m didn't make a difference for me with smart sorry.

Can you give me the commands it would be calling so I can test them against my system?

My disks are sata but I need to use the scsi flag to make them detect (both explicitly listed in smartdconf) and parsed with a smarts -q onecheck.

Cheers
Paul

I read smart values in script
/var/web-gui/data/napp-it/zfsos/_lib/get-disk.pl
in sub get_mydisk_smart line 798 for getting temp

if you uncomment the following print command and the print html-header in line 722, you will see the return value
 
Last edited:
Thanks Gea, I have run the commands manually at the command line and results are below.

Smartctl 5.4 - again not sure if this is just me or others.

Code:
paul@Indiana:~$ [B]sudo smartctl -l scttempsts -d scsi /dev/rdsk/c2t0d0s0[/B]
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

paul@Indiana:~$

Code:
paul@Indiana:~$[B] sudo smartctl -H -d scsi /dev/rdsk/c2t0d0s0[/B]
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

SMART Health Status: OK
paul@Indiana:~$

Code:
paul@Indiana:~$ [B]sudo smartctl -i -T permissive /dev/rdsk/c2t0d0s0[/B]
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

/dev/rdsk/c2t0d0s0: Unable to detect device type
Smartctl: please specify device type with the -d option.

Use smartctl -h to get a usage summary

paul@Indiana:~$ sudo smartctl -i -T permissive /dev/rdsk/c2t0d0s0

not used that I can see in get-disks.pl but needed the type specified.
Code:
paul@Indiana:~$ [B]sudo smartctl -i -T permissive /dev/rdsk/c2t0d0s0 -d scsi[/B]
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Serial number:       E3X31X63KEUGXN
Device type: disk
Local Time is: Mon Dec 12 07:56:19 2011 EST
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
paul@Indiana:~$

again not used anywhere that I can see in get-disks.pl
Code:
paul@Indiana:~$ [B]sudo smartctl -a -T permissive /dev/rdsk/c2t0d0s0 -d scsi[/B]
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Serial number:       E3X31X63KEUGXN
Device type: disk
Local Time is: Mon Dec 12 07:57:00 2011 EST
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
SMART Health Status: OK

Current Drive Temperature:     43 C
Manufactured in week 00 of year 0000
Specified cycle count over device lifetime:  100
Accumulated start-stop cycles:  0

Error Counter logging not supported
No self-tests have been logged
paul@Indiana:~$
 
not used that I can see in get-disks.pl but needed the type specified.
Code:
paul@Indiana:~$ [B]sudo smartctl -i -T permissive /dev/rdsk/c2t0d0s0 -d scsi[/B]

[/QUOTE]

i use the same in line 771

[QUOTE]
[I]again not used anywhere that I can see in get-disks.pl[/I]
[CODE]paul@Indiana:~$ [B]sudo smartctl -a -T permissive /dev/rdsk/c2t0d0s0 -d scsi[/B]

[/QUOTE]

i do not request all values (-a) but only temperature in line 798 via
`sudo /usr/sbin/smartctl -l scttempsts -d $type /dev/rdsk/$disk`

can you try
sudo smartctl -l scttempsts -d scsi  /dev/rdsk/c2t0d0s0
 
Yo,

Still need some help in the powermanagement section...
I want my HDD's to spin down if they are not in use for 15min/900s, so I typed 900 in the disks powermanagement setting. But my disks don't seem to be spinning down after that amount of time!
So could use some help!

gr33tz

check if you have set autopm from default to enable and read
http://nexenta.org/boards/1/topics/1414#message-1450
 
i use the same in line 771



i do not request all values (-a) but only temperature in line 798 via
`sudo /usr/sbin/smartctl -l scttempsts -d $type /dev/rdsk/$disk`

can you try
sudo smartctl -l scttempsts -d scsi /dev/rdsk/c2t0d0s0

No luck

Code:
paul@Indiana:~$ sudo smartctl -l scttempsts -d scsi /dev/rdsk/c2t0d0s0
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

paul@Indiana:~$
 
No luck

Code:
paul@Indiana:~$ sudo smartctl -l scttempsts -d scsi /dev/rdsk/c2t0d0s0
smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

paul@Indiana:~$

ok, i will go to back -a command
(i have used it in former versions but switched because -a was slower)
thanks
 
I'm working on piecing together a new esxi 4.1 AIO.

It'll be awhile before I'm ready. I want to use a xeon e5-1650
Ive got a norco 4224.

I'm focusing on the storage currently.

These are the drives I have
6x 2tb Hitachi 5k3000
1x 2tb Seagate ST2000DL003
4x 1.5tb Seagate 7200.11
2x 64gb SSD Kingston v100's (willing to purchase more with the current deal)
3x 500gb Western Digital wd5000aaks
1x 500gb Seagate 7200.10

I have about 7TB of data currently

I'm wanting to have 500GB or more of storage for VM's (i'm guessing 10-15 VM's depending on what I'm testing for work. Id love to be able to run more than this amount.

Id like to make use of both ssds, and my 4 500GB drives to store the VM's

I was planning on raidz2 for both my 2tb Hitachis, and the 1.5tb Seagates. Those will make up the Media pool.

I'm currently running a esxi 4.1 box, and windows home server for the media.

Any suggestions?
 
If you simply want to access one share from different hosts via smb, afp or nfs4, the user id is irrelevant because you access the share via username/pw.

How do you specify the username/pw when mounting on NFS?
 
What's the difference between the "status" and "alert" e-mails?

I have status set to e-mail me every day at 6:00 am (so I can make sure everything looks OK).

However, I'm confused about "alert". If there a problem with the arrays I'd like to be notified every 6 hours if possible...
 
What's the difference between the "status" and "alert" e-mails?

I have status set to e-mail me every day at 6:00 am (so I can make sure everything looks OK).

However, I'm confused about "alert". If there a problem with the arrays I'd like to be notified every 6 hours if possible...

you get an alert on next time-intervall (up to 1 min) after a disk error is recognized.
If you do not like 1min, you can set alert check every 6 hours

These alert mails are independant from regular status-mails
 
I'm working on piecing together a new esxi 4.1 AIO.

It'll be awhile before I'm ready. I want to use a xeon e5-1650
Ive got a norco 4224.

I'm focusing on the storage currently.

These are the drives I have
6x 2tb Hitachi 5k3000
1x 2tb Seagate ST2000DL003
4x 1.5tb Seagate 7200.11
2x 64gb SSD Kingston v100's (willing to purchase more with the current deal)
3x 500gb Western Digital wd5000aaks
1x 500gb Seagate 7200.10

I have about 7TB of data currently

I'm wanting to have 500GB or more of storage for VM's (i'm guessing 10-15 VM's depending on what I'm testing for work. Id love to be able to run more than this amount.

Id like to make use of both ssds, and my 4 500GB drives to store the VM's

I was planning on raidz2 for both my 2tb Hitachis, and the 1.5tb Seagates. Those will make up the Media pool.

I'm currently running a esxi 4.1 box, and windows home server for the media.

Any suggestions?


I would use a raid-10 for VM's and disable sync write (no write cache needed)
- eventually use a ssd as read cache
- think about a hotfix

-use other SSD as read cache for the media Pool build from 2TB disks and one Raid-Z2
-use the smaller 1.5TB disks to build a backup pool
or build one large Raid-Z2 and use only 1,5 TB from the 2 TB disks

(I would not use Raid-Z1 with 4 disks and up)
 
I'm at a total loss here...

I've tried editing the powerconfig and tried to reboot..then the server just stopped rebooting..it powers on and does nothing,no postscreen,bios,no beeps,nada..
Tried removing the 3 IBM M1015 cards but nothing...

Supermicro X9 SCM-F
Intel E-1230 xeon

I can connect with iPMI but it sits idle,fans spinning at max
plz help!
:mad:

EDIT1: I removed the battery on the motherboard and am back able to start up,but bootdrive won't boot...says no OS!
EDIT2: Managed to get the server back up again, could editing powerconfig (setting autopm to enable) have to do anything with my problem?
EDIT3: Server keeps rebooting for no apparent reason now and then when I fiddle in the napp-it gui...faulty bootdrive? using an older 2.5" WD I had laying around!
 
Last edited:
siam: You may want to start a new thread on your issues. ;)

In the meantime, try replacing the PSU with another one. After building many PCs over the decades, I find that the PSU is the easiest component to replace and test, so is the VERY FIRST THING I check when something goes wrong.
 
I'm at a total loss here...

I've tried editing the powerconfig and tried to reboot..then the server just stopped rebooting..it powers on and does nothing,no postscreen,bios,no beeps,nada..
Tried removing the 3 IBM M1015 cards but nothing...

Supermicro X9 SCM-F
Intel E-1230 xeon

I can connect with iPMI but it sits idle,fans spinning at max
plz help!
:mad:

EDIT1: I removed the battery on the motherboard and am back able to start up,but bootdrive won't boot...says no OS!
EDIT2: Managed to get the server back up again, could editing powerconfig (setting autopm to enable) have to do anything with my problem?
EDIT3: Server keeps rebooting for no apparent reason now and then when I fiddle in the napp-it gui...faulty bootdrive? using an older 2.5" WD I had laying around!

sounds like bad hardware. reset bios to default and
start with the minimalst config/ram and a new hd and install os on it,.
If its working add other parts unless you hopely find the problem
change ram/ power supply if error remains
 
Yo,

Thanks..thought it was hardware related
Anyways got it back up and running and will be testing integrety over the next few days!
If problems still persist I'll try changing hardware!

gr33tz
 
ok, i will go to back -a command
(i have used it in former versions but switched because -a was slower)
thanks

Code:
[B]sudo smartctl -l scttempsts -d scsi /dev/rdsk/c2t0d0s0[/B]

smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

^ This doesn't work for me either.

Why are some of my drives reporting status and some aren't?

snap2.png


As per my earlier posts #2143, the drives have only started displaying smart info since I physically removed then reconnected it. Whats slightly annoying is after doing the same procedure with all 6 drives, 3 drives have now begun showing smart, while the other 3 don't. This seems very odd? Below you can see that the drive does infact support smart.

Code:
[B]smartctl -a /dev/rdsk/c4t1d0s0 -d scsi[/B]

smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Serial number:       S2H7J9JB602452
Device type: disk
Local Time is: Wed Dec 14 15:52:08 2011 WST
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
SMART Health Status: OK

Current Drive Temperature:     30 C
Manufactured in week 00 of year 0000
Specified cycle count over device lifetime:  100
Accumulated start-stop cycles:  0

Error Counter logging not supported
No self-tests have been logged
 
Hello!

I'm thinking installing an ESX on my current OI server and change SAN to ESXi-all-in-one. I have some questions though.

I will install ESX on a hard drive and in esx, I will create a VM with OI installed in it. I would like to host all other VMs on a SAN share. Is that possible?

So, 1st i would boot up esx, then OI inside ESX and when that booted up, esx would attach iscsi datastore from OI and boot the rest of the VMs.Is that doable or do I have to host all images on the hard drive where I have ESX?

lp, Matej
 
Code:
[B]sudo smartctl -l scttempsts -d scsi /dev/rdsk/c2t0d0s0[/B]

smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

^ This doesn't work for me either.]

Its a pity that smartctl -i and -l scttempsts does not work everywhere.
These two command are much faster than a smartctl -i which delivers all informations.

I have switched to -a from napp-it 0.6n and hope it will work for all now
(ata, sata and scsi disks)
 
Hello!

I'm thinking installing an ESX on my current OI server and change SAN to ESXi-all-in-one. I have some questions though.

I will install ESX on a hard drive and in esx, I will create a VM with OI installed in it. I would like to host all other VMs on a SAN share. Is that possible?

So, 1st i would boot up esx, then OI inside ESX and when that booted up, esx would attach iscsi datastore from OI and boot the rest of the VMs.Is that doable or do I have to host all images on the hard drive where I have ESX?

lp, Matej

yes, you need a boot disk example on onbaord Sata to boot ESXi with a local datastore on it.
On this local datastore, you must install der OI SAN-VM . No other VM needs to be on this local datastore.

You need a second disk-controller (best LSI HBA with 2008 chipset). Set this disk-controller to pass-through
and assign it to your OI VM. All disks are directly and only accessable from OI

Set autostart to start this SAN VM at first place
Create a ZFS datapool on your disks and create a ZFS folder, I would disable sync write property on this ZFS for performance

Share this folder via NFS (iSCSI is not a good option at all with all-in-one)
Share this folder via SMB for easy access for snaps/ copy/ move/ clone/ backups

Set permission of the share to 777 and ACL to everyone@=modify with inheritance on
(or set root access for the ip of ESXi)

Create new VM's as desired on this NFS SAN datastore only

For best performance, you may activate VMCI betwwen your VM's or try the ESXi vmxnet3 network adapter
which can deliver Network performance up to 10 Gb/s between VM's an the ESXi virtual switch.
But there are some having problems doing this. With e1000 driver and vmci you can get several Gb/s datarate also.
 
Hi all, thumbs up for the great work in this thread!
Following this article for a while now, I have finally decided to build my own ZFS NAS taking into account the tips in here... Unfortunately, it seems as if I even do not manage to do the part that's supposed to be easy, the installation of Solaris 11. :eek: I hope someone can help me!

Since Solaris 11 Express is not available anymore, I have prepared myself a Solaris 11 LiveCD to install my system. The DVD works, as I have tested it on an old HP DL380 G4 here at work. The hardware I have at home are:
- Supermicro X8SIE-F mainboard
- 12 GB ECC RAM
- Xeon L3426 CPU
- LSI 9211-8i Controller flashed with the IT firmware (got that working, at least ;))
- At the moment, 2 x 320 GB HDD's in RAID-1 (mainboard) for OS

I think I pretty much followed the recommendations in this article, at least in terms of hardware.

The LiveCD starts up and presents me the GNU Grub menu. I tried both, the first and the second option ("Solaris 11 11/11" and "Solaris 11 11/11 VESA"), but it is always the same. It starts with the dots, then changes to commandline. It takes a pretty long time and never asks me for a keyboard layout or language selection. All I see is:

Remounting root read/write
Probing for device nodes...
Preparing image for use
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
Enter user name for system maintenance (Control-D to bypass):

I then can login using root and solaris (or even with a blank password, for that matter), and it spits me out on a commandline in which I cannot do anything useful. I googled a lot about this and also used this forums' search function, but I cannot seem to find anything that helps me.

I suspect one of my hardware parts is incompatible, so my question is: is there any way to find out which part is not compatible? Can I add the relevant driver to the DVD somehow? Supermicro offers some Solaris drivers, so I might be lucky.

Any other ideas despite of hardware incompatibility?

Any help would be greatly appreciated! Thanks!

Cap'
 
Yo,

Do you see your LSI device in post screen when booting?
Did you press ctrl+c and disabled bootsupport in the LSI menu?
Is your ram Unbuffered?

gr33tz
 
With your hardware i would first burn another DVD or try another DVD drive.
It should boot completely Solaris 11 Live from DVD independantly from disks and disk controller.

You may also try OpenIndiana 151a.
 
With your hardware i would first burn another DVD or try another DVD drive.
It should boot completely Solaris 11 Live from DVD independantly from disks and disk controller.

You may also try OpenIndiana 151a.

Also with his hardware he doesn't even have to reburn the dvd!
His board has Ipmi,you just load the solaris iso in ipmi,boot the server,hit 'del' and choose ipmi cdrom from boot options!
Install solaris.

Gr33tz
 
Can anyone give advice on how to resize a partition in OpenIndiana? I can't seem to get it to work.

I started with a 16gb partition under ESXI 5.0 for Openindiana on a 100gb SSD. It is now full and I want to resize it to about 32gb instead. I've changed the setting in ESXI to 25gb however I cant seem to get it to merge and create a new rpool for openindiana.

Where do i go from here to resize the OI solaris partition?
 
To tag on to the existing hardware discussion, I've decided to kick my aging server hardware to the curb and also switch from Amahi to OpenIndiana. The OP was fantastic and contained lots of great info, but I just wanted to see if I could get a final opinion on the hardware I landed on for this build.

The use case for me is a simple media server. I currently have 5 2TB WD Green drives (I know Greens aren't ideal, I'll replace them when HD prices drop again) and I wasn't planning on virtualizing. I'll be wanting to simply access my shares via NFS over a Gigabit network.

I'll also need to get a system drive (should I be shooting for 2 drives in a raid 1 configuration for system), but unless an SSD would be of major benefit there, I'll likely just go with cheaper 160GB drive(s).

Here's a link to the hardware I decided on thanks to recommendations in this thread:

http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=25469268
 
Guys,
Thanks for your quick replies.

Just to take them all into account:
Do you see your LSI device in post screen when booting?
Did you press ctrl+c and disabled bootsupport in the LSI menu?
Is your ram Unbuffered?

-> Yes, I can see the LSI device, it is also stating correctly it has the IT firmware. I did not see any bootsupport option in the menu (I think it's Ctrl-I to get there), but I'll check again tonight. I'll also have to check if my RAM is buffered - what's your hint here?

Also with his hardware he doesn't even have to reburn the dvd!
His board has Ipmi,you just load the solaris iso in ipmi,boot the server,hit 'del' and choose ipmi cdrom from boot options!

-> Thanks for the hint, I did not think about that! I'll give it a try tonight when I get home. I would also rather try Solaris first...

So I also read from your replies that the hardware I have chosen *should* normally be working, right?

I'll post again asap if it worked.
Thanks,
Cap'

PS: I forgot to mention that the same thing happens to me if I boot from the text based boot CD...
 
To tag on to the existing hardware discussion, I've decided to kick my aging server hardware to the curb and also switch from Amahi to OpenIndiana. The OP was fantastic and contained lots of great info, but I just wanted to see if I could get a final opinion on the hardware I landed on for this build.

The use case for me is a simple media server. I currently have 5 2TB WD Green drives (I know Greens aren't ideal, I'll replace them when HD prices drop again) and I wasn't planning on virtualizing. I'll be wanting to simply access my shares via NFS over a Gigabit network.

I'll also need to get a system drive (should I be shooting for 2 drives in a raid 1 configuration for system), but unless an SSD would be of major benefit there, I'll likely just go with cheaper 160GB drive(s).

Here's a link to the hardware I decided on thanks to recommendations in this thread:

http://secure.newegg.com/WishList/PublicWishDetail.aspx?WishListNumber=25469268

1) Nice components on your list...BUT...your MB and CPU do not go together. X8 series board is socket 1156 for previous gen such as i3-540 CPU. i3-2100 is socket 1155 for newer gen boards. Either change CPU down to 1156, or choose newer X9 series board. For example the X9SCL+-F is an perfect choice as replacement.

2) Make sure you reflash your SASUC8I to IT firmware. Alternately, consider some of the cheaper ebay alternatives to this card, IBM BR10i is equivelent LSI 1068e, or IBM M1015 is newer SAS2008 controller.

3) Regular 2.5" or 3.5" notebook/desktop drives are fine for the boot drives. If using Solaris 11 or OpenIndinana you will need to load the system on one of the drives and manually create the mirrored rpool later. It's not too difficult, see instructions here:
http://constantin.glez.de/blog/2011/03/how-set-zfs-root-pool-mirror-oracle-solaris-11-express
 
For 3), what is the advantage to mirror using ZFS instead of hardware mirroring using the onboard controller? Sorry if this question is dumb, I just don't see it :)
Thanks!
Cap'
 
Two things: with software mirroring, zfs can check and detect issues (possibly fixing them in the process). Also, reading from a mirror with zfs will often be faster than HW since it can round-robin between the disks (some HW does this some does not.)
 
1) Nice components on your list...BUT...your MB and CPU do not go together. X8 series board is socket 1156 for previous gen such as i3-540 CPU. i3-2100 is socket 1155 for newer gen boards. Either change CPU down to 1156, or choose newer X9 series board. For example the X9SCL+-F is an perfect choice as replacement.

2) Make sure you reflash your SASUC8I to IT firmware. Alternately, consider some of the cheaper ebay alternatives to this card, IBM BR10i is equivelent LSI 1068e, or IBM M1015 is newer SAS2008 controller.

3) Regular 2.5" or 3.5" notebook/desktop drives are fine for the boot drives. If using Solaris 11 or OpenIndinana you will need to load the system on one of the drives and manually create the mirrored rpool later. It's not too difficult, see instructions here:
http://constantin.glez.de/blog/2011/03/how-set-zfs-root-pool-mirror-oracle-solaris-11-express

Aiee! Thanks for the catch on the socket difference. I'm assuming the only difference between the X9SCL+-F and the X9SCL-F-O is that the latter *only* supports ECC DIMMs?

It looks like the BR10i is just a straight rebrand of the LSI card, so I'll probably go with that and save myself $100.

Thanks again!
 
Has anyone tried building an all in one using XenServer? I'd be curious what the pros and cons have been vs doing this with ESXi.
 
Aiee! Thanks for the catch on the socket difference. I'm assuming the only difference between the X9SCL+-F and the X9SCL-F-O is that the latter *only* supports ECC DIMMs?

It looks like the BR10i is just a straight rebrand of the LSI card, so I'll probably go with that and save myself $100.

Thanks again!

On the X9SCL+-F vs X9SCL-F: no, they both support exactly the same memory configurations. The only difference is the NIC chips. On the "-" version they use two different kinds of NIC chips (Intel 82579LM and 82574L). On the "+-" version both NICs are the same (both Intel 82574L). This is the only material difference between the two boards. The difference matters most to users of ESXi, which does not have native support for the 579LM NIC.
 
On the X9SCL+-F vs X9SCL-F: no, they both support exactly the same memory configurations. The only difference is the NIC chips. On the "-" version they use two different kinds of NIC chips (Intel 82579LM and 82574L). On the "+-" version both NICs are the same (both Intel 82574L). This is the only material difference between the two boards. The difference matters most to users of ESXi, which does not have native support for the 579LM NIC.

Ah, thanks very much!
 
Back
Top