OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

How do you currently have disks organized?
Never mind, I see...

You know, you could also create a new RAIDz vdev with 3 disks and add it to the current zpool. You would loose 2TB, but then again, RAIDZ arrays bigger than 5 disks are not recommended because of the rebuild time. It's better to have many small RAIDZ vdevs in one pool because of better security. If you have a large vdev, it takes a long time to rebuild and there are grater chances another drive will crash.

Matej
 
Stilgars said:
All 2 TB+ drives on the market using Advanced Format still emulate 512 B sectors. Therefore the issue is still ongoing, and users still need to use the known tricks to avoid problems with ZFS (modified zpool executable with hardcoded ashift=12 or gnop command under BSD environments).

For WD Green, you should use the WDIDLE3 utility to set spinning downtime to max value. If not, disks lifespan would surely take a huge hit.
Click to expand...

WD Black 2TB are not 4k
 
liam137 said:
Well, Captain, I may have some bad news for us. I feel the problem is with the samsung drives. I replaced the motherboard with new-old-stock (with Sun stickers on it no less) and a new 9211-8i hba. Before the install was even finished I saw errors with one of the drives and the install wouldn't progress. I'm going to play with different firmware versions before moving on with the 'great parts swap'.

Cheers,
Liam
Click to expand...

Hey Liam,
Thanks for the heads up... that would indeed be bad news. I just got my Mobo a few days ago, and as I was away until today, I can only now start putting it back together... I'll keep you updated about my progress.
Cheers,
Cap'
 
Liam,
Assembled it and reinstalled Napp-It. I could import the datapool, so I didn't have to copy it all over again. It's currently running a scrub, but so far, no errors visible.
I'll check back tomorrow to see if it still has no errors.
Cheers,
Cap'
 
Updated my server yesterday, i rather much like the subtle changes to the web gui, looks nice! Now if i can figure out why my 3rd M1015 won't work so that i can get my storage pools back online :(
 
So I've created a pool, raidz with 5 1TB drives:

Then I created a zfs-folder:
ZFS%20%281%29.PNG


Q1. Why have I lost 360GB just creating a zfs-folder? I haven't used 369GB - this is a fresh install (notice next to 'tank' used).

Then I shared the file using ACL on folders:
ZFS%20%282%29.PNG


All works fine - 360gb is confusing though. My 5tb raidz is only 3.2usable
 
Last edited:
DlStreamnet said:
So I've created a pool, raidz with 5 1TB drives:

Then I created a zfs-folder:

Q1. Why have I lost 360GB just creating a zfs-folder? I haven't used 369GB - this is a fresh install (notice next to 'tank' used).
Click to expand...

If you create a pool with napp-it defaults, a 10% reservation is set to the parent pool to avoid a total fill-rate of the summarized filesystems above 90% of the pool capacity
(you should not go with your filesystems above)

If you do not like, you can delete or modify this reservation.
 
_Gea said:
If you create a pool with napp-it defaults, a 10% reservation is set to the parent pool to avoid a total fill-rate of the summarized filesystems above 90% of the pool capacity
(you should not go with your filesystems above)

If you do not like, you can delete or modify this reservation.
Click to expand...

Sounds good to me, thanks for letting me know, Gea.

Edit: Another happy customer here Gea. Works awesome, had burst writes of 163MB/s - now copying 400+ gb of data at 70MB/s which is great.

Q. Do Jumbo frames exist on OpenIndiana and is there any need?

Q. My hard drives seem to be spooling down once they have written, and then almost immediately have to spool back up for the next write - does this sound normal?
 
Last edited:
DlStreamnet said:
Do Jumbo frames exist on OpenIndiana and is there any need?
Click to expand...

Yes, but both the driver and the hardware has to support them.
To get jumbo frames working you also need your other machines to support them as well as any switch(es) in between.

As to there being any need......your mileage may vary but I don't see much of an increase in performance. The CPU usage is a bit lower though (not that it's particularly taxed anyway).
Others have reported larger performance improvements though, and a few others have reported it's actually reduced performance!

The best advice would be to try them and see for yourself!
You won't get much more than around 110MB/s max over a gigabit link though, no matter what size frames you use!
 
A Smart OK means nothing.
A ZFS error means, that there IS a problem on disk,. cabling, controller or controller driver
 
It sounded like it didn't have power. It's a certified repaired seagate drive. It has passed seatools as well.

I have re-added the drive, what command do I run to re-add the drive?
 
Billy_nnn said:
Others have reported larger performance improvements though, and a few others have reported it's actually reduced performance!
Click to expand...
for 1gbit networks jumbo frames always reduces IOPs due to coalescence. instead of sending 4 1500 MTU packets as fast as possible the network stacks waits to send a single 9000MTU packet. A single packet that also has slightly higher latency. Not a lot, but there IS a difference.

the difference in IOPs is not 4:1 mind you but every test i've run and all the tests i've seen show a slight decrease in IOPs for both NFS and iscsi. throughput for gig is a wash either way.

10gbit and above though is a bit different because you can write larger frames to the wire MUCH faster (10x faster than 1gig) so the latency overhead between 1500 and 9000MTU packets are really quite similar. You can also handle many more packets per second on the storage array since you're only writing 1/4 the amount of packets.
 
DlStreamnet said:
It sounded like it didn't have power. It's a certified repaired seagate drive. It has passed seatools as well.

I have re-added the drive, what command do I run to re-add the drive?
Click to expand...

napp-it menu disk replace: FAULTED with working one

If the working disk is in the same slot now than the FAULTED and you have set pool property autoreplace=on,
this will happen automatically (if your controller supports hot plug), otherwise select only the FAULTED disk and klick on replace

if your controller does not support hotplug, you need to reboot.
 
Last edited:
_Gea said:
napp-it menu disk replace: FAULTED with working one

If the working disk is in the same slot now than the FAULTED and you have set pool property autoreplace=on,
this will happen automatically (if your controller supports hot plug), otherwise select only the FAULTED disk and klick on replace
Click to expand...

invalid vdev specification
the following errors must be manually repaired:
/dev/dsk/c4t3d0s0 is part of active ZFS pool tank. Please see zpool(1M).
 
You may try to reboot to check if this disk is reused automatically because it was a former pool member,
otherwise you need to re-partition/ delete label (ZFS does not allow use of active ZFS disks).

You can do this at CLI with format. I do usually on Windows or my Mac (with an Sata to USB converter) and reformat to HFS or NTFS.
 
_Gea said:
You may try to reboot to check if this disk is reused automatically because it was a former pool member,
otherwise you need to re-partition/ delete label (ZFS does not allow use of active ZFS disks).

You can do this at CLI with format. I do usually on Windows or my Mac (with an Sata to USB converter) and reformat to HFS or NTFS.
Click to expand...

Re-added the drive after formatting to ntfs...now OpenIndiana won't boot. Is there a way to view the log at the OpenIndiana screen.

Edit:
Could not proceed due to an error. Please try again later or ask your sysadmin.
Maybee a reboot after power-off may help.
cannot replace c4t3d0 with c4t3d0: c4t3d0 is busy

edit2:

nvalid vdev specification
the following errors must be manually repaired:
/dev/dsk/c4t3d0s0 is part of active ZFS pool tank. Please see zpool(1M).

Second attempt gives me the same error. This is after formatting with NTFS in Windows...hmm

Edit3:
action: Replace the faulted device, or use 'zpool clear' to mark the device repaired.
I guess I need to do this?

Edit4:
I ran zpool clear tank - the error is now gone, the device isn't faulted.

However I have formatted it as ntfs - so now I assume I need to resliver?

Edi5:
c4t3d0 ONLINE 0 0 0 (resilvering)
 
Last edited:
DlStreamnet said:
Re-added the drive after formatting to ntfs...now OpenIndiana won't boot. Is there a way to view the log at the OpenIndiana screen.

Edit:
Could not proceed due to an error. Please try again later or ask your sysadmin.
Maybee a reboot after power-off may help.
cannot replace c4t3d0 with c4t3d0: c4t3d0 is busy
Click to expand...

i suppose:
ZFS startet a resilvering here automatically (reason of device busy)
you interupted the resilvering and needed to restart replace

ps
if a disk failes and the same disk comes back, you need to confirm/ clear the pool error manually
 
_Gea said:
i suppose:
ZFS startet a resilvering here automatically (reason of device busy)
you interupted the resilvering and needed to restart replace

ps
if a disk failes and the same disk comes back, you need to confirm/ clear the pool error manually
Click to expand...

Maybe, but the disk still shown as faulted. Maybe just didn't update. It's resilvering anyway.

Can I view progress?
 
DlStreamnet said:
Maybe, but the disk still shown as faulted. Maybe just didn't update. It's resilvering anyway.

Can I view progress?
Click to expand...

menu pools

ps
a failed disk keeps faulted until it is fully replaced (and then removed).
 
Last edited:
Captainquark said:
Liam,
Assembled it and reinstalled Napp-It. I could import the datapool, so I didn't have to copy it all over again. It's currently running a scrub, but so far, no errors visible.
I'll check back tomorrow to see if it still has no errors.
Cheers,
Cap'
Click to expand...

Captain,

Good to hear you're in the home-stretch - best of luck to you. My box has been up for two days now. I won't say anything either way as I don't want to tempt the wrath of the whatever from high atop the thing.

For what it's worth, I flashed P7(IT) onto the 9211. It came with P7 in IR so I figured go with what's on it. I've also not touched any bios/firmware on the motherboard.

One thing I wanted to mention is I have disabled multipathing. You'll see in /var/adm/messages (dmesg) warnings about multipath status degraded. As there's nothing to multipath, off it goes. The command is stmsboot -d along with a reboot. If you do disable it, do so before importing the pool.

I finally made a script for the install. The longest part was the installation itself. One quick script and everything's installed and configured. I highly recommend making one, though I'm not quite sure how much napp-it automates. I've looked at it and for me the cli is much quicker.

I also forgot to ask, what release are you using? Are you an OS/OI guy or are you using Solaris proper?

I'm going to give it a week or so of uptime and see how it goes.

Cheers,
Liam
 
Hi all,

Another question in the efforts of fine tuning my zfs-share, would adding more processors to ESXi 5 assist?
 
madrebel said:
for 1gbit networks jumbo frames always reduces IOPs due to coalescence. instead of sending 4 1500 MTU packets as fast as possible the network stacks waits to send a single 9000MTU packet. A single packet that also has slightly higher latency. Not a lot, but there IS a difference.

the difference in IOPs is not 4:1 mind you but every test i've run and all the tests i've seen show a slight decrease in IOPs for both NFS and iscsi. throughput for gig is a wash either way.
Click to expand...

It would seem that it doesn't always reduce IOPS

http://www.boche.net/blog/index.php...mparison-testing-with-ip-storage-and-vmotion/


I think it depends on several factors - try and see in each situation is the best advice IMO - but if for some reason your current setup doesn't support jumbo frames, then don't worry about it (in all probability, you won't be missing much)
 
Guys,
Just trying to bring my (newly assembled) system back to the state it was before. Had no troubles installing net-ssleay to use TLS Email, but when I send a test message, I get

Code: 
invalid SSL_version specified at /usr/perl5/site_perl/5.12/IO/Socket/SSL.pm line 308

For help, please send mail to this site's webmaster, giving this error message and the time and date of the error.
[Sun May 20 17:26:09 2012] admin.pl: invalid SSL_version specified at /usr/perl5/site_perl/5.12/IO/Socket/SSL.pm line 308

I have to admit I first installed net-ssleay 512 and 584, then installed the TLS packages. Saw my mistake later and uninstalled 584, as the installed perl version is 512, but it did not solve the problem... any ideas?

Thanks!

Cap'
 
I'm having some trouble - when my LSI-1068E controller is connected via passthrough, the smartinfo page gets stuck on "processing, please wait.." and does not load. smartctl -a /dev/rdsk/c4t0d0p0 works in the terminal, though.

Also, when I try to set a drive to offline under hotswap, I get "cfgadm: Hardware specific failure: failed to unconfigure SCSI device: I/O error"

Does anyone know why this is happening?
 
Last edited:
Liam,

Yeah finally I got my new mobo, I am now using a Supermicro X8SI6-F. Main difference is that it has an onboard SAS Controller with 8 ports that I plan to use with SAS disks for the root and VM pool (though I don't have the disks yet, only 2 boot disks of 36 GB size). So my new setup is now:
  • Supermicro X8SI6-F
  • Intel L3426 Xeon
  • 12 GB DDR3 registered (Kingston)
  • 2 x 36 GB SAS connected to onboard SAS Ctrl to boot from
  • LSI 9211-8i Controller
  • 7 x 1TB SATA Samsung drives on 9211-8i Ctrl

I got my X8SIE-F back from Supermicro with a note that they ran long-term tests and did not find any problems, though for them, everything is within the specs. The LSI 9211-8i however was exchanged by RMA from my vendor. So I am not sure what exactly caused the problems before, as I use a different mobo and Controller now.

However, I have not seen any errors anymore since I run the new config. I wrote and read several hundred Gigs with no errors whatsoever. I ran 3 scrubs and they all completed within 3 hours (each), whereas it took me 14+ hours before to run one single scrub on the same pool.

Only problem I see at the moment is performance, which does not go over 35-40 MB/s when I copy data onto it from my Windows 7 machine. I know there is a lot of information in this thread related to Windows/CIFS performance, so I will read through it when I have more time and try out what's suggested there.

liam137 said:
For what it's worth, I flashed P7(IT) onto the 9211. It came with P7 in IR so I figured go with what's on it. I've also not touched any bios/firmware on the motherboard.
Click to expand...

I updated everything: X8SI6-F BIOS and IPMI firmware, LSI 9211-8i BIOS and firmware, onboard SAS BIOS and firmware. The LSI stuff, I updated to P13.5 (IT, of course), which is the latest. As far as I can see, everything works perfectly there.

One thing I wanted to mention is I have disabled multipathing. You'll see in /var/adm/messages (dmesg) warnings about multipath status degraded. As there's nothing to multipath, off it goes. The command is stmsboot -d along with a reboot. If you do disable it, do so before importing the pool.
Click to expand...
I have to check this when I have more time, however, my pool is already imported back and running productive... anything I need to do differently in that case?

I finally made a script for the install. The longest part was the installation itself. One quick script and everything's installed and configured. I highly recommend making one, though I'm not quite sure how much napp-it automates. I've looked at it and for me the cli is much quicker.
Click to expand...
I was hoping that once installed, I don't have to touch it again :) so I guess I'll skip that part.

I also forgot to ask, what release are you using? Are you an OS/OI guy or are you using Solaris proper?
Click to expand...
I am using Solaris 11 11/11 natively.

Well, good luck to you, too, I will update regularly on how it goes with my box.

Cheers,

Cap'
 
Hey guys, for the noobs amongst us I made some videos:
How to make a bootable ESXi 5 USB -
http://www.youtube.com/watch?v=C1EvNBJ3Zb8

How to license ESXi -
http://www.youtube.com/watch?v=AN_-ybGOtzc

How to configure vt-d -
http://www.youtube.com/watch?v=CzcWd9veVuQ

How to add a datastore -
http://www.youtube.com/watch?v=mmzkHCtuWaA

How to upload ISO's -
http://www.youtube.com/watch?v=c6v8GsWJ9dc

How to install OpenIndiana (ZFS File system/Raidz) -
http://www.youtube.com/watch?v=9DeFUVfv_aQ

How to set a static IP on OpenIndiana -
http://www.youtube.com/watch?v=yOz4-ORawl0

How to install napp-it on Open Indiana -
http://www.youtube.com/watch?v=Li1Y9nI9v2g

Configuring ZFS -
http://www.youtube.com/watch?v=u70MlGpQTRM

Configuring Windows Share on ZFS -
http://www.youtube.com/watch?v=KPrgdFsE104

Auto start virtual machines in ESXi 5 -
http://www.youtube.com/watch?v=Kj3Zt_FaYdk

------------------------------------------------------------------------------

Edit: Would my write speeds go up considerably if I installed an SSD drive for L2ARC?
 
Last edited:
There is something weird for my SMB permission.

I created a Folder with SMB access to root, and created a "allusers group" on OI. Everything is default.

Then from windows I went as root with SMB, and created 3 user folder inside, such as:

\\OI\Users\User1
\\OI\Users\User2
\\OI\Users\User3

For each of the UserX folder I went into advance permission, and removed the inheritance, removed the "OI\AllUsers" group, and added OI\UserX user with full permission.

This should effectivly let All users go to \\OI\Users\ and see all user folder, however it shouldn't let them do anything to these folder except if it's their own folder.

However, I seem that all users can still do anything to any of thoses folder even if they have no permission (as shown in windows permission tab).

The only permission on the UserX folder is: root and UserX ..

Any idea?
 
DlStreamnet said:
Edit: Would my write speeds go up considerably if I installed an SSD drive for L2ARC?
Click to expand...

Hey,
I don't think so... isn't L2ARC short for "Level 2 Advanced Read Cache", designed to speed up random reads?

Write Cache is ZIL, if I remember correctly. If you meant ZIL, then I think it is depending on the scenario you're using it (small files, large files etc.). Fastest will always be RAM.
 
Even then, it depends on the type of writes you are doing - the ZIL is only used for synchronous writes, so using a cache device for it will only speed up synchronous writes - it won't affect asynchronous write speed.
 
Billy_nnn said:
Even then, it depends on the type of writes you are doing - the ZIL is only used for synchronous writes, so using a cache device for it will only speed up synchronous writes - it won't affect asynchronous write speed.
Click to expand...

Mostly large files. I'm doing it purely because SSD's are quite cheap now, and I assume i'd only need a 30GB.

At present I see 65MB/s sustained, but it can burst at 100MB/s for a little while.
 
How are you writing to the pool? If you are using SMB then a ZIL wont help. If youre using NFS a ZIL can help, provided your ZIL is fast enough (cheap old SSDs are not a help here).
 
thedge said:
How are you writing to the pool? If you are using SMB then a ZIL wont help. If youre using NFS a ZIL can help, provided your ZIL is fast enough (cheap old SSDs are not a help here).
Click to expand...

ZFS raidz + ACL. Just mapping it as \\oizfs01\data\movies
 
x-cimo said:
There is something weird for my SMB permission.

I created a Folder with SMB access to root, and created a "allusers group" on OI. Everything is default.

Then from windows I went as root with SMB, and created 3 user folder inside, such as:

\\OI\Users\User1
\\OI\Users\User2
\\OI\Users\User3

For each of the UserX folder I went into advance permission, and removed the inheritance, removed the "OI\AllUsers" group, and added OI\UserX user with full permission.

This should effectivly let All users go to \\OI\Users\ and see all user folder, however it shouldn't let them do anything to these folder except if it's their own folder.

However, I seem that all users can still do anything to any of thoses folder even if they have no permission (as shown in windows permission tab).

The only permission on the UserX folder is: root and UserX ..

Any idea?
Click to expand...

All you need is:
\\OI\Users: group everyone@ = read + root=full without inheritance
\\OI\Users\User1 : User1=full or modify with inheritance
\\OI\Users\User2 : User2=full or modify with inheritance
\\OI\Users\User3 : User3=full or modify with inheritance

About inheritance of a folder: it affects newly created files or folders within.
If you have full access to Users with inheritance, you have access to all User folder,
but not folders below.

Check/set these ACL best with napp-it ACL extension where you have full control
of user ACL, inheritance and ACL order (setup of local user-ACL is free)
 
You must log in or register to reply here.
Back
Top