OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

[Varg]

So I'd format the OI with ZFS and then when install Debian on top of that it's ok to format the vm as ext3/4 or whatever? To me it seems unelegant.

That's correct. If you want to have zfs penetrating directly into the vm, you would have to use zones instead of vmware.

 

[smellyeagle]

That's correct. If you want to have zfs penetrating directly into the vm, you would have to use zones instead of vmware.

Would I be able to keep a small partition for the Linux OS and then mount a native ZFS NFS share on top of it?

 

[TheLastBoyscout]

So I'd format the OI with ZFS and then when install Debian on top of that it's ok to format the vm as ext3/4 or whatever? To me it seems unelegant.

That's the way this SAN works. Remember, that you have to have some sort of file system that ESXi can work with and allocate your disks on (1). ESXi uses that allocated space as an empty virtual hard disk for the new VM. When you start the VM and install on OS, you install another file system (2). Even just running VMware Workstation on on you favorite OS works this way.
The only option to avoid this is to pass the HDD directly to the VM so that ESXi can't see it.

-TLB

 

[e2g]

I installed only one hard drive (one of the two 2.5" 4K drives i recently bought. None of the 2TB drives plugged in). I install OI, and after the few screens where it gets to load the OS, it doesn't. It keeps restarting my computer. So, i swapped the first hard drive with the second one (again, I bought 2 4K 2.5" laptop drives to be used as a rpool mirror) and same thing happened.

Just to follow up if anyone cares. I got one of my Samsung F4 drives and installed OI without a problem. When I restart/power cycle the machine, it actually boots into the OS rather than keeps restarting or does not boot into grub>

Again, the only thing I can see that is different is that these 2.5" 4K drives report 4096 as their stripe size whereas my other drives (my Samsung f4 drives and a non-4K drive, all report sector size 0)

Do you think I got two bad drives?

 

[_Gea]

pobably not
But why do you want to use 2 TB drives for booting?
You cannot create datapools on it

Do you have any small non 4k disks to check against

 

[_Gea]

So I'd format the OI with ZFS and then when install Debian on top of that it's ok to format the vm as ext3/4 or whatever? To me it seems unelegant.

Maybe you missed the problem you want to solve with this solution

About the problem:
If you want to virtualize on a barebone type-1 system like ESXi, all hardware for guests
is usually emulated, (cpu, ram, video, nics, disk-controller and disks)

Disks for guests are emulated by ESXi. In reality they are files either on a local ESXi datastore
or a SAN Server. Because of minimal to non-existent storage comfort of ESXi itself on local datastores,
a lot of persons use external SAN storage to store these virtual disks. ZFS-SAN server adds a lot of functionality
like unlimited snaps without delay, better performance, replication, easy access to copy/move/clone/restore/backup and
a lot of other SAN features.

Mostly a SAN server is a separate server system. SAN-server and VM-Server are connected with expensive
FC or IP Adapters and switches to deliver performance.

The All-In-One concept integrates this SAN server and the hardware switch as a virtual SAN with a virtual Switch
together with internal virtual high-speed interconnect based on virtual nics.

From a admins/ user view, the All-In-One works exactly like a conventional configuration with a ESXi-Server,
a separate SAN-Server and a comfortable vlan-hardware-switch to connect them.
But all is software


And to come back to your Debian guest
Didn't you like the idea to format as ext2/3 and to have all ZFS goodies below

 

[e2g]

pobably not
But why do you want to use 2 TB drives for booting?
You cannot create datapools on it

Do you have any small non 4k disks to check against

I don't want to use 2TB drives for booting. Here is the scenario:

I have 5 (well 6, but not using the 6th one yet) 2TB Samsung F4 drives. For my boot drive, I had a 500GB Desktop drive (Western Digital Blue non-4K drive). Why did I get such a big hard drive as my boot drive? Well, I purchased it for $25, and I am sure I can use that extra space for temporary files or something else. Then i wanted to have two drives for my boot drive in a mirrored config.I also didn't want to fill any of the hot swappable bays in my Norco. So i purchase 2.5" sized drives. I bought 500GB 2.5" drives as the price difference between smaller sized drives was very small ($5?).

So now I have my 5 samsung drives to be configured in raidz, and 2 laptop drives (4K drives) to be configured in mirrored mode. I am just trying to install OI on one of those drives. Since I couldn't install it on any of the two 2.5" drives I just bought (last page I mentioned it kept going to "grub>" or the latest now is that it restarts constantly), I wanted to know if it has anything to do with OI installing to a 4K drive. So as a test this morning/evening I used one of my Samsung f4 drives to see if would install just fine.

it installed just fine.

Another test I did (right now), is I just installed Windows 7 on one of the 2.5" drives just fine. That tells me nothing is wrong with this drive. It is looking more and more with OI. I am about to install Ubuntu to see if it will install correctly on the drive as well.

Does this make a little more sense?

To top things off, currently, I am using IPMIviewer on a windows 7 machine to control and install things on my server. With an OI installation on my server, I could not get my mouse to work over the viewer. I had to have one plugged into the server. My keyboard worked however. I just installed Windows 7 on my server..and mouse and keyboard work just fine.

 

[smellyeagle]

And to come back to your Debian guest
Didn't you like the idea to format as ext2/3 and to have all ZFS goodies below

I think my problem was conceptualizing file systems. I wanted all the ZFS goodies and thought I'd lose them by formatting a guest with ext. I realized that this isn't an issue in the slightest--I can retain all the neat features of ZFS with no penalty.

 

[_Gea]

The only similar grub problem i have heard about is a problem with usb that must be disabled completely with OI.
Your problem is in any way new to me. Does anyone other has 4k boot-problems with OI (I do not have free 4k disks)?

or
Could you try another controller to boot from

 

[smellyeagle]

c204 based motherboards like the supermicro X9SCM have 2x 6gbps and 4x 3gbps sata ports. is it possible to pass through the 3gbps ports to OI and use the 6gbps as a boot drive for esxi?

 

[danswartz]

I believe the 6gb and 3gb ports are on the same controller, so no luck

 

[e2g]

The only similar grub problem i have heard about is a problem with usb that must be disabled completely with OI.
Your problem is in any way new to me. Does anyone other has 4k boot-problems with OI (I do not have free 4k disks)?

or
Could you try another controller to boot from

I just install Ubuntu no problem using onboard SATA ports. I connected them to a different controller (the onboard LSI controller) and tried to install OI, and same thing: keeps restarting.

I just disabled USB ports, and same behaviour.

 

[smellyeagle]

I believe the 6gb and 3gb ports are on the same controller, so no luck

How do I confirm they are on the same controller? I don't have the hardware to test.

 

[_Gea]

I just install Ubuntu no problem using onboard SATA ports. I connected them to a different controller (the onboard LSI controller) and tried to install OI, and same thing: keeps restarting.

I just disabled USB ports, and same behaviour.

Then i suppose there is really a incompatibility

 

[e2g]

great.....thanks a lot for all your suggestions.

 

[mrgstiffler]

great.....thanks a lot for all your suggestions.

You could try something else like Nexenta.

 

[danswartz]

How do I confirm they are on the same controller? I don't have the hardware to test.

download mobo manual and read it?

 

[e2g]

You could try something else like Nexenta.

Yeah, I thought some more about using a different OS. I guess I will be able to export my pool from OI and import to Nexenta just fine..or I may run into some trouble as it seems the nexenta is running an older solaris kernel?

yup, can't do it. I may give Nexenta core a go next time. In the mean time, I am returning the hard drives as one was showing signs up failure. I will just stick to using one drive for now.

 

[_Gea]

Yeah, I thought some more about using a different OS. I guess I will be able to export my pool from OI and import to Nexenta just fine..or I may run into some trouble as it seems the nexenta is running an older solaris kernel?

yup, can't do it. I may give Nexenta core a go next time. In the mean time, I am returning the hard drives as one was showing signs up failure. I will just stick to using one drive for now.

current OpenIndiana 151 is already based on Illumos just like Nexenta will be
with next version. They are about to share the same Kernel.

I would not do new installations with current Nexenta based on OpenSolaris 134


[ame="http://en.wikipedia.org/wiki/Illumos"]Illumos - Wikipedia, the free encyclopedia@@AMEPARAM@@/wiki/File:Illumos_logo.png" class="image"><img alt="Illumos logo.png" src="http://upload.wikimedia.org/wikipedia/en/a/af/Illumos_logo.png"@@AMEPARAM@@en/a/af/Illumos_logo.png[/ame]

 

[jwinsor566]

Hello Everyone,

I have implemented an all in one box using your guide and Nexenta 3.0.5. I set up a RAIDz1 with 3 SAS Seagate Constellation 1 TB drives and Added a Corsair force 40GB SSD for Log and 40 GB Corsair Force SSD for Cache.

I then created an NFS share on that volume for ESX to store VM's. My issue is that all the vm's using this datastore have Write Latency in the thousands of milliseconds!!!!

Can anyone point me in the right direction as to how to troubleshoot this latency? Has anyone else run into this issue using the all in one box? I am wondering if there is an issue in how the all in one box works as far as possible going in circles to write and read data.

Thanks ahead of time for any info!!

 

[danswartz]

ESXi does all writes synchronously. Do this for that share 'zfs set sync=disabled XXXX'.

 

[jwinsor566]

ESXi does all writes synchronously. Do this for that share 'zfs set sync=disabled XXXX'.

Thanks Dan,

Do you need to reboot the Nexenta box in order for that to take effect?

Also would there be any improvement if I switched to Jumbo Frames MTU for VMWare connections or would that not matter in Your opinion?

 

[gremlin190]

You guys installing apcupsd, how did you get it to work on openindiana? i can't seem to get the ugen driver to install. I try to go thru package manger and it just says everything is up to date on install.
When I try to run apctest, I get an error saying FATAL ERROR in generic-usb.c at line 636

 

[danswartz]

Thanks Dan,

Do you need to reboot the Nexenta box in order for that to take effect?

Also would there be any improvement if I switched to Jumbo Frames MTU for VMWare connections or would that not matter in Your opinion?

AFAIK, jumbo frames is a minor win in certain corner cases. The big lose here is waiting for the disk write to post. I don't think you need to reboot, no.

 

[danswartz]

You guys installing apcupsd, how did you get it to work on openindiana? i can't seem to get the ugen driver to install. I try to go thru package manger and it just says everything is up to date on install.
When I try to run apctest, I get a segfault and an error saying FATAL ERROR in generic-usb.c at line 636

I gave up and made apcupsd a network connector to a different box (my pbx) which has has the usb cable on it. OI seems flaky for USB in general.

 

[gremlin190]

You guys installing apcupsd, how did you get it to work on openindiana? i can't seem to get the ugen driver to install. I try to go thru package manger and it just says everything is up to date on install.
When I try to run apctest, I get an error saying FATAL ERROR in generic-usb.c at line 636

I may be off when I say I can't install the ugen, I have tested this software with linux and it worked great with my ups.

Prtconf -v

input, instance #2
Driver properties:
name='pm-components' type=string items=3 dev=none
value='NAME= hid2 Power' + '0=USB D3 State' + '3=USB D0 State'
Hardware properties:
name='driver-minor' type=int items=1
value=00000000
name='driver-major' type=int items=1
value=00000002
name='low-speed' type=boolean
name='usb-product-name' type=string items=1
value='TRIPP LITE SMART1050SLT '
name='usb-vendor-name' type=string items=1
value='Tripp Lite '
name='usb-serialno' type=string items=1
value='9610CY0SM637500364'
name='usb-raw-cfg-descriptors' type=byte items=34
value=09.02.22.00.01.01.00.e0.00.09.04.00.00.01.03.00.00.00.09.21.10.01.00.01.22.61.04.07.05.81.03.08.00.28
name='usb-dev-descriptor' type=byte items=18
value=12.01.10.01.00.00.00.08.ae.09.12.30.0c.02.01.02.03.01
name='usb-release' type=int items=1
value=00000110
name='usb-num-configs' type=int items=1
value=00000001
name='usb-revision-id' type=int items=1
value=0000020c
name='usb-product-id' type=int items=1
value=00003012
name='usb-vendor-id' type=int items=1
value=000009ae
name='compatible' type=string items=9
value='usb9ae,3012.20c' + 'usb9ae,3012' + 'usbif9ae,class3.0.0' + 'usbif9ae,class3.0' + 'usbif9ae,class3' + 'usbif,class3.0.0' + 'usbif,class3.0' + 'usbif,class3' + 'usb,device'
name='reg' type=int items=1
value=00000001
name='assigned-address' type=int items=1
value=00000003
Device Minor Nodes:
dev=(223,4)
dev_path=/pci@0,0/pci15d9,605@1d/hub@1/input@1:hid_-124_4
spectype=chr type=minor
dev_link=/dev/usb/hid5


prtconf -D


pci15d9,605, instance #0 (driver name: ehci)
hub, instance #0 (driver name: hubd)
device, instance #1 (driver name: usb_mid)
mouse, instance #0 (driver name: hid)
keyboard, instance #1 (driver name: hid)
pci8086,3b42, instance #2 (driver name: pcieb)
pci1000,3140, instance #0 (driver name: mpt)
sd, instance #4 (driver name: sd)
sd, instance #5 (driver name: sd)
sd, instance #6 (driver name: sd)
sd, instance #7 (driver name: sd)
sd, instance #8 (driver name: sd)
sd, instance #9 (driver name: sd)
sd, instance #10 (driver name: sd)
sd, instance #11 (driver name: sd)
pci8086,3b4a, instance #3 (driver name: pcieb)
pci15d9,605, instance #0 (driver name: e1000g)
pci8086,3b4c, instance #4 (driver name: pcieb)
pci15d9,605, instance #1 (driver name: e1000g)
pci15d9,605, instance #1 (driver name: ehci)
hub, instance #1 (driver name: hubd)
input, instance #0 (driver name: ugen)
mouse, instance #3 (driver name: hid)

 

[s0rce]

You guys installing apcupsd, how did you get it to work on openindiana? i can't seem to get the ugen driver to install. I try to go thru package manger and it just says everything is up to date on install.
When I try to run apctest, I get an error saying FATAL ERROR in generic-usb.c at line 636

don't know about OI but I installed it on solaris 11 express and it worked. Did you enable-usb when you ran configure?

 

[gremlin190]

yeah man sure did

/etc/opt/apcupsd/sbin/apctest


2011-07-21 14:51:02 apctest 3.14.8 (16 January 2010) sun
Checking configuration ...
Attached to driver: usb
sharenet.type = DISABLE
cable.type = USB_CABLE

You are using a USB cable type, so I'm entering USB test mode
mode.type = USB_UPS
Setting up the port ...
apctest FATAL ERROR in generic-usb.c at line 636
Cannot find UPS device --
For a link to detailed USB trouble shooting information,
please see <http://www.apcupsd.com/support.html>.
apctest error termination completed

 

[smellyeagle]

As a way to save a little bit of money on a home setup, would a Silicon Image SiI3124 based card work well hooked up to a 3gb last gen ssd for an OI boot drive? I'm looking at using a c202/c204 mb with a xeon e3 cpu with the onboard sata used for 4 wd green drives.

 

[davros123]

don't know about OI but I installed it on solaris 11 express and it worked. Did you enable-usb when you ran configure?

same here...installed from source with no issues on OpenSolaris and Solaris 11 Express for an APC UPS over ethernet and serial.

 

[TheLastBoyscout]

You guys installing apcupsd, how did you get it to work on openindiana? i can't seem to get the ugen driver to install. I try to go thru package manger and it just says everything is up to date on install.
When I try to run apctest, I get an error saying FATAL ERROR in generic-usb.c at line 636

It's been a while, but I vaguely remember running into a similar problem on OI 148b (among other things since I'm a total UNIX noob ...).
IIRC, I ended up going into the package manager and searched for USB; enabled a few unselected packages and also enabled gcc3. After a few more systems, I ended up enabling only one of thee missing USB packets, but I don't have access to the systems right now.

-TLB

 

[anthem00]

As a way to save a little bit of money on a home setup, would a Silicon Image SiI3124 based card work well hooked up to a 3gb last gen ssd for an OI boot drive? I'm looking at using a c202/c204 mb with a xeon e3 cpu with the onboard sata used for 4 wd green drives.

I don't think you need an SSD for OI/SE/Nexenta. It's not i/o intensive so any drive would work (even an old sata drive is fine). You would be better served by using an ssd as the read cache drive (though I would hesitate to put old ssd drives as ZIL cache)

 

[anthem00]

AFAIK, jumbo frames is a minor win in certain corner cases. The big lose here is waiting for the disk write to post. I don't think you need to reboot, no.

I second this. For home use, I wouldn't even bother with jumbo frames. It's a very minor win and doing so means you have to have jumbo capability from source to end (through all switches, etc). The performance improvements are minimal unless you're really pushing performance.

In an enterprise environment where you have ESXi/10gbE, multiple pNICS, Vswitches, etc then it /might/ be worth it to enable Jumbo's to gain the few percentage bandwidth involved there.

 

[Freak1]

Now i tried adding a user in XP pro didn't get any security tap and windows 2003 enterprice its the same as windows 7 enterprice it can't find the user.

Now i joined it to my domain loged in a windows 7 enterprice as domain administrator. But that user dosen't have access to the shares on the ZFS server, how do i give the administrator of the domain rights?

 

[anthem00]

Hey Gea,

I just found this thread (and site) and think its great work you've done here. I've been using zfs for several years and various platforms (OpenSolaris, Nexenta, and now SE11). Anyhow, since I'm very familiar with zfs and use the command line, the napp-it gui is not quite as helpful for me as I'm sure it is for people new to *nix/zfs. . .

However, I did drop a nice 100euro donation to napp-it.org as I did install napp-it onto one of my test bed machines (for my own personal use). Eventhough I don't /need/ it per se, it is 100x times easier while I'm doing some benches and need to rapidly create/destroy pools (obviously not necessary once into production). But clicking checkboxes is a lot easier than issuing the zfs create with a list of c0t5000C5003437E47Bd0, etc, etc. . . So, that alone was worth my donation to you guys .

A couple of my zfs notes for people as I'm still reading through the length of this thread -
-stripe multiple vdevs to get performance. Whether thats striping mirrors or raidz/2 vdevs
-pools are expandable by vdevs (groups of disks). Seems like many people still misunderstand this. You can keep expanding your pool, just by vdevs (groups of disks, just not 1 disk).
-for your all-in one, I would argue that you don't need fast raid 10 vdevs. Chances are the all-in-one home/media server is not hungry for iops and having it shared on the raidz2 pool is more than adequate. This is especially so with more memory and L2ARC.
-If you really want to speed things up, look at SSD L2ARC for reads. This is combination of vmdk's on the main tank works very very well for home/media setups.
-For writes (which is almost never the case in the home/media market), a fast ZIL works wonders. We've deployed a few DDRdrives and they are blistering fast for iops random writes.
-SAS Expanders work fine if you know which combinations work. We have had the Supermicro E1/E2 chassis series with SAS expanders (LSI based) paired with LSI1068/2008 without any issues (except for a LSI firmware v8/v9 recently where the controller reported 2x/4x the amount of actual drives - a firmware update was released to fix that). We've only used the HP expander once and didn't bother as we have had issues with it playing nice with others. So I will wholeheartedly recommend the SuperMicro chassis 846/847 with expanders and LSI IT (9211/2008) controllers working with ESXi, ZFS variants. That being said - I will recommend that you get the SAS versions of drives to use with an expander. We have come across SATA connected to SAS expanders where all the drives stopped responding on a scrub. It was pretty ugly. Its only happened once and we're not sure if it was related to expanders, sata->sas, or what not - but the SAS versions of the drives have been rock solid (albeit 30% more costly),
-ZFS will use EVERY bit of memory you want to throw at it. For home media use, I agree that 4gb is probably fine if you don't plan on using dedup. However, if you have ESXi hosts, then throw whatever memory you have into the machine and it will effectively use everything you give it (another reason why VM ESXI hosts on the regular pool are fine with adequate memory). With an SSD cache as L2ARC, you can have a lot of read cache to play with. 48GB of memory with a pair of 240gb SSD cache gives you essentially 500gb of super fast cache. . .


This bench isn't that accurate as it involves some read cache devices and a ZIL, but this is a 3 x vdev (raidz2 8 drives). Kind of interesting to decode when you have SSD with paired with regular drives.


write 10.24 GB via dd, please wait...
time dd if=/dev/zero of=/tank/dd.tst bs=1024000 count=10000

10000+0 records in
10000+0 records out

real 10.1
user 0.0
sys 4.1

10.24 GB in 10.1s = 1013.86 MB/s Write


read 10.24 GB via dd, please wait...
time dd if=/tank/dd.tst of=/dev/null bs=1024000

10000+0 records in
10000+0 records out

real 2.5
user 0.0
sys 2.5

10.24 GB in 2.5s = 4096.00 MB/s Read


Bonnie
tank
File 48G
Seq-Wr-Chr 103 MB/s
Seq-Write 627 MB/s
Seq-Rewr 378 MB/s
Seq-Rd-Chr 97 MB/s
Seq-Read 1014 MB/s
Read Seeks 2846.5/s
Files 16
Seq-Create ++++/s (out of range)
Rnd-Create 32630/s

 

[_Gea]

hello anthem00

Thank you again for your donation.
napp-it is currently done in my free time and it helps to do further development.

Thank you also for your insights

 

[_Gea]

Now i tried adding a user in XP pro didn't get any security tap and windows 2003 enterprice its the same as windows 7 enterprice it can't find the user.

Now i joined it to my domain loged in a windows 7 enterprice as domain administrator. But that user dosen't have access to the shares on the ZFS server, how do i give the administrator of the domain rights?

- you have to add a user to Solaris not to winXP
- you must connect as root or a SMB user mapped to root to change ACL share-settings from Windows
(does not work with every Windows version like home or ultimate edition)
- or set permissions with napp-it ACL extension for local users (under development)

 

[kegler]

Hi gea


ive joined my oi to my AD, no issue for windows7 or windows server 2008 in or not within the domain. windows environment all ok

but for mac os lion which i got last night was only able to access to the root folder of my smb share. can't go any further.... keep prompting no access to the folders in smb share

tried root or a user created in oi or even windows domain admin account

any idea?

 

[_Gea]

no Lion here yet

 

[gremlin190]

It's been a while, but I vaguely remember running into a similar problem on OI 148b (among other things since I'm a total UNIX noob ...).
IIRC, I ended up going into the package manager and searched for USB; enabled a few unselected packages and also enabled gcc3. After a few more systems, I ended up enabling only one of thee missing USB packets, but I don't have access to the systems right now.

-TLB

Seems it was mostly that I was trying a different brand ups then APC. Plugged in a APC and no problems. Oddly enough the other one worked great in Ubuntu. Oh well, I'm happy now