OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

aha, I fixed the hostname issue:

I removed 12 the line:

PARAM_REQUEST_LIST=1,3,6,12,15,28,43

and changed REQUEST_HOSTNAME=no to yes

in /etc/default/dhcpagent

and added the file /etc/hostname.rge0 (my inteterface)
which contained inet hostname

it works now!
 
aha, I fixed the hostname issue:

I removed 12 the line:

PARAM_REQUEST_LIST=1,3,6,12,15,28,43

and changed REQUEST_HOSTNAME=no to yes

in /etc/default/dhcpagent

and added the file /etc/hostname.rge0 (my inteterface)
which contained inet hostname

it works now!
 
_Gea said:
currently, auto-service has 1min and 15min timeframe as a root-cronjob
the autojob service itself is only reading the os-disk which you cannot use
for energy management ever.

Datapools are only affected by autojobs if you have set an status, alert, snap, scrub or replication job.
-> set these jobs to a daily timeframe

another thing is the Solaris fault management, see
http://nexenta.org/boards/1/topics/1414#message-1450
if its running, your disk will never sleep but Solaris is always up to date with pool errors.

Gea
Click to expand...

Thanks, working fine now.
One of the jobs was wrongly configured.
And fmd was interfering too, this fixed it:

Code: 
> You should also be able to configure the scan interval by adding
> an entry to /usr/lib/fm/fmd/plugins/disk-transport.conf file. i.e.
>
> setprop interval 24h
 
Huh? No, go to the command line and do:

smartctl -a /dev/rdsk/c3t0d0 -T permissive -d scsi -H

look at the command options to see what you can dig for...
 
go to the command line and do:

smartctl -a /dev/rdsk/c3t0d0 -T permissive -d scsi -H

look at the command options to see what you can dig for...
 
Metaluna said:
To get the best performance out of 4k drives, I would destroy the pool and then reformat it using a FreeBSD/ZFSGuru liveCD, which can create properly 4k-aligned pools. Then you can import the pool back into OI and upgrade it, and it will maintain the 4k alignment. I've done this a couple of times and it works fine so far (knock on wood).

My only caveat is that I don't know if _Gea or any other ZFS/OpenSolaris experts have officially "blessed" this technique, or if there are any hidden problems with it.
Click to expand...

I just want to confirm with everyone else as I just built a server with 6 Samsung F4 and now I am reading proper alignment for these 4k drives hasn't made its way into OpenIndiana and the like? So doing so above is the work around..?
 
Is there a way to check that the imported 4k drives alignment is correct? With ReadyNas systems I think that you divided the sectors by 16 to confirm correct alignment. is there a similar way on OI

I have imported my 4k drives from a FreeNas V13 system and all looks to be ok but I don't feel that I can trust it at this stage. after upgrading to V28 in Napp-it the pools show that they are V28 but when you look in "Poolinfo" they still show as V13

Even though the import looks to have worked ok from FreeNas I am still tempted to scrub the drives and try the ZFSGuru route
 
danswartz said:
go to the command line and do:

smartctl -a /dev/rdsk/c3t0d0 -T permissive -d scsi -H

look at the command options to see what you can dig for...
Click to expand...

This is the output, i only see temperature.


smartctl 5.40 2010-10-16 r3189 [i386-pc-solaris2.11] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Serial number: xxxxxxxxxx
Device type: disk
Local Time is: Sun Jun 12 11:33:54 2011 CEST
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
SMART Health Status: OK

Current Drive Temperature: 26 C

Error Counter logging not supported

[GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
No self-tests have been logged
Long (extended) Self Test duration: 17460 seconds [291.0 minutes]
 
I found this in the changelog option: send alert via TLS ex for Googlemail (TLS module must be installed manually)

Can anyone tell me how i set that up?
 
I found this in the changelog option: send alert via TLS ex for Googlemail (TLS module must be installed manually)

Can anyone tell me how i set that up?
 
I did say you might have to look at the various command line options to see what you can dig out of the SMART info...
 
e2g said:
I just want to confirm with everyone else as I just built a server with 6 Samsung F4 and now I am reading proper alignment for these 4k drives hasn't made its way into OpenIndiana and the like? So doing so above is the work around..?
Click to expand...

I haven't checked to verify if the latest OI and SE11 are still doing it the "wrong" way, but as far as I know the problem has not been fixed. If a drive lies about its sector size and reports 512b, most ZFS implementations (even the FreeBSD one) will align the pool improperly. The reason FreeBSD saves you is not related to ZFS directly, but because their GEOM I/O framework allows you to create "virtual" drives that report true 4K sectors. It's not really a bug in ZFS. If a drive lies about its sector size, there's no way for ZFS to know what the real setting should be.

Here are some references describing the issue:
http://www.solarismen.de/archives/4-Solaris-and-the-new-4K-Sector-Disks-e.g.-WDxxEARS-Part-1.html
http://www.solarismen.de/archives/5-Solaris-and-the-new-4K-Sector-Disks-e.g.-WDxxEARS-Part-2.html

http://opensolaris.org/jive/thread.jspa?threadID=135737

The FreeBSD workaround:
http://blog.monsted.dk/?q=node/1

And an alternative workaround using a hacked zpool binary:
http://www.solarismen.de/archives/12-Modified-zpool-program-for-newer-Solaris-versions.html

The downside of the FreeBSD workaround is that you have to be using a storage controller supported by FreeBSD, otherwise it wont be able to see the drives. For small pool sizes (5-6 drives) you can probably use your motherboard controller then boot FreeBSD from a USB-connected CD drive or USB stick. For larger pools, LSI 1068e-based controllers are the the easiest and will work with just about any OS version without swapping cables and controllers around. FreeBSD may now also have LSI 2008 support but I haven't tried it.

Also, if you're using the ZFSGuru LiveCD, do NOT format the drives with GPT partitions, as Solaris doesn't like it and won't see the pools. Use the GEOM option in the webGUI.


Tropheus said:
Is there a way to check that the imported 4k drives alignment is correct? With ReadyNas systems I think that you divided the sectors by 16 to confirm correct alignment. is there a similar way on OI
Click to expand...

You can check the alignment with

zdb -C poolname | grep ashift

If the ashift value is 12, (2^12 = 4096) then all is well. If it's 9 (2^9 = 512) then the pool will be misaligned. It should be okay to use ashift = 12 even on true 512b drives, as this will allow you to use 4k drives to replace/upgrade the pool in the future.
 
Last edited:
Interesting....

zdb -C reports my imported FreeNas 4k drives as 12, So does that mean that the alignment is ok?
 
Last edited:
danswartz said:
I did say you might have to look at the various command line options to see what you can dig out of the SMART info...
Click to expand...

After some digging around in the help file i found tries changing scsi to sat and it shows much more now. Thanks a lot for your help. And I'm sorry for the questions, ZFS and Linux is very new to me, and i want to make sure i got it right before i set the system in "production".
 
Freak1 said:
I found this in the changelog option: send alert via TLS ex for Googlemail (TLS module must be installed manually)

Can anyone tell me how i set that up?
Click to expand...

I tries google and reading some pages in this thread. But i can't really find anything i can understand as how i do this. And how to test if it is working.
 
The patched zpool executable works fine for creating an ashift=12 pool.
Its much easier than screwing around with freebsd, moving your array and upgrading it.
 
Loto_Bak said:
The patched zpool executable works fine for creating an ashift=12 pool.
Its much easier than screwing around with freebsd, moving your array and upgrading it.
Click to expand...

Would this hacked binary work on OpenIndiana as this is what I chose to install and having a hard time figuring why I can't move my mouse on IPMI View but that's another story.

Seems like it would since he mentions he had to pull the original src files from the openindiana website but may be wrong.
 
Loto_Bak said:
The patched zpool executable works fine for creating an ashift=12 pool.
Its much easier than screwing around with freebsd, moving your array and upgrading it.
Click to expand...


I agree the patched zpool is easier if you can find the right binary for your OS. When I was first researching it, I remember reading some debate over whether the patch was good, and some people were claiming it made their pool unstable. But it sounds like all of that is resolved now and a lot of people have used it successfully.
 
Now, with regard to Ashift, if I already have a pool of ashift=9, can I modify it to be ashift=12 for the 4k sector drives? Additionally, what kind of performance boost will I gain and is it really worth it?
 
unclerunkle said:
Now, with regard to Ashift, if I already have a pool of ashift=9, can I modify it to be ashift=12 for the 4k sector drives? Additionally, what kind of performance boost will I gain and is it really worth it?
Click to expand...

Unfortunately I don't think it's possible to modify an existing pool that way :(

As for performance, it probably depends on the type of accesses you do. From some of the benchmarks I've seen I get the impression that the performance boost is somewhere around 20%. So, it's significant, but not earth-shattering. I suspect that long sequential reads/writes will be less affected. Also, IIRC, if the number of drives in your pool is in the "sweet spot" (something like 5 drives for raidz, 6 drives for raidz2) there will be less performance loss because of the way the stripe sizes work out or something like that.
 
I finally created a zpool with ashift=12
I have 6 Samsung F4s with raidz (not the sweet spot but trying to maximize capacity here) and was getting 500MB/s sequential Write....530MB/s sequential reads

I tried to run the benchmark on the napp-it GUI by clicking start, but it never refreshes with the benchmark info after refreshing the page.

Also, Here is crystalDisk benchmark from my Windows 7 box
serversmbperformance.png
 
Anyone want to recommend 1TB and 2TB hard drives for ZFS? I already have a 1TB Samsung F3 and 1.5 Caviar Green that I could use, but my desktop need at least one conventional HD and I would prefer it to be 7200rpm. Right now I am thinking 2x1TB mirror for vm datastores and 3x2TB RAIDZ for media and files. I don't know if I should avoid 4K drives but they are really cheap.
 
should I stay away from 4k drives? So far the only 2TB I have found that is not 4k is the 5k3000 unless I missed something.
 
I honestly don't know. No experience with them, but I get the impression concerns about performance are overblown - unless you have some kind of high-speed, streaming requirement, it's hard to imagine it being a big deal, if the cost savings is good enough.
 
shiznit said:
should I stay away from 4k drives? So far the only 2TB I have found that is not 4k is the 5k3000 unless I missed something.
Click to expand...

I'm running 4x 2TB 5k3000 in my little ZFS server and they have been great (other than the 2 DOA drives from newegg)
 
Does anyone have a sample AppleVolumes.default file I can use. I use to have Time Machine working correctly, but after multiple rebuilds, I can't seem to get it going again.

Here's what my AppleVolumes.default looks like:

:DEFAULT: options:acls,upriv perm:0777
# The "~" below indicates that Home directories are visible by default.
# If you do not wish to have people accessing their Home directories,
# please put a pound sign in front of the tilde or delete it.
# End of File
/nas/macbook1 macbook1

When I open Time Machine, it doesn't show the disk. AFP is enabled for the macbook1 folder, and netatalk is installed. Also, are there any afp/avahi logs to look at?

Gea, thanks for all the help.
 
I've been using iSCSI and the global SAN initiator (free) for Time Machine backup - it works like a champ. (Doesn't answer your question though, but is another way that works well)
 
capt_zman said:
Does anyone have a sample AppleVolumes.default file I can use. I use to have Time Machine working correctly, but after multiple rebuilds, I can't seem to get it going again.

Here's what my AppleVolumes.default looks like:

:DEFAULT: options:acls,upriv perm:0777
# The "~" below indicates that Home directories are visible by default.
# If you do not wish to have people accessing their Home directories,
# please put a pound sign in front of the tilde or delete it.
# End of File
/nas/macbook1 macbook1

When I open Time Machine, it doesn't show the disk. AFP is enabled for the macbook1 folder, and netatalk is installed. Also, are there any afp/avahi logs to look at?

Gea, thanks for all the help.
Click to expand...

I'm a noob in this area, but I think at a minimum you at least need to have 'options:tm' appended to the last line, i.e.
/nas/macbook1 macbook1 options:tm


Here's how napp-it configured my AppleVolumes.default (haven't tested it yet though)

Code: 
:DEFAULT: options:upriv,usedots

# The "~" below indicates that Home directories are visible by default.
# If you do not wish to have people accessing their Home directories,
# please put a pound sign in front of the tilde or delete it.
#~

# End of File

/tank/Backups/timemachine timemachine options:tm allow:@staff volsizelimit:10000
 
capt_zman said:
Does anyone have a sample AppleVolumes.default file I can use. I use to have Time Machine working correctly, but after multiple rebuilds, I can't seem to get it going again.
Click to expand...
Code: 
:DEFAULT: options:acl,usedots,invisibledots,upriv perm:0777
/storage/TimeMachine TimeMachine options:tm allow:@staff volsizelimit:600000
/storage/Backups Backups  allow:@staff
The 2nd line is what you need to look at, the part "options:tm" is all you need to add to the end your line.

[edit] Looks like Metaluna beat me to it.
 
I have been trying to get napp-it to connect to my domain but without luck.

This is what i did:
name of domain "Domain name"
ad-domain-server "DNS name"
domainadmin "User"
password "Password"

I tried setting the nameserver ip from my router to AD server in /etc/resolv.conf, didn't work, tried restarting OI after and then its back to the router again, and still doesn't work.

I'm not sure what to check for in krb5.conf, but time is set right.

Here is the error message
Current state of SMB/CIFS Server: online
Current membership: workgroup [WORKGROUP]

Try to join domain :

step 1: Timesync with ad-server adds: Error : node name or service name not known
14 Jun 17:05:09 ntpdate[1979]: can't find host adds

14 Jun 17:05:09 ntpdate[1979]: no servers can be used, exiting
step 2: set dns entry in /etc/resolv.conf:
step 3: set /etc/krb5/krb5.conf:
step 4: try to join via: smbadm join -u administrator freak - please wait ....



Joining freak ... this may take a minute ...


failed to find any domain controllers for freak
 
Question,

Does Opensolaris provide HA clustering? I have read that there is an HA open project but wondering if anyone has experience with setting it up.
 
Freak1 said:
I have been trying to get napp-it to connect to my domain but without luck.

This is what i did:
name of domain "Domain name"
ad-domain-server "DNS name"
domainadmin "User"
password "Password"

I tried setting the nameserver ip from my router to AD server in /etc/resolv.conf, didn't work, tried restarting OI after and then its back to the router again, and still doesn't work.

I'm not sure what to check for in krb5.conf, but time is set right.

Here is the error message
Current state of SMB/CIFS Server: online
Current membership: workgroup [WORKGROUP]

Try to join domain :

step 1: Timesync with ad-server adds: Error : node name or service name not known
14 Jun 17:05:09 ntpdate[1979]: can't find host adds

14 Jun 17:05:09 ntpdate[1979]: no servers can be used, exiting
step 2: set dns entry in /etc/resolv.conf:
step 3: set /etc/krb5/krb5.conf:
step 4: try to join via: smbadm join -u administrator freak - please wait ....



Joining freak ... this may take a minute ...


failed to find any domain controllers for freak
Click to expand...

There's a bug/design flaw in napp-it that takes the dns entry you input for your domain server and replaces the ip in resolv.conf with that dns name. You'll need to place the ip in resolv.conf again and run the last step manually.

You can use this page, check each step to verify and you should be good to go:

http://blogs.oracle.com/timthomas/entry/configuring_the_opensolaris_cifs_server
 
ChrisBenn said:
I've been using iSCSI and the global SAN initiator (free) for Time Machine backup - it works like a champ. (Doesn't answer your question though, but is another way that works well)
Click to expand...

Thanks for all the help guys. I tried the iSCSI option and it worked great, so we'll stick with that.
 
Astronot said:
There's a bug/design flaw in napp-it that takes the dns entry you input for your domain server and replaces the ip in resolv.conf with that dns name. You'll need to place the ip in resolv.conf again and run the last step manually.

You can use this page, check each step to verify and you should be good to go:

http://blogs.oracle.com/timthomas/entry/configuring_the_opensolaris_cifs_server
Click to expand...

It seems like no matter what i do it fails.

Now time sync works but thats it.

Try to join domain :

step 1: Timesync with ad-server adds: 15 Jun 07:56:28 ntpdate[2966]: adjust time server 192.168.1.19 offset -0.094527 sec
step 2: set dns entry in /etc/resolv.conf:
step 3: set /etc/krb5/krb5.conf:
step 4: try to join via: smbadm join -u administrator freak - please wait ....



Joining freak ... this may take a minute ...


failed to find any domain controllers for freak

this is the dns it makes:
search freak
domain freak
nameserver adds
(I tried changing freak to freak.local i tried changing adds to to IP)

Here is the krb5
[libdefaults]
default_realm = FREAK

[realms]
FREAK = {
kdc = adds
admin_server = adds
kpasswd_server = adds
kpasswd_protocol = SET_CHANGE
}

[domain_realm]
.freak = FREAK

(I tried changing adds to ip, freak, freak.local)

When i do it manually in terminal i always get failed to find any domain controllers for .... (i tried both freak and freak.local)
 
capt_zman said:
Does anyone have a sample AppleVolumes.default file I can use. I use to have Time Machine working correctly, but after multiple rebuilds, I can't seem to get it going again.

Here's what my AppleVolumes.default looks like:

:DEFAULT: options:acls,upriv perm:0777
# The "~" below indicates that Home directories are visible by default.
# If you do not wish to have people accessing their Home directories,
# please put a pound sign in front of the tilde or delete it.
# End of File
/nas/macbook1 macbook1

When I open Time Machine, it doesn't show the disk. AFP is enabled for the macbook1 folder, and netatalk is installed. Also, are there any afp/avahi logs to look at?

Gea, thanks for all the help.
Click to expand...

do it like this:
in services-afp-volumes, you must set defaults like :
DEFAULT: options:acl,upriv perm:0777

in menu zfs-folder, you must afp-share the folder with the time-machine option
thats all you need

Gea
 
Freak1 said:
It seems like no matter what i do it fails.

Now time sync works but thats it.

Try to join domain :

step 1: Timesync with ad-server adds: 15 Jun 07:56:28 ntpdate[2966]: adjust time server 192.168.1.19 offset -0.094527 sec
step 2: set dns entry in /etc/resolv.conf:
step 3: set /etc/krb5/krb5.conf:
step 4: try to join via: smbadm join -u administrator freak - please wait ....



Joining freak ... this may take a minute ...


failed to find any domain controllers for freak

this is the dns it makes:
search freak
domain freak
nameserver adds
(I tried changing freak to freak.local i tried changing adds to to IP)

Here is the krb5
[libdefaults]
default_realm = FREAK

[realms]
FREAK = {
kdc = adds
admin_server = adds
kpasswd_server = adds
kpasswd_protocol = SET_CHANGE
}

[domain_realm]
.freak = FREAK

(I tried changing adds to ip, freak, freak.local)

When i do it manually in terminal i always get failed to find any domain controllers for .... (i tried both freak and freak.local)
Click to expand...

AD server must also be the dns-server who translates names to ip.
Therefor you must use the IP of your AD server when you join with napp-it

(or if you do it manually, use ip at dns entry)

check also if you ad server has already a host with your solaris hostname,
if so delete it andtry rejoin (try at least twice to join).

Gea
 
You must log in or register to reply here.
Back
Top