ZFS testing and monitoring

R

Ripley

Limp Gawd
Joined
Nov 4, 2004
Messages
240
I'm finishing up the setup on my 36TB ZFS setup. I have two things that I could use some help with.

1.) I'd like to do some performance testing to make sure I'm getting acceptable performance. But I'm not sure how to do it. How do you guys do testing? I'm not building for top speed but I want to make sure I don't have anything misconfigured that's causing trouble.

2.) I'm running Ubuntu Server 14.04LTS. How do you guys monitor the status and health of your setups? I don't want to run into a situation where a drive fails and I don't know about it.
 
I already run icinga on a ubuntu VM to monitor other things. There is a check_zpool plugin I use to check any data pools. For performance, a reasonable start (if you have a windows workstation) is to export a filesystem with sharesmb=on (with whatever samba juju you need to do) and use crystaldiskmark on the windows box. Run with the default (IIRC) 4GB data file. It will test sequential read/write, random, etc...
 
How difficult was the setup of icinga? Does it monitor SMART info as well?

For the testing, when you say export a filesystem do you mean a Samba share? Or do you mean something else?
 
Icinga is a little tricky, since it is all set up (AFAIK) through the command line, but once it works, it just works. It can monitor smart info if you have a plugin for that (I don't know if it does or not.) Yes, that is what I meant - create a filesystem, and export it with samba using the sharesmb property for the filesystem...
 
danswartz said:
Icinga is a little tricky, since it is all set up (AFAIK) through the command line, but once it works, it just works. It can monitor smart info if you have a plugin for that (I don't know if it does or not.) Yes, that is what I meant - create a filesystem, and export it with samba using the sharesmb property for the filesystem...
Click to expand...

If you hadn't already Icinga setup would you think it was worth the effort to monitor your ZFS server?
 
depends. there are cron jobs that will monitor your zfs pools and send mail but since i was already set up, it was easier to do it that way.
 
ZFS on Linux supports the ZED (ZFS Event Daemon).

Simply modify the file /etc/zfs/zed.d/zed.rc

Uncomment "ZED_EMAIL=" and fill in your email address.

Don't forget to restart the service then.

Now you will be notified of any zpool status changes (such as the pool going degraded when a disk fails) as well as any zpool read/write/cksum errors the moment they happen. You can also see all past events by running "zpool events"

Also, you just need any mailer service set up on the server for the emails to work. Something like postfix, ssmtp, or sendmail will do.


Furthermore to get notified of any direct disk issues via SMART you can configure the smartd to email you when SMART event errors happen.
 
Last edited:
I use OMDistro with Nagios/CheckMK, it needs a little configuration but it works great
 
SirMaster said:
ZFS on Linux supports the ZED (ZFS Event Daemon).

Simply modify the file /etc/zfs/zed.d/zed.rc

Uncomment "ZED_EMAIL=" and fill in your email address.

Don't forget to restart the service then.
Click to expand...

I'm running ZoL 0.6.3 on ubuntu 12.04 and do not have a zed.d directory. Do you know if there is a way to check if ZED is installed? I don't see a package and google hasn't been very helpful.
 
Hmm, I thought it was part of ZoL 0.6.3 release.

I'm running Debian and using the standard ZoL Debian package in the repo.

Do you have the daemon?

"service zed status"

I would go in #zfsonlinux in freenode and ask about it for ubuntu perhaps.

Sorry I don't know more about the details of it other than it just exists by default for my zfs install for me and just works haha.
 
Last edited:
For monitoring the ZFS pool and SMART, I wrote this python script a while back:
Code: 
# Import smtplib for the actual sending function
import os
import sys
import smtplib
import mimetypes
from email.Encoders import encode_base64
from email.MIMEBase import MIMEBase
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

def sendAlertMail(subj, body):
        #print("sending mail...")
        FROM = '[email protected]'
        TO = '[email protected]'
        MAILSERVER = 'smtp.mail.com'
        PORT = 465
        MLOGIN = 'mylogin'
        MPASSWORD = 'mypassword'

        try:
                s = smtplib.SMTP_SSL(MAILSERVER, PORT)
                #s.set_debuglevel(1)
                s.login(MLOGIN, MPASSWORD)

                msg = MIMEMultipart()
                msg['From'] = FROM
                msg['To'] = TO
                msg['Subject'] = subj
                msg.attach(MIMEText(body))

                #send it
                s.sendmail(FROM, TO.split(";"), msg.as_string())
                #print("sending successful")
                s.quit()
                s.close()
        except Exception, e:
                print e
                sys.exit(1)

        sys.exit(0)

# test emailing
#sendAlertMail('ZFS alert test', os.popen("/sbin/zpool status").read().strip())

# zpool checking
if os.popen("/sbin/zpool status -x").read().strip() != "all pools are healthy":
        sendAlertMail("ZFS alert: zpool error", os.popen("/sbin/zpool status").read().strip())

# SMART checking
# include disks to be checked in the list
disks = ['/dev/disk/by-id/ata-WDC_WD...', '/dev/disk/by-id/ata-WDC_WD...']
for disk in disks:
        s = os.popen('/usr/sbin/smartctl -H ' + disk + ' | grep "SMART overall-health self-assessment test result"').read().strip()
        if s != "SMART overall-health self-assessment test result: PASSED":
                sendAlertMail("ZFS NAS alert: SMART error", os.popen('/usr/sbin/smartctl -a ' + disk).read().strip())

sys.exit(0)
Adapt as required. You could use the ZED and the smartmontools daemon instead, ofc.

For local performance testing people use iozone and bonnie++. Testing over a samba share will not show pool peak performance, but it can uncover network/SMB issues.
 
bleomycin said:
I'm running ZoL 0.6.3 on ubuntu 12.04 and do not have a zed.d directory. Do you know if there is a way to check if ZED is installed? I don't see a package and google hasn't been very helpful.
Click to expand...

So I just spoke with the Debian ZoL package maintainer and he said this:

<SirMaster> I have a question, so i use your packages and I found that I have the ZED, is that something you had to include?

<FransUrbo> "had to" is to strong a word :). It's included in ZoL. But I DID write a init script for it...

<SirMaster> Yeah I thought it was included in 0.6.3, but he says he doesn't have /etc/zfs/zed.d/ directory

<FransUrbo> No, it's probably in /usr/{lib,share} somewhere...

So you will have to search for zed.rc or something and then probably write an init script for it for Ubuntu. I can't show you mine cause Ubuntu uses upstart and Debian uses init so its not going to be the same. Maybe there is already an upstart init script somewhere. See if you can contact the ZoL Ubuntu PPA maintainer.
 
SirMaster said:
So I just spoke with the Debian ZoL package maintainer and he said this:

<SirMaster> I have a question, so i use your packages and I found that I have the ZED, is that something you had to include?

<FransUrbo> "had to" is to strong a word :). It's included in ZoL. But I DID write a init script for it...

<SirMaster> Yeah I thought it was included in 0.6.3, but he says he doesn't have /etc/zfs/zed.d/ directory

<FransUrbo> No, it's probably in /usr/{lib,share} somewhere...

So you will have to search for zed.rc or something and then probably write an init script for it for Ubuntu. I can't show you mine cause Ubuntu uses upstart and Debian uses init so its not going to be the same. Maybe there is already an upstart init script somewhere. See if you can contact the ZoL Ubuntu PPA maintainer.
Click to expand...

Ahhh thank you for the follow up! Makes sense, I do have a zed daemon but no init script as you say. I'll have to poke around for one/make one or just upgrade to 14.04 which I believe has one available.
 
Here's another option that I found and am currently using - http://zfswatcher.damicon.fi/

I still need to test the email functionality but setup and install was super easy. Its working on Ubuntu 14.04 with no problem.
 
You must log in or register to reply here.
Back
Top