FreeBSD ZFS NAS Web-GUI

Status
Not open for further replies.
S

sub.mesa

2[H]4U
Joined
Feb 16, 2010
Messages
2,508
Latest Download link (via Bit-Torrent)
Stable LiveCD: http://zfsguru.com/files/ZFSguru-0.1.7-livecd-amd64.iso.torrent
Experimental LiveCD: http://zfsguru.com/files/ZFSguru-0.1.7-preview3.iso.torrent (contains experimental ZFS v28 system image)


Original message
Wouldn't that be great: a FreeBSD distribution that really is FreeBSD, but comes with a nice Web-GUI to manage your disks for use with ZFS. Still the power of FreeBSD with full shell access and the ability to run almost anything, paired with the convenience of a Web-GUI to do most basic tasks.

Most people will know the FreeNAS GUI; i want to make something much easier and less complex. This topic serves to discuss what features this GUI should have, and any other ideas that may be inspiring.

I'm currently in the works with PHP to setup a script that automates ZFS tasks; making SSH shell access less often required. It can also be used to display nice graphs for example about used storage space or bandwidth. The idea is to make a package with FreeBSD pre-installed and most things setup; only the Web-GUI being required to actually use it as NAS.

The advantages versus FreeNAS would be that you're running full-fledged FreeBSD (can install torrent stuff for example) and you will have the most recent stable ZFS version.

I'm currently trying to come up with a list of basic functionality. Could use some help with that. Assume you are a Windows-user looking at linux/BSD alternatives to build a nice NAS for use with ZFS. What features do you need? Once you login to the Web-GUI, what do you like to see? What should be implemented by the Web-GUI?

To kick-off, a small list of basic functionality:
  • Display connected disks and their corrent occupation
  • Display active ZFS arrays and possible options (scrub)
  • front-end to Samba/NFS/iSCSI configuration (including username/password stuff)
  • access information, bandwidth status
  • easy listing of snapshots and rollback/clone options
  • operations such as ZFS expansion should be possible through the GUI

Would love to get more feedback on this; anyone waiting for a project like this? I think FreeNAS is too much stripped down; too much GUI. I like to build something that works with native FreeBSD; so you in essence get the best of both world. But at the cost of reduced web-GUI functionality; i'm not going to recreate the entire FreeNAS GUI! Just those relevant to ZFS NAS.
 
Last edited:
This would be ideal... I would like to see all aspects of ZFS administration available. The ability to set up drives into virtual devices, virtual devices into pools, pools into datasets. Easy setup for samba and nfs. Also all the other goodies like replacing bad drives, automating scrub, adding drives. Also some kind of diagnostic page with I/O specs, bandwith, LAN speeds/CPU. An advanced feature which could check the configuration and suggest better configuration based on good practices.

Can I beta test it tomorrow? :)

WF
 
I would say rather than using FreeBSD, use OpenSolaris. It has a much better & up-to-date ZFS implementation. If you could give me a pre-built platform that just did ZFS really well ( in regards to performance, features and easy of use), I would (and others, I would imagine) jump at the opportunity to use it.
 
I'll be writing the scripts in PHP; so it just needs FreeBSD + Apache + MySQL + PHP5.

Yes diagnostic information about each harddrive would be useful; SMART data for each disk and the ability to send an alert email if the SMART data suggests a failing disk.

Still; i don't want to make it too complicated. I was thinking about 10 pages or so, that should cover all functions. The idea is to replace the command line interface for all common tasks/maintenance.

Would this allow Windows users to run FreeBSD + ZFS? Installing FreeBSD is a pain so i probably create one installed image that one only needs to copy to a system disk and boot from it; much easier i think. But i also will release the PHP scripts for those who want to install themselves or use on their existing system.

Though i'm already coding part of the stuff, its just an idea at this stage. I could use some creative ideas, and discussion of how to make this really usable to people not familiar with BSD or open source software for that matter.
 
Rakinos said:
I would say rather than using FreeBSD, use OpenSolaris. It has a much better & up-to-date ZFS implementation.
Click to expand...
Sorry, i don't agree. Any bugs in ZFS are OpenSolaris bugs; the FreeBSD implementation is a real implementation and nothing inferior to it.

Both stable ZFS versions on FreeBSD and OpenSolaris is at version 14 right now; though with OpenSolaris you can run the bleeding edge v24; but its not stable.

I also like to use something that is truely open source; Oracle could be changing ZFS' future; but FreeBSD would still have their own ZFS import and can do changes as they see fit independent from Oracle. So ZFS on FreeBSD might actually have more future than on OpenSolaris itself; though that's a huge guess into the future.

As i'm very familiar with FreeBSD and think its generally the best server OS of all, that's where i would want to work on. FreeBSD imports the best technologies, and exports its own unique selling points. Both the Linux kernel and Firefox benefited from FreeBSD innovations.

Perhaps the scripts could be tuned easily to work on OpenSolaris as well, but at this stage i want to focus on FreeBSD instead. The more open source; the better. :)

If you could give me a pre-built platform that just did ZFS really well ( in regards to performance, features and easy of use), I would (and others, I would imagine) jump at the opportunity to use it.
Click to expand...
Well be careful when it comes to your data. While this project is intended to be useful, i do not want people to use it too quickly and suddenly be hit with some bug that kills their data; just like the recent FreeNAS thread on this board. ;-)

So i would like to test it a lot. I guess it would take at least a month to produce something that is actually usable; a first stable release. Now i just need a name. :)

FreeZFS? ;-)
 
sub.mesa - I would love to use what you make, but the reason I use opensolaris is also because of dedup

http://blogs.sun.com/jsavit/entry/deduplication_now_in_zfs

$ zpool upgrade -v
This system is currently running ZFS pool version 22.

The following versions are supported:

VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit
15 user/group space accounting
16 stmf property support
17 Triple-parity RAID-Z
18 Snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties

For more information on a particular version, including supported releases, see:

http://www.opensolaris.org/os/community/zfs/version/N

Where 'N' is the version number.
Click to expand...
 
Yes, but i would point out that you would require running a 'bleeding edge' version of ZFS to get full deduplication support; it is NOT yet stable.

Basic de-duplication already works on FreeBSD 9.0, scroll down on this page:
http://www.freebsd.org/news/status/report-2010-01-2010-03.html

As soon as OpenSolaris stable release gets newer ZFS version, likely FreeBSD wouldn't be long to follow. But it would first be integrated in FreeBSD 9-CURRENT (the bleeding edge development branch). Later probably MFC'ed to 8.2 and branched 9.0. Possibly 9.0 will release sooner than 8.2.
 
FreeNAS is a project dedicated to a NAS, with many FreeBSD elements removed.
I want something that you can add to FreeBSD; not the other way around.

That way you still have a normal FreeBSD setup and can normally use that to recover or investigate or install other apps such as torrent stuff as people frequently want that.

Also, i dislike the way FreeNAS treats the disks, and leads to buggy behavior. I want a clean design based on geom_label. That also makes it easy to distinguish your disks, in case something goes wrong.

So i guess i'm just inventing a smaller yet better wheel. ;-)
 
It's not much yet, but i thought a 'sneak preview' would be nice, here it is:

mesa-sneakpreview1.png


Currently the status page works, and i'm working on the Disks page to recognise disks used for ZFS etc. And the Disks page would also show disk health status (bad sectors; cable problems).

Comments are welcome! Especially regarding functionality on the pages. I really like this project. ;-)
 
sub.mesa - you hit the nail on the head.
although FreeNAS will soon have proper ZFS functionality, it does feel like naked OS.
the only thing that attracts me more in Windows based NAS solution is the functionality and wide application avaliability.
by making FreeBSD more user friendly to the beginner, I really think you'll make it a very attractive platform (for me/us - windows users).

about OpenSolaris, another con is that FreeBSD seems to support much more hardware at the moment, this is a really good reason (atleast for me) to stay away from it.

one suggestion about the GUI though - Adding a temprature display to each hard-drive would be great.
i'll keep adding suggestions along the thread as i'll see your progress.

very nice initiative, thanks :)
 
sub.mesa said:
It's not much yet, but i thought a 'sneak preview' would be nice, here it is:

mesa-sneakpreview1.png


Currently the status page works, and i'm working on the Disks page to recognise disks used for ZFS etc. And the Disks page would also show disk health status (bad sectors; cable problems).

Comments are welcome! Especially regarding functionality on the pages. I really like this project. ;-)
Click to expand...

I think it's a great idea sub.mesa, it will help a lot of people especially the one's that don't like the cli approach.

My suggestion would be, you see where your tabs are at the top, maybe you could set these in order of usage at initial set up: what I mean for example tab 1 would be the welcome system page, the next tab could be where you need to add disks from, the third tab where you need to create you virtual devices and so on.

Someone has done a similar thing for the Nexenta/eon OS's, see here (not really that pretty, as long as it works as suppose: :D

http://www.napp-it.org/index_en.html
 
So far your sneak peak looks good to me from a functionality standpoint. A few thoughts:
- Add a way to get shell access via the GUI on the off chance that the user can't use PuTTY or your scripts have failed.
- After all of the major work is done, maybe do something about the color scheme?
 
I dont think temperature output would be possiable? Atleast not without smartmontools or some equiv.
 
@nbat58: i can see this screenshot:
http://www.napp-it.org/doc/napp-it/top/comstar_pre.png

But i'm not sure what exactly you mean. Are you talking about the user able to change the order of the tabs? Don't see how that is particularly useful; the idea is to have perhaps 6 top-level menus and perhaps sub-pages for each top-level menu. The subpages could act like real tabs.

@Danny: yes i will definately add shell access, and yes the visual aspects will be improved; though i prefer using as much as CSS for markup and as little images as possible. But clickable image-icons and stuff like that; yes i might add that, but not before basic functionality works.

I probably have to add a "System" top-level menu where preferences, reboot/shutdown, access control (password) and shell access reside. Also it would be where you can update the portstree and install software by browsing the portstree visually. Wouldn't that be nice? ;-)

And temperature readouts are possible, as i can simply query smartctl (smartmontools) when installed.
 
sub.mesa said:
@nbat58: i can see this screenshot:
http://www.napp-it.org/doc/napp-it/top/comstar_pre.png

But i'm not sure what exactly you mean. Are you talking about the user able to change the order of the tabs? Don't see how that is particularly useful; the idea is to have perhaps 6 top-level menus and perhaps sub-pages for each top-level menu. The subpages could act like real tabs.

@Danny: yes i will definately add shell access, and yes the visual aspects will be improved; though i prefer using as much as CSS for markup and as little images as possible. But clickable image-icons and stuff like that; yes i might add that, but not before basic functionality works.

I probably have to add a "System" top-level menu where preferences, reboot/shutdown, access control (password) and shell access reside. Also it would be where you can update the portstree and install software by browsing the portstree visually. Wouldn't that be nice? ;-)

And temperature readouts are possible, as i can simply query smartctl (smartmontools) when installed.
Click to expand...

Thats what i figured.

would a GUI based file manager be to much to ask for? something like norton commander :D would be handy so we dont have to wear out our fingers so much.

Also Email reports are a must have IMO, and being able to pick what data is reported and when would be awsome. Email results of scrubs, etc.
 
Been working almost straight. :p

Some things going slowly, but i'm getting there. Still no real functionality; only information gathering/structuring. I've done the status page, network page; now want to implement the disk page functionality; formatting disks for use with ZFS. The next step would be the ZFS page, where you can actually make an array and manage filesystems.

Once basic functionality works, i'll release a downloadable version of the scripts for basic testing. But i can't give an ETA yet. Once i did most of the work i'll release a series of beta releases that will likely stay beta until a reasonable test group and test period assures no obvious bugs in the scripts that can eat your data. ;-)

So don't expect a .zip url tomorrow or something. :D
Will take a bit longer, still.

But i do have new.... Screenshots!

mesa-sneakpreview2.png


mesa-sneakpreview3.png


mesa-sneakpreview4.png
 
sub...
I have finished playing with opensolaris and have picked up freeBSD again based upon your preliminary work. By the time you have a somewhat stable alpha I will have learned enough to be very comfortable with the system. I hope you know what your getting into :)

Just set a new SUPERMICRO MBD-X7SPA-H-O on fire... It was fine for several hours until I plugged in a video card... I love the smell of burning electronics in the morning..
 
Looking very nice. I'm assuming we would have to move our data, and create a new pool to upgrade to a ZFS v24 right?

Possibility of saving your settings for use in FreeBSD9? Just curious what the migration procedure would be like.
 
Upgrading a pool would be done with a 'zpool upgrade' command; this is something that will be integrated in the GUI probably. Note however; you cannot downgrade a pool! So if ZFS v24 proves unstable and not usable for real stuff yet; then you would have to destroy the array to go back to ZFS v13/14 which is the stable ZFS version in both FreeBSD and OpenSolaris currently.

New Screenshot:

mesa-sneakpreview5.png


Some more progress:
- formatting disks now works (destroys any existing geom labels, partitions, BSD labels)
- writing new geom label works
- encryption not yet

With encryption, the idea is to encrypt whole disks then pass the encrypted disk to ZFS. zfs-crypto which adds encryption to ZFS itself on the dataset level, is not yet finished and integrated. So use GEOM_ELI for now. As consequence, after rebooting user needs to enter password in order to mount the zfs array.

I'm using sudo right now to execute shell scripts with root privilege. I do need something relatively secure; i did not got setuid on shell scripts to work, but it appears that is disabled by design. Only binaries can be setuid/setgid. I also need a way to enter password 'interactively' when dealing with geli encryption utility; as it wouldn't accept it as a command line option.

Probably i can look at FreeNAS how they do things. First want things to work, then i'll perfect the scripting system to avoid any unnecessary security risks.
 
It's going quite well so far. :)

Perhaps i can release a first test version pretty soon; i could use help testing that and general feedback (improvements). I think the scripts could be easily extended by people who need additional functionality.

For now, this is a small project; if it becomes more serious i'll host it somewhere safe. :)

New additions:
- creating zpools

mesa-sneakpreview6.png


Currently creating ZFS arrays through the Web-GUI only works if the physical member disks are formatted with a geom_label. By always using geom_label we can prevent some weird accidents and configuration mockups.

Still a lot of work to be done; the pages are all single page pages; i want each top-menu to have subpages as well. Then it'll start being more complex, but the basic functionality is already there. :)

Before i release a test version, the GUI should be able to:
- recognise and present disks (works)
- format disks, destroying any labels/attachments (works)
- create new geom label (works)
- create new zpool (almost works)
- destroy zpool (not implemented)

So i guess that would be pretty soon already. :)
 
What kind of performance impact will using the geom_eli have? Would this be pretty much invisible assuming a sufficiently powerful cpu? Big reason I've been holding off on ZFS is the encryption, but if this works... :)

Also in your screenshots, you have it listed as "RAID6 (double parity)". Wouldn't RAIDZ2 be appropriate? Eventually RAIDZ3 will be integrated and there's no corresponding triple parity name.
 
Ruroni said:
Also in your screenshots, you have it listed as "RAID6 (double parity)". Wouldn't RAIDZ2 be appropriate? Eventually RAIDZ3 will be integrated and there's no corresponding triple parity name.
Click to expand...

(just opinion/thought)

Depending on the user, say someone like myself, I'm very interested in this project. I will want a storage server soon for a small business I plan on starting. I know what RAID is, and it's different levels, but I know nothing about the levels of ZFS' RAIDZ. Maybe more people will accept it if it relates to what they know more... rather then using terms they aren't familiar with which might cause they to go else where and get an inferior product? Just to get them in, then they'll actually start learning the rest... I dunno...

I used to think FreeBSD was just another linux... till I really started reading this section of the forum and getting a real idea of what it is...

there was a post about levels of linux and like what kind of person uses it or what it's ment for...

and someone said FreeBSD is like linux but without the noobs... at that point I stopped looking at linux for my solution... still cracks me up....


So yeah... hope this works out... there will be a spot on my rack when your done... thx
 
Encryption done by GEOM_ELI is multithreaded, and can use hardware encryption if available; some Core i5 support this i believe; not sure if that works yet. VIA Padlock and other encryption acceleration is supported though.

The performance is very good since you can use quadcore CPU. Though you require multiple queue depth to satisfy all cores; for single queue depth I/O only one core will be used for encryption. Generally with quadcore you can go up to 200MB/s and beyond; far above the gigabit threshold. However, keep in mind that latencies are increased. This will not impact sequential performance, but it would impact random read performance. For example, when using iSCSI to supply other PCs with their system disk, you would have lower boot performance; though still on the threshold of being noticeable; i guess about ~25% slower in these cases, with partial random and some sequential I/O.

So other than random read blocking I/O, the encryption would not really be 'felt' by the user. By the way, in my tests several 'slower' AMD cpus outperform my Intel E8400, so it appears AMD has better performance on GELI than Intel, as for other tasks the Intel definitely is faster than my slower AMDs.

The encryption works at disk level though; so before you enter the encryption key the array would be UNAVAIL / FAULTED because the target disk device (with .eli suffix) is not available. You can opt to encrypt only part of your member disks. Imagine a RAIDZ2 or RAID6 with 2 parity disks; you can encrypt only 3 disks letting the other disks unencrypted. This would impact performance less heavily and would make regular recovery impossible without the password. You would be able to read snippets from the disks that are not encrypted, though ZFS is not something that has many utilities to recover files from such as with NTFS and FAT which are very simple filesystems in comparison. So, for casual protection partial encryption might be a good compromise.

But you can go ahead and encrypt all your 8 disks in a RAID-Z for example, the only 'catch' would be that you would need to enter your password for all 8 disks after each reboot. I can probably group the authentication so you can use only one password for all 8 disks, not sure what that means from a cryptographic point of view, though.

About the naming convention: i have triple parity currently listed as RAID7 (triple-parity). Though it is not an 'official' level i would argue the RAID convention is flawed anyway: if RAID0 is RAID, then JBOD is as much RAID and we are talking about multi-disk arrays instead of redundancy. The Linux software RAID driver calls itself the 'multiple disk' driver or md, that's a pretty straight forward and quite correct name in my eyes.

So bottom line, RAID7 might just make sense and would be easy to understand for regular people who are still new to ZFS. I would like to include an overview about ZFS features on the help pages, to learn casual people how ZFS is different and how its power can be fully utilized. ZFS is not just another filesystem to drop your files to, it's another way of thinking in terms of storage. Everything has space shared and many old wisdoms are obsolete with an integral package that combines both filesystem and RAID or multidisk-engine.

I'm currently working on a ZFS execution engine, which executes 'dangerous' commands that the user must first verify and can modify if necessary. That would also be very useful in terms of debugging. Later i'll probably remote the extra 'command confirmation page' if it works without problems.
 
New additions:
- destroy pools
- script now detects disks part of a zpool array, and refuses to use them to create a new array
- visual changes to let user know which disks are in use when creating a new array
- pool query page with scrub/stop scrub options (see screenshot)

New screenshot:

mesa-sneakpreview7.png


Todo list:
- extra page should have terminal window and some options to set
- need zfs filesystem management (for things like compression/NFS sharing)
- i want to have another navigation layer which act like tabs. so the ZFS page would see tabs like "pools" "filesystems" etc
 
I also had a different top-level menu layout in mind:

STATUS - NETWORK - DISKS - POOLS - FILES - SERVICES - EXTRA

Does that make any sense?
 
Now I'm waiting for a public release to try it out

How easy will upgrades be? Even to "development" versions
Your entire setup seems very lightweight, I can imagine it would be nice on 9.0 -- It'd be genius for anyone trying osol and suffer solaris pains
 
sub.mesa said:
@nbat58: i can see this screenshot:
http://www.napp-it.org/doc/napp-it/top/comstar_pre.png

But i'm not sure what exactly you mean. Are you talking about the user able to change the order of the tabs? Don't see how that is particularly useful; the idea is to have perhaps 6 top-level menus and perhaps sub-pages for each top-level menu. The subpages could act like real tabs.
Click to expand...

What I meant is that it would be good that the tabs at the top are set up in order of initial set up (so you start at the first tab and work your way towards the end to finish the set up). Also it would be very helpful that you could have on the welcome page for example a quick set up guide.

Sometimes new users spend more time trying to figure out how to use an aplication than the actual time it takes to install it set it up and use it.
 
vraa said:
Now I'm waiting for a public release to try it out

How easy will upgrades be? Even to "development" versions
Your entire setup seems very lightweight, I can imagine it would be nice on 9.0 -- It'd be genius for anyone trying osol and suffer solaris pains
Click to expand...
First i thought you meant updates of my scripts; which i plan to do in the Extra menu. That way you can start with my development version and update as i release new versions of the scripts. I would note that any release before 1.0 should be considered experimental and unstable; don't trust your data without solid backup to it yet. It can already do harmful things like destroying pools and formatting disks.

But since you probably meant upgrades to a newer FreeBSD version; yes it should work. Since you don't have one integral stripped-down OS anymore like FreeNAS is, you can upgrade FreeBSD to 9.0-CURRENT if you wish. I've already got triple-parity raidz3 "RAID7" in the GUI. Perhaps with some tweaks it will also run on OpenSolaris. However, it does require sudo access on the www group without password at this moment, which could be a security concern. I would need root access one way or another, though.

tigerwolf753 said:
Would it be possible to use disk labels to graphically map out the drives to the physical slots like they do here: http://www.nexenta.com/corp/nexentastor-faq-table/238-how-can-i-locate-drive-in-the-appliance-is-there-a-map-of-physical-slots ?
Click to expand...
Well looking at my screenshot; you already can see the disk device name, the device string, the size and the geom disk label which you can choose. The only thing missing there is the serial number of the device. Is that what you're talking about, or did i miss something?

The idea is that you have a rack with 8 disks and you call your disks disk1, disk2, disk3, etc. So you identify the disk by the label name, not by the device name. So one time labeling then you can continue to use the label name and ZFS will use it too in its output. The idea is that since you can choose the label you can use a scheme that works nice with your disk setup. If you have a large rackmount with 24 drives, you sure want the rack label that says "disk 16" to correspond to a similar label. So you know which disk is, physically.

nbat58 said:
What I meant is that it would be good that the tabs at the top are set up in order of initial set up (so you start at the first tab and work your way towards the end to finish the set up). Also it would be very helpful that you could have on the welcome page for example a quick set up guide.
Click to expand...
Well, that's already the layout isn't it?

First you enter the status page where only information gets displayed. Then there's the network page, vital to checking if settings are right (and dhcp enable/disable and ip changes work now). As for storage, now we get Disks, Pools and Files. On the Disk menu page we format our physical disks and place a GEOM label on them. On the Pools page we add those labeled disks to a ZFS pool (array). Then finally on the Files page we create filesystems and can see how our data is structured.

At least i thought that would be a logical sequence. :)
 
@sub.mesa:

I'm too lazy to actually put a label on each tray so I just think it would be neat to be able to map out the drives onto a picture of the case.
 
Yes but unless i sell this together with hardware, how will i know how your case looks like, or how the cables are internally connected?

You could either hot-plug them and label them straight away so you start with 0 disks, connect one, hey its called ad4, label it as disk1. Etcetera.

Or if you got a nice drive-cage which has LEDs which work on the SATA-level (detects datastream) then you could read from the device and you can see which LED is shining. I could create a 20-second workload that makes the led go on via the GUI, but if you don't have this feature you still have to do the label name -> physical disk mapping manually yourself.
 
Another screenshot, this time of the Extra menu. As you can see i added another navigation layer, the 'navigation tabs' that should feel like tab pages. Not sure if the feeling really coveys. Any feedback on the visuals? Want to keep things simple, without images at this point. But it should still feel like a small yet clear interface where you can instantly find what you want.

I'll probably add a lot of 'hints' on the way. Such as if the user clicks the pools button without any disks being formatted yet, it would tell the user to format his disks first as an extra paragraph at the top, with colours or something. Like a notify box.

The preferences shown in the screenshot don't work yet; though it's just a start; i took most time for the navtabs due to a nasty css issue which needed lots of debugging. :p

mesa-sneakpreview8.png
 
Okay, I see what you mean, there is not really any easy way out of manually labeling each drive, no biggie.

I like the clean interface and especially the idea of pop-up hints. That way you could have helpful tips and warning without cluttering the place with text. Though, depending on how prevalent they are, you might want to have an option somewhere to disable them for advanced users.

Also, would it be possible to have an option to power off the machine through the interface?
 
Status
Not open for further replies.
Back
Top