FreeBSD ZFS NAS Web-GUI

No login/authentication, though the option that limits access to LAN interface will be in the first release. And the password option probably in the release after that. That would require using PHP sessions.

Powering off would need an extra 'Shutdown' navtab in the Extra menu, that has both shutdown and reboot buttons. Currently working on command execution, which is kewl:



New stuff:
- command line works
- root access works

Interface seems clean and uncluttered.. Very intuitive so far..
Can't wait to test .. You have no idea how excited I am. Once the disenchanted Vail Beta testers get to try this I predict mass desertions from WHS v2 especially for anyone having large data shares. I'll use Vail for backup purposes only as WHS has saved my a** on many occasions.

Well don't put too much 'publicity' about it yet -- or i might be slash-dotted faster than i thought.

I could use help from you guys to test it through its early stages before its usable for the mass people.

Here's another screenshot (yes we all love screenshots! )

Well, I've been messing around in NexentaStor's web GUI for the past few hours and I have noticed a few things missing in theirs that would be awesome to have in this implementation. One thing is that if the array is degraded or something similarly major, there should be a big warning box on the status page saying so.

I mention this because I created a RAID-Z array in my NexentaStor VM, turned it off, deleted one of the virtual hard drives, turned it back on and went to the status page on their GUI. All that showed up on the status page was a line of text at the bottom saying something about the Fault-Management Summary. Since I had just created a problem with the array, I was already looking for something like that so I just clicked through and saw what I already knew, that the array is degraded. But if I hadn't already known that, I probably wouldn't have paid it much thought and that seems to defeat the purpose of a status page. One idea would be to have a warning indicator in the top-right of the interface or something to quickly alert the user to a problem so they can respond.

The other thing I noticed is that there is no way to replace a failed drive from the GUI in NexentaStor's implementation. Not only did it fail to give meaningful warning in the GUI that a disk failed, but then there is no other way to replace the disk but to go to the console, which makes having a GUI silly if critical tasks like drive replacement can only be done with text commands. I'm not sure if maybe there is some technical limitation here because it seems to me that the process of figuring out which disk failed, selecting it, selecting the replacement, and then keeping track of the re-silvering process is better suited to a GUI than the console.

That aside, it's pretty sweet how ZFS can automatically online the hot spare to compensate for the failure and then put it on standby again once the failed drive is replaced.

Well, I've been messing around in NexentaStor's web GUI for the past few hours and I have noticed a few things missing in theirs that would be awesome to have in this implementation. One thing is that if the array is degraded or something similarly major, there should be a big warning box on the status page saying so.

Click to expand...

Maybe change the background color or something. Something that will either pop out or totally catch attention.

Loving the screenshots. It's looking nice to the eyes. And is laid out logically.

What ports will need to be installed for this to work on a minimal freebsd 8 install?

lang/php5
lang/php5-extensions (not sure which yet; but at least PCRE)
www/apache24 (or another apache/webserver)
security/sudo (and with root access without password to the www group configured with visudo)

optionally:
- iSCSI-target daemon (istgt)
- FTP-server
- Samba
- Smartmontools (SMART disk info utility)

The rest is all installed by default.

sub.mesa, perhaps the only option for Apache service should be "Restart". Less you condone GUI assisted service suicide. lol

sub.mesa,

Are you using any code in the php scripts that requires apache? I prefer to use lighttpd, as this would be the only web based application running from the array and i don't want the overhead or advanced configuration of apache.

Or even better in my case, if there was a way to install the scripts on a separate webserver that would run the commands via ssh against the storage box.

What you are doing is a wonderful idea. Thank you for helping out the community.

Just for comparison, I would want you to look at the gui on Amahi Home Server and see if there is anything you like. Their gui just went through an upgrade.

Ben

This is certainly a neat project sub.mesa! I look forward to a source code release, so I could perhaps re-tool some of it for use with my Linux server setup. (Leaving credit to you in-tact in the source code, of course)

@Kritter - yes good idea; the apache will only have Restart button; probably i'll use a shell script to restart apache.

khaoohs said:

sub.mesa,

Are you using any code in the php scripts that requires apache? I prefer to use lighttpd, as this would be the only web based application running from the array and i don't want the overhead or advanced configuration of apache.

Click to expand...

Yes the idea is that this should work, though i never used something other than apache myself. The only thing i can think of is that i'm using $_SERVER['REMOTE_ADDR'] which comes from the webserver; but if that works on other HTTP engines as well, yes you can use a lighter engine.

And i might opt for that option in the 'default install' or setup. I also do not need SQL support; i'll only have to write a small preferences file so do not need database.

Or even better in my case, if there was a way to install the scripts on a separate webserver that would run the commands via ssh against the storage box.

Click to expand...

Even the most low-end systems should be able to run the PHP scripts; they are not that CPU intensive at all; just some regular expression work and such. Assuming authentication/security is not a problem, i wouldn't see a real reason to let another computer do the work for the scripts.

What you are doing is a wonderful idea. Thank you for helping out the community.

Click to expand...

Yes, well, i felt there was no real good alternative to FreeNAS for 'windows users', who are not too familiar with a shell or command line. I can imagine you would be hesitant to store your files on an OS which requires mainly shell or command line access which you are unfamiliar with. For those users, providing a web-interface will greatly lower the learning curve to software beyond Windows. It's also a great 'first step' into a world that is not Windows, for many, and an opportunity to learn new things while getting the most basic stuff to work with ease using the web-interface.

So if i can make a package that people can use without really having to know FreeBSD, then it would potentially benefit a lot of people. They wouldn't need to learn FreeBSD (which would exclude most potential users) while still able to use the advanced ZFS filesystem and all of its power; since it's just awesome for many home users.

Might even be using the web-interface myself, to structure ZFS data differently and more visually. I am kind of worried i have no idea what i get myself into, though. For all i know, once i put something online that works and is easy to install, it might start circulating like crazy. At that stage there will be a lot of user feedback, and people will be using it for real. Any bugs could kill people's data. So i'm kind of hesitant and need to think this through.

Perhaps it needs a website and support forum as well, for those to ask questions. And i need a proper name still.

But yes, i'm quite excited as well. After all, with not too much work i already got something that does most basic stuff. Once the web-interface is finished from a core functionality point of view, i would need to think about how to package it.

Probably i can release it in various formats:

• 100KB Do-It-Yourself package; scripts only; requires you to install and setup FreeBSD
• ~400MB Complete 8.0/8.1 system image with everything pre-installed; can be written straight to a physical disk which copies everything including boot sector and partition table. Could also be written to USB stick though USB booting on FreeBSD is not problem-free in my experience.

sub.mesa said:

But yes, i'm quite excited as well. After all, with not too much work i already got something that does most basic stuff. Once the web-interface is finished from a core functionality point of view, i would need to think about how to package it.

Probably i can release it in various formats:

• 100KB Do-It-Yourself package; scripts only; requires you to install and setup FreeBSD
• ~400MB Complete 8.0/8.1 system image with everything pre-installed; can be written straight to a physical disk which copies everything including boot sector and partition table. Could also be written to USB stick though USB booting on FreeBSD is not problem-free in my experience.

Click to expand...

A VMware appliance would be cool too. That would make it real easy to take out for a test drive. And kudos for all your hard work so far, it's amazing how far you've got in just a few days.

Yes, though i would rather opt for Virtualbox instead; being an open source VM solution - though now also in the hands of Oracle; but at least it's GPL, so Oracle's options are limited. There already is a virtualbox-ose fork; that will continue to live on as GPL software.

Can i create a VMware appliance with the free release? I won't be buying a license.

Virtualbox would work and would probably be better since it is open. I just automatically think of VMware when I think of virtualization because that is what I work with most of the time. And yes, you can create VMs with VMware Player 3 (which is free), though you couldn't with the older versions.

I just got it working under Lighttpd+fcgi+php5. It needs the following php.ini tweaks versus the development sample configuration:
- short tags = on
- magic quotes gpc = off
- cgi.fix_pathinfo = on

The proper order to install is:
ports/www/lighttpd (do not select spawn fcgi option)
ports/www/fcgi
ports/lang/php5 (select CGI)
edit files /usr/local/etc/lighttpd.conf and /usr/local/etc/php.ini
create your /www document root where you will store the scripts, and adjust server config accordingly.

Then the last thing you need is sudo:
ports/security/sudo
then do this under a bash shell:
export EDITOR=ee
visudo

and add a rule like this:
%www ALL=(ALL) NOPASSWD: ALL

This will allow root access to all "www" users, including your web-server. This might be a security concern if your server is directly connected to the internet. Currently this is the way i gain root access on my scripts, but since i use a single execute function i can change that if possible.

Will still take a couple of days for a pre-alpha release; still much to do.

Sub.mesa,

Good to hear you already got it working with lighttpd, we certainly all appreciate your efforts.

I do not feel a usb installation is the way to go for this project, a full featured hard drive install of FreeBSD would most likely be best. FreeNAS already has an embedded version for people who feel the need to go that route.

One thing to take away from FreeNAS though is the ability to export a single configuration xml file with all the system settings. It would be nice to have a way to export your rc.conf, smb.conf, etc., to be able to restore your config easily in the case of an OS crash.

Exporting/Importing system configuration is indeed an excellent feature; but i will probably do these 'wishlist' features after a first beta release which focuses on basic functionality.

Been coding a lot, resolved some bugs, added some sanitization at some places and best of all:

Expansion now works! - you can now add arrays to existing arrays!

What exactly does that mean... add an array to an array? so array A is bigger now because array B was added?

Greatone123 said:

What exactly does that mean... add an array to an array? so array A is bigger now because array B was added?

Click to expand...

I think this quote answers that question rather well:

sub.mesa said:

Can i expand my ZFS RAID array?
Yes, but some restrictions apply.

What you cannot do, is expand an existing RAID-Z (RAID5) or RAID-Z2 (RAID6) array with one or more disks.
But, you can add new disks or RAIDs to an existing pool. So if you have a 4-disk RAID-Z, you can add another 4-disk RAID-Z so you have 8 disks. The second array would share free space with the first; in essence it would be a RAID0 of two RAID5 arrays. ZFS can expand this way.

What you can do, is expanding mirrors and RAID0's. In the example above that's what actually happened: a new array is RAID0-ed with the existing array. New created files will be written to both devices, for additional speed. Setting copies=2 would make files in that directory be stored on both RAID arrays; for extra redundancy.

Click to expand...

This looks great!!! And now that I get to the end of the thread, I can think of a handful of things that would be useful. This is probably more of a wishlist for the 1.0 release... Maybe you could edit the first post to include the wishlist of features?

Raw smart status display and simplified smart status display... for the [H] users and new users... with big RED FLASHING box on top when a drive is bad... FLASHING Yellow when it's dying... Think star trek :-D
Email/sms notification to multiple users/email address, sms that does not require an email server, similar to the nagios setup,
autoinstall disc image- download iso, burn, insert and have it install all packages and setup/config in a nice easy turnkey setup. goal: to be up and running in 15 minutes.
I would still like to vote for a usb drive setup for those of us who want to use every single sata port for storage... maybe a post 1.0 feature, but i would love to have it load into ram and only save settings to the thumb drive.
i second the import/export config file feature.

i picture starters using the basic easy/usb setup, then decide they want some auto torrenting/sabnzbd setup or more features. Export config, install full freebsd, to system drive, then start breaking things in freebsd, last move, format c:\windows :-p

Danny Bui said:

I think this quote answers that question rather well:

Click to expand...

I like it.. can't wait to try this, I don't know much about any of this except the basic ideas behind it... I'd gladly be the one that proves someone that know little about can use it

Sub..
It's been a few days of trying to set up freebsd following your step by step document. I have tried to install the system at least 6 times, always running into some problem. Today I spent most of the day going step by step and I'm having permission problems with samba. The windows -7 system can see the bsd "computer" and asks for name and password. I have tried every combination possible. Is there a way to turn off security to my client computers and video devices (Popcorn Hour) on my lan? (192.168.1.xx). I am waiting for your project anyway but I would like to test the functionality and capability of zfs while I wait. My whs is in pieces at the moment with drives and new equipment scattered all about . I have not set anything on fire since the SM atom motherboard.

Thank You again,
Jeff

Disregard

Make this and make it work with a CF card pl0x pl0x pl0x

No new screenshots; sorry!

But i'm working on upgrading the scripts. If that works well, i will consider a release. Just need to do more work on the Files page and need the Preferences to actually work and be saved in a preferences file in the docroot. Once those two main things work, i'll release a test version for you guys to crunch.

So it shouldn't take long now; but i also have to do real work so it might take a few days still.

I think it's pretty neat what i got thus far. If the updates work well, you could update via my website or by http file upload; the latter already works, the former not yet. So once it works you can update as i release new versions and keep track of the development and potential issues.

After the first release i will address wishlists and prioritize them; difficult tasks such as xbox plugin or whatever; i can't do that; at least not in the timeframe i have. But things that easily work in BSD (at least with my knowledge) such as a torrent-client, yes that would be a good wishlist item. And the 'norton commander' clone was a nice idea as well; though i'm not sure i can restore its former glory. ;-)

@MetricGnome: i have FreeBSD installed to 1GB CF (actually its 8GB but 1GB primary partition with everything on it). So you could install it on a USB pendrive, when i release it in binary format.

@wingfat: open a new thread that works best. i'll answer you there. Post the zpool status output and the contents of the /usr/local/etc/smb.conf file if you can copy-paste them.

You can ask me any FreeBSD question in there as well.

After all, some things cost you 1 hour which cost me 1 minute. ;-)

Okay new screenshot:



I now implemented configuration file reading/writing; so the preferences should be saved. I also implemented notify boxes; the screenshot shows the error popup when you have no configuration file. This works on any page you visit. So in some cases you could have multiple of those boxes.

It's not exactly how i want it yet, but basically i think those popups are quite user friendly. The idea is that you get these automatically when something is wrong, or to notify you something actually worked or was changed.

I'm also working on the update feature; file upload works already now want to do remote upgrade. And it checks what your currently installed version is, and compares it to the latest version online. Pretty neat. ;-)

A release is coming near.

Very cool! I look forward to seeing a release coming in the near future!

Yes it should not take long. Only the Files page misses core functionality.

So what i want to do:
- implement more sanity checks (sudo) and enable read only mode when sudo not available or permission denied
- implement creating Filesystems on the Files page
- implement remote update
- publish release notes on website
- publish first test release

im keeping an eye on this so i can test this as a storage medium for my esxi box. NFS on 20+disks...

cant wait for the project page and test release!

Alright all core functionality except file sharing (Samba/NFS) is currently implemented.

New additions:
- Filesystem page allows creating new filesystems, displays current
- allows viewing of filesystem properties (zfs get all)
- allows destroying filesystems
- New File Browser which can navigate directories and show its contents; not file contents yet

So points remaining are:
- sanity checks, sudo check, read only mode
- remote update
- last global source code check
- release notes, publishing on website

Before i publish on website, i'd like at least some of you guys to pre-test it; but only those who can test it without putting any data at risk! It's still very much untested.

So i guess those interested in running the bleeding edge first pre-release can start sending me a PM on this board. For now, i don't want the URL to be public. Depending on your feedback, the first public release might be a week later or so.

It is still missing file sharing as core functionality; without that it wouldn't actually allow the system to be used as NAS. This should be addressed before releasing it to the mass people. Once i have a website running i have to be prepared for lots of user feedback; and have a support forum in place. So still lots of work, aside from the scripts themselves.

But by all means - send me a PM! After all, you want to experience it for yourself. ;-)

Update: remote updating works! Quite exciting, actually. ;-)

Don't forget to send me a PM.

Great work sub.mesa, I wish I had not started my system yet so that I could give it a whirl, ah well...

By the way have you thought of a name for this yet?

The drives on my Tyan B26 "Tank" have been swapped out with new 2TB drives awaiting this day. The Norco 4220 is waiting in the wings to attach to the Tyan. Bring on the alpha!

sub.mesa said:

Alright all core functionality except file sharing (Samba/NFS) is currently implemented.


But by all means - send me a PM! After all, you want to experience it for yourself. ;-)

Click to expand...

Have sent PM...Hope you can give us a peek this weekend as this is my weekend off.

N.B. I reached 85+ MB/s transfers between Win7 and FreeBSD. System would crash until I used your ZFS tweaks. (3 x 2TB raidz) test bed. Opensolaris would reached a top but steady rate of 55MB/s.

Yea i have to do one more thing then i'll send you guys the URL by PM.
Will be tonight; in about 4 hours or so.

pm sent, many thanks for the link.

Okay URL soon - last code overview starting now.

sudo checks and other sanity checks are implemented.
And it got a new look and logo too - though its not final yet.