ZFS File server for department

T

tocket

n00b
Joined
Oct 15, 2011
Messages
60
I'm in the process of putting together equipment for a file server based on ZFS for my university department. It will replace our old file server (which was actually used more like a backup server), which is an aging Pentium 4 workstation, running Windows 2k3 with two 500GB consumer grade SATA disks in RAID1. We are roughly 40 people at the department and currently using about 400GB of storage.

My budget is fairly tight ($2200), but here's what I had in mind:

HP ProLiant DL180 G6 (641363-425)
16 GB DDR3-10600 ECC Reg memory
2x Seagate Constellation ES 1 TB SAS-2 (ZFS mirroring)
Intel 311 series SSD 20GB (as ZLOG)

I guess I will also need some boot drive to put the OS (leaning towards OpenIndiana) on as well. Perhaps some small cheap SATA drive would do here, but I'm not sure if the server would be OK with mixing SAS-2 and SATA on what I guess is the same backplane. Also I'm wondering if it's worth making a RAID1 array with two boot disks and if you can have both hardware and software mirroring on the same controller with ZFS.

I'd be very grateful for any feedback you can provide on this setup.
 
I would recommend two smaller ssds and mirror them, since having the zil die can be a hassle. ZFS recommendations are for 1/2 your ram for the zil, so anything over 8gb is wasted here.
 
tocket said:
I'm in the process of putting together equipment for a file server based on ZFS for my university department. It will replace our old file server (which was actually used more like a backup server), which is an aging Pentium 4 workstation, running Windows 2k3 with two 500GB consumer grade SATA disks in RAID1. We are roughly 40 people at the department and currently using about 400GB of storage.

My budget is fairly tight ($2200), but here's what I had in mind:

HP ProLiant DL180 G6 (641363-425)
16 GB DDR3-10600 ECC Reg memory
2x Seagate Constellation ES 1 TB SAS-2 (ZFS mirroring)
Intel 311 series SSD 20GB (as ZLOG)

I guess I will also need some boot drive to put the OS (leaning towards OpenIndiana) on as well. Perhaps some small cheap SATA drive would do here, but I'm not sure if the server would be OK with mixing SAS-2 and SATA on what I guess is the same backplane. Also I'm wondering if it's worth making a RAID1 array with two boot disks and if you can have both hardware and software mirroring on the same controller with ZFS.

I'd be very grateful for any feedback you can provide on this setup.
Click to expand...

some thoughts
- once you have discovered snapshots to go back to a former state based on files,
you will use it. If you currently use 400GB, buy 2 TB disks

- for Raid-1, buy an extra hotfix-disk

- if you use it as a filer and not for databases, NFS or ESXi datastore, a log device
is quite worthless. Use a bigger SSD ex 128 GB as read cache instead

- The SAS Hardware Raid-Controller Smart Array P410/256MB seems quite useless for ZFS
(maybee system-disk via Hardware Raid may be an option if you need to buy, do not use on datapools)
be prepared to buy an extra HBA (Software Raid) controller like LSI 9211

- I do not know the Nic
be prepared to buy an Intel Nic

- There is no real advantage buying SAS drives, you may use three good Sata disks
beside that you can mix Sata and SAS
 
danswartz said:
I would recommend two smaller ssds and mirror them, since having the zil die can be a hassle. ZFS recommendations are for 1/2 your ram for the zil, so anything over 8gb is wasted here.
Click to expand...
Thanks for the tip. Perhaps I should start without putting the ZIL on SSDs and add them later if necessary. I think the 311 series is pretty nice for this purpose though, even if it's larger than needed.


_Gea said:
some thoughts
- once you have discovered snapshots to go back to a former state based on files,
you will use it. If you currently use 400GB, buy 2 TB disks

- for Raid-1, buy an extra hotfix-disk

- if you use it as a filer and not for databases, NFS or ESXi datastore, a log device
is quite worthless. Use a bigger SSD ex 128 GB as read cache instead

- The SAS Hardware Raid-Controller Smart Array P410/256MB seems quite useless for ZFS
(maybee system-disk via Hardware Raid may be an option if you need to buy, do not use on datapools)
be prepared to buy an extra HBA (Software Raid) controller like LSI 9211

- I do not know the Nic
be prepared to buy an Intel Nic

- There is no real advantage buying SAS drives, you may use three good Sata disks
beside that you can mix Sata and SAS
Click to expand...
Thanks Gea,

Yeah, I have discovered the beauty of snapshots, but perhaps not realized how much more space they require :)

There won't be any databases on it, at least not heavily used ones, so using a L2ARC would seem the better option then. However, I think I will evaluate the system without L2ARC first and add one later if required.

The NIC is a HP NC362i Integrated Dual Port Gigabit Server Adapter and the server is supposed to support Solaris 10, so I hope it will work. But I will definitely go with Intel if there are problems.

I thought the P410 was OK to use with ZFS on HP servers though? I heard that it's flaky if used on other platforms, but solid on HP hardware. Is it the same with all HP Smart Array controllers?
 
Last edited:
tocket said:
I thought the P410 was OK to use with ZFS on HP servers though? I heard that it's flaky if used on other platforms, but solid on HP hardware. Is it the same with all HP Smart Array controllers?
Click to expand...

Its not a good idea to use a hardware-Raid controller with ZFS at all.
Its much better to use onbaord Sata via AHCI or a LSI HBA without any Raid functions.

Reasons for ZFS Software Raid:
Faster with modern CPU, full ZFS error control, no write hole, ZFS pools are controller independent and movable
 
Skip hardware raid - that is a no-no with ZFS. Read the ZFS article on wikipedia.
 
Ah, so you can mirror rpool using ZFS? I hadn't thought about that, but that's a much better solution.

I'm guessing the backplane of the DL180 is connected to the P410 via a SFF-8087 cable though, so I can't use the normal SATA contacts on the mobo (I'm not even sure there are any!). That means I'd have to get an additional controller card (for which something based on the LSI2008 would be my first choice).
 
Consulting the manual for the server I gathered that it does have an ICH-10R controller onboard and 6 SATA connectors. That should mean I could use those ports instead of the P410. You can use reverse SFF-8087 to SATA breakout cables to connect the ICH-10R to the backplane right?

I still have not managed to find out if this particular server is a 4 or 8 drive model. According to the picture on HPs site it's 8 and the quickspecs say that 8 drive models use P410 while 4 drive models use B110i.

There is supposed to be an upgrade from the 4 drive to 8 drive which is decently priced though (506924-B21).
 
If you must use a hw-raid card, then you need to flash away the raid functionality. Use JBOD-mode or IT-mode. Get rid of hardware raid functionality before experimenting.
 
tocket said:
Ah, so you can mirror rpool using ZFS? I hadn't thought about that, but that's a much better solution.
Click to expand...

Definately mirror your boot drive, you can use small sata 2.5" HDDs. They are really cheap, just get two and mirror them.
 
I went ahead and ordered the system, which now looks like:

HP ProLiant DL180 G6 (641363-425)
3x 4GB DDR3-10600 ECC Reg
2x Seagate Constellation ES 2 TB SATA
2x Intel 320 Series 40 GB (mirrored for rpool)
HP Dedicated management port kit

I will then use a 4x SATA to SFF-8087 cable to connect the ICH-10R to the backplane. I plan to hide the SSD boot drives away inside of the chassis, so that I can use all front slots for the main storage pool.

What OS would you recommend for this application? My gut tells me OpenIndiana, since I don't need encryption, but I have never used it (I run Solaris 11 at home) so I'm not sure.
 
OpenIndiana is a fork of the Solaris 11 source code. Solaris 11 Express is build 151a of the S11 source code. OpenIndiana is build 147 of the S11 source code - in other words there are only minor differences between S11E and OpenIndiana. Solaris 11 is build 173, so there are larger differences.
 
Solaris 11 is obviously the most up-to-date version, but from what I understand you need to pay to get updates for it. That's why I was thinking OpenIndiana is probably the better choice since I will most likely need security fixes for a production server like this.
 
tocket said:
Solaris 11 is obviously the most up-to-date version, but from what I understand you need to pay to get updates for it. That's why I was thinking OpenIndiana is probably the better choice since I will most likely need security fixes for a production server like this.
Click to expand...

you not only need to pay for Solaris 11 if you want updates, you must pay if you want to use it.
Its only free for noncommercial and developer use. But universities get discounts
 
You can also use nexentacore - based on OS kernel with debian userland code. It is actively maintained and free for personal use too.
 
danswartz said:
You can also use nexentacore - based on OS kernel with debian userland code. It is actively maintained and free for personal use too.
Click to expand...

I would not use Nexentacore for a new config. It is end of live.
The successor Illumian will be out in a few weeks (based on Illumos, just like OpenIndiana but with Debian-like packaging)
and will replace current Nexentacore and will be the base for next NexentaStor. Because there is no update-path and
tools and handling is quite different, i would skip NexentaCore.
 
danswartz said:
Has nexenta said this, or is this your guess?
Click to expand...

some links about future plans

nexenta.org
http://www.nexenta.org/boards/1/topics/3376

OpenIndiana developer meetings
http://openhatch.org/meeting-irc-logs/oi-meeting/2011-11-29.log.html

Illumos-discuss archiv
http://www.listbox.com/member/archive/182180/2011/sort/time_rev/


Its not only a guess. It is the next step for Nexenta - moving from last free OpenSolaris 134 Kernel
(base of current Nexenta) to Illumos (the free fork of OpenSolaris Kernel development) with a current Kernel 151+

Currently the only unclear topic is the relation between Illumian (Illumos + Debian packaging - driven by Nexenta)
and the OpenIndiana project (Illumos and IPS packaging) - if they compete or cooperate or even merge in building a distro - i hope for the last.

But of course, Nexenta did not say - skip NexentaCore and wait - or use OpenIndiana/ Solaris instead.
That is my opinion.
 
Last edited:
Yes sorry, I was unclear. You make it sound as if their community NCP is dead going forward, which is a complete guess, IMO. I don't really care whose kernel they are using, it is everything else about it that is relevant to many people making this choice.
 
danswartz said:
Yes sorry, I was unclear. You make it sound as if their community NCP is dead going forward, which is a complete guess, IMO. I don't really care whose kernel they are using, it is everything else about it that is relevant to many people making this choice.
Click to expand...

Its not only a kernel-update. Its a break and a OS-move from the former "Ubuntu with a Solaris kernel idea" to a pure Solaris derivate with Debian-like handling to keep look and feel for current customers.

All apps from NexentaCore came from Debian/Ubuntu where the Illumian ones are the same like the OpenIndiana ones - same as Solaris. This means that not only setup procedures like network settings are different. There are some apps no longer available and others to come. Repository is completely different. Only handling keeps Debian-like.

So its a complete new OS for Nexenta. Its like
"The old king is dead, long live the new king"

I appreciate this because it strengthen the free Solaris derivates and concentrates development efforts to fewer options.

The real successor of Nexenta (Debian with Solaris Kernel) may be StormOS.
(If development is not stopped, developer is working at Nexenta now)
 
The file server is now up and running! I decided on OpenSolaris, which appears to be working great. I did not need to install any drivers, so the installation was super easy. The first thing I did when I got the server was to rip the P410 out, as per your recommendation :)

Thanks to everyone who helped!

I now know that this particular server only has 4 HDD bays, but it looks like it's easy to upgrade it to 8 or 12 (just change the backplane). I was surprised that the only available power connectors were one Berg connector and one 8-pin connector. No molex or SATA! I needed SATA power connectors for the boot drives, which are not mounted directly in the chassis, so I had to make my own cable here.

I was also a little bit disappointed with how HP forces you to buy their disks, since the server doesn't come with any empty caddies and you can't buy them either. Luckily we had some old drives that were no longer being used that I could take the caddies from.

The integrated NIC was automatically detected and installed with Intel drivers and seems to work fine, so I suspect it's just a rebranded Intel NIC.

Now, I have one more question. I'm about to set up the ZFS folders and I'm wondering what the best way to do this is. Do I make a ZFS folder in napp-it for each user or do I just create a users folder and then use Windows to create the individual user folders? I'm expecting 30-40 users and I want to be able to set a quota for everyone. I also expect some Mac users, if that changes anything.
 
tocket said:
Now, I have one more question. I'm about to set up the ZFS folders and I'm wondering what the best way to do this is. Do I make a ZFS folder in napp-it for each user or do I just create a users folder and then use Windows to create the individual user folders? I'm expecting 30-40 users and I want to be able to set a quota for everyone. I also expect some Mac users, if that changes anything.
Click to expand...

You may go both ways. If you use user-quotas, you must set via CLI see
http://docs.oracle.com/cd/E19253-01/819-5461/gigaz/index.html
All user folders are below the same share.

If you create a ZFS folder for each user, each folder is then a share and can have
independant ZFS properties

about Macs
Easiest is SMB only with the Problem that SMB is currently not working with OSX Lion.
(Problem is already fixed in Illumos). AFP is needed for TimeMachine only.
 
Thanks Gea, that's very useful information. I think I will try with user quotas first.

Upon further investigation I found out the NIC is actually Intel 82576, that's good right?

On a side note, this place has a pretty good internet connection:


and I get 842/476 Mbit/s using bredbandskollen.se :)
 
If you have only 40 users, then I would create an individual zfs filesystem for each user. If you have 1000s of ZFS filesystems then it might get slow. But you have less than 100, which is no problem.
 
I have run into an unexpected problem. The SMB/CIFS service stops working after a period of time. When this happens it is no longer possible to access the shares until the service is restarted.

Any idea what could be causing this and how to fix it? I couldn't find anything in the system log file viewer.

Of course I could periodically restart the service, but that's hardly a good solution to this problem.
 
Is the service not running, or shares just got marked as down? I see the latter occasionally, and clicking on them in the client restores them. Only happens occasionally so I haven't been motivated to root cause it.
 
I also see the shares getting marked as down temporarily, and sometimes they quickly switch back and forth between online and offline (several times a second). That is mostly annoying when it happens, but ideally I don't want to see it at all.

Unfortunately the main problem is when SMB becomes unresponsive for extended periods of time (or until I restart the service).

I think the service is still running when it happens, but I have to check the next time I see it. Is "svcs | grep smb" a good way to do it?
 
tocket said:
Is "svcs | grep smb" a good way to do it?
Click to expand...
To see its current status, yes

If necessary svcs -x to see why it offlined.

What OS did you go with? Ever since OpenSolaris b134b (which, IIRC is the only publically available build of OS now) I've had issues with using concurrent connections using DNS aliases. Are you doing that? When this occurs, can you access the shares from another machine, maybe one that would not have active connections to this box already (are mapping drives via script/GP?).

I've also had issues with authentication to the SPN (DNS alias is fine...), but that's more obvious since you get an access denied. I think there's a bug report about that. This was both for OI 148-151a and Solaris 11 Express.

PS It's "normal" for drives to be marked disconnected after an idle period. IIRC default is 15 minutes. There's an MS KB on it if you'd like to change it.

*skimming thread... at work*
 
It's running OpenIndiana (oi_151a).

I'm accessing it via its local hostname (needed to set ACLs) and IP directly, but it doesn't seem to matter which way I do it - when the shares go offline they can't be accessed either way. I've only tried accessing the shares via "Map network drive...", the Run dialog and network browsing though.

It seems that it started working again now without a restart, so I have to wait for it to break again to do more tests.
 
Now it's done it again. I get a whole bunch of different errors when trying to connect to the shares from windows.

An error occured while reconnecting O: to \\carbon13\users
Microsoft Windows Network: The local device name is already in use.
Click to expand...

O:\test is not accessible.
The file is currently not available for use on this computer.
Click to expand...

\\x.x.x.x is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.
The network address is invalid.
Click to expand...

If I do an nmap scan of the server it shows two ports as being filtered:
Code: 
PORT     STATE    SERVICE
22/tcp   open     ssh
81/tcp   open     hosts2-ns
111/tcp  open     rpcbind
139/tcp  filtered netbios-ssn
445/tcp  filtered microsoft-ds
515/tcp  open     printer
548/tcp  open     afp

Yet, the smb service appears to be running:
Code: 
erik@carbon13:~$ svcs | grep smb
online         Jan_03   svc:/network/shares/group:smb
online         Jan_04   svc:/network/smb/server:default
online         Jan_04   svc:/network/smb/client:default

So why did it stop responding?
 
Anything in dmesg on OI?

Packet sniff on client box and see what is going on?
 
tocket said:
Now it's done it again. I get a whole bunch of different errors when trying to connect to the shares from windows....
?
Click to expand...

I use a OI 151a SMB filer (W2008 AD Domain) with hundreds of students without problems.
I would first try to replace the network adapter (use Intel) and keep all network settings per default.
(ex no special mtu or link aggregation).
 
I reinstalled OI and will try to set it up again, hopefully without the same problem. I also had some issues where the napp-it webserver shut down randomly.

The NICs are Intel 82576 (just relabeled by HP). I haven't made any changes there.
 
Nope, got the exact same problem again. There is something that might be interesting in dmesg. Last thing before it stops working:

Code: 
Jan  9 16:43:30 carbon sshd[802]: [ID 800047 auth.error] error: accept: Software caused connection abort
Jan  9 16:43:30 carbon napp-it-mhttpd[1166]: [ID 895811 daemon.crit] accept - Software caused connection abort
Jan  9 16:43:33 carbon inetd[713]: [ID 702911 daemon.error] accept: Software caused connection abort
After that both smb and napp-it stop working. I think I will have to try a pure Intel NIC, because it may indeed be the problem.
 
So I brought an Intel NIC from home to install in the server, but when I open the server up I realize that it only has PCI-E slots (1 x16 and 3 x8). That's too bad since I only have PCI NICs.

Now I'm looking at buying an Intel NIC and notice there's a new one called I350-T2. Does anyone know if it works in oi_151a?
 
Here's a new error message, seems to have coincided with SMB stopping to respond quite well. What does it mean? A quick google search on "nbns: packet decode failed" only comes up with the source code for opensolaris.

Code: 
Jan 10 16:33:34 carbon smbd[856]: [ID 592507 daemon.error] nbns: packet decode failed

I have seen on a couple occasions that napp-it stops responding a couple of minutes after SMB. Strange coincidence?
 
I decided to go with an Intel Gigabit ET Dual Port Server Adapter (82576). Unfortunately I'm still experiencing the same problem with the SMB service as before, so the problem does not appear to be the network interface.

I haven't tried booting from a Solaris 11 live CD, since that would mean I'd have to run the server without napp-it, and I don't know how to configure SMB from a CLI.

I really can't understand what's wrong with the server.

When I did the nmap this time it's only port 139 that shows up as filtered, 445 is still open. I thought both were SMB?

Code: 
PORT    STATE    SERVICE
22/tcp  open     ssh
111/tcp open     rpcbind
139/tcp filtered netbios-ssn
445/tcp open     microsoft-ds
515/tcp open     printer
548/tcp open     afp
 
I'm giving up on OpenIndiana and switching to Nexenta. So far it's working well. Hopefully it stays that way.

Reading the user guide for Nexenta I'm thinking that I should set up a domain controller, since there are so many users in this department. It makes sense to me to have all the user accounts in one place, since it's a pita having to manage them otherwise.
 
You must log in or register to reply here.
Back
Top