Building your own ZFS fileserver

yea, I tried dedup, it worked badly. I didn't have enough ram (only 4GB of ram on my 5x750 raidz2 array) and it caused hard drive thrashing and terrible performance. I tried a pretty huge CPU upgrade (2ghz athlon64 x2 to 3ghz athlon II x4) and no difference in speed. I ended up disabling it.

I will just stick with solaris until FreeBSD stable release with version higher then mine is released.
 
You need about 32 gigabyte RAM and an SSD to use de-duplication effectively. It also can corrupt your pool; it is not yet stable!

Solaris 9/10 release has this functionality disabled.
 
This has probably been answered somewhere, but I haven't been able to find it..

Can you use different size drives in a ZFS pool? How about different mfg / models of the same capacity?
 
Yes you can, but you should keep disks in the same vdev the same.

Bad config:
vdev1: RAID-Z 500G disk 250G disk 250G disk
vdev2: RAID-Z 500G disk 250G disk 500G disk

Good config:
vdev1: RAID-Z 500G 500G 500G
vdev2: RAID-Z 250G 250G 250G

For performance reasons, you would want the performance of each vdev to be about the same, so don't try anything too extreme like:
vdev1: RAID-Z2 of 10 disks
vdev2: single disk or mirror

You can do that, but it would lower the performance of your 10-disk RAID-Z2 array, it wouldn't make much sense to do it like this.

But a common practice is to start with RAID-Z1/2 of 1TB disks, then later buy a batch of 2TB disks, and in future you may want to add 3TB or 4TB disks. That's possible, but you have to think about what config:

- mirrors of 2 disks
- raid-z of 5 disks
- raid-z2 of 6/10 disks

Then you start with your first batch of HDDs:
vdev1: RAID-Z of 5 disks

Then later you buy 5 more 2TB disks and add to the pool:
vdev1: RAID-Z of 5 disks (1TB)
vdev2: RAID-Z of 5 disks (2TB)
 
Hi guys, nice thread.

Freebsd8 STABLE can run zfsv28

On my VM

root@vm zfsnap # uname -a
FreeBSD vm.xxx.org 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0 r216674M: Fri Dec 24 00:24:34 UTC 2010 [email protected]:/usr/obj/usr/src1/sys/VM80 amd64

ZFS filesystem version 5
ZFS storage pool version 28

and ahci :)

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 8192MB (16777216 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 8192MB (16777216 512 byte sectors: 16H 63S/T 16383C)

I am runnign a 2 disk mirror.

FreeBSD in terms of filesystem and i/o did stall in terms of development, between v5 and v6 was stuck with stagnant ata code and ufs code. Both outdated in my view, no jounaling support etc.
V7 got zfs which was a massive breath of fresh air and by the time 8.0 was released it matured enough to be considered stable.
v8 then got ahci which is a entirely freshly written bit of code not on by default but very easy to enable, which gives NCQ support on sata.

zfs is gettng tons of attention on freebsd as its a way to make freebsd stand out (currently) and bring more modern filesystem features to the OS. Considering a patch has been made for STABLE I wouldnt be surprised at all if 8.3 gets zfsv28. As far as I know zfsv28 includes the metaslab patch but would be nice to have that confirmed for peace of mind.

To upgrade a freebsd8 system to zfsv28 the patch is in the url below along with instructions, you need experience in compiling world/kernel from src. Best to use src code from late december only to ensure compatability.

http://lists.freebsd.org/pipermail/freebsd-stable/2010-December/060643.html

That is the patch I used, I reccomend not doing a reboot between the installkernel and installworld for this.

Issues so far?

There is a zfs warning during boot on the console, dont know the exact nature of it yet, its a syntax error warning, so I assume a zfs command in one of the boot rc.d scripts is incorrect syntax for v28. Also the py-zfs port gets marked as ignore on v28, so that either is broken on v28 or v28 brings support of those functions to the core os files.
 
Python is being stripped out; so soon you won't need those py-zfs ports anymore but it would be native C applications without depending on Python; the IllumOS project has rewritten these userland utilities in C instead.

About UFS and journaling: multiple methods are used to ensure data consistency on UFS:
- SoftUpdates (since 4.x i think); not journaling but something a little more cool; especially since this can improve performance rather than tax performance.
- geom_journal provides both metadata and data journaling to UFS
- SU+J combines the best of soft updates and UFS journaling and would be the recommended setup if you still want to use UFS

ZFS of course is much better and uses a journaling design; which does have a tax on performance. This you can test by disabling the ZIL; not recommended for real data!

Please keep in mind the ZFS v28 patch is highly experimental and could eat your data. One possibility is that a bug in ZFS v28 changes data on your pool, and reverting back to V15 won't work anymore and it would say corrupted metadata; in that case all your data is gone on the pool; everything. That's why you shouldn't use ZFS v28 for important stuff just yet; only for testing.

The ZFS v28 patch is nearing maturity though; it should indeed be merged to 8-STABLE after a while, after which is can be implemented in 8.3. Not sure whether 9.0 will come before 8.3, but that's certainly a possibility. Think about summer 2011 when ZFS v28 is considered production ready and matured.
 
sub.mesa said:
Please keep in mind the ZFS v28 patch is highly experimental and could eat your data. <snip> Think about summer 2011 when ZFS v28 is considered production ready and matured.
Click to expand...

This is what rubs me the wrong way about the idea of running a ZFS implementation that's been hacked into a non-solaris O/S. I'd say if anyone cares about ZFS being "production ready", rather than wait 6+ months they would try Solaris Express 11 with ZFS v31, which adds encryption, has broad hardware support and runs beautifully virtualized (EXSi 4.1 w/ LSI SAS2008 hardware passthrough in my case).

Not to discount the fine work sub.mesa has done, and I know FreeBSD has its merits, but so far I haven't found a downside to SEX11 and overall pretty impressed.
 
Last edited:
bear in mind zfsv28 is a new path so yes whilst he said there is a risk the risk is tiny and is of course a alpha patch at current, once its merged it will probably be considered stable code. I have never lost data on zfs, and here is what my VM has been through (on zfsv28).

1 - virtualbox freaked out and I had ahci errors essentially causing a hdd to timeout and dissapear, zfs reverted to a degraded state until I rebooted, nothing noticeable happened no crash or panic. after a reboot it auto checked for errors and found none, I ran a scrub and that was fine also.
2 - I then resized my drives, 4 gig to 8 gig. booted up with one drive removed, and a empty 8 gig drive added. replaced the old drive in the pool with the 4 gig and it resilvered the data no problems, repeated procedure for other drive. then zpool auto grew to the new size.
zfs has a lot of data integrity protection so losing data on a mirror setup is safer than older types of raid including hardware raid. and in my view it remains faster than ufs due to superior queuing and caching algorithms. so dont get scared because someone freaks out about a patch that hardly anyone is using.

the VM has no important data on, I am notusing v28 on any production machines.
 
odditory said:
This is what rubs me the wrong way about the idea of running a ZFS implementation that's been hacked into a non-solaris O/S.
Click to expand...
I would not use the word hack; some very knowledgeable developers are working on ZFS integration, and have improved it beyond what is possible on Solaris; for example booting on RAID-Z and RAID-Z2. The implementation is very clean, kernel-level and shouldn't be defined as 'hackery'; a term that may apply to the FUSE layer, trying to get ZFS to work that way.

By the way; FreeBSD 9.0 would be released around summer 2011, and most likely feature ZFS v28 at that time. But ZFS v28 will be merged to 8-STABLE before that time; perhaps in few months from now. At that point it would be less experimental, but i still would recommend caution about using a too new/beta ZFS release; you did not choose ZFS to corrupt your pool on some obscure bug.

That said, this is the second ZFS v28 patchset with most/all features done, meaning the maturing process has commenced and lots of people will start testing this. Wouldn't it give you a safe feeling that your filesystem has been tested thoroughly when you trust your valuable data to it?
I'd say if anyone cares about ZFS being "production ready", rather than wait 6+ months they would try Solaris Express 11 with ZFS v31, which adds encryption, has broad hardware support and runs beautifully virtualized (EXSi 4.1 w/ LSI SAS2008 hardware passthrough in my case).
Click to expand...
Except that Solaris Express 11 is not a stable production release at all, according to wikipedia it is "licensed for testing only"; hardly to be called a matured production-ready release.

Instead, the Solaris 10/9 release has ZFS v22 with v21 - deduplication - disabled. So you could say v20/22 is the stable version on Solaris platform that commercial clients run.

The whole idea of using ZFS is protection for your data. So why would you want to sacrifice that by running an alpha/beta/testing version of that filesystem? Do you want to fly in an airplane that runs beta-quality software? No i don't think so; your data should equally need protection if you want serious protection that ZFS can offer.

Besides, not that many cool features between V15 and V31; the zfs-crypto integration is nice and other features as well; but the best ZFS features are stable and as such that's what you should use. V15 should be fine, until V28 is matured.

Generally you could say the more days have passed since a release and the more people using a piece of software tells something about possible bugs/issues. If you use something that's not tested widely then it has the potential of very dangerous bugs that only in specific situations present themselves. Such corruption bugs in de-duplication did occur in ZFS; and i think people should not experiment with their real pools but instead use the alpha/beta ZFS versions for testing and keep stable ZFS for their real data.

Not to discount the fine work sub.mesa has done, and I know FreeBSD has its merits, but so far I haven't found a downside to SEX11 and overall pretty impressed.
Click to expand...
Solaris should be a decent OS and if it works for you then i see no reason for you to switch to FreeBSD.

Being a true open source project that also means i can trust that ZFS in FreeBSD will stay without being dependent on a company with different interests. For the long term FreeBSD is quite a safe place to be as ZFS enthusiast and for myself as developer of a ZFS-related distribution; if i had used OpenSolaris i would be in trouble right now.

I think several OS implementing ZFS should give the user much better choice and better chance the user finds a project to his liking.
 
Last edited:
sorry sub.mesa I should have warned others the v28 patch is testing only and not marked as stable.
 
@sub.mesa good points, but mine wasn't that solaris express 11 was a production O/S - i was quoting your previous post. i meant the readiness of ZFS in SolarisExp11 versus the readiness of the latest patch/port of ZFS in FreeBSD. good point about later pool revs not being all that compelling in terms of featureset. regardless I do think SolarisExp11 at least appears viable in its current state, even for hobbyists/enthusiasts/home users, based on my my own testing plus all the blogs I keep reading about admins migrating from OpenSolaris to SolarisExp11 (FWIW).

I understand people don't like Oracle in the same way some people don't like Apple or Microsoft. Yes, key ZFS people left after the Oracle takeover, but if they ever manage to put out killer features like block rewrite/pool resizing/defrag, I won't care. And if I had to guess I'd say we'll sooner see those types of enhancements and features come to ZFS come from Sun/Oracle than from the opensource community. I'd love to be wrong. Granted, the barrier to entry for community contributions is high due to lack of documentation.

To SolarisExp11's credit, while it may be labeled as "for testing", that may well have more to do with marketing/political purposes than a reflection of a "beta" status, and nonetheless its still a fully supported O/S, being worked on by people with a financial incentive to get it right.

Anyway, like you said, if we're going to talk about ZFS let's keep an open mind and consider every significant platform so users *can* make an informed decision. Now if you'll excuse me I'm off to try the latest zfsGURU LiveCD.
 
Last edited:
I tend to agree with odditory - forking a production code base and expecting the open source efforts to effectively take over and add features is a tall undertaking. Not impossible, but hard.

However, Oracle has it's advanced filesystem in development, BTRFS, and is integrating code in linux in a better way than Sun did. So I'm not sure that we can expect a lot of new development from them on ZFS. I would be really happy if they did (I think ZFS is way better than BTRFS), but I wouldn't depend on it. Oracle bought Sun, not the other way around.

Opensolaris has a few major advantages over FreeBSD, the main one being that CIFS support is in the kernel and and works WAY better than SAMBA, which has been the weak link in many linux based servers performance wise. CIFS blows SAMBA away in performance and consistency, and only opensolaris has it.

Ultimately, the best situation would be for a real production effort by Oracle to integrate ZFS into the linux kernel for wide distribution. FreeBSD has better driver support than Opensolaris, but way less than for Linux. But I am pretty sure this is not going to happen.

Such is life.
 
You guys make very very very good points

But hatred of MSFT and AAPL is nothing like the hatrd for Oracle?
 
odditory said:
I understand people don't like Oracle in the same way some people don't like Apple or Microsoft.
Click to expand...

there is no comparison really. Oracle went as far as suing apple for using java (which was open sourced by Sun and supposed to be safe against lawsuits). A huge subsection of the FOSS community swears by java, and now Oracle put it in jeopardy. At their darkest hour MS and Apple weren't as bad as oracle is.

but if they ever manage to put out killer features like block rewrite/pool resizing/defrag
Click to expand...
You seem to be under the mistaken impression that ZFS is not open source, and that open source is random unemployed programmers who do it as a hobby.
The vast majority of open source programming is done by people employed by corporations. Corporate backing of open source is huge. Google is a major player in open source, both directly (google chrome, google video codecs, etc), and indirectly (main financiers of mozilla, google code repository, and more).

@mikesm: An alternative to open solaris is illumos (and its various build flavors). Illumos is a fork of opensolaris that is meant to be binary compatible with solaris, and remove all closed source dependencies in opensolaris. Oracle shut down opensolaris, but it cannot shut down illumos. Illumos was created by several forward looking companies who were using opensolaris technologies and did not want the risk of being kneecapped by Sun or anyone who bought them, and it was a wise move because that is exactly what happened a few months later.
 
Last edited:
taltamir said:
You seem to be under the mistaken impression that ZFS is not open source, and that open source is random unemployed programmers who do it as a hobby.
Click to expand...

I wasn't under that impression, but feel free to put words in my mouth. I wasn't talking about opensource in general, but about what I've seen from the opensource community for ZFS specifically - which doesn't appear to be much beyond porting efforts. GreenBytes has done some interesting things but they're a commercial third party.

When FOSS creates a significant new feature or enhancement to ZFS with the opensource code, feel free to post a link so we can read about it. That wasn't sarcastic.
 
Last edited:
odditory said:
I wasn't under that impression, but feel free to put words in my mouth. I wasn't talking about opensource in general, but about what I've seen from the opensource community for ZFS specifically - which doesn't appear to be much beyond porting efforts. GreenBytes has done some interesting things but they're a commercial third party
Click to expand...
my apologies for jumping to conclusions then. But, the development by Sun IS open source development. I think you meant development that is "not first party" rather then "open source community"
because the open source community is a bunch of corporations, mostly.

When FOSS creates a significant new feature or enhancement to ZFS with the opensource code, feel free to post a link so we can read about it. That wasn't sarcastic.
Click to expand...
Versions 1 through 28 of ZFS were developed by the FOSS community, specifically, by the corporation known as Sun microsystems under the CDDL license (pardon my tautology).
 
Hi guys! My first ZFS-setup really fucks with my brain here.
Hardware:
- Athlon XP 2200+
- Asus A7N266-VM (nForce 220)
- 1 GB RAM
- STLab A-224 4-port SATA/RAID-controller (SiI 3114)
- WD 80 GB IDE
- 3x Samsung F4EG 2TB, HD204UI (ad4, ad6, ad 8)
- PCI GbE NIC
- FreeBSD 8.1
The SATA-controller is only able to run in legacy IDE-mode.

Code: 
dd if=/dev/zero of=/dev/ad4 bs=10M count=100
gives me 55 MB/s, which I think is OK.
Code: 
dd if=/dev/ad4 of=/dev/null bs=10M count=100
gives me 88 MB/s, which is the maximum speed of the drives. Also OK.

But when I go
Code: 
zpool create lagring raidz /dev/ad4 /dev/ad6 /dev/ad8
dd if=/dev/zero of=/lagring/fil.bin bs=10M count=100
I get a lousy 8 MB/s and strange rythms from the drives.
Code: 
dd if=/lagring/fil.bin of=/dev/null
gives me 22 MB/s which is also bad.
I also tried gnop with (destroyed the zpool first):
Code: 
gnop create -S 4096 /dev/ad4
...
zpool create lagring raidz /dev/ad4.nop /dev/ad6.no /dev/ad8.nop
dd if=/dev/zero bs=/lagring/fil.bin bs=10M count=100
resulting with the same lousy 8 MB/s.

So now I have run out of ideas. I don't know how to figure out whether it is a software- or hardware-problem. My first guess was limitations in the PCI-bus, but that doesn't make sense with those raw single-speeds. Secondly I though the SATA-controller was too cheap, but Google found lots of positive experiences with SiI3114.
Is it the lack of 64-bit-support? Or not enough memory? I have used the 1GB-memory-tuning-example from the FreeBSD handbook in /boot/loader.conf.
Code: 
vm.kmem_size="330M"
vm.kmem_size_max="330M"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="5M"

I don't have use for extreme RAID-speed, but I want a decent 50-60 MB/s read and write-speed. I guess I won't be able to push IO-speed much futher than that with both SATA-controller and NIC running on the PCI-bus. Is 50-60 MB/s possible with this hardware?
 
You got both too few RAM and you're using a PCI controller; which totally destroys performance especially in RAID-Z where all disks are accessed at the same time; but only PCI does not allow this and thus you would get a disk 1-2-3-4 serial access pattern instead.

If you want to reuse old hardware, FreeNAS with UFS may be a more higher performance choice. But still PCI is going to be very bad. Too bad it's still in wide usage, PCI-express has been out for at least 6 years but PCI storage for reason is still very popular. It performs very bad on ZFS, however.

ZFS also runs quite poor on 32-bit hardware. ZFS runs great on cheap but modern hardware, though.
 
Ok, I see. A HP PL ML110 G6 is on its way (ordered a week ago), but I forgot to order extra memory. Is it possible to achieve about 80-90 MB/s on such modern HW, 3xHD204UI and 1 GB RAM?
 
For three disks 1GB could do actually. But you should know that ZFS performance basically scales with RAM; 1GB RAM means the lowest-end ZFS server; generally i recommend 6GiB minimum and 4GiB absolute minimum. ZFS can crash with too few memory.

Ideally you would want at least 8GiB RAM in a new ZFS server. That's just 110 euro for 8GiB DDR3 ECC; about the price of a dualcore cpu. Many systems used for ZFS are memory-starved.

But you can try with 1GiB; it should run, but it may not be very fast; still you should get most of your gigabit capped speeds i think; random I/O is another story though. And stability may also be in question. I believe 1GiB RAM systems were around 2000; now we're 2010 and 8GiB could be considered normal this age.

Trying a Solaris-based distribution might get you higher speeds; since these appear to work better with few memory (i.e. more is given to ZFS).
 
Thanks! I will give it a try, very aware of the memory issue. The load is almost 100% sequential and only used by one or two users simultaneously.

It is ridiculous to ship a "professional" server with only 1 GB of RAM these days, but the order was made in a bit of a hurry. The prize isn't that much of an issue (~150 EUR for 8 GB ECC), but orders, mailing and vacation is a bit of a mess.
 
I know, besides it's just ZFS that has much different hardware requirements. 8GB RAM for a software RAID or hardware RAID fileserver would be excessive by traditional standards. ZFS is different; it eats RAM.

The idea is that because you can buy non-TLER disks (Samsung F4), non-RAID controller and forget about UPS/BBU for data integrity, you save up money to spend on disks and RAM; which is what ZFS wants. :)

That said, i run ZFS on 32-bit 500MHz CPU with 1GB RAM; it is possible, but with careful tuning. Without tuning it's easy to crash such a system. amd64 (meaning 64-bit) already works much better in that regard.

A very low-power ZFS platform would be the upcoming 9W/18W AMD Zacate platform with Ontario/Brazos CPU; like the AMD competitor to Atom but then faster and lower platform power, since it includes alot of stuff and the chipset is nothing more than an I/O hub. This gives you 6x SATA plus PCI-express x16 slot; so another HBA = 14 disks. I posted a picture of that board, a couple pages back in this thread. It should be available in a couple of months.

If you can't do home-built stuff but want/need to buy via OEM vendors, then you should take care to look at your priorities: performance, scalability, stability (ECC), ease of use, cost and power consumption.
 
odditory said:
@sub.mesa good points, but mine wasn't that solaris express 11 was a production O/S - i was quoting your previous post. i meant the readiness of ZFS in SolarisExp11 versus the readiness of the latest patch/port of ZFS in FreeBSD.
Click to expand...
Readiness? What exactly does that mean. :D

If you mean that Oracle will always be ahead of ZFS versions, then YES! They will release it prior to anything else, as test release.

If you mean stable releases usable for real data, then FreeBSD lags decently behind Oracle, but not that much. The ZFS v28 support is being finalized. It may still take Oracle some time to push beyond the current v20/22 stable versions. So if Oracle gets v31 to production releases and FreeBSD manages to get v28, i would be very content!

The v28 patch on FreeBSD has been available for some months, though. But during that time bug-fixing and code review has matured the code to the 'beta' phase. Now it would be a matter of weeks or a few months before it reaches 8-STABLE; production quality. The first real release with production ZFS v28 may not be until summer 2011 though. But i'll ship stable v28 when it is deemed stable by the FreeBSD devs.

regardless I do think SolarisExp11 at least appears viable in its current state, even for hobbyists/enthusiasts/home users, based on my my own testing plus all the blogs I keep reading about admins migrating from OpenSolaris to SolarisExp11 (FWIW).
Click to expand...
It's fun to play with, and you can use it on your precious data as well, and it works fine! So you keep using it, until one day some bizar issue triggered a corruption bug and your essential metadata is corrupted; byebye pool!

This happens; de-duplication corruption bugs are also noteworthy.

We should realise that:
  • A filesystem is a crucial part of any system, and can wreck havoc when showing signs of corruption or other malfunction.
  • ZFS is a relatively new and highly complex filesystem and volume manager-hybrid. This means new theories, new code, and added complexity which means alot of things COULD fail, if there was a bug hidden in the code. With a simple filesystem like FAT or even NTFS that would be much less likely.
  • The risk of using an experimental version of ZFS is very 'binary'; with that i mean, it either works or it kills your data. That may be an overstatement, since there are grades of stability and maturity of the code. But the danger is that you cannot accurately predict the real danger without signs of failure. If all goes perfect for 4 months and next day it tells you the pool is corrupt; well bummer! And you may not even really know whether it was a bug in recent ZFS or something else. But that could mean your data is just gone.

If you do want to run experimental ZFS versions, then by all means make sure you invest in a really good backup; a separate system containing identical data. Incremental backups would be best. If you do that, then you can sleep well knowing that even there is a small risk of experimental ZFS eating your data, you still have invested in a true backup that will protect your data in such event. Sleep well! :)

So i guess my point is: be careful with your precious data. Not much sense using ZFS for its reliability and then lose your data due to some stupid bug in experimental ZFS code. If you got a fail-safe for that issue, fine, if your data is not that important (can be redownloaded); fine, but don't think ZFS is a magic bullet; ZFS's weakness lies in the complexity in some of the code. That increases the theoretical chance for corruption bugs to occur. Perhaps only in very specific circumstances, which escape routine testing.

Anyway, like you said, if we're going to talk about ZFS let's keep an open mind and consider every significant platform so users *can* make an informed decision. Now if you'll excuse me I'm off to try the latest zfsGURU LiveCD.
Click to expand...
I think it's important for consumers to have choice. Vendor-lockins can be nasty and if there are no good open-source ZFS projects out there that are a viable alternative to Oracle it may compel Oracle to play the bullyboy again. So choice is good, for us consumers; users of software!

Regardless of all these discussions, let's not forget what actually matters. It does not matter whether it supports V15, V22 or V28. It does not matter whether it has kernel level CIFS. It does not matter whether it can boot from RAID-Z2. It does not matter what license the project is.

What actually matters, for the real end-user, is:
  • Availability (you have to know about something to be able to choose it)
  • Features (it can do what you want from it, to a certain degree)
  • Usability (it is pleasureful/convenient to operate; includes level of expertise required to operate)
  • Documentation and Support (should you need help with anything)
  • Marketing (to get you in the right mood, to differentiate from other solutions)
  • Price and effort required to purchase/download (if it only accepts credit card it may put you off; free for download is much better of course)

Now of course 'features' is a very big issue; it's not just a matter of yes or no. Likewise the 'usability' would encompass the total experience of interacting with the device/product. Still i think we should consider technical arguments as inferior to the real important things that makes a person decide for one solution over the other.

Each project has their own merits and thus there is no 'best choice'; except for my project, of course. :D

Oh pssst; i got a new preview3 iso up for download; with ZFS v28 system image. Very experimental, of course. ;)
 
Last edited:
mikesm said:
I tend to agree with odditory - forking a production code base and expecting the open source efforts to effectively take over and add features is a tall undertaking. Not impossible, but hard.
Click to expand...
"Take over" may be too strong words, but we can see code sharing between several projects like IllumOS and FreeBSD for instance. This means that all the open source initiatives that have a strong focus on ZFS, can share their work on ZFS.

The only problem would be when there would be changes to the on-disk format, making it incompatible with Oracle ZFS. We'll have to see what Oracle will publish as CDDL code; if post-v28 versions will be released as CDDL, then the open source projects could import the changes and migrate to their codebase, such as been happening to date.

However, Oracle has it's advanced filesystem in development, BTRFS, and is integrating code in linux in a better way than Sun did.
Click to expand...
Btrfs is just a filesystem, not a volume manager. You could hook it up to md-raid just fine, though. But that's not really the same power as is possible on ZFS. It's also still very experimental and not suitable for production usage, unlike ZFS which is in use by commercial vendors and actively supported by Oracle.

Btrfs would be more like the successor to Ext4, than the nephew of ZFS. That's not good, that's not bad, it's just apples vs. oranges.

Opensolaris has a few major advantages over FreeBSD, the main one being that CIFS support is in the kernel and and works WAY better than SAMBA, which has been the weak link in many linux based servers performance wise. CIFS blows SAMBA away in performance and consistency, and only opensolaris has it.
Click to expand...
It's indeed a feature specific to the Solaris platform, but it's not always a salvation. Some users even reported they consider migrating to Linux because it had Samba and OpenSolaris does not. The cifs kernel driver was also broken in b134 release, if i recall correctly.

I agree a kernel CIFS driver is sleek, though! But as bad project Samba is, the recent Samba 3.4 and 3.5 performs much better already, and support Asynchronous I/O which was lacking on FreeNAS versions for example.

So my point is, this is not that compelling an argument as it once used to be. I still would prefer a kernel-implemented over a bloated program like Samba, though, but recent versions do appear to do a decent job at both performance and features.

The booting from RAID-Z/Z2 might be a sleeker option by comparison, or the use of processor-accelerated encryption on ZFS.
 
zfs will work on low ram but it will have severe impact on performance, I now ignore threads about performance from users with less than 4 gig of ram, but I will help them to make things at least stable.

also raidz in my view will be slower than raid mirror as its basically software raid5 which has never been fast.
 
I haven't played around much with the latest SAMBA versions, but I have seen lots of people complain about performance and inadequate AD integration. If the goal behind a ZFS server is high performance windows fileserving (not necessarily required for everyone, but a pretty widespread requirement), the having a filesystem that can deliver 300 MB/s all day long limited buy a crappy SAMBA code base that can't crack 40 MB/s on teamed NIC's is kind of a problem.

That's part of the reason that Sun built CIFS server code that was designed from the ground up for performance. SAMBA 4.0 is supposed to be a lot better, but has been delayed and it's not clear the code is that good.

I also believe the illumos folks have decided to use SAMBA instead of CIFS, which further compounds the misery. FreeBSD isn't doing anything interesting about this problem and from my POV is mostly focused on porting things from linux and solaris and other OS's, and not innovating itself.

I use pfsense, which is based on the freebsd platform, but I have been unimpressed with it's reason for existence. It has a different open source licensing regime than linux, which enables porting code from opensolaris and other OS"s than cannot be ported to linux. But this is not about innovation.

thx
mike
 
mikesm said:
I haven't played around much with the latest SAMBA versions, but I have seen lots of people complain about performance
Click to expand...
Without hardware bottlenecks, you should now get proper Samba performance. You can test yourself with my ZFSguru preview3 release, which uses Samba 3.5. You can use the default tmpfs memory filesystem share, so you will test only the network and Samba performance. It should get you 100MB/s-like sequential throughput.

Samba's performance has increased, but if you use Windows XP which does not support Async I/O then you may still have poor gigabit performance.

FreeBSD isn't doing anything interesting about this problem and from my POV is mostly focused on porting things from linux and solaris and other OS's, and not innovating itself.
Click to expand...
Don't you think you're saying things which may - in retrospect - not be true?

FreeBSD is the progenitor of alot of software innovations. I will name a few:
  • progenitor of TCP/IP stack, of which parts are also used by Microsoft Windows
  • SMP performance innovations; FreeBSD is known for its advanced SMP multiprocessing performance, fine-grained locking and CPU scheduling.
  • Parts of the SMP_NG work have been ported to Firefox project, Linux kernel project and MySQL database project.
  • Mac OSX is basically built on the FreeBSD OS, they forked it as DarwinOS; part of the reason Mac OSX users had access to ZFS; but probably Apple removed it because it was unclear whether relying on Oracle-owned ZFS could be a liability. I think Steve Jobs preferred control over early access to advanced technology.
  • FreeBSD jails are a lightweight alternative to virtualization, without any performance loss normally associated when virtualizing (emulating) services.
  • FreeBSD supports booting from RAID-Z and RAID-Z2, even while degraded. An improvement over Solaris, which only supports booting off a single disk or mirror; no intelligent boot code is present. This would be the first software RAID6 implementation that is fully bootable without special BIOS requirements.
  • FreeBSD designed the MBR/GPT hybrid partition scheme in which you use a GPT partition which is made bootable by normal MBR methods. Thus, you would use a modern GPT partition with manually assigned label while still being bootable by BIOS systems without EFI, which is normally required to boot from GPT.
  • FreeBSD has GEOM I/O framework; the best I/O framework i've seen in any OS. It allows storage to be completely pluggable. Examples:
  • You could put encryption on a zvol, using either conventional AES hardware-accelerated or software multi-threaded implementation with maximum performance, or use the DARPA funded special encryption GEOM_BDE which is advanced in the sense that it completely anonymises I/O traffic; so is safe even when the I/O is being snooped. This would be preferred if you do I/O over the internet, even if its a secure channel.
  • You can use software RAID0, RAID1, RAID3, RAID5 and JBOD
  • RAID1 has most advanced implementation to date i've seen with load balancing algoritms and alot of features
  • RAID5 has seen sequential writes faster than my Areca controller; 560MB/s where my Areca stops at 420MB/s-440MB/s. Write that is, not read. Read is easy; write is hard.
  • Uses pseudoRAID driver which recognises your onboard RAID arrays; like Intel onboard RAID or AMD onboard RAID arrays.
  • Can chain several GEOM modules together; so they are stackable and can be combined in any combination.

It has a different open source licensing regime than linux, which enables porting code from opensolaris and other OS"s than cannot be ported to linux.
Click to expand...
The liberal BSD license allows any BSD code to be used in any product, commercial, closed source, or whatever. That's the reason Windows uses parts of BSD. It couldn't if it were GPL.

So anyone, including the Linux people with the more restricted GPL license, can bollow/reuse/steal/whatever code from BSD. But the other way around is not possible, FreeBSD can't benefit from any GPL code, at least not in the kernel and they try to keep other stuff non-GPL as well, such as the rewritten compiler, trading GCC for the new Clang compiler.

So i would say it's more the other way around; FreeBSD innovates important stuff for operating systems that get distributed in any modern OS running today. That mean that virtually every human that operates a computer has used FreeBSD code. Other way around, there are no Linux innovations that make it to FreeBSD; none that i'm aware of.

Perhaps you should look at Wikipedia to learn more about FreeBSD. Here's a good start:
http://en.wikipedia.org/wiki/FreeBSD
http://en.wikipedia.org/wiki/GEOM
 
sub.mesa said:
That said, i run ZFS on 32-bit 500MHz CPU with 1GB RAM; it is possible, but with careful tuning. Without tuning it's easy to crash such a system. amd64 (meaning 64-bit) already works much better in that regard.
Click to expand...
What kind of platform is that? Do you run RAIDZ or just a simpler ZFS RAID?

A very low-power ZFS platform would be the upcoming 9W/18W AMD Zacate platform with Ontario/Brazos CPU; like the AMD competitor to Atom but then faster and lower platform power, since it includes alot of stuff and the chipset is nothing more than an I/O hub. This gives you 6x SATA plus PCI-express x16 slot; so another HBA = 14 disks. I posted a picture of that board, a couple pages back in this thread. It should be available in a couple of months.
Click to expand...
I have looked at Super Micro X7SPA-H for a while because of its excellent ICH,small form factor, low power consumption and dual core CPU. But 300 EUR for that kind of low performance and scalability is a bit too much. Then I stumbled upon the little bit more tempting HP MicroServer (dualcore Athlon II Neo N36L), but found ML110 G6 to be just slightly more expensive.
 
endrebj said:
What kind of platform is that? Do you run RAIDZ or just a simpler ZFS RAID?
Click to expand...
It's an AMD Geode LX-800 or something, which runs at 500MHz and is i386 32-bit x86 core, it's so low power it doesn't even have a heatsink! How's that. And it runs great, but the I/O is very light. It runs on compact flash and only sees regular reads. I use it for specific light tasks (DHCP, firewall, boot image for my workstations).

I have looked at Super Micro X7SPA-H for a while because of its excellent ICH,small form factor, low power consumption and dual core CPU. But 300 EUR for that kind of low performance and scalability is a bit too much. Then I stumbled upon the little bit more tempting HP MicroServer (dualcore Athlon II Neo N36L), but found ML110 G6 to be just slightly more expensive.
Click to expand...
Why not use any of the SuperMicro core i3/i5/i7 boards? The ones with integrated 8x LSI HBA offer alot of value for all features provided. Would be a great choice for any big fileserver with more than 10 disks. The AMD Neo is quite low performance as well.

If you want very low power and reasonable performance, AMD Ontario/Zacate is what you may want to look for. more info here + screenshot
 
chrcoluk said:
also raidz in my view will be slower than raid mirror as its basically software raid5 which has never been fast.
Click to expand...

I contend raidz will outperform a mirror in sequential reads, sequential writes, and random writes. It won't outperform a mirror on small random reads, but you could always add an l2arc to alleviate that issue.
 
ok the boot error is I just had to copy the updated rc.d/zvol file as doesnt get picked up by mergemaster :)
 
Hi guys,

since OpenSolaris is now closed and honestly, I feel a bit more familiar to FreeBSD if it's a good idea to create a file server based on the current FreeBSD 8.1 and ZFS v14. It will be for a SoHo (Home Office & private) file server, which will also contain a huge amount of video data (video cut using Final Cut Studio). I would like to put those 6x 1TB HDs into a RAIDz1, and put in an Intel SASUC8i with 8 more drives later (4x 1.5TB + 4x 1TB that are currently in 2 slow external RAID5 enclosures).

I wonder, because I tried ZFS back 2 years ago and it worked very that time, but the machine was a bit underpowered, so it was slow. The current machine is a:

- AMD Athlon X2 245e (2.9GHz DualCore)
- 2x2GB + 2x1GB ECC RAM
- 6x Hitachi 24/7 drives for ZFS + 1x WD 160GB IDE drive for FreeBSD

As it seems OpenIndiana and ALSO Solaris 11 Express are NOT mentioned to be used on a "production server" - whatever that means - I would rather use an older ZFS version and have save data than "playing" with unstable stuff.

Could you guys give me hint, please?

Many thanks in advance :)
 
Last edited:
@DieTa: know that UFS+ZFS doesn't work well on FreeBSD 8.x; for performance reasons you would want a ZFS-only system. A Root-on-ZFS installation an help with this. You can try the mfsBSD distributino which includes such a script, or use my ZFSguru 0.1.7 livecd which can also do just that.

The result would also be that you may not need a separate system disk anymore. You could put your system on your RAID-Z pool and let it enjoy the same protection as well, and saves power since these old PATA disks use about 7-9W idle.

6GiB RAM is great; make sure you do /boot/loader.conf tuning of kmem, arc_min and arc_max. Using FreeBSD 8.2-RC1 instead of 8.1 might also be an option, the ZFS code in there is quite stable and has several fixes compared to V14.
 
@chrcoluk: Sure - as ZFS needs 64bit :)

@sub.mesa: Meanwhile found your website on submesa.com and reading for and hour or so on it. I just found ZFSguru 0.1.5 version - but I guess I will find version 0.1.7 using Google ;)

Having the OS itself on ZFS sounds good, as long as I don't kill anything. I will find those kmem, arc_min & arc_max commands in your documentation, no?

By the way - also struggled over OpenIndiana in the maintime - hard for me to understand if it's better to use FreeBSD or OpenIndiana (original Solaris 11 Express is out because of Oracle honestly).
 
Hi sub.mesa,

just found time to play with ZFSguru and I'm pretty much impressed. But 2 questions came up while testing:

- can I install it as a "regular" FreeBSD 8.2 and install other stuff onto this zpool (like netatalk for example via ports)? Would prefer that because it directly boot ZFS.
- How can I create a new SMB share in the WebGUI?

Thanks!
 
You must log in or register to reply here.
Back
Top