• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

BTRFS vs ZOL?

T

tycoonbob

Limp Gawd
Joined
Jul 29, 2012
Messages
157
Hey guys.

I'm doing a new build and trying to decide between using BTRFS or ZOL, since I'll be running CentOS 7.

Yeah, I know BTRFS is still not production ready, but basic features I'm sure are working well enough and I'm looking to build a system that will last 4-5 years with minimal maintenance.

Build:
Chassis: SuperChassis 846E1-R900B
Motherboard: SuperMicro X8DTE-F
CPU's: x2 Xeon L5640
RAM: 96GB (12x8GB)
PSU: x2 900W
Backplane/SAS Expander: BPN-SAS-846EL1
HBA: LSI 9211-8i
OS Drives: x1 128GB SSD
VM Drives: x1 500GB SSD (maybe 1TB)
Storage Drives: x8 Toshiba PH3500U-1I72 (may add more to start, or in the future)

So yeah. Doing a CentOS 7 build, for both storage and virtualization, using KVM + Kimchi. Originally I had planned to use ZoL, and probably start off with a RAIDz3 with 10 drives, and later add 10 more drives in another z3 vdev in the same zpool.

Since btrfs is up and coming, I am considering giving it a go. I know literally nothing about it though, so it may be worth not considering it at all.

KVM guest will live on the 500GB (or 1TB) SSD, with backups to the main storage array, and that storage array will also be used for NFS shares. Other minor things, I will put the 1-2 OS drives in PCI caddies to not consume drive bays up front, and may well put the VM drive(s) also in PCI caddies in the rear of the case. If I recall, this chassis has 7 PCI slots, 2 of which will be used with the HBA and additional NIC's (quad GbE card, possibly), leaving 5 open. I have two Syba PCI slot caddies currently that I plan to re-use.

As far as the VM drives, I am considering 2 500GB SSDs in a vdev mirror, instead of just a single drive formatted ext4.


So with the information provided, I'm looking for thoughts or opinions? Am I doing something completely wrong? What would you do different?

Thanks! (and sorry for the rambling post)
 
Ok, I will venture my opinion :)

Try to decide first how you will handle the VM images. Will they be files, or block devices?

If they will be block devices, you should go ZFS because then you can use ZVOL's as the block devices. This gives you the ability to snapshot, clone and thin-provision your VM images.

I have this running on my setup, and I have 10 VM's each thinking they have a 1TB OS disk, where in reality they use 4 gig on disk. I installed my base system on a thin-provisioned ZVOL of 1TB, then made 10 clones. The clones only differ in some config files from the base, so only the differences are saved. BONUS: 4 gig is nothing, all read requests of all VM's to their OS disk are cached from RAM. Very fast.

If you are going to go for files. I would go BTRFS because BTRFS lets you do the things like snapshots and clones on the file level.

Either way, I would not use EXT4 on the disk for the VM images. Whatever you choose, it will be better then EXT4.

BTRFS is definitely worth considering at this point. And I say this as a long time ZFS user. (Been using it before it even existed on Linux.) I am still on ZFS, but I will move to BTRFS once they have the ZVOL equivalent.

One thing to keep in mind though: for BTRFS you better have a recent kernel. 3.19 or 4.0 would be ideal. I don't know what Centos 7 ships with, I think it is one of the more conservative distros out there.

Good luck with the project.
 
rsq said:
Ok, I will venture my opinion :)

.....
One thing to keep in mind though: for BTRFS you better have a recent kernel. 3.19 or 4.0 would be ideal. I don't know what Centos 7 ships with, I think it is one of the more conservative distros out there.

Good luck with the project.
Click to expand...

that is a must for btrfs runninx 3.19.5 or greater, since < 3.1X kernel has a defect on kernel lock while running btrfs.
as I remember, centos 7 kernel > 3.19.5

since centos is piggy-ing back rhel source code. they are super duper conservative . they just need stability 24/7 :D
this is the reason for me to deploy centos 6.X (previous years back) and centos 7.X(at least early this year)

I am running btrfs raid 0 on filesystem and ZoL on data storage all together.
they seem Happy, had been replaced 3 failed drives on ZoL...
 
Unfortunately, CentOS 7.1 is based on kernel 3.10.x, so it doesn't appear to be a good candidate for btrfs. I can definitely give Fedora 22 a go, since it just dropped. I think F23 will have btrfs as the default filesystem, so it should definitely be available in F22, and F22 has kernel 4.0. I suppose it'd be pretty brave of me to move ALL of my home resources to one box based on F22, but it would be a good chance for me to spend some time learning Docker, and maybe moving a lot of my stuff there.

Thoughts on F22?

As far as my storage layout goes, I'm not really concerned about where I'm storing my KVM images (or docker files), and mostly concerned with how to layout my storage drives, since all of my data will be living there (with CrashPlan backing it all up, along with USB external backups). I've never used zfs, so it's not like I have an advantage with it over btrfs, but I want something stable that I can rely on.

For what it's worth, I also have a LSI MegaRAID 9261-8i controller (that's used in my current storage box), and I may consider sticking with that since I'm familiar with it. I wanted to give a chance to non-hardware RAID alternatives, but I'm just not sure this is the right time. BTRFS isn't 100% complete, and ZFS on Linux, well it's solid and it works, but Linux isn't native for ZFS and all that. Not sure if that matters, but I just don't feel confident with that yet.

EDIT: And yet another idea comes up. I've been using Proxmox for the past year or two, and I just seen where the latest version now ships with ZFS. It may be worth me setting that up and going that route. I just wish Proxmox was CentOS based instead of debian, but that's a minor personal preference.
 
Last edited:
tycoonbob said:
Unfortunately, CentOS 7.1 is based on kernel 3.10.x, so it doesn't appear to be a good candidate for btrfs. I can definitely give Fedora 22 a go, since it just dropped. I think F23 will have btrfs as the default filesystem, so it should definitely be available in F22, and F22 has kernel 4.0. I suppose it'd be pretty brave of me to move ALL of my home resources to one box based on F22, but it would be a good chance for me to spend some time learning Docker, and maybe moving a lot of my stuff there.

Thoughts on F22?

As far as my storage layout goes, I'm not really concerned about where I'm storing my KVM images (or docker files), and mostly concerned with how to layout my storage drives, since all of my data will be living there (with CrashPlan backing it all up, along with USB external backups). I've never used zfs, so it's not like I have an advantage with it over btrfs, but I want something stable that I can rely on.

For what it's worth, I also have a LSI MegaRAID 9261-8i controller (that's used in my current storage box), and I may consider sticking with that since I'm familiar with it. I wanted to give a chance to non-hardware RAID alternatives, but I'm just not sure this is the right time. BTRFS isn't 100% complete, and ZFS on Linux, well it's solid and it works, but Linux isn't native for ZFS and all that. Not sure if that matters, but I just don't feel confident with that yet.

EDIT: And yet another idea comes up. I've been using Proxmox for the past year or two, and I just seen where the latest version now ships with ZFS. It may be worth me setting that up and going that route. I just wish Proxmox was CentOS based instead of debian, but that's a minor personal preference.
Click to expand...

just correction, centos 7.X is ok as long as stick with 0 and 1 for btrfs.
by default installation. you can pick btrfs 0 or 1 insted of lvm

stay away with Fedora, not stable for server environment...

btrfs is zfs alike..... only two solid usable on btrfs -> 0 and 1.

I am using proxmox 3.4, they ported ZoL. performance is pretty solid,
debian is good and very stable. do not get wrong... many low power server or embedded server are running debian flavor.

as I said, I am using raid 0 for OS and ZoL on centos 7 since early this year,,, the server is running 24/7. some server for non 24/7 are running centos 6.2 with ZoL and ext4 for OS.

if you no idea on where are you gong in limux world. pick centos 7.X for server 24/7 with btrfs 0/1 and ZoL for data storage. latest ZoL is 0.6.4 and I am using 0.6.3.1.
 
Fedora has been pretty solid for me. If you need the newer kernel, I say give it a try. They even have a server oriented spin.
 
Darakian said:
Fedora has been pretty solid for me. If you need the newer kernel, I say give it a try. They even have a server oriented spin.
Click to expand...

Fedora has new update and eye candies...
you are ok on non 24/7 with Fedora, they did not promise not breaking something during the update..

as I said, centos is stable and robust for 24/7 and security updates.
rhel actually do a lot of backported too.
when I said kernel 3.19... this is for vanilla linux kernel, not rhel/centos kernel since they backported many from newer kernel.
 
cantalup said:
Fedora has new update and eye candies...
you are ok on non 24/7 with Fedora, they did not promise not breaking something during the update..

as I said, centos is stable and robust for 24/7 and security updates.
rhel actually do a lot of backported too.
when I said kernel 3.19... this is for vanilla linux kernel, not rhel/centos kernel since they backported many from newer kernel.
Click to expand...

There are two definitions of stability here.
1) Stable is leaving things where they are and how they are. That is; a keyboard is a stable device since the design has not changed in the last 20+ years.
2) Stable is not crashing.

Fedora is stable under definition 2 and mostly under 1. For any given release they commit to being stable under definition 1, but between releases all bets are off. I agree that centos is great, but for a fileserver fedora is equally great and if a new kernel is needed then fedora is the way to go.

Edit: There are benefits to not meeting definition 1 btw. Gnome 3 for instance broke from 2 and is much better for it.
 
Last edited:
Darakian said:
There are two definitions of stability here.
1) Stable is leaving things where they are and how they are. That is; a keyboard is a stable device since the design has not changed in the last 20+ years.
2) Stable is not crashing.

Fedora is stable under definition 2 and mostly under 1. For any given release they commit to being stable under definition 1, but between releases all bets are off. I agree that centos is great, but for a fileserver fedora is equally great and if a new kernel is needed then fedora is the way to go.

Edit: There are benefits to not meeting definition 1 btw. Gnome 3 for instance broke from 2 and is much better for it.
Click to expand...

stable as defined for server......
your definition is not defined fedora as a server 24/7 stability :p
 
cantalup said:
stable as defined for server......
your definition is not defined fedora as a server 24/7 stability :p
Click to expand...

That would be definition #2 I believe. Do you have another definition of 'stable'? If so could you share it?
 
Darakian said:
That would be definition #2 I believe. Do you have another definition of 'stable'? If so could you share it?
Click to expand...

your definition is not related with stable in server environment
stable in server minded: performance, stability, and security

please tell me, fedora or centos that people always choose for server 24/7? just a simple question.

What is CentOS?
CentOS Linux is a community-supported distribution derived from sources freely provided to the public by Red Hat for Red Hat Enterprise Linux (RHEL). As such, CentOS Linux aims to be functionally compatible with RHEL. The CentOS Project mainly changes packages to remove upstream vendor branding and artwork. CentOS Linux is no-cost and free to redistribute. Each CentOS version is maintained for up to 10 years (by means of security updates -- the duration of the support interval by Red Hat has varied over time with respect to Sources released). A new CentOS version is released approximately every 2 years and each CentOS version is periodically updated (roughly every 6 months) to support newer hardware. This results in a secure, low-maintenance, reliable, predictable and reproducible Linux environment.

if you need definition of fedora versus rhel, go to redhat site -> https://www.redhat.com/en/technologies/linux-platforms/articles/relationship-between-fedora-and-rhel
...
Developers and Linux enthusiasts flock to Fedora for the latest features and the opportunity to directly collaborate with Red Hat engineering. Banks, stock exchanges, hospitals, and businesses that run the world's leading websites choose Red Hat Enterprise Linux for the platform's performance, stability, and security, which lets them implement mature and well-organized IT infrastructures across the enterprise.
....

those should help to understanding on fedora versus rhel(fee version fo rhel: centos)


one example on ZoL on fedora 21 issue: https://forums.servethehome.com/ind...t-add-drives-because-of-partition-issue.4919/

if you are planning to run btrfs (0/1) and ZoL, get centos that already has battle proven that follow rhel GPL opensource repo.

I think, this is more than enough for a little education and this is up to you to pick flavor for server purpose.
 
Last edited:
There is no indication that the issue in question is specific to fedora. I watch this list regularly, and most of the distro-specific bugs/issues are related to installation, management scripts, build errors, etc... The stuff you're talking about is very unlikely to have anything to do with fedora (and no, I'm not a fedora user or fanboy, just trying to point out a very misleading post.)
 
If you want to fool around maybe BTRFS is worth messing with, but if you're setting up some server and you just want things to work, ZFS is the way to go. It just works. You're not missing out on anything feature-wise and you're not worrying it might end up bitting you on your ass at some point with odd bugs/problems.

But you know, maybe you want to fool around.

CentOS vs Fedora... unless you have some specific reason to go with Fedroa, I'd stick with CentOS. A year ago or whatever, when CentOS meant CentOS 6, that was getting a bit long in the tooth. But now is ideal time to jump on CentOS, fairly up-to-date and looking forward to something like a decade of support. I mean Fedora 19 came out less than 2 years ago?--- And its now dropped from support. I think its like next release + 13 months for support? That goes by fast for a fileserver.
 
Last edited:
ZFS is like a satellite guided missile: it's setup-and-forget. (Ok, sure, there's maintenance but every system requires some TLC). Go with ZFS.
 
cantalup said:
those should help to understanding on fedora versus rhel(fee version fo rhel: centos)


one example on ZoL on fedora 21 issue: https://forums.servethehome.com/ind...t-add-drives-because-of-partition-issue.4919/
Click to expand...

awesome, my issue got quoted in a thread, I'm internet famous. In all seriousness, I don't think the above issue is a RedHat (Fedora) specific issue. I was trying to import from FreeBSD and it was a pain in the ass that's for sure. Maybe it was an issue, still wouldn't change the fact that I think it's the only real FS to use for large datasets.
 
For a server, I'd normally use CentOS. I've been working with CentOS since v5, and now I can say I've been working with CentOS 7 for over 9 months. I'm really liking v7, and the only reason I suggested Fedora was for the newer kernel (in reference to using btrfs). Someone mentioned that RHEL likely back-ported some features from a newer kernel into their v 3.10 kernel, but it seems more and more likely I will be going with CentOS 7.

I should be ordering the new (to me) box this week, so the build won't actually start until next week (at the earliest). Right now I'm pretty sure I'll be going with ZFS, but unsure the route I want to take. I've been using Proxmox for the past year or two, and while I'm not in love with it, I love the features that are there in the webui. If I need something, it's right there and easy to do. Seeing how the latest version of Proxmox now includes ZoL, I could see myself going in this direction.

The alternative for me right now is to go CentOS 7, ZoL, KVM, and likely WebVirtMgr (tested Kimchi; didn't care much for it -- lacks too many features at this stage).

I'm more comfortable with yum/rpm's, so I'd prefer to go the CentOS route...but Proxmox is basically ready to go out of the box. KVM + OpenVZ + ZoL...all right there. Somewhat dated webUI, but has tons of features. Decisions, decisions.
 
gigatexal said:
awesome, my issue got quoted in a thread, I'm internet famous. In all seriousness, I don't think the above issue is a RedHat (Fedora) specific issue. I was trying to import from FreeBSD and it was a pain in the ass that's for sure. Maybe it was an issue, still wouldn't change the fact that I think it's the only real FS to use for large datasets.
Click to expand...
I planned to help your issue,
but seeing fedora... , this was useless since fedora lifecycle is pretty fast, aka next please on fedora 2X :p

get centos! you save your misery life on ZoL :D

I am happy centos with built-in btrfs 0/1 and ZoL.than other distro for server purpose
 
Last edited:
danswartz said:
There is no indication that the issue in question is specific to fedora. I watch this list regularly, and most of the distro-specific bugs/issues are related to installation, management scripts, build errors, etc... The stuff you're talking about is very unlikely to have anything to do with fedora (and no, I'm not a fedora user or fanboy, just trying to point out a very misleading post.)
Click to expand...

if you follow fedora, they focus on bleeding package and technology.
they do not intend to solve many bugs. please check their bug tracker....

when goes to rhel realm, they do seriously fixing bugs and backporting

please do installation on centos versus fedora on ZoL..
when you hit the wall, you mostly on your own in fedora. since fedora lifecycle is pretty fast...
centos is different world, once you hit the wall, their mailing list is very helpfull and mostly will migrate to ZoL bugtracking.

I am not a fanboy, only a regular techie person that try to pick the best tools for me.
 
Darakian said:
I've been using linux forever dude, I know why centos and fedora both exist. You don't need to quote marketing material at me. I'd still like to hear your definition of stable.
Click to expand...

I already mention "stable", if you insist again, do I have to follow your words to make you happy :D.

those not marketing material!. I have been in centos, fedora, rhel (at work), suse (prev work), debian, ubuntu, and slackware (was..)

if you happy with fedora, this is your choice

I try to explain to first starter to pick the best distro for server enviroment

simple on centos/rhel. they never add bleeding edge package. fedora does.
 
Last edited:
tycoonbob said:
For a server, I'd normally use CentOS. I've been working with CentOS since v5, and now I can say I've been working with CentOS 7 for over 9 months. I'm really liking v7, and the only reason I suggested Fedora was for the newer kernel (in reference to using btrfs). Someone mentioned that RHEL likely back-ported some features from a newer kernel into their v 3.10 kernel, but it seems more and more likely I will be going with CentOS 7.

I should be ordering the new (to me) box this week, so the build won't actually start until next week (at the earliest). Right now I'm pretty sure I'll be going with ZFS, but unsure the route I want to take. I've been using Proxmox for the past year or two, and while I'm not in love with it, I love the features that are there in the webui. If I need something, it's right there and easy to do. Seeing how the latest version of Proxmox now includes ZoL, I could see myself going in this direction.

The alternative for me right now is to go CentOS 7, ZoL, KVM, and likely WebVirtMgr (tested Kimchi; didn't care much for it -- lacks too many features at this stage).

I'm more comfortable with yum/rpm's, so I'd prefer to go the CentOS route...but Proxmox is basically ready to go out of the box. KVM + OpenVZ + ZoL...all right there. Somewhat dated webUI, but has tons of features. Decisions, decisions.
Click to expand...

Indeed, rhel does much backported (which pick up by centos stream) including kernel backported
you can find detail on redhat bugtrack. rhel 6 is the most active tracks, follows by 7.
example: https://access.redhat.com/documenta..._Linux/6/html/6.4_Technical_Notes/kernel.html
rhel 6: some fixed (important), mostly fixed in security, and some backported, and stability( memory/IO/KVM and others)
* I am still running centos 6.X :p....

I am using proxmox 3.4.aka minimal debian flavor for replacing free esx1 5.5

centos 4.X was the first start on my centos world.
at that time, I used slackware for all my 24/7 server purposes.

Honestly, linux KVM is missing polished web GUI, and easier maintain by commandline.
this is a big plus proxmox on GUI.

I believe when GPL polished KVM web GUI is already available. proxmox big plus advantage would diminish.

BTW, I tested with minimal centos 7 installation, the space required is very small.
this is the good candidate for installig KVM packages and ready form VMs...
the bad thing is..... good GUI based for maintain KVM does NOT exist, and all must be done via commandline.

I believe you know the meaning of "centos plus" :D, if you have to....
 
You must log in or register to reply here.
Back
Top