FreeNAS ZFS Box - Questions

[ethraax]

I've been thinking about expanding my desktop's storage capacity with a hardware RAID card, but I've decided that I could accomplish the same thing (and much more) using a separate file server.

I've been playing around with FreeNAS and ZFS on a virtual machine for a bit, and I'm pretty sure that my new file server will use that, unless someone comes up with a better alternative.

I should also clarify what I'd like to use my file server for. I'd like to be able to backup all of my computers roughly once a week (perhaps slightly more often, if it's not too expensive). I'd also like to centralize storage for all of my media files (comprising of video and audio files). To that end, I'd like to be able to, at the least, stream HD video to 2 different devices at once.

In addition, I already have two 1.5 TB drives in my desktop, set up in a RAID 0 array on my motherboard (yes, it's not safe - that's one of the reasons I'd like to upgrade my storage system). I'd like to use these drives in the new file server, but I need to be able to copy the contents over before using them.

Here's the hardware I'm planning on using:

Motherboard: Gigabyte GA-P55-USB3
CPU: i3-530 (2.93 GHz)
RAM: G.Skill 3x2GB DDR3 1333
PSU: COOLER MASTER GX Series - 750W
Case: NZXT GAMMA Classic Series (Mid-Tower)

At the moment, I'm planning on buying six more 1.5 TB drives. I'd like to use a storage pool with two 4-drive raidz1 vdevs. Basically:

zpool create tank raidz1 da0 da1 da2 da3 raidz1 da4 da5 da6 da7

One reason to do this is that I can create the first four-disk vdev, copy the files on my existing RAID 0 setup over, and then add another four-disk vdev (I realize that I must copy all the files on the disk to get them striped across the entire storage pool, but I can set it up to do that gradually over a week).

Anyways, I had a few questions.

1) Is the CPU/RAM enough? I'd like to be able to compress certain filesystems (for example, the backup filesystem) without any trouble. I've heard that ZFS eats RAM like a fat kid eats cake, which is why I've decided to get 6 GB, but if I can bring that down to 4 GB without significant performance loss, I'd rather do that (I can always upgrade at a later date).

2) One of the other reasons I wanted to go with 4-disk raidz1 vdevs was because it seems easily upgradable. From my understanding, I can upgrade the storage pool at any time by adding another 4-disk vdev in raidz1. For example, I can buy four 2TB drives and upgrade the capacity by another 6 TB. [2TB * (4 - 1)]. Is this true? Has anyone upgraded ZFS storage pools in this way (specifically, on the FreeBSD implementation, since FreeNAS runs on FreeBSD).

3) Can I encrypt my storage so that I have to "log in" to the server after it starts up before it loads the hard drives? I was going to run FreeNAS off a USB flash drive inside the case. Preferably, once the file server is plugged in and started up, I should be required to SSH in and enter a password before it can read the drives (similar to TrueCrypt). Is this possible, and is this feasible (with the hardware I've mentioned)? Will it incur a significant overhead to reading and writing data?

4) I was going to use the Gigabit Ethernet that comes integrated into the motherboard for communication with the server. If I expect the server to get lots of traffic, should I invest in a NIC, or are the on-board NICs pretty good (like on-board sound)?

5) Also, it would be nice if the device could broadcast data over wireless. I've considered buying a wireless access point which could be placed on top of the server. Will an access point alone be enough to provide this functionality, or is a router required? To clarify, I'm not concerned with accessing the Internet. Instead, I simply would like the server to be accessible to any wireless device (running Windows, if it worked with macs that would be an added benefit) within range, even if they aren't connected to a network (kinda like plugging lots of computers into a switch, without having an uplink, except with wireless). If what I'm describing is impossible, or horrendously expensive, that's fine - it's not a required feature.

6) I understand that I can SSH into the server, but I'd also like a way to directly connect to it. Can I run a serial cable from my server to my desktop and run some software that makes my desktop appear as a monitor+keyboard dumb terminal? The key here is that I want something that would function essentially the same as plugging a monitor and keyboard directly into the server (I'd like to be able to see what it's doing after it starts, but before it boots). I guess the one-sentence version of this question would be: What are my out-of-band management options?

7) One of the reasons I picked the PSU that I did was because it had lots of SATA power connectors. However, it still has barely enough. Also, while my case has plenty of places for me to mount hard drives, it doesn't have enough for an upgrade of 4 more (I could probably ghetto-mount them somewhere else inside the case, because I'm not using many expansion cards, but I'd rather not). If I decide to upgrade with more hard drives, can I place the drives in an external enclosure but keep them part of my file server? It would be great if someone had information about upgrading this kind of system.

Sorry about the horrifically long post, but I had lots of questions and I wanted to be thorough. If anyone can answer any of the questions above, or suggest different hardware or a different setup, I'd very much appreciate the input.

Thanks in advance for your replies!

-- Ethan

 

[sub.mesa]

I've been playing around with FreeNAS and ZFS on a virtual machine for a bit, and I'm pretty sure that my new file server will use that, unless someone comes up with a better alternative.

Have you read about my Mesa web-interface yet? It's not ready for real usage for the moment, but if you would like to use it in the future you could prepare for that already.

I should also clarify what I'd like to use my file server for. I'd like to be able to backup all of my computers roughly once a week (perhaps slightly more often, if it's not too expensive). I'd also like to centralize storage for all of my media files (comprising of video and audio files). To that end, I'd like to be able to, at the least, stream HD video to 2 different devices at once.

Streaming over network generally works fine; but sometimes is tricky due to latency. If your program can increase the buffer/precaching size, then you should have no problems.

Sometimes, multiple streams totally crash HDD performance causing them to seek excessively and be really, really slow. So the MB/s may be misleading; but hey test it out! With just two streams this shouldn't really be a problem.

In addition, I already have two 1.5 TB drives in my desktop, set up in a RAID 0 array on my motherboard (yes, it's not safe - that's one of the reasons I'd like to upgrade my storage system). I'd like to use these drives in the new file server, but I need to be able to copy the contents over before using them.

You have 2x 1.5TB and you buy 6x 1.5TB more. So this is what you do:

1. create 4-disk RAID-Z from new 1.5TB drives
2. copy data from desktop RAID0 to new RAID-Z
3. now add a new raidz to your existing pool; voila now you have 8x 1.5TB drives in two RAID-Z pools all sharing the same free space.

Note that the stored files before adding the second array will only have the performance of that of a single array; it won't re-distribute the data. So only newly created/modified data will get speed enhancement from the second RAIDZ pool.

Generally your setup is good; two RAIDZ pools are better than one. Consider also using copies=2 for data really important, especially if that data isn't all that large. This would make both copies land on different arrays as well.

1) Is the CPU/RAM enough? I'd like to be able to compress certain filesystems (for example, the backup filesystem) without any trouble. I've heard that ZFS eats RAM like a fat kid eats cake, which is why I've decided to get 6 GB, but if I can bring that down to 4 GB without significant performance loss, I'd rather do that (I can always upgrade at a later date).

You're right about the fat kid!

A quadcore could be worth considering if you do compression, otherwise the Core i3 is fine. ZFS can use all the cores you have; so a 64-core monster without other hardware bottlenecks would scale really well with ZFS and compression. So a low-power, low-frequency quadcore chip might be preferable.

AMD has the X4 610e or something; not sure about the number. But it's a quadcore with 45W TDP. If you're keen on saving power this might be a win-win on both?

To be honest, your core i3 might be just enough. But if you want to safe and like some additonal CPU power for future features then a quadcore is highly recommended. Or even the 6-core from AMD as its rather cheap.

2) One of the other reasons I wanted to go with 4-disk raidz1 vdevs was because it seems easily upgradable. From my understanding, I can upgrade the storage pool at any time by adding another 4-disk vdev in raidz1. For example, I can buy four 2TB drives and upgrade the capacity by another 6 TB. [2TB * (4 - 1)]. Is this true? Has anyone upgraded ZFS storage pools in this way (specifically, on the FreeBSD implementation, since FreeNAS runs on FreeBSD).

True and correct.

In fact, you can add anything: a single disk, a mirror, a RAID-Z2. Any number of disks too. What you CAN'T do, is add a disk to an existing vdev; for example making your 4-disk RAID-Z a 5-disk RAID-Z.

The advantage here is:
- you do not need same capacity of disks; you can use 3TB disks in the future and share free space with your smaller capacity disks. What RAID allows you to do this?
- you can expand in a safe way, without putting your data at risk in a complex procedure; and it happens instantly; not 24+ hours.
- you can add vdevs with a different redundancy. Want to add just two disks? Then add a mirror instead. Note that you can not remove vdevs, and you can't transform a mirror into a raid-z either.

So actually i prefer the ZFS way of expansion. But block-pointer rewrite will make things like expanding RAID-Z and alot of other things possible in the future. So it's just the current versions that do not allow this; no theoretical impossibility.

3) Can I encrypt my storage so that I have to "log in" to the server after it starts up before it loads the hard drives?

Yes, geom_eli allows you to do this. It is integrated in FreeNAS and FreeBSD. It has an option that requires you to enter a passthrase at boot time or once the system has started. This allows encrypting both your system disk and data disks.

Note that my project will also support encryption in the future, but encrypting ZFS disks instead of the usual UFS as i believe FreeNAS uses.

I was going to run FreeNAS off a USB flash drive inside the case. Preferably, once the file server is plugged in and started up, I should be required to SSH in and enter a password before it can read the drives (similar to TrueCrypt). Is this possible, and is this feasible (with the hardware I've mentioned)? Will it incur a significant overhead to reading and writing data?

FreeBSD supports exactly this. Not sure how FreeNAS implemented it. My project will allow you to encrypt/decrypt driver via the web-interface; requiring you to supply the password over HTTP connection (presumed safe if its your wired LAN).

4) I was going to use the Gigabit Ethernet that comes integrated into the motherboard for communication with the server. If I expect the server to get lots of traffic, should I invest in a NIC, or are the on-board NICs pretty good (like on-board sound)?

Use the onboard NIC, unless it gives you problems. My experience is a cheap (but modern) Realtek is just as fast as my Intel PRO/1000 PT PCI-express x1 adapter. But sometimes the drivers of other chips are less good (bugs) or the hardware is unsupported (still an issue with FreeNAS sometimes as it uses older FreeBSD versions).

5) Also, it would be nice if the device could broadcast data over wireless. I've considered buying a wireless access point which could be placed on top of the server. Will an access point alone be enough to provide this functionality, or is a router required?

In FreeBSD you can configure mostly any wireless adapter to function as access point, where other wireless clients can connect to (with WPA2 or without). It would require time to set this up though, it's likely not implemented in FreeNAS.

Perhaps you need base FreeBSD, with all bells and whistles available, rather than FreeNAS which has everything it does not 'need' shaved away; making it hard to configure it for things it was not designed for.

If going that route, the step to my web-interface is really small, as both can be used inter-dependently; they don't get in eachothers way. So you could go in SSH and create your disks and zpool and filesystems by hand, and then install/start/navigate to my web-interface and it would work as if you did it via the web-interface. I have to add that, if you do things manually, be sure to use geom_labels to your disks, which makes management better since my interface uses that. But even if you don't you can still use all ZFS functions.

So you could use both; simple things via the web-interface for your convenience and more advanced things like your access point and other stuff on normal command line using SSH (windows application: PuTTY).

7) One of the reasons I picked the PSU that I did was because it had lots of SATA power connectors. However, it still has barely enough. Also, while my case has plenty of places for me to mount hard drives, it doesn't have enough for an upgrade of 4 more (I could probably ghetto-mount them somewhere else inside the case, because I'm not using many expansion cards, but I'd rather not). If I decide to upgrade with more hard drives, can I place the drives in an external enclosure but keep them part of my file server? It would be great if someone had information about upgrading this kind of system.

Are you using your 5,25" (cdrom-size) bays already to store HDDs? If not buy some 4-in-3 HDD bays, storing 4 3,5" desktop-class drives at the height of three cd-roms. Or something similar. If you can't, you can try (powered) eSATA controller, or some other SAS/SATA-based connection.

Cheers!
-sub

 

[Dangman]

PSU: COOLER MASTER GX Series - 750W

Ditch that crappy PSU:
http://www.jonnyguru.com/modules.php?name=NDReviews&op=Story&reid=188

In that price range, you should be able to find the better quality Antec Neo Eco 620C 620W PSU or for about $20 more, you can find the higher quality Antec TP-750 750W PSU which is currently on sale.

 

[ethraax]

First off, thanks for your detailed response!

Have you read about my Mesa web-interface yet? It's not ready for real usage for the moment, but if you would like to use it in the future you could prepare for that already.

Although I like the WebGUI in FreeNAS, I'm not likely to heavily rely on it. It has issues with ZFS. The biggest issue is that it won't let you add vdevs to an existing pool. I could, of course, be using it wrong, but the ZFS commands are incredibly easy, so I don't mind.

I'd like to have the server up and running within 20 days. I assume I can switch to your web-interface without losing all my data though, as my data is stored on separate disks.

I've also considered using full FreeBSD. Not only does it have better hardware support (being newer and all), but I can probably get updates and help more quickly than with FreeNAS. Luckily, I should be able to change my OS as long as it supports ZFS.

Streaming over network generally works fine; but sometimes is tricky due to latency. If your program can increase the buffer/precaching size, then you should have no problems.

Is this something that an add-on NIC could help with? I feel like the integrated NIC probably doesn't have some traffic profiling features that could prove useful if I have multiple computers access the server at once. I could be wrong though - I'm no expert on NIC's.

Sometimes, multiple streams totally crash HDD performance causing them to seek excessively and be really, really slow. So the MB/s may be misleading; but hey test it out! With just two streams this shouldn't really be a problem.

Since streaming HD video still doesn't require a whole lot of bandwidth, couldn't I set it up to take turns caching chunks of 10-20 seconds of video and take turns among all the streams? I assume if it's possible, it's already implemented in ZFS, but that's just an assumption.

You have 2x 1.5TB and you buy 6x 1.5TB more. So this is what you do:

1. create 4-disk RAID-Z from new 1.5TB drives
2. copy data from desktop RAID0 to new RAID-Z
3. now add a new raidz to your existing pool; voila now you have 8x 1.5TB drives in two RAID-Z pools all sharing the same free space.

That's my plan exactly.

Note that the stored files before adding the second array will only have the performance of that of a single array; it won't re-distribute the data. So only newly created/modified data will get speed enhancement from the second RAIDZ pool.

Is there some sort of script that will re-distribute the data, preferably at a scheduled time (like 4 AM - 6 AM every morning) until it's finished? It seems simple enough that I could probably write it myself.

Generally your setup is good; two RAIDZ pools are better than one. Consider also using copies=2 for data really important, especially if that data isn't all that large. This would make both copies land on different arrays as well.

Most of the data is unoriginal, in that I could reobtain it if I had to. However, the sources are scattered and I don't have any of them documented, so doing so would be a pain in the ass. My point is that I'm fine with only having one drive of redundancy for each set of four drives.

A quadcore could be worth considering if you do compression, otherwise the Core i3 is fine. ZFS can use all the cores you have; so a 64-core monster without other hardware bottlenecks would scale really well with ZFS and compression. So a low-power, low-frequency quadcore chip might be preferable.

AMD has the X4 610e or something; not sure about the number. But it's a quadcore with 45W TDP. If you're keen on saving power this might be a win-win on both?

To be honest, your core i3 might be just enough. But if you want to safe and like some additonal CPU power for future features then a quadcore is highly recommended. Or even the 6-core from AMD as its rather cheap.

I've thought about going with a Core 2 Quad, but I also wanted DDR3 RAM, and I didn't think that there were LGA-775 boards that supported DDR3 RAM. Also, power is not a concern for me - I spend most of the year where there are no electric bills (it's a flat rate). I'll look at the hex-core from AMD, although I must admit, I've never used AMD processors before.

In fact, you can add anything: a single disk, a mirror, a RAID-Z2. Any number of disks too. What you CAN'T do, is add a disk to an existing vdev; for example making your 4-disk RAID-Z a 5-disk RAID-Z.

The advantage here is:
- you do not need same capacity of disks; you can use 3TB disks in the future and share free space with your smaller capacity disks. What RAID allows you to do this?
- you can expand in a safe way, without putting your data at risk in a complex procedure; and it happens instantly; not 24+ hours.
- you can add vdevs with a different redundancy. Want to add just two disks? Then add a mirror instead. Note that you can not remove vdevs, and you can't transform a mirror into a raid-z either.

So actually i prefer the ZFS way of expansion. But block-pointer rewrite will make things like expanding RAID-Z and alot of other things possible in the future. So it's just the current versions that do not allow this; no theoretical impossibility.

I was under the impression that storage pools with the same classification of vdevs performed better.

Yes, geom_eli allows you to do this. It is integrated in FreeNAS and FreeBSD. It has an option that requires you to enter a passphrase at boot time or once the system has started. This allows encrypting both your system disk and data disks.

Note that my project will also support encryption in the future, but encrypting ZFS disks instead of the usual UFS as i believe FreeNAS uses.

I'm a little confused - does it support disks that are used with ZFS? I assume I'll have to set it up from the shell and not WebGUI, but that's fine with me. Also, what kind of overhead does it require? Should I upgrade my CPU if I plan on encrypting my drives?

In FreeBSD you can configure mostly any wireless adapter to function as access point, where other wireless clients can connect to (with WPA2 or without). It would require time to set this up though, it's likely not implemented in FreeNAS.

Perhaps you need base FreeBSD, with all bells and whistles available, rather than FreeNAS which has everything it does not 'need' shaved away; making it hard to configure it for things it was not designed for.

This may push me to run FreeBSD instead of FreeNAS, but let me just clarify what I want to be able to do (I wasn't terribly precise). Let's say that my server was in the middle of a desert, far away from any wireless networks, with no connections other than power, and that it had a wireless adapter attached to it. I want someone with a laptop near my server to be able to connect to it as if the server was its own network with a single device. Obviously, I wouldn't want the network driver on the server to expose any users on the network to each other (if two people have laptops and are connected to it, neither person should be able to detect if the other user is connected, or, at least, not detect the identity of the other user.)

Are you using your 5,25" (cdrom-size) bays already to store HDDs? If not buy some 4-in-3 HDD bays, storing 4 3,5" desktop-class drives at the height of three cd-roms. Or something similar. If you can't, you can try (powered) eSATA controller, or some other SAS/SATA-based connection.

The case has 7 3.5" internal bays. It has a few external 5.25" bays, and I was going to use 1-to-1 converters. Even if I use 3-to-4 converters, I'm more concerned about the power situation, since there certainly won't be enough SATA connectors for more hard drives, even if there is physical space.

In this case, I was thinking of a smallish box containing perhaps 8 SATA drives that plugged into the back of the server.

Thanks for your time!

-- Ethan

 

[leagle]

Sub.Mesa sort of touched on it, but don't use a realtek nic with FreeNAS. The driver support is either bad or doesn't exist. I learned this the hard way. Now I'm a fan of only using Intel Pro 1000 nics in any server that I build.

 

[sub.mesa]

Although I like the WebGUI in FreeNAS, I'm not likely to heavily rely on it.

The problem with FreeNAS is that it has part of FreeBSD stripped down to save space; thus you can't (easily) do everything with FreeNAS like you can with FreeBSD even with SSH access and doing it on the command line.

Note that i do not know FreeNAS very well; i never really used it just looked at it and tested it a little bit in a VM.

It has issues with ZFS. The biggest issue is that it won't let you add vdevs to an existing pool. I could, of course, be using it wrong, but the ZFS commands are incredibly easy, so I don't mind.

They are easy, but setting up your disks properly is some work too:

identify name of drive to be formatted
overwrite device with zeroes partially
apply a geom_label to the drive
create zpool

FreeNAS creates partitions on your drives and no geom labels as far as i recall. As you use FreeBSD+ZFS, it's only logical to utilize the additional benefits FreeBSD offers you over OpenSolaris; one of those is geom_labels which can identify your disks with 'ZFS-disk1' for example, regardless of how it was connected (SCSI device or SATA device; different SATA ports, etc).

My interface does exactly that; setup the disks in the best way possible as i believe, and creating the zpool array. It also allows expansion, adding a second array, by the way. That already works.

Basically, if you want to use my interface, you have two options:
1. use the pre-installed image on a USB-stick or small HDD
2. install FreeBSD yourself, then install my web-interface next to it (install instructions)

Even for those who are perfectly fine with command lines and ZFS commands, my interface can be useful: it represents data visually better, allows quick shortcuts and perhaps can prevent minor but deadly mistakes. You can also review each 'dangerous' command (like creating/expanding your array) before executing it, so you can confirm the command is correct or learn how it is done if you're not yet familiar with the command syntax.

Even if you don't want to use my interface, setting up the disks and such and using normal FreeBSD might be recommended over using FreeNAS, with its restrictions.

I'd like to have the server up and running within 20 days. I assume I can switch to your web-interface without losing all my data though, as my data is stored on separate disks.

Is this something that an add-on NIC could help with?

The latency would be coming from disks; not due to your network. In fact, onboard NICs supplied directly by the chipset as usually the lowest latency. Most gigabit NICs have receive/send checksums and other features.

Since streaming HD video still doesn't require a whole lot of bandwidth, couldn't I set it up to take turns caching chunks of 10-20 seconds of video and take turns among all the streams? I assume if it's possible, it's already implemented in ZFS, but that's just an assumption.

You can enable prefetching. Prefetching is enabled by default for 64-bit systems with more than 4GB memory. Sometimes, prefetching can cause lower performance due to it consuming RAM otherwise available to ZFS' ARC cache.

So test what works best for you.

Is there some sort of script that will re-distribute the data, preferably at a scheduled time (like 4 AM - 6 AM every morning) until it's finished?

Not at the ZFS level. But you could copy all data to a new directory/filesystem, duplicating all data. That would require enough free space. Such things can be done on the command line, jobs can control whether to pause or resume this job.

I've thought about going with a Core 2 Quad, but I also wanted DDR3 RAM, and I didn't think that there were LGA-775 boards that supported DDR3 RAM. Also, power is not a concern for me - I spend most of the year where there are no electric bills (it's a flat rate). I'll look at the hex-core from AMD, although I must admit, I've never used AMD processors before.

For these purposes it may make sense, compression/encryption is FPU-heavy and that's AMD's strong suit; while Intel dominates ALU performance if i understand CPU-performance well enough. Core 2 is still using FSB-architecture; i prefer a Core i3 over a Core2Quad! ZFS uses alot of memory so memory latency is an issue (throughput not that much). So slower frequency RAM with lower CAS shouldn't hurt performance.

The AMD option is worth investigating, its also a lot cheaper. But Intel has nice chips too, a core i5 quadcore would be great; but hey the core i3 option could be just fast enough if that's more convenient to you.

I was under the impression that storage pools with the same classification of vdevs performed better.

True yes, for performance reasons you should keep the performance levels of each vdev about the same so you don't create too many bottlenecks.

I'm a little confused - does it support disks that are used with ZFS? I assume I'll have to set it up from the shell and not WebGUI, but that's fine with me. Also, what kind of overhead does it require? Should I upgrade my CPU if I plan on encrypting my drives?

You can encrypt whole disks with a ZFS filesystem on it, or you can create ZVOLs on a regular ZFS filesystem and encrypt those, containing a UFS filesystem instead. Essentially, a 20GB file that is encrypted and has a UFS filesystem on it. I never tried a ZFS filesystem on ZVOL though; but that's an interesting option as well if that works properly.

I want someone with a laptop near my server to be able to connect to it as if the server was its own network with a single device.

Connect as in: connect to wireless network 'FreeBSD-NAS', enter WPA2 password, now open Firefox and navigate to http://<ip of server> and you get access to the web-interface, or use PuTTY to connect using SSH protocol. That all should work with a BSD setup, functioning as access point (and perhaps NAT router if you like, or DHCP server).

Obviously, I wouldn't want the network driver on the server to expose any users on the network to each other (if two people have laptops and are connected to it, neither person should be able to detect if the other user is connected, or, at least, not detect the identity of the other user.)

Hmm, wireless communication can be detected and potentially identify a user or reveal sensitive information. This is another topic that falls outside of ZFS; though perhaps someone else could comment on this.

The case has 7 3.5" internal bays. It has a few external 5.25" bays, and I was going to use 1-to-1 converters. Even if I use 3-to-4 converters, I'm more concerned about the power situation, since there certainly won't be enough SATA connectors for more hard drives, even if there is physical space.

So you buy some splitter cables so you can connect 4 HDDs on one SATA connector. That shouldn't be a big problem.

In this case, I was thinking of a smallish box containing perhaps 8 SATA drives that plugged into the back of the server.

Possible too, but do mind a good solution! A good SAS with two external Mini-SAS connectors would allow 8 drives to be connected in an external appliance, cable length about 1.0 metres i think. The controller is cheap, but the external housing with Mini-SAS likely a little more expensive.

 

[ethraax]

Yeah, sorry for getting slightly off-topic - I guess I should take up my FreeBSD/Networking questions on a FreeBSD forum.

So you buy some splitter cables so you can connect 4 HDDs on one SATA connector. That shouldn't be a big problem.

This might sound silly, but I never knew those existed. I think I'll get a different PSU then - the primary reason I picked the one that I did was because of it's number of SATA connectors. Perhaps I can put the savings from that into upgrading to an i5 or a nice hex-core AMD processor.

Possible too, but do mind a good solution! A good SAS with two external Mini-SAS connectors would allow 8 drives to be connected in an external appliance, cable length about 1.0 metres i think. The controller is cheap, but the external housing with Mini-SAS likely a little more expensive.

I don't mind spending a little bit of extra money (in addition to more disks) when upgrading my system's capacity. I just want to make sure that there's a reasonably elegant solution. The one you presented seems to cover that. My biggest requirement is that the extra hard drives be elegantly removed from the server when it is shutdown for relocation (which will happen roughly twice a year, possibly four times a year).

I have another question that completely slipped my mind when I made this post:

8) What kind of hard drives should I get? Currently, I have two 1.5 TB Samsung EcoGreen drives. They run a bit slow (5400 rpm) and have a 32MB cache. I was looking on Newegg at Western Digital Caviar Greens, mostly because they have a 64MB cache, but I don't really have a good reason for buying those drives over any other. Should I be concerned with using "green" drives in a RAID configuration, and are there any other hard drives that I should look at? Also, should I buy two more Samsung EcoGreen drives so that second raidz1 vdev has identical drives?

9) What kind of PSU should I look at? Now that I know I don't need one with many SATA connectors, I feel like I can save a significant amount of money. I don't plan on running a PCIe 2.0 video card, so I should only need a 500W (or so) PSU to cover the whole thing, and then some. Is there anything I should look for in a PSU with regards to running a file server?

Thanks for your help.

-- Ethan

 

[aaronspink]

For these purposes it may make sense, compression/encryption is FPU-heavy and that's AMD's strong suit; while Intel dominates ALU performance if i understand CPU-performance well enough. Core 2 is still using FSB-architecture; i prefer a Core i3 over a Core2Quad! ZFS uses alot of memory so memory latency is an issue (throughput not that much). So slower frequency RAM with lower CAS shouldn't hurt performance.

Um...

Compression/encryption are NOT FPU heavy unless you want LOSSY compression and encryption which if you act fast you can probably get some worthless patent on. Lossless compression by necessity is all integer math. Same for encryption. If you want to accelerate encryption then you want either a hardware accelerator or instructions that accelerate encryption which are available in VIA C7 and 32nm Intel CPUs.

 

[ethraax]

My question still stands though - will using an i3 with an encrypted ZFS drive likely cause a bottleneck in the CPU?

 

[adi]

I just set up freeNAS as my backup's backup, here are my results

Set to use gzip on all but video files. When copying over, I easily sustained 90-100mb/sec with compression on. Load averages were in the 9-11 range, but server was still quite responsive. This is on a Core2Duo 3.0ghz. I am very tempted to drop in a quad core, but since I don't access it regularly, I'll probably just leave it alone.

With 8gb memory, it was sitting at 35-50% memory usage. Since you are looking at doing encryption/more roles, I would say 6-8gb is the minimum you would want.

I am also not using raidz, but just a spanning pool. With real ZFS raidz/z2, I would think it would also be more resource intensive as well since it will be doing parity.

My main server uses TrueCrypt, and it uses very little resources (2-3% CPU) when doing work on files. I don't know how efficient ZFS/freeNAS encryption is though.

ETA: onboard realtec nic would hard lock 30 minutes into large transfers. Tossed in an Intel nic, rock solid since.

 

[sub.mesa]

Um...

Compression/encryption are NOT FPU heavy unless you want LOSSY compression and encryption which if you act fast you can probably get some worthless patent on. Lossless compression by necessity is all integer math. Same for encryption. If you want to accelerate encryption then you want either a hardware accelerator or instructions that accelerate encryption which are available in VIA C7 and 32nm Intel CPUs.

Well i apologise if i missed the mark on FPU/ALU; that was just a guestimate from me. But this page does show AMD is quite well suited for both compression and encryption with low power versions (low TDP); it also implies that encryption is rather FPU based, but if you say that's not true i'll take your word for it.
http://www.tomshardware.co.uk/mini-itx-amd-785g-intel-h55,review-31943-7.html

As far as i know the VIA C7 AES-acceleration ("Padlock") does only 70MB/s and thus a single core AMD/Intel CPU would do better without acceleration already; at the cost of increased CPU utilization, though.

On my own tests done on FreeBSD without hardware acceleration, the speed grows linearly with the number of cores. At least on my AMD quadcore, and AMD dualcore. I also tested and compared with Intel E8400 dualcore (3.0GHz S775 FSB based). My 2.0GHz quadcore gets about twice the speeds the Intel does, with AES-128 encryption. But those tests are old and don't have the exact results at hand.

Since this depends solely on your CPU and OS (crypto-driver), we could do some comparisons in the future, as we all have different hardware but encryption speeds if tested properly should be very comparable against eachothers CPU.

My main server uses TrueCrypt, and it uses very little resources (2-3% CPU) when doing work on files. I don't know how efficient ZFS/freeNAS encryption is though.

Somehow i doubt that. If you're transferring at 60-100MB/s you should be seeing alot higher CPU utilization on the client/desktop. Likely you won't see kernel CPU usage attributed to the CPU usage of the TrueCrypt process, in the windows task manager. Looking at the total system CPU usage might be a better idea.