• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

NAS upgrade - looking for advices

zrav said:
If you don't need to virtualize the storage OS, do you still need device passthrough? Yes, it seems trickier/pickier on QEMU/KVM than on ESXi, no I haven't tried it (no need). Depending on what you need, LXC might be a solution.
Click to expand...
At that point the only things I'd like to pass through are Intel NICs (dual/quad ports). Is that possible? Or if there is a 1:1 NIC dedicated mapping in KVM I may also be able to live with that.
For other hosts (HTPC) I'd like to passthrough my nvidia card if possible.

zrav said:
Also do you actually need iSCSI? I would avoid adding unnecessary complexity/tech stacks. Local storage certainly is faster.
Click to expand...
It's not like I really need iSCSI but centralized VM storage would be better in my opinion. The performance of that compared to CIFS (SMB) / NFS would be much better for such a scenario.
If I could get sufficient performance over NFS I don't really see a need for iSCSI although I doubt that NFS to 4-5 VM hosts will provide high enough performance ....

zrav said:
The benefit of running Linux as the host is its great hardware support, software availability and community.

VMware will start to show its advantages on big installations where you can make use of the advanced features. On few, more or less independent hosts I have yet to see a real advantage compared to KVM.
Click to expand...
Advanced features such as? (besides vMotion / storageMotion which cost a lot of $$)


zrav said:
As for performance and features of ZFS implementations, see here: http://www.open-zfs.org/wiki/Features
You'll hear contradictory things about ZoL: Some say that only zvols are fast as that's what it's main developer uses, others say the opposite. I would setup the pool in each of the OSes/configurations you are considering and test performance with your loads.
We have not had any stability issues whatsoever since ZoL 0.6.3. Same goes for performance as long as the pools are setup/dimensioned properly.
Click to expand...
Agreed. Most ZFS reviews on the web only tend to make me more confused :(
 
luckylinux said:
At that point the only things I'd like to pass through are Intel NICs (dual/quad ports). Is that possible? Or if there is a 1:1 NIC dedicated mapping in KVM I may also be able to live with that.
For other hosts (HTPC) I'd like to passthrough my nvidia card if possible.
Click to expand...
Why do you need to passthrough the NICs? The Virtio NICs are fast. Setup network bridge on host and be done.
GPU passthrough should be possible and performance looks favorable on KVM compared to ESXi: http://grids.ucs.indiana.edu/ptliupages/publications/Cloud_2014 (1).pdf

It's not like I really need iSCSI but centralized VM storage would be better in my opinion. The performance of that compared to CIFS (SMB) / NFS would be much better for such a scenario.
If I could get sufficient performance over NFS I don't really see a need for iSCSI although I doubt that NFS to 4-5 VM hosts will provide high enough performance ....
Click to expand...
Your opening post didn't say you were planning on serving other hosts with storage. In that case iSCSI is of course faster than the other options. And you'll definitely want to run mirrors and not RAIDZx.

Advanced features such as? (besides vMotion / storageMotion which cost a lot of $$)
Click to expand...
Exactly.
 
zrav said:
Why do you need to passthrough the NICs? The Virtio NICs are fast. Setup network bridge on host and be done.
GPU passthrough should be possible and performance looks favorable on KVM compared to ESXi: http://grids.ucs.indiana.edu/ptliupages/publications/Cloud_2014 (1).pdf
Click to expand...
OK that's good news ;)

zrav said:
Your opening post didn't say you were planning on serving other hosts with storage. In that case iSCSI is of course faster than the other options. And you'll definitely want to run mirrors and not RAIDZx.

Exactly.
Click to expand...
I didn't because back there I didn't think about putting VM storage on SSD. That came out after many post and a discussion with _Gea (I think it was here , post #22).

I though about putting a:
2 striped of (2 x mirrored 1TB Crucial M550) for 2TB usable. Should that be enough and give good performances?

Is there any guide about setting up iSCSI or SAN for ZoL? The results I found on google seem more to configure the GNU/Linux box to use iSCSI as a client, not a server/host. Do you have any first-hand experience with iSCSI or similar on ZoL?
 
2x2 SSDs should be plenty fast (be aware of lower endurance on consumer SSDs, but you can monitor that via SMART).
With an all SSD pool you don't even need slog device or L2ARC as there's nothing to gain (unless you used a much faster SSD). Do you have at least 10GE to get that speed to the other hosts?

There's plenty of tutorials for setting up an iSCSI target under linux. You simply configure the target to point to the zvols you create.
 
zrav said:
2x2 SSDs should be plenty fast (be aware of lower endurance on consumer SSDs, but you can monitor that via SMART).
With an all SSD pool you don't even need slog device or L2ARC as there's nothing to gain (unless you used a much faster SSD).
Click to expand...
Agreed. I save some money on ZIL/L2ARC and spend 1600€ on SSD for VM storage :D
Write endurance of consumer SSDs is quite low, although I think 72TB warranted endurance (for each SSD) should be enough, don't you think?

zrav said:
Do you have at least 10GE to get that speed to the other hosts?
Click to expand...
I'm planning it although 10GE is quite expensive. I'm also considering Infiniband since both switches and adapters are like 3-4 times less expensive than 10Gbe (although second hand).

That speed (10-20GE) is only required for the biggest hosts IMO. Smaller hosts should be happy enough with 4 separate 1gbps links (so that each VM uses one, not using Link Aggregation as that doesn't increase the overall bandwidth). Bonding may also be an option.

zrav said:
There's plenty of tutorials for setting up an iSCSI target under linux. You simply configure the target to point to the zvols you create.
Click to expand...
iSCSI configuration for ZoL is not specific to ZFS?
 
72TB is low (just 72 full drive writes), but most MLC consumer SSDs seem to live much longer than rated.

Correct, you create a zvol (virtual zfs backed block device), then you configure the target (which will take any block device as backstore). http://linux-iscsi.org/wiki/ISCSI
 
zrav said:
72TB is low (just 72 full drive writes), but most MLC consumer SSDs seem to live much longer than rated.

Correct, you create a zvol (virtual zfs backed block device), then you configure the target (which will take any block device as backstore). http://linux-iscsi.org/wiki/ISCSI
Click to expand...
So which SSD would you suggest I buy?

Samsung seems to always screw up their firmware.

SandForce-based SSDs also have a mess of firmware issues and don't operate well when the disk if >50% full.

Intel S3500 may be an option but it's like 2-3 times more expensive than Crucial M500/M550 (275TB rated endurance IIRC).
Intel S3700 may also be an option but it's like 5 times more expensive than Crucial M500/M550 (>2000TB write endurance).

I think a test (here for instance) showed that up to 300TB any 20-25nm MLC will do just fine. The Crucial M550 is rated at 72TB for 1TB as it is for 128GB. That doesn't make any sense IMO :confused: If it's rated 72TB for the 128GB version it should be more like ~720TB for the 1TB version. I don't think they over-provisioned the 128GB with 900% spare sectors of flash "for free" ;)

Any suggestion on alternatives is welcome. I just want the SSD to be "good" and feature power loss protection (which for instance Samsungs don't - not even the PRO version).
 
Last edited:
luckylinux said:
Samsung seems to always screw up their firmware.
Click to expand...
I'm only aware of the 840 Evo issue.
SandForce-based SSDs also have a mess of firmware issues and don't operate well when the disk if >50% full.
Click to expand...
Agreed.

The Crucial M550 is rated at 72TB for 1TB as it is for 128GB. That doesn't make any sense IMO :confused: If it's rated 72TB for the 128GB version it should be more like ~720TB for the 1TB version.
Click to expand...
Yeah, i noticed that too. I think thats a marketing mistake or erring on the side of caution.

Any suggestion on alternatives is welcome. I just want the SSD to be "good" and feature power loss protection (which for instance Samsungs don't - not even the PRO version).
Click to expand...
Well with this constraint there aren't many options, so go with the 550's. Independent of that, with 5 VM hosts you should prolly be looking at UPS's...
 
zrav said:
Well with this constraint there aren't many options, so go with the 550's. Independent of that, with 5 VM hosts you should prolly be looking at UPS's...
Click to expand...
I have an UPS in any case. And redundant PSU on that server too ;)
I'm not sure if I need power loss protection for the SSD as well. Any insights?

If you lift that requirement, which alternatives would you suggest? Intel S3700 has a good price / endurance ratio. Others?

EDIT: Samsung 850 PRO 1TB seems to have very good performance. That's 50% more expensive than Crucial M550 though :(
 
Last edited:
Well, the UPS doesn't help against data corruption when accidentally pulling the power cable or the wrong drive out of an array (been there). But it depends on what your requirements are.

The S3700 is excellent of course (we are using it as L2ARC), but as pool drive you need the proper budget...
I was going to suggest the 850 Pro. You could start with a single pair of those, activate LZ4 compression on the ZFS pool and see what mileage you get before (if needed) adding a second mirror.
 
zrav said:
I was going to suggest the 850 Pro. You could start with a single pair of those, activate LZ4 compression on the ZFS pool and see what mileage you get before (if needed) adding a second mirror.
Click to expand...
So 2 x 850 PRO 1TB. Mirror or striped for ZFS?
One thing I'm concerned about Samsung is GNU/Linux support, but I may be wrong ...
 
If you mean the Samsung app to install new firmware, yes you'd have to move that drive to a Windows PC to flash. Otherwise I'm unaware of any incompatibilities, why wouldn't the Samsung work if other SSDs do?
 
zrav said:
Otherwise I'm unaware of any incompatibilities, why wouldn't the Samsung work if other SSDs do?
Click to expand...
Just remembering a bad day trying to update my Samsung firmware on a Windows PC with AMD hardware ... I'm just a little skeptic about it working well with GNU/Linux :rolleyes:

By the way do you use LUKS of your pools (general or for VM storage)? I'd be interested to see if performance is highly affected by it.
 
zrav said:
I have never used LUKS.
Click to expand...
Ah OK.
However about Samsung 850 PRO: here they estimate lifetime at 6000 TB for the 1TB model.

However going by manufacturers specs:
- Crucial M500/M550/MX100 is rated 72TB for every size
- Samsung 850 PRO is rated 150TB for every size

So I'm not sure about it ... I think both value are quite conservative, especially for the 1TB drive. Out of the two Samsung 850 PRO may perform much better when the disk is nearly full, but at the same time it doesn't feature power loss protection.

This seems like a tough call to choose between the 2 solutions :(
 
Or to put another choice: SANDISK Extreme Pro
However it seems that if doing lot of writes both the Samsung 850 PRO and the Crucial MX100 (so supposedly the M500 and M550 as well) run into heavy performance degradation.

Any suggestion on which one of these would be best suited for VM storage then?
 
I use a lot of Sandisk Pro Extreme, mainly because of the build-in overprovisioning
that keeps write performance high under load from stock and because they are quite
cheap with 10 years warranty.

I would expect, that a Samsung Pro is equal or faster under heavy load if you manually
add a HPA on a new disk.
 
_Gea said:
I use a lot of Sandisk Pro Extreme, mainly because of the build-in overprovisioning
that keeps write performance high under load from stock and because they are quite
cheap with 10 years warranty.

I would expect, that a Samsung Pro is equal or faster under heavy load if you manually
add a HPA on a new disk.
Click to expand...

Well Sandisk Pro Extreme are a bit more expensive per GB compared to Samsung 850 PRO.

The drawback is that neither of them provides power loss protection (although the Sandisk Pro Extreme has "partial" power loss protection due to an SLC buffer AFAIK).

Strangely enough the Crucial MX100/M500/M550 features power loss protection using supercapacitors.
Now if such a feature is needed for storage disks I'm not sure. Since I don't plan at this point to use a ZIL (no need since VM storage is already SSD) ideally they should have power loss protection. Any tips here?

If I buy either of them, how much "manual" over-provisioning should I allocate from the disk 960GB/1TB capacity? 10% should be enough or it would be better more?

EDIT: just saw a deal for the Sandisk Pro Extreme 960GB (415€) which I could get for the same price as the Crucial M550. Compare that to Samsung 850 PRO for 550€ ...
 
Last edited:
A 960 TB Sandisk and a 1TB Samsung have 1024 MB flash memory.
The Sandisk is "smaller" because of the large built-in overprovisioning.

With the Sandisk you do not need extra overprovisioning.
 
_Gea said:
A 960 TB Sandisk and a 1TB Samsung have 1024 MB flash memory.
The Sandisk is "smaller" because of the large built-in overprovisioning.

With the Sandisk you do not need extra overprovisioning.
Click to expand...

So you'd reccomend the Sandisk for VM storage even though it doesn't have power loss protection? The deal seems to be good in my opinion ;)

BTW I have dual redundant PSU each connected to a separate UPS. There is always the slight chance that there may be a backplane power supply failure though ...
 
I'd like a confirmation / further advices since I'm about tu bite the bullet and take 4 x Sandisk Ectreme Pro 960GB right now for VM storage :rolleyes:

If anyone here has advices on other drives for VM storage I'd gladly hear their opinion ;)
 
All a matter of price. There will be always something that is faster or more safe like a Intel 3500/ 3700 enterprise SSD. What you can do is adding a 100 GB Intel 3700 as a ZIL device to reduce small random writes on your pool, to improve overall sync write performance and to add a power-fail safe log device
 
_Gea said:
All a matter of price. There will be always something that is faster or more safe like a Intel 3500/ 3700 enterprise SSD. What you can do is adding a 100 GB Intel 3700 as a ZIL device to reduce small random writes on your pool, to improve overall sync write performance and to add a power-fail safe log device
Click to expand...
Mmm ... then I run out of HDDs / SSDs bays;
- 6 x HDD for main storage (RAIDZ2)
- 2 x SSD for OS (mirror)
- 4 x SSD for VM storage (2 x mirror of 2)
= no space left for a ZIL device :S

What to do now? Should I "dowgrade" the main storage to a 4 disk RAIDZ2?
Other solutions? I'd rather not change the chassis since a 3u 16 bays chassis with redundant PSU may be an option but like 1000€ more (+ CPU / RAM / Motherboard ~ 2000€) :(
 
Last edited:
Do you really need that level of security with a boot mirror + a 2 TB Raid-10 ultrafast SSD pool with a dedicated enterprise ZIL?

The chance of a corrupted filesystem on a powerloss without a dedicated ZIL but with sync enabled (where log is placed on the regular SSD pool) is quite minimal. Many production systems work like this for years and backups are needed anyway.
 
_Gea said:
Do you really need that level of security with a boot mirror + a 2 TB Raid-10 ultrafast SSD pool with a dedicated enterprise ZIL?

The chance of a corrupted filesystem on a powerloss without a dedicated ZIL but with sync enabled (where log is placed on the regular SSD pool) is quite minimal. Many production systems work like this for years and backups are needed anyway.
Click to expand...
In fact I though I didn't need the ZIL but I'm open to suggestions at this point.
With the current case I can have either 6 x 3.5'' + 6 x 2.5'' devices or 8 x 3.5'' + 4 x 2.5'' devices.

Otherwise I could go the cheap way (by sacrificing redundant PSU) and get (for instance):
- Case: 3U 16 x 3.5'' + 2 x 2.5'' @ 260€
- Power supply: 2U seasonic PSU SS-500L2U or second hand redundant Zippy Emacs PSU (although I think better of Seasonic) @ 130€
- Motherboard: Supermicro X10SRL-F 2011 V3 @ 250€
- RAM: 2 x 16GB ECC Registered DDR4 RAM @ 400€
- CPU: INTEL Xeon E5-2620 v3 @ 340€

Total is ~ 1400€ which is reasonable, although I'd like to avoid spending more money if possible :rolleyes:
Any tips?
 
Last edited:
luckylinux said:
Mmm ... then I run out of HDDs / SSDs bays;
Click to expand...

If you're not too picky on "bays" and you have some spare room in your case, you could look at picking up a couple of these:


And if "hot swap" is a must, there are options for that as well (though typically a little more spendy):


They are an easy way to turn your unused PCI slots into 2.5" HDD/SSD slots.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
dsumike said:
If you're not too picky on "bays" and you have some spare room in your case, you could look at picking up a couple of these:


They are an easy way to turn your unused PCI slots into 2.5" HDD/SSD slots.

And if "hot swap" is a must, there are options for that as well (though typically a little more spendy):

Click to expand...
I'm afraid I'm also low on PCI(e) slots once I'll have the HBA controllers, Gigabit and 10GE NICs, ... :(

EDIT: Furthermore the Startech adapter won't work: it's a full PCI bracket, not a low profile one :(
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
dsumike said:
Guess it's time to get a new case.

I like this one: SC846TQ-R900B Chassis
Click to expand...

I already have 2 x (Supermicro SC847E16-R1400UB: 24 front bays + 12 rear bays) hot swap units but the point is not to use them (yet). They will do media storage in the future ;)

The point was to optimize the current configuration which unfortunately doesn't seem to be possible :(
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Maybe I should not put the ZIL then, simple as that ;) Not sure if I can otherwise put the ZIL inside the chassis and not fasten it :confused: I wish there would be a low profile 2.5'' bay holder :(

EDIT: found one here (from Germany)

Alternatively I could just use the second chassis I own (I currently own 2 x SC825TQ-740LPB) as JBOD for the HDDs. That may be another option (avoiding the need - for now - to buy a new chassis + Motherboard + CPU + RAM).
 
@_Gea: what do you suggest to do (if possible with the hardware I currently own)?
Do you think a ZIL would be needed? I could sacrifice one PCI slot for that (see adapter in my previous post). In that case should I buy the Intel S3700 100GB or 200GB version?

Or what are you suggestingme to "sacrifice" in favor of it?

Thank you for your help ;)
 
You must find your own compromise
If its basically a soho build, you can

- use the 100 GB Intel 3700 or
- skip the dedicated ZIL and use the onpool ZIL (not as fast and secure as with a dedicated enterprise ZIL)
- use the PCI-e mounting set or a double sided tape to mount the ZIL elsewhere in the case
- or use a Raid-Z1 of 3 SSDs instead the Raid-10
 
_Gea said:
You must find your own compromise
If its basically a soho build, you can

- use the 100 GB Intel 3700 or
- skip the dedicated ZIL and use the onpool ZIL (not as fast and secure as with a dedicated enterprise ZIL)
- use the PCI-e mounting set or a double sided tape to mount the ZIL elsewhere in the case
- or use a Raid-Z1 of 3 SSDs instead the Raid-10
Click to expand...
And between the 100GB and 200GB versions of the ZIL, which one would you pick (budget aside)? I don't think even with 10GbE I would need 200GB of ZIL but I think I can get more IOPs with the 200GB. Opinions on this?

I think I'll put this in the PCI bracket adapter then ;)
 
A ZIL must not be larger that the data that can be transferred within 10s.
With 10 GbE you have about 1 GB x 10 what means less than 10 GB.

Even "the best of all enterprise ZILs", a Dram based ZeusRAM has only 8 GB.
So the only difference between the two Intels is a higher IOPS value at the double price
what will result in a slightly better write performance with sync write.
 
_Gea said:
A ZIL must not be larger that the data that can be transferred within 10s.
With 10 GbE you have about 1 GB x 10 what means less than 10 GB.

Even "the best of all enterprise ZILs", a Dram based ZeusRAM has only 8 GB.
So the only difference between the two Intels is a higher IOPS value at the double price
what will result in a slightly better write performance with sync write.
Click to expand...

100GB it is then (need to save some money). Would it be better if I do over-provisioning or with 2000TiB write endurance there is no need?
 
The 100 GB Intel has 128 GB Flash so a build in 28GB (about 30%) overprovisioning.
If you like you can add more but I doubt you see a difference.
 
_Gea said:
The 100 GB Intel has 128 GB Flash so a build in 28GB (about 30%) overprovisioning.
If you like you can add more but I doubt you see a difference.
Click to expand...

So 100GB without any overprovisioning. What would happen if ever I were to bottleneck this 100GB ZIL (say I have 4 x concurrent 10GbE connections or more)?
 
Your pool becomes quite slow on sync writes unless you add more and/or faster ZIL devices
 
You must log in or register to reply here.
Back
Top