Long story short, I have this box:
Chassis: Supermicro SC846E16-R1200B (Replaced PSU's with PWS-920P-SQ)
Motherboard: Supermicro X8DTE-F
CPU's: x2 Intel Xeon L5640 (Hexa-core, 2.26GHz, HT)
RAM: 96GB DDR3 (12 x 8GB PC3-10600, Registered ECC)
HBA: Adaptec 5805 (All drives in JBOD passthrough)
SAS Expander/Backplane: Supermicro BPN-SAS-846EL1 (The SAS Expander is the backplane of the chassis)
OS Drives: x2 Crucial MX100 128GB SSD (ZFS Mirror)
Storage Drives: x8 Toshiba PH3500U-1I72 5TB 7200RPM (ZFS Mirror)
This box is running Proxmox, and is my primary hypervisor, and storage server (basically Debian + KVM + ZoL). Right now I have ~10TB of data on the main ZFS pool, including the 15 or so VM's I have running. My plan is to create a new ZFS Mirror to use for VM storage, and looking to do ~500GB SSD's for that. Right now I'm looking at the MX200 500GB.
Also, I'm looking to add L2ARC and a SLOG to this box. I have my ARC limited to 48GB of RAM, but intent to raise that to 64B before adding a L2ARC. I limited the ARC to half the system memory to see what performance I got, and to leave plenty open for my VM's. Right now the system is consuming ~66GB of RAM (48GB for ARC, 18GB for VM's) leaving 32GB free. Increasing my ARC limit to 64GB will still leave me 14-16GB free RAM, leaving me room for a few more VM's, while keeping at least 8GB free.
Performance on the 8 x 5TB drives is pretty good already:
Drive Test Results
Sequential Writes:
Sequential Reads:
But when I am doing large file copies (sequential files, above 20GB), I notice a decrease in VM performance. Obviously moving my VM's onto a dedicated SSD pool will invalidate this concern, I still want to add a SLOG and L2ARC.
SLOG: It's my understanding that ZFS flushes the ZIL to the pool every 5 seconds. Even though I have 4 GbE NIC's, rarely will I be saturating all 4 from multiple devices at the same time. Theoretically, I shouldn't have more than 500MB of data in the SLOG before it's flushed to my ZFS pool. Am I interpreting this correctly? Of course I want to over-provision for write tolerance, I really won't need an SSD larger than 8GB. Now the problem with smaller SSD's is that the write performance is always lower. It seems you don't get solid write performance unless you get at least a 400GB SSD, which is definitely overkill for what I need. I don't want to spend more than $400 on a SLOG device, and only spend $400 if there is a performance reason to do so. I'm considering just going with an Intel S3710 200GB ($300), or S3510 480GB ($365), or possibly some PCIe alternative (limited to PCIe 2.0 bandwidth, I believe).
L2ARC: Want to stick under $400 for this one as well. Intel S3510 480GB?
These Intel drives definitely give the endurance, but their performance isn't quite up there with the Samsung 850's, Crucial MX200's, etc.
Questions:
1) VM drives: (two in ZFS mirror) 500GB Crucial MX200 seems like a solid choice at $200/ea. Are there any others I should be considering around that price range?
2) SLOG: What are some good options under $400? Intel DC SSD offerings? Any NVMe offerings I should be considering? How about NVRAM options? Just found a "Curtiss Wright 5453 1GB NVRAM PCIe Card" which seems to be a battery backed NFS accelerator, and can be bought used for under $40. 1GB should be plenty for a SLOG, and I'd assume that'd be fast. Or how about a Fusion-IO 320GB PCI-e drive? I can get that on eBay for under $300.
3) L2ARC: Intel S3710 480GB sufficient? Are there any concerns with size for L2ARC? Better options under $400?
As you can clearly see, I am wiling to put money into this build. It's my only box; and really the only critical use it gets other than backing up files, is storing my fiancee's RAW images as she edits. She works on her pictures directly on the server, and this is what I would consider mission critical.
Thanks!
Chassis: Supermicro SC846E16-R1200B (Replaced PSU's with PWS-920P-SQ)
Motherboard: Supermicro X8DTE-F
CPU's: x2 Intel Xeon L5640 (Hexa-core, 2.26GHz, HT)
RAM: 96GB DDR3 (12 x 8GB PC3-10600, Registered ECC)
HBA: Adaptec 5805 (All drives in JBOD passthrough)
SAS Expander/Backplane: Supermicro BPN-SAS-846EL1 (The SAS Expander is the backplane of the chassis)
OS Drives: x2 Crucial MX100 128GB SSD (ZFS Mirror)
Storage Drives: x8 Toshiba PH3500U-1I72 5TB 7200RPM (ZFS Mirror)
This box is running Proxmox, and is my primary hypervisor, and storage server (basically Debian + KVM + ZoL). Right now I have ~10TB of data on the main ZFS pool, including the 15 or so VM's I have running. My plan is to create a new ZFS Mirror to use for VM storage, and looking to do ~500GB SSD's for that. Right now I'm looking at the MX200 500GB.
Also, I'm looking to add L2ARC and a SLOG to this box. I have my ARC limited to 48GB of RAM, but intent to raise that to 64B before adding a L2ARC. I limited the ARC to half the system memory to see what performance I got, and to leave plenty open for my VM's. Right now the system is consuming ~66GB of RAM (48GB for ARC, 18GB for VM's) leaving 32GB free. Increasing my ARC limit to 64GB will still leave me 14-16GB free RAM, leaving me room for a few more VM's, while keeping at least 8GB free.
Performance on the 8 x 5TB drives is pretty good already:
Drive Test Results
Sequential Writes:
Code:
root@mjolnir:~# time sh -c "dd if=/dev/zero of=/tanks/tank_data_01/dd_test_8g_128k.tmp bs=128k count=62500"
62500+0 records in
62500+0 records out
8192000000 bytes (8.2 GB) copied, 6.23245 s, 1.3 GB/s
real 0m6.236s
user 0m0.024s
sys 0m3.748s
root@mjolnir:~# time sh -c "dd if=/dev/zero of=/tanks/tank_data_01/dd_test_8g_128k.tmp bs=128k count=62500 conv=fdatasync"
62500+0 records in
62500+0 records out
8192000000 bytes (8.2 GB) copied, 42.656 s, 192 MB/s
real 0m43.282s
user 0m0.035s
sys 0m6.443s
root@mjolnir:~# time sh -c "dd if=/dev/zero of=/tanks/tank_data_01/dd_test_256g_128k.tmp bs=128k count=2000000 conv=fdatasync"
2000000+0 records in
2000000+0 records out
262144000000 bytes (262 GB) copied, 448.095 s, 585 MB/s
real 7m28.098s
user 0m1.417s
sys 2m38.794s
root@mjolnir:~# time sh -c "dd if=/dev/zero of=/tanks/tank_data_01/dd_test_256g_1m.tmp bs=1M count=256000 conv=fdatasync"
256000+0 records in
256000+0 records out
268435456000 bytes (268 GB) copied, 461.685 s, 581 MB/s
real 7m41.688s
user 0m0.288s
sys 2m34.050s
Sequential Reads:
Code:
root@mjolnir:~# time sh -c "dd if=/tanks/tank_data_01/dd_test_8g_128k.tmp of=/dev/null bs=128k count=62500"
62500+0 records in
62500+0 records out
8192000000 bytes (8.2 GB) copied, 12.0287 s, 681 MB/s
real 0m12.031s
user 0m0.010s
sys 0m2.722s
root@mjolnir:~# time sh -c "dd if=/tanks/tank_data_01/dd_test_256g_128k.tmp of=/dev/null bs=128k count=2000000"
2000000+0 records in
2000000+0 records out
262144000000 bytes (262 GB) copied, 413.405 s, 634 MB/s
real 6m53.411s
user 0m0.435s
sys 1m22.216s
root@mjolnir:~# time sh -c "dd if=/tanks/tank_data_01/dd_test_256g_1m.tmp of=/dev/null bs=1M count=256000"
256000+0 records in
256000+0 records out
268435456000 bytes (268 GB) copied, 387.86 s, 692 MB/s
real 6m27.866s
user 0m0.130s
sys 1m18.439s
But when I am doing large file copies (sequential files, above 20GB), I notice a decrease in VM performance. Obviously moving my VM's onto a dedicated SSD pool will invalidate this concern, I still want to add a SLOG and L2ARC.
SLOG: It's my understanding that ZFS flushes the ZIL to the pool every 5 seconds. Even though I have 4 GbE NIC's, rarely will I be saturating all 4 from multiple devices at the same time. Theoretically, I shouldn't have more than 500MB of data in the SLOG before it's flushed to my ZFS pool. Am I interpreting this correctly? Of course I want to over-provision for write tolerance, I really won't need an SSD larger than 8GB. Now the problem with smaller SSD's is that the write performance is always lower. It seems you don't get solid write performance unless you get at least a 400GB SSD, which is definitely overkill for what I need. I don't want to spend more than $400 on a SLOG device, and only spend $400 if there is a performance reason to do so. I'm considering just going with an Intel S3710 200GB ($300), or S3510 480GB ($365), or possibly some PCIe alternative (limited to PCIe 2.0 bandwidth, I believe).
L2ARC: Want to stick under $400 for this one as well. Intel S3510 480GB?
These Intel drives definitely give the endurance, but their performance isn't quite up there with the Samsung 850's, Crucial MX200's, etc.
Questions:
1) VM drives: (two in ZFS mirror) 500GB Crucial MX200 seems like a solid choice at $200/ea. Are there any others I should be considering around that price range?
2) SLOG: What are some good options under $400? Intel DC SSD offerings? Any NVMe offerings I should be considering? How about NVRAM options? Just found a "Curtiss Wright 5453 1GB NVRAM PCIe Card" which seems to be a battery backed NFS accelerator, and can be bought used for under $40. 1GB should be plenty for a SLOG, and I'd assume that'd be fast. Or how about a Fusion-IO 320GB PCI-e drive? I can get that on eBay for under $300.
3) L2ARC: Intel S3710 480GB sufficient? Are there any concerns with size for L2ARC? Better options under $400?
As you can clearly see, I am wiling to put money into this build. It's my only box; and really the only critical use it gets other than backing up files, is storing my fiancee's RAW images as she edits. She works on her pictures directly on the server, and this is what I would consider mission critical.
Thanks!
Last edited: