![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Hi, am doing some research here on what options I have for putting together a new storage unit, thinking in terms of rack space, cost, hardware, and 2 nodes for HA, etc. Except for the last one shown, only those with redundant power supplies are chosen. Not sure if this is required if there are 2 nodes already. All units are priced for 32GB RAM per node and 2 x 1.0TB SATA disk for the OS (to be run via motherboard SW RAID 1) along with 3Y on site warranty.
Ranking from most expensive to least expensive, in terms of $/core
6047R-E1R36N 4U 36 bay unit with 2 x E5-2609 v2
http://www.thinkmate.com/system/superstorage-server-6047r-e1r36n
- This is an all in one storage unit.
- 4x RJ45 Gigabit Ethernet LAN ports.
- 3 x PCIe 3 x 16 + 1 x PCIe 3 x 8
$4,947.00 - Total
$1,236.75 - $/core
5018D-MTRF 1U 1 node unit with 1 x E3-1220 v3 3.10GHz
http://www.thinkmate.com/system/superserver-5018d-mtrf
- Need two of these 1U units (total 2 CPUs).
- Integrated 2x RJ45 Gigabit Ethernet LAN ports
- 1 x PCIe 3
- Cheapest 1U unit offering from SuperMicro on this site and it's a Haswell!
$4,620.00 - Total
$577.50 - $/core
Dell R3210 1U 1 node unit with 1 x E5-1410 2.80GHz
- Need two of these 1U units (total 2 CPUs)
- Integrated 2x RJ45 Gigabit Ethernet LAN ports
$3,957.40 - Total
$494.68 - $/core
6027TR-DTRF 2U 2 nodes unit with 2 x E5-2609 v2 per node
http://www.thinkmate.com/system/hdx-xs12-3260
- 2 nodes cluster in a 2U box (total 4 CPUs)
- 2 x RJ45 Gigabit Ethernet LAN ports
- 3 x PCIe 3 x 8 FH + 1 x PCIe 3 x8 LP
$5,523.00 - Total
$345.19 - $/core
6027TR-HTRF 2U 4 nodes unit with 2 x E5-2609 v2 per node
http://www.thinkmate.com/system/superserver-6027tr-htrf
- 4 nodes cluster in a 2U box (total 8 CPUs).
- 2 x RJ45 Gigabit Ethernet LAN ports.
- 1 x PCIe 3.0 x 16 LP
$9,724.00 - Total
$303.88 - $/core
6017TR-TF 1U 2 nodes unit with 2 x E5-2609 v2 per node
http://www.thinkmate.com/system/superserver-6017tr-tf
- 2 nodes cluster in a 1U box (total 4 CPUs).
- 2x RJ45 Gigabit Ethernet LAN ports
- 1x PCI-E 3.0 x16 LP
- No PSU redundancy.
$4,832.00 - Total
$302.00 - $/core
Ideally I want cluster in a box (CiB) solution to save rack space. It seems between SuperMicro and Dell, only SuperMicro offers this type of products. Dell has this new VTRX box but it's only for Windows so it's out of the question.
Observations
1) While these CiB solutions from SuperMicro doesn't have the lowest starting costs, they work out to be the lowest per core. I know the 5018D-MTRF and the Dell R320 have different CPUs but still. The Dell is even using E3 vs a E5 for most of the SuperMicros.
2) With the exception of the all in one storage unit 6047R-E1R36N offering 4 x 1Gbps ports = 4Gbps (teaming), rest are all 2 x 1Gbps ports = 2Gbps (teaming).
3) Not an observation but more of a marketing thing but looks like Yahoo is favoring SuperMicro over others. Interesting story.
http://cdn.thinkmate.com/static/supermicro/superserver/pdf/Yahoo_SMCI_Case_Study.pdf
4) The drive bays that come with the SuperMicro CiB solutions are partitioned for the 2 nodes. So neither of the node is able to access all the drives. This is unlike the DataOn solutions like http://www.dataonstorage.com/cluste...u-12-bay-cluster-in-a-box-storage-system.html.
Questions
1) Would a 2Gbps effective speed (possible to drop to 1Gbps if one of the connections isn't working) be sufficient if these are going to run as storage heads, offering iSCSI and NFS services to a few web, dB, mail servers? I think so because web clients are going to be connecting a much slower speed over the WAN than 1Gbps. However I also notice that 1Gbps Ethernet is slower than the common SAS 3Gbps speed. Hack it's even slower than 1st gen SATA!
2) Noticed during pricing that they offer the Intel® TrueScale Fabric Host Adapter QLE7340 - 40Gbps InfiniBand (1x QSFP) for $327. This is cheaper than some of the 10Gbps Ethernet NIC but not sure of costs for adding an InfiniBand QDR switch, and whether it's even needed.
3) Can we virtualize everything for the 6027TR-DTRF with the following approach:
a) Run one ESXi server on each of the two nodes.
b) For each ESXi server, run a few VMs serving web, mail, and also the storage head.
c) Have a network load balancer to split traffic to either one of the ESXi server for incoming traffic.
d) The dB VM will only run in one of the nodes for simplicity sake.
Why I'm thinking of this?
a) May be it's faster and have lower latency for the web / mail / dB VMs inside each ESXi to communicate to the storage head VM via a vNetwork Distributed Switch?
b) If one of the nodes fail, the other one would still be entirely up and running in good order. If the node fails happen to be the one running the dB VM, we can start it up one the other ESXi node, and configure the other servers to connect to this backup dB (with a different IP).
c) I can start with less start up costs.
d) If 2 x 4 cores per nodes isn't enough, we can use 2 x 6 cores or 2 x 8 cores. It's still going to be cheaper than buying these many cores on stand alone servers.
4) Now the big question is how do we run HA active-active for a file system like ZFS without Nexanta, which uses High Availability? I don't know and have the answer but I have read people porting Packer Maker CRM to ZFS on this blog http://zfs-create.blogspot.se/. So if someone here has done it, and don't mind offering help, please PM me or something.
Ranking from most expensive to least expensive, in terms of $/core
6047R-E1R36N 4U 36 bay unit with 2 x E5-2609 v2
http://www.thinkmate.com/system/superstorage-server-6047r-e1r36n
- This is an all in one storage unit.
- 4x RJ45 Gigabit Ethernet LAN ports.
- 3 x PCIe 3 x 16 + 1 x PCIe 3 x 8
$4,947.00 - Total
$1,236.75 - $/core
5018D-MTRF 1U 1 node unit with 1 x E3-1220 v3 3.10GHz
http://www.thinkmate.com/system/superserver-5018d-mtrf
- Need two of these 1U units (total 2 CPUs).
- Integrated 2x RJ45 Gigabit Ethernet LAN ports
- 1 x PCIe 3
- Cheapest 1U unit offering from SuperMicro on this site and it's a Haswell!
$4,620.00 - Total
$577.50 - $/core
Dell R3210 1U 1 node unit with 1 x E5-1410 2.80GHz
- Need two of these 1U units (total 2 CPUs)
- Integrated 2x RJ45 Gigabit Ethernet LAN ports
$3,957.40 - Total
$494.68 - $/core
6027TR-DTRF 2U 2 nodes unit with 2 x E5-2609 v2 per node
http://www.thinkmate.com/system/hdx-xs12-3260
- 2 nodes cluster in a 2U box (total 4 CPUs)
- 2 x RJ45 Gigabit Ethernet LAN ports
- 3 x PCIe 3 x 8 FH + 1 x PCIe 3 x8 LP
$5,523.00 - Total
$345.19 - $/core
6027TR-HTRF 2U 4 nodes unit with 2 x E5-2609 v2 per node
http://www.thinkmate.com/system/superserver-6027tr-htrf
- 4 nodes cluster in a 2U box (total 8 CPUs).
- 2 x RJ45 Gigabit Ethernet LAN ports.
- 1 x PCIe 3.0 x 16 LP
$9,724.00 - Total
$303.88 - $/core
6017TR-TF 1U 2 nodes unit with 2 x E5-2609 v2 per node
http://www.thinkmate.com/system/superserver-6017tr-tf
- 2 nodes cluster in a 1U box (total 4 CPUs).
- 2x RJ45 Gigabit Ethernet LAN ports
- 1x PCI-E 3.0 x16 LP
- No PSU redundancy.
$4,832.00 - Total
$302.00 - $/core
Ideally I want cluster in a box (CiB) solution to save rack space. It seems between SuperMicro and Dell, only SuperMicro offers this type of products. Dell has this new VTRX box but it's only for Windows so it's out of the question.
Observations
1) While these CiB solutions from SuperMicro doesn't have the lowest starting costs, they work out to be the lowest per core. I know the 5018D-MTRF and the Dell R320 have different CPUs but still. The Dell is even using E3 vs a E5 for most of the SuperMicros.
2) With the exception of the all in one storage unit 6047R-E1R36N offering 4 x 1Gbps ports = 4Gbps (teaming), rest are all 2 x 1Gbps ports = 2Gbps (teaming).
3) Not an observation but more of a marketing thing but looks like Yahoo is favoring SuperMicro over others. Interesting story.
http://cdn.thinkmate.com/static/supermicro/superserver/pdf/Yahoo_SMCI_Case_Study.pdf
4) The drive bays that come with the SuperMicro CiB solutions are partitioned for the 2 nodes. So neither of the node is able to access all the drives. This is unlike the DataOn solutions like http://www.dataonstorage.com/cluste...u-12-bay-cluster-in-a-box-storage-system.html.
Questions
1) Would a 2Gbps effective speed (possible to drop to 1Gbps if one of the connections isn't working) be sufficient if these are going to run as storage heads, offering iSCSI and NFS services to a few web, dB, mail servers? I think so because web clients are going to be connecting a much slower speed over the WAN than 1Gbps. However I also notice that 1Gbps Ethernet is slower than the common SAS 3Gbps speed. Hack it's even slower than 1st gen SATA!
2) Noticed during pricing that they offer the Intel® TrueScale Fabric Host Adapter QLE7340 - 40Gbps InfiniBand (1x QSFP) for $327. This is cheaper than some of the 10Gbps Ethernet NIC but not sure of costs for adding an InfiniBand QDR switch, and whether it's even needed.
3) Can we virtualize everything for the 6027TR-DTRF with the following approach:
a) Run one ESXi server on each of the two nodes.
b) For each ESXi server, run a few VMs serving web, mail, and also the storage head.
c) Have a network load balancer to split traffic to either one of the ESXi server for incoming traffic.
d) The dB VM will only run in one of the nodes for simplicity sake.
Why I'm thinking of this?
a) May be it's faster and have lower latency for the web / mail / dB VMs inside each ESXi to communicate to the storage head VM via a vNetwork Distributed Switch?
b) If one of the nodes fail, the other one would still be entirely up and running in good order. If the node fails happen to be the one running the dB VM, we can start it up one the other ESXi node, and configure the other servers to connect to this backup dB (with a different IP).
c) I can start with less start up costs.
d) If 2 x 4 cores per nodes isn't enough, we can use 2 x 6 cores or 2 x 8 cores. It's still going to be cheaper than buying these many cores on stand alone servers.
4) Now the big question is how do we run HA active-active for a file system like ZFS without Nexanta, which uses High Availability? I don't know and have the answer but I have read people porting Packer Maker CRM to ZFS on this blog http://zfs-create.blogspot.se/. So if someone here has done it, and don't mind offering help, please PM me or something.
Last edited: