![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
I'm amazed by the thoughts that go through my head when I'm doing my normal daily tasks (shower, brushing teeth, laying in bed, etc). Lately I've been thinking about possible ways to achieve redundant storage plus compute resources (cpu/ram, for VM's), and I think I have it figured out. Basically I want to share my thoughts and get some opinions, and looking for folks to poke holes in my possible plan.
Hardware:
x3 Dell C2100s:
*x2 Intel Xeon L5630
*24GB RAM
*Redundant PSU's
*x2 OS drives
*x8 4TB 7200 RPM drives
*x4 120GB SSD's
*Quad Gigabit NICs
x1 Dell PowerConnect 2824 Switch
x1 mini PC (think Intel NUC, or some other low powered small form factor PC with x86 CPU)
Software stack (high level):
*Ceph + Calamari
*QEMU/KVM
*OpenNebula
What's the plan? Well, I've been looking for a way to use a few servers as both compute and storage nodes, while maintaining plentiful storage I/O for VM's, and also redundancy of storage with a replication of at least 2. With the hardware listed above, I would (ideally, but not initially) load each Dell C2100 with 8 4TB 7200 RPM drives (HGST Deskstar NAS, possibly), 4 120GB SSD's, and 2 more 120GB SSD drives (internal to the chassis).
The 2 internal 120GB SSD drives would be in a RAID 1 configuration to use for the OS, which for me would likely be CentOS 7. I would configure a Ceph cluster using each Dell C2100 as a monitor node, and using the 8 4TB drives as OSD's. The remaining 4 120GB SSD's (Intel Pro 2500?) would be used for journal of the OSD (2 OSD's per journal drive).
Alternatively, I've been exploring the idea of a separate SSD OSD pool with 512GB SSD's (Crucial MX100s?), with 2 per host and configure tier caching for additional performance. With this, I'd drop to only 2 journal drives per host, which is still manageable.
With the above, and assuming all 3 servers had 8 4TB drives populated, and a replication factor of 2, I would have a total of ~48TB of usable storage. Performance on this alone would be pretty nice, but the optional SSD caching pool would just be the icing on the cake (while adding another ~1.5TB of available storage). From there I would install Calamari on the mini PC, as a way to view and monitor my Ceph environment, but I'm not sure if Calamari can view/manage a tiered setup correctly.
Network-wise, I would have 2 LACP groups, each with 2 GigE links. One group would be purely for Ceph traffic, and the other group would be used for LAN/Management traffic.
***If a dual GigE LACP doesn't keep up with the Ceph traffic, I would consider NetGear's new SMB 10GigE switches and add a single 10GigE NIC to each server for Ceph traffic.
Next up would be to install OpenNebula onto the mini PC, alongside Calamari. I would also install Open vSwitch and KVM/QEMU on my Dell C2100's. Using OpenNebula, add the servers to my OpenNebula frontend server, creating a OpenNebula cluster. I would configure OpenNebula to create a new Ceph pool for VM storage. From this point I have a fully functioning OpenNebula environment with VM's living on Ceph storage. Great.
The last thing is getting NAS-like storage (i.e. NFS and CIFS), and deciding the best way to do this. There are ways to configure NFS over RBD, so I could create a new Ceph pool and share that out as NFS storage which would definitely do what I need. Most of my servers are Linux, so compatibility is there. I do have two Windows PC's (likely to be replaced with Mac's in the future), and possibly a couple Windows VM's (I'm debating on moving my DNS and DHCP off to my EdgeRouter Lite), so that would only leave Windows servers for Active Directory, which may be an opportunity to explore OpenLDAP and Samba3 as alternatives. Ergo, the need for CIFS may not exist.
The other option I have been considering is running FreeNAS or OpenMediaVault in a VM on OpenNebula, creating disks on rbd, and letting OpenMediaVault do a software RAID, or FreeNAS do ZFS from here. However, that's doing raid ontop of a distributed file system, so I wouldn't think that's ideal.
Alternatively, I may consider oVirt instead of OpenNebula, as it has nearly the same infrastructure requirements. Still undecided, but leaning toward OpenNebula.
As a bonus, I would also like to build a VDI environment at home, based on Linux and SPICE. I would like to have a few thin clients throughout the house, all connecting to a pool of VM's running in my cluster, which would be ideal for kids (in the future, since I have no kids now), in the kitchen, garage, etc. Where ever I would like to have access to a computer, without buying a full computer to deal with and manage.
There is definitely a learning curve, but that is fine. Right now if my PSU in my storage box died, I would be completely down as all my VM's live there via NFS. This is the primarily scenario that I want to avoid. This new possible setup would also allow for unlimited expanding opportunities by simply adding a new node whenever I need additional storage, or compute resources. New nodes can utilize large drives as they come available, and also upgrade to newer hardware effectively replacing old/out-dated hardware, etc.
So with this high level overview, does anyone have anything they could add? Anyone see any problems with a plan like this? All feedback is appreciated.
Hardware:
x3 Dell C2100s:
*x2 Intel Xeon L5630
*24GB RAM
*Redundant PSU's
*x2 OS drives
*x8 4TB 7200 RPM drives
*x4 120GB SSD's
*Quad Gigabit NICs
x1 Dell PowerConnect 2824 Switch
x1 mini PC (think Intel NUC, or some other low powered small form factor PC with x86 CPU)
Software stack (high level):
*Ceph + Calamari
*QEMU/KVM
*OpenNebula
What's the plan? Well, I've been looking for a way to use a few servers as both compute and storage nodes, while maintaining plentiful storage I/O for VM's, and also redundancy of storage with a replication of at least 2. With the hardware listed above, I would (ideally, but not initially) load each Dell C2100 with 8 4TB 7200 RPM drives (HGST Deskstar NAS, possibly), 4 120GB SSD's, and 2 more 120GB SSD drives (internal to the chassis).
The 2 internal 120GB SSD drives would be in a RAID 1 configuration to use for the OS, which for me would likely be CentOS 7. I would configure a Ceph cluster using each Dell C2100 as a monitor node, and using the 8 4TB drives as OSD's. The remaining 4 120GB SSD's (Intel Pro 2500?) would be used for journal of the OSD (2 OSD's per journal drive).
Alternatively, I've been exploring the idea of a separate SSD OSD pool with 512GB SSD's (Crucial MX100s?), with 2 per host and configure tier caching for additional performance. With this, I'd drop to only 2 journal drives per host, which is still manageable.
With the above, and assuming all 3 servers had 8 4TB drives populated, and a replication factor of 2, I would have a total of ~48TB of usable storage. Performance on this alone would be pretty nice, but the optional SSD caching pool would just be the icing on the cake (while adding another ~1.5TB of available storage). From there I would install Calamari on the mini PC, as a way to view and monitor my Ceph environment, but I'm not sure if Calamari can view/manage a tiered setup correctly.
Network-wise, I would have 2 LACP groups, each with 2 GigE links. One group would be purely for Ceph traffic, and the other group would be used for LAN/Management traffic.
***If a dual GigE LACP doesn't keep up with the Ceph traffic, I would consider NetGear's new SMB 10GigE switches and add a single 10GigE NIC to each server for Ceph traffic.
Next up would be to install OpenNebula onto the mini PC, alongside Calamari. I would also install Open vSwitch and KVM/QEMU on my Dell C2100's. Using OpenNebula, add the servers to my OpenNebula frontend server, creating a OpenNebula cluster. I would configure OpenNebula to create a new Ceph pool for VM storage. From this point I have a fully functioning OpenNebula environment with VM's living on Ceph storage. Great.
The last thing is getting NAS-like storage (i.e. NFS and CIFS), and deciding the best way to do this. There are ways to configure NFS over RBD, so I could create a new Ceph pool and share that out as NFS storage which would definitely do what I need. Most of my servers are Linux, so compatibility is there. I do have two Windows PC's (likely to be replaced with Mac's in the future), and possibly a couple Windows VM's (I'm debating on moving my DNS and DHCP off to my EdgeRouter Lite), so that would only leave Windows servers for Active Directory, which may be an opportunity to explore OpenLDAP and Samba3 as alternatives. Ergo, the need for CIFS may not exist.
The other option I have been considering is running FreeNAS or OpenMediaVault in a VM on OpenNebula, creating disks on rbd, and letting OpenMediaVault do a software RAID, or FreeNAS do ZFS from here. However, that's doing raid ontop of a distributed file system, so I wouldn't think that's ideal.
Alternatively, I may consider oVirt instead of OpenNebula, as it has nearly the same infrastructure requirements. Still undecided, but leaning toward OpenNebula.
As a bonus, I would also like to build a VDI environment at home, based on Linux and SPICE. I would like to have a few thin clients throughout the house, all connecting to a pool of VM's running in my cluster, which would be ideal for kids (in the future, since I have no kids now), in the kitchen, garage, etc. Where ever I would like to have access to a computer, without buying a full computer to deal with and manage.
There is definitely a learning curve, but that is fine. Right now if my PSU in my storage box died, I would be completely down as all my VM's live there via NFS. This is the primarily scenario that I want to avoid. This new possible setup would also allow for unlimited expanding opportunities by simply adding a new node whenever I need additional storage, or compute resources. New nodes can utilize large drives as they come available, and also upgrade to newer hardware effectively replacing old/out-dated hardware, etc.
So with this high level overview, does anyone have anything they could add? Anyone see any problems with a plan like this? All feedback is appreciated.
