• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

CEPH vs TrueNAS for storage setup?

C

cyclone3d

Fully [H]
2FA
Joined
Aug 16, 2004
Messages
18,234
I'm working on setting up a small Proxmox cluster. Hardware is as follows:
3x HP micro PCs with:
i5-8500
512GB nvme
32GB RAM
1Gb NIC (Will be using USB 1GB NICs for needed additional NICs)

These systems have a second nvme slot but with the SATA bay / cooler install the second nvme slot is unable to be used. I can mod this if needed, will just need to space the SATA bay up slightly for clearance.

This is what we ant to run for the VMs:
1. Windows Server 2022 Std for the DCs.
2. OPNsense for firewall / security
3. FreePBX for the phones.

Looking at the documentation for CEPH, it looks like it wants a minimum of 10Gb for the connection as well as a stupid amount of RAM and CPU cores.
This seems completely unnecessary to me for such a small setup, but what do I know?

I've also seen reports that CEPH can wear SSDs very quickly and we do not want that.

Would running TrueNAS be better for the storage this type of setup?

We have a standalone bare metal Windows 2019 server for the file share with 20Gb connection to the switch and that will not be going away but will be upgraded to 2022.
 
Last edited:
Maybe I'm suffering from heat stroke or something (been working in 100F all day for two days)...

Buut, I'm trying to figure out what the question is regarding truenas and where it fits in the question. I am running truenas on two machines. However, they are both running on top of proxmox. I didn't even try using the hypervisor in truenas if that was what you were considering.
 
So Proxmox has CEFS natively / installable by default.

I am wanting to use the hosts as the storage for the VMs as well and it was suggested to me to use CEFS but after reading up on it, it seemed like TrueNAS might be better for my use case.

Maybe my google-fu was failing me but I was not finding any real good info on performance or requirements for good performance.

I'm used to running VMWare with real SAN hardware so just trying to figure out what is a good choice and if it will do what I need.

Basically what I want is to be able to use the second nvme slot or maybe even the SATA ports and have the 3 hosts have a SAN setup as well / redundant and be able to tell the host to use the local storage / local SAN node for the VM it is hosting instead of looking to one of the other hosts for the data storage.

I would think to would do that by default but not sure.
 
As TrueNAS and Proxmox use the same Open-ZFS, the simpelst, fastest and most resource efficient method would be using ZFS directly on Proxmox without a storage VM. With only 32GB RAM this seems the only practical method as a TrueNAS storage VM should have 8-16GB alone. Only negative aspect is that the Proxmox GUI offers only minimal ZFS management options what means that you either must use zfs/zpool commands at console or add one of the ZFS web-gui options to extend the Proxmox Gui functionality with missing ZFS options (Replication, Snap, Pool, Filesystem, Disk, Volume management, Clones etc) like Cockpit, Poolsman or my upcoming napp-it cs port from Solaris that even offers replication between OS platforms in the GUI.
 
Last edited:
applegrcoug said:
Maybe I'm suffering from heat stroke or something (been working in 100F all day for two days)...

Buut, I'm trying to figure out what the question is regarding truenas and where it fits in the question. I am running truenas on two machines. However, they are both running on top of proxmox. I didn't even try using the hypervisor in truenas if that was what you were considering.
Click to expand...
This would be my advice also. Proxmox as the host and run truenas and whatever else within.
 
Are you trying to run a business/something important off of this? Do you have any performance or reliability needs?

No matter what, shared storage over 1Gbps seems like a really bad idea.
 
If this is a home lab are you looking to doing any Clustering? If so Use Proxmox and built in CEF. It should make things simpler. If no clustering I would recommend a separate dedicated NAS with at least 2.5gbe or 10gbe and then 1gbe or 2.5gbe nodes using an NFS mount for the 3 nodes. My 2 cents. If this commercial consult an expert :)
 
The we in the:
We have a standalone bare metal Windows 2019 server for the file share with 20Gb connection to the switch and that will not be going away but will be upgraded to 2022.

Sentence make it look as if it is for a workplace and that file sharing part will be on a different machine with very fast ethernet, if the storage being talked about is just for the local (and small) amount of data used by FreePBX, OPNsense, couple of domain controller windows servers session, probably that the 10gb requirement and giant ram does not apply.
 
LukeTbk said:
The we in the:
We have a standalone bare metal Windows 2019 server for the file share with 20Gb connection to the switch and that will not be going away but will be upgraded to 2022.

Sentence make it look as if it is for a workplace and that file sharing part will be on a different machine with very fast ethernet, if the storage being talked about is just for the local (and small) amount of data used by FreePBX, OPNsense, couple of domain controller windows servers session, probably that the 10gb requirement and giant ram does not apply.
Click to expand...
Good catch but Windows OS not good IMO for storage. SMB iSCSI and 20gbit sounds like a bonded 2 10GB SFP+ connections. I would rather setup and run ProxMox on the 2019 box and move the windows OS to a VM. Pass Storage directly to a FreeNas box(via HBA in IT mode) and share out NFS DataStores to your cluster via NFS. Maybe use CEF for OS Boot Drives only and store those their(nvme drives) and Storage+Data drives on the FreeNas VM. I mean without knowing all the other details its hard to know what will work and what wont.
 
Windows Server can be a quite good OS for storage when you skip the normal software raid mode and use Storage Spaces. Storage Spaces can be very fast if you format them with 64-512K instead the default 4k and avoid raid-5 redundancy for Spaces. With .vmdk based virtual disks you can even build SMB based clusters. You can use shares over SMB direct/RDMA (requires Windows server) that gives a performance near to local NVMe with an ultra low CPU load. No other solution comes near with 40-100 G nics. And the best, ZFS on Windows has reached a state where I would say we are near to usable.

When porting my ZFS web-gui to Windows i found that Windows can be a very good NAS OS if you are willing to understand the Storage Spaces concept and learn Powershell (or use a web-gui like my napp-it where I added Storage Space management to ZFS management)
 
Last edited:
_Gea said:
Windows Server can be a quite good OS for storage when you skip the normal software raid mode and use Storage Spaces. Storage Spaces can be very fast if you format them with 64-512K instead the default 4k and avoid raid-5 redundancy for Spaces. With .vmdk based virtual disks you can even build SMB based clusters. You can use shares over SMB direct/RDMA (requires Windows server) that gives a performance near to local NVMe with an ultra low CPU load. No other solution comes near with 40-100 G nics. And the best, ZFS on Windows has reached a state where I would say we are near to usable.

When porting my ZFS web-gui to Windows i found that Windows can be a very good NAS OS if you are willing to understand the Storage Spaces concept and learn Powershell (or use a web-gui like my napp-it where I added Storage Space management to ZFS management)
Click to expand...
I'm really curious... do you have data or anything you can share showing SMB on windows being nearly equal to local nVME? I'm having a very hard time believing that, especially where it matters (latency).
 
I only skimmed it but that doesn't seem to support your assertions and I didn't see anything about latency there. Looks like at least a 10% throughput loss, and I'm sure latency is way worse than local. I'll read it more in-depth tomorrow though.
 
It remains a network traffic but it makes a huge difference if a nic can write directly to RAM of the remote server via RDMA instead involving the OS, RAM and CPU.

Especially low latency is one of the key advantages of RDMA transfers over ip transfers
https://community.fs.com/article/roce-vs-infiniband-vs-tcp-ip.html

RDMA is not a Windows thing but SMB + RDMA (SMB direct) outside Windows ex via SAMBA is not as ready for general use.
 
Last edited:
LukeTbk said:
The we in the:
We have a standalone bare metal Windows 2019 server for the file share with 20Gb connection to the switch and that will not be going away but will be upgraded to 2022.

Sentence make it look as if it is for a workplace and that file sharing part will be on a different machine with very fast ethernet, if the storage being talked about is just for the local (and small) amount of data used by FreePBX, OPNsense, couple of domain controller windows servers session, probably that the 10gb requirement and giant ram does not apply.
Click to expand...
Yes, the file share is on the bare metal box and the Proxmox boxes will only run FreePBX, OPNsense and a couple Windows DCs.

I want the VMs to use the storage on the Proxmox boxes and be set up for HA.
 
Note: changed CEFS to CEPH as it should have been in th title and OP.

Looking around some more, CEPH should do the replication of storage between the nodes but also seeing that it may not actually work properly?
 
I guess I just need to try this out.

Trying to look up stuff is almost as bad as trying to follow MS or Cisco documentation and everybody seems to have completely different opinions and experiences of what works or not.

Not even sure if HA with Proxmox will work without a physical fencing device per their documentation which may have not been updated since 2019.
 
We already have a working setup with older hardware.

Everything will be tested thoroughly before being put into place.

I've just never used Proxmox before. I'm used to esxi and higher end SAN hardware such as NetApp and the older HP SANs.
 
cyclone3d said:
We already have a working setup with older hardware.

Everything will be tested thoroughly before being put into place.

I've just never used Proxmox before. I'm used to esxi and higher end SAN hardware such as NetApp and the older HP SANs.
Click to expand...
I’m learning proxmox right now and it’s actually pretty user friendly. Most questions were answered without snarky comments which was something I was not accustomed to after using Linux for the last few years.
 
cyclone3d said:
We already have a working setup with older hardware.

Everything will be tested thoroughly before being put into place.

I've just never used Proxmox before. I'm used to esxi and higher end SAN hardware such as NetApp and the older HP SANs.
Click to expand...
I still noted dedicated box and separate cluster. What requirements do you need from this storage?
 
I just went through building a proxmox cluster with SSD, HDD, 10gb lan on ceph.

My understanding is that ceph is block storage vs file storage as you would get with a NAS. Meaning it works like you are writing to a hard drive locally on your box by read/writing sectors vs locking entire files. In the end that works better for me as I do alot of databse work. Ceph transmits the sector over the ethernet to the other devices before returning to the client. That way things are replicated in real time for disaster recovery. I initially put my ceph on the main 1gb network with a regular switch, then I bought 10gb ethernet dual ports to replicate on its on back end by having each machine point to the other nodes in the cluster on its own back end. But this configuration only works with 3 machines. If you start putting more nodes in, then you'd need a 10gb switch which are spendy. Ceph or anything will wear out hard drives fast, but make sure you buy enterprise rated SSDs. Do not buy the $50 Samsung 970 evo 2tb SSD... Those are not enterprise rated. I've been running my cluster for 6 months (low utilization) and my proxmox is reporting 0% wear out on my SSDs.
 
If you're considering running TrueNAS as a SAN make sure you do a deep dive into properly configuring block level storage. I set this up for an ESXI home lab last year and the system requirements as far as RAM, high endurance ARC drives, necessary storage capacity to maintain performance, etc. are pretty steep. This is a good overview https://www.truenas.com/community/resources/the-path-to-success-for-block-storage.197/

CEPH ( or StarWind vSAN, I'm not positive of your plans between the dedicated server and cluster. ) sound like your best fits now that Broadcom took esxi \ VMware out of the equation for most shops.

The 10gb connection requirement is pretty common, the Lenovo mini's are popular choices for home labs because several models can take an internal NIC upgrade. The servers \ storage will run over 1gb but it's a pretty big bottleneck.
 
I can give you my personal experience running TrueNAS as an NFS target for a proxmox box with 5 or so VMs.

I would *NOT* use 1gbps networking. My proxmox box and truenas box talk to each other on a dedicated 10Gbps direct link using older Intel 550 10 gig NICs. Front end network on Proxmox is 1Gbps.

My Truenas uses 4 x Samsung 870 EVO SATA SSDs in a whatever the ZFS version of RAID 10 is, I forget what it's called. 32GB of RAM, on a quad core 7500T Intel. Zero issues. The performance damn near seems like local storage.

My proxmox box is a Intel 8600T also with 32GB ram. Again, the VMs are snappy, responsive, no issues. My unifi controller and OpenVPN cloud connexa VPN "connector" reside on that proxmox box.

Would I run 100 VMs on this set up? No, based on my CPU, network and RAM usage on my Proxmox and TrueNAS boxes, I would say easily a 12+ VPNs would run just fine on this setup. Maybe more ram on the Proxmox box in that scenario, that's it.
 
Ceph is a bigger deal. Not that you can't make a mutli-node ZFS thingy, but with Ceph, you're taking a minimum of 3 nodes and likely more.

Ceph is more like a VSAN or storage seen in hyperconverged scenarios. Talking about "erasure coding" instead of RAID, etc.

Ceph storage is viewed as pools for objects to be spread across nodes for redundancy, rather than mere striping concepts. It's abstract enough to not care about differences between nodes or storage areas provided to it. So, you can build pretty crazy stuff using Ceph.

Not necessarily a tuber endorsement, but his might help


View: https://www.youtube.com/watch?v=jJrnJ9rj6fs
 
cjcox said:
Ceph is a bigger deal. Not that you can't make a mutli-node ZFS thingy, but with Ceph, you're taking a minimum of 3 nodes and likely more.

Ceph is more like a VSAN or storage seen in hyperconverged scenarios. Talking about "erasure coding" instead of RAID, etc.

Ceph storage is viewed as pools for objects to be spread across nodes for redundancy, rather than mere striping concepts. It's abstract enough to not care about differences between nodes or storage areas provided to it. So, you can build pretty crazy stuff using Ceph.

Not necessarily a tuber endorsement, but his might help


View: https://www.youtube.com/watch?v=jJrnJ9rj6fs
Click to expand...

I was really interested in this until I saw the setup complexity. Once someone packages this like Nutanix, this will take over the world.
 
OFaceSIG said:
I can give you my personal experience running TrueNAS as an NFS target for a proxmox box with 5 or so VMs.

I would *NOT* use 1gbps networking. My proxmox box and truenas box talk to each other on a dedicated 10Gbps direct link using older Intel 550 10 gig NICs. Front end network on Proxmox is 1Gbps.

My Truenas uses 4 x Samsung 870 EVO SATA SSDs in a whatever the ZFS version of RAID 10 is, I forget what it's called. 32GB of RAM, on a quad core 7500T Intel. Zero issues. The performance damn near seems like local storage.

My proxmox box is a Intel 8600T also with 32GB ram. Again, the VMs are snappy, responsive, no issues. My unifi controller and OpenVPN cloud connexa VPN "connector" reside on that proxmox box.

Would I run 100 VMs on this set up? No, based on my CPU, network and RAM usage on my Proxmox and TrueNAS boxes, I would say easily a 12+ VPNs would run just fine on this setup. Maybe more ram on the Proxmox box in that scenario, that's it.
Click to expand...


i am temporarily using a 1Gb link right now for my TrueNAS core, I do NFS share to my desktop (mint linux) and run the VMs via vmware workstation, using the NFS shares on my truenas for the disks, via my BrocadeICX and it is like local storage. You could run 10VM's over 1Gb, it all depends what your I/O load is..

How often are your VM's taxing more than 120MB/s constantly at any given time....

Yes, 10Gb is better and things will be faster, but you could run it on 1Gb if it is all you had


Now, OP, why are you planning to use old "desktop" equipment to run critical infra like AD and FreePBX? If you have old servers already, why not use those?

Do you have a backup plan in place?
 
Last edited:
MrGuvernment said:
i am temporarily using a 1Gb link right now for my TrueNAS core, I do NFS share to my desktop (mint linux) and run the VMs via vmware workstation, using the NFS shares on my truenas for the disks, via my BrocadeICX and it is like local storage. You could run 10VM's over 1Gb, it all depends what your I/O load is..

How often are your VM's taxing more than 120MB/s constantly at any given time....

Yes, 10Gb is better and things will be faster, but you could run it on 1Gb if it is all you had
Click to expand...
Oh it will work, for sure. However when building VMs, doing patching, and the like, I regularly see well in excess of 1Gbps being used.
 
You must log in or register to reply here.
Back
Top