I have been using Hermes for a month and a half now and it doesn't cease to impress me.
At home I have it hosted on my main Linux PC while the LLM lives on a DGX Spark cluster. Either running 2 or 4 nodes at the moment depending on what I'm doing. It will be permanently 4 nodes down the road.
Current LLM is Deepseek v4 Flash 0731 running FP8 KV cache with a 1M context length.
At work I have a similar setup in our test lab, a cluster of 2 DGX Sparks running DSv4. I have just had Hermes migrate itself from a VM to a bare metal host that I originally had as a worker node.
The worker node used to have one RTX 5090, as of last week it has two of them. I had Hermes sort out the fix to enable P2P between the cards and then evaluate what backend was the faster for serving Qwen 3.8 27B FP8 with TP=2. SGLang turned out to be a lot faster than vLLM that originally was used.
The 5090s are used for delegations, need to see if I also can make them work as a fallback LLM next in case I take down the Sparks for maintenance.
Fun times ahead and so far I have only scratched the surface of what these can do.
At home I have it hosted on my main Linux PC while the LLM lives on a DGX Spark cluster. Either running 2 or 4 nodes at the moment depending on what I'm doing. It will be permanently 4 nodes down the road.
Current LLM is Deepseek v4 Flash 0731 running FP8 KV cache with a 1M context length.
At work I have a similar setup in our test lab, a cluster of 2 DGX Sparks running DSv4. I have just had Hermes migrate itself from a VM to a bare metal host that I originally had as a worker node.
The worker node used to have one RTX 5090, as of last week it has two of them. I had Hermes sort out the fix to enable P2P between the cards and then evaluate what backend was the faster for serving Qwen 3.8 27B FP8 with TP=2. SGLang turned out to be a lot faster than vLLM that originally was used.
The 5090s are used for delegations, need to see if I also can make them work as a fallback LLM next in case I take down the Sparks for maintenance.
Fun times ahead and so far I have only scratched the surface of what these can do.