Hi Martian,
I use pfSense and Untangle into a prod envionment. You're right when you say that pfSense are a security suite and Untangle a UTM system.
I successfully use both, back to back to provide me a solution like a Sonicwall TZ-210 with CGSS enable.
It's working well en ESXi with...