simple signal/vonage VOIP = major issues, firewall related?

so, about 6 months ago we got away from our super old 3COM PBX and phone system (multicast hell anyone?) and went with simple signal VOIP solution. we loved that it used polycom phones, hosted in cloud (we have tons of remote offices), and wasn't outrageously priced. that is about where the love affair ended. we have been plagued with outages, phone issues, no voice over phone calls, TERRIBLE customer service, random shit like phones becoming unregistered and people at simple signal (now vonage) have no explanation, voicemails from other numbers showing up in other people's phones ... i could go on and on.

lately we have been having a very consistent issue where phones will send and receive calls to each other but no voice or audio comes through. HOWEVER, if you call from a cell phone, the phone works. all other types of calls do not have any audio/voice.

vonage/simple signal of course blames our watchguard firewalls but we have been all through our policies with everyone and nothing is blocking the traffic. and the thing is, sometimes everything works great, most times it doesn't.

nailing down simple signal/vonage is like nailing jello to the wall. their support is the worst. they NEVER can explain their issues and are very quick to point the finger elsewhere. i have had 4 different very experienced network professionals from all walks of life examine this shit closely and they all come to the same conclusion: SIMPLE SIGNAL/VONAGE service is just flat out unreliable.

so, before our legal team gets involved and we start to get out of our contract (trust me, we have piles and piles of evidence, logged crap, calls, tickets, and other stuff to break contract purely on SLA), i want to reach out and ask the people here ... DOES ANYONE HAVE ANY SUGGESTIONS OR IDEAS AS TO WHAT IS HAPPENING THAT COULD POSSIBLE BE OUR FAULT AND NOT VONAGE/SIMPLE SIGNAL? an no, its not a SIP-ALG issue.

Sounds like a routing issue to me.

Could also be packets being dropped... no necessarily by the firewall.

For quite a while - months- after switching over to a Cisco VOIP phone setup we would regularly (every single day) have issues of no sound one or both ways, braking up, distortion, dropped calls, etc.

Our ISP had to fix some stuff.. they actually ran some new lines, the people in charge of the phone system had to change some QOS settings until they figured out what would work good. They also had to mess with firewall settings.

The best way to track it down what is causing the problem is to set up a dedicated wireshark computer and plug it into a mirror port and then capture all the data going through when a call is having problems.

It was a huge, huge, huge pain to get stuff working properly at my site whereas the other sites they switched over at the same time didn't have any of these problems.

In my experience SIP ALG breaks stuff, ie the phones handle NAT just fine by themselves. That said, I've seen some issues using pf (similar to yours) whereas iptables has been doing fine.

If I were you, put a few phones on a separate VLAN. Run them though vanilla OpenWRT/Miktrotik box (anything will do fine for SIP/VoIP) or something that uses the same rulset and or course use QoS on the WAN connection so you don't end up with bandwidth starvation. Does that work fine? Also make sure that these phones have a recent fw running.
//Danne

SIP ALG is disabled is what i meant by my comment.

we have done a tone of wireshark captures and nothing we see leads us to believe the firewall is blocking any of the traffic. we even tried adding very liberal open policies and nothing helped.

we run cisco 3560 switches and that is my next step but not sure where to begin. i have never done any QoS on cisco switches and a lot of stuff i read online pertains to cisco VOIP systems.

sounds like a NAT issue to me... the SIPs coming through but the RTP is dropping, doing VoIP over NAT is not easy and requires proper configuration on both ends...

i would try re-enabling the sip helper to see if it will fix some of the configuration issues...

i mean if it's configured right you shouldn't need it (you wouldn't want it), but there is a lot that could be wrong...

Hence is why I recommended using something that is known to work, finding a compatible OpenWRT box and/or Mikrotik box isn't hard. I have several offices running this setup with Snom and/or Gigaset phones just fine. pf (in FreeBSD) which is the same as in pfsense and opnsense didn't work out very well in that regard.

You have a firewall issue, it is not your provider.

I will preface this with I work for Nextiva, a competitor in the same hosted VOIP space.

The issue you describe where calls have no audio when originating from landline or other VOIP handsets but have audio when originating from cell phones sounds like an upstream inbound carrier issue. That would be a very rare issue and would need packet captures to diagnose properly.

We have seen issues with the Watchguard firewalls and recommend that our customers create rules that allow all traffic from our data center public subnets. If you can I would try another firewall preferably a Cisco ASA or Dell Sonicwall. Also try switching your phones to TCP instead of UDP if Vonage allows that.