In this case, Iam assuming you have verified the firewall on either side, and the network conditions are not an issue (eg. no packet loss, no crazy jitter and normal latency).
In some phones such as Grandstream or on your PBX, make sure you do NOT use the "adaptive jitter buffer" or similar settings. If possible disable that or change it to "fixed".
If you do, you may wonder why when you first start a call the user cannot hear you or the voice s........
I used to believe that for Desktops especially that the "ondemand" CPUfrequency changing that kernels included with Ubuntu and Debian based distros have would be sufficient for snappy performance.
However, you can feel the lack of performance on the fastest computer if you have ondemand. A lot of times even under high load 100% of your CPUfrequency in MHz will not be used.
For example a 2.8Ghz CPUmay only run at 1.8MHz or even .9GHz. Now........