In this case, Iam assuming you have verified the firewall on either side, and the network conditions are not an issue (eg. no packet loss, no crazy jitter and normal latency).
In some phones such as Grandstream or on your PBX, make sure you do NOT use the "adaptive jitter buffer" or similar settings. If possible disable that or change it to "fixed".
If you do, you may wonder why when you first start a call the user cannot hear you or the voice s........