

Apologies for the delay. On the VPN termination point, you have to set the allowed IP addresses. On the case of a client, a /32 is enough. It means that only this IP would be receiving responses. A client with a different IP address would be able to inly send packets, not to get any back, thus not able to get a TCP session. I think it is enough and rhat no additional FW rule is needed.
On you server:
tcpdump -ni any tcp port <your server port>
You will know if your traffic reaches your server and to what it has to respond to.
It feels like NAT issue to me. Or DNS :D