Like cam said, depending on what router you are using, its more thank likely due to your NAT (network address translation)setting being set to "strict".
What you have to do is adjust some of the settings on your router, by browsing to it's ip address in your browser (IE, Firefox etc.)
Most routers have an web-based administrative interface that's located at "http://192.168.0.1". (This address does depend on your model. Consult your router user guide for more info.) or alternativly go to
this website which has all the information on each router you could possibly think of, and provides a guide for each on what ports to open, just confirm the make/model of your router and click on the link for it on the website, and follow the steps
Once you have done this, HOPEFULLY it should have cured your connection problems!
Let me know if the problem persists.
Joe