I am having some odd issues with my remote desktop connection. I have a vista desktop running with remote desktop enabled. It works fine locally and if I connect to my public IP from home. I have triple checked the port fowarding and I know its right. However whenever I try to connect from anywhere else (IE Work, my parents place, ect) it fails. I can access my router from anywhere so I know the connection is working. It has nothing to do with the versions of remote desktop as I use the same version in all places (original windows XP client). For the life of me I can't figure out why the connection won't go. Any help would be appreciated.

Side note: I am using a non-standard port # for security reasons so I don't think its getting block by my isp.