SSH on non-standard port?

Check your firewall. It's probably blocking that port.

EDIT: If you're using Ubuntu, your Firewall is probably UFW. Then do this:

sudo ufw status

To check if UFW is running, if it is then open the port like this:

sudo ufw allow 2222/tcp

Obviously, replace 2222 with your SSH port.