simonw over at debian-administration.org seems to have a problem with SSH Auth on busy servers. I've had a few hosts out in the wild with ssh on port 22 and was suffering similiar problems.
ssh_exchange_identification: Connection closed by remote host
My solution was to have these options in sshd_config:
These 2 options solved the problems for me. On a busy host you might want to look at the sshd_config manpage.
Another helpful options is: ControlMaster from the ssh_config manpade
What this does it that for a given hostname and username combination it reuses the connection so that on subsequent logins you can login without entering the passwort and in most cases almost immediately as sshd doesn't need to run another process since it reuses the one already existing.Server!/Horror