Run ssh-agent and ssh-add on login via SSH ¶

You can try adding this:

eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

This way the ssh-agent does not start a new shell, it just launches itself in the background and spits out the shell commands to set the appropriate environment variables.

As said in the comment, maybe you do not want to run the agent at all on the remote host, but rather on the box you are working from, and use:

ssh -A remote-host

to forward the services of your local ssh agent to the remote-host.

For security reasons you should only use agent forwarding with hosts run by trustworthy people, but it is better than running a complete agent remotely any time.