sign_and_send_pubkey: signing failed: agent refused operation
This happens when you don't manually add your ssh key with ssh-add it is some weird new feature in SSH or Ubuntu/Debian that causes this weird problem.
Solution:
ssh-add
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)........