Saying that – passwordless authentication over ssh is already quite good security. Provided no-one gets your private key (and you don’t reuse keys over different machines), IMO you don’t need to worry too much.
Have you actually been logging attempted ssh logins? I would be inclined not to worry too much about the security implications – if you’ve disabled passwords for ssh, people aren’t going to be able to login without a key.
Put it another way – if someone could access my computer via passwordless ssh, I’d be much more worried about my bank being hacked than me!