Refactor SSH key loading so we don't duplicate keys