Alternative solution to ssh-agent is p11-kit - see Server stuck in POST when Nitrokey HSM is inserted
I am not sure it is possible to configure properly the ssh-agent for the first use case, specifically whether it was foreseen in its design. @nitroalex: ideas?