For interactive use it is easy to enter the user PIN when required. However, what is the best practice to „cache“ a PIN in your application?
Ideally I would enter once the PIN during startup of the application and it is then protected by the kernel keyring instead of having an environment variable.
How is this usually done? Or does it get entered remotely from a script running on another server?
That depends on your security requirements and ranges from hard-coded PIN to n-of-m Public Key Authentication .
Usually you authenticate towards the HSM and the authentication state is maintained until you logout, reset or remove the device. Obviously that leaves the opportunity to hijack the open connection and ride on the already authenticated communication channel. That is usually mitigated by opening a secure messaging channel with the HSM before (or after) you perform user authentication. That secure messaging channel is then bound to the authentication state, so that an adversary can not intercept the authenticated session.
An application that is using this approach does not need to cache the PIN, as re-authentication happens with each command send to the HSM. You only need to re-enter the PIN if the session terminates for whatever reason. Secure messaging keys are held in memory, so they should disappear when the application terminates.
Secure messaging is supported in the Java middleware (OCF, JCE-Provider) or the PKI-as-a-Service system.
Instead of using a PIN you can also use Public Key Authentication, which is a challenge-response protocol that uses an ECDSA key on a HSM to authenticate towards the other HSM. That can be combined with an n-of-m threshold scheme, so that multiple key custodians need to authenticate before the target HSM is logged-in.