Looking into the documentation, I could not find any mention of the way the session is secured between the host and the HSM.
Did I miss something ?
Could someone indicate which is the protocol used to establish the key and secure the communication ?
Ok. If the information in the developer portal does not provide enough detail, then @sc-hsm maybe could help out?
AFAIK the secure channel is established over Global Platform features and protocols like TR-03110 which were developed for specific government usecases. For general use, the RAMoverHTTP might be worth looking into.
The SmartCard-HSM supports ISO Secure Messaging as defined in TR-03110 and used in e-Passports and e-ID cards. The secure channel establishment is backed by a 3-tier PKI using Card Verifiable Certificates (CVC). For each SmartCard-HSM (or Nitrokey HSM) a Device Certificate is issued by a Device Issuer CA (DICA), that itself is certified by the Scheme Root CA (SRCA). A terminal will first validate the device certificate via the DICA certificate against a trusted SRCA and then perform Chip Authentication V2 as defined in TR-03110. This creates symmetric session keys that initialize Secure Messaging.
Secure Messaging basically encrypts the payload and integrity protects Command- and Response APDUs that are used to communicate with the HSM.
One implementation of the protocol can be found in the openSecureChannel() method of the SmartCard-HSM class in OpenSCDP. OpenSC also has support for this protocol, but that must be specifically enabled during compile.