Nitrokey HSM - use of RSA-PKCS-OAEP results in CKR_MECHANISM_INVALID

I’m trying to decrypt a message witch was encrypted with bearssl (RSA, 2048, OAEP). I tested the decryption with openssl, witch worked, but I have no luck with the NitroKey. What I have done:

generated a Keypair:
pkcs11-tool -l --pin XXXXXX --keypairgen --key-type rsa:2048 --id 10

downloaded the public Key:
pkcs11-tool -l --pin XXXXXX --id 10 --read-object --type pubkey --output-file pubkey.spki

now I encrypted a message with this bublic key and tryed to decrypt it with the Nitrokey:
pkcs11-tool --id 10 --decrypt -p XXXXXX -m RSA-PKCS-OAEP --input-file data.crypt

But the result is:
Using slot 0 with a present token (0x0)
Using decrypt algorithm RSA-PKCS-OAEP
OAEP parameters: hashAlg=SHA-1, mgf=MGF1-SHA1, source_type=0, source_ptr=0000000
000000000, source_len=0
error: PKCS11 function C_DecryptInit failed: rv = CKR_MECHANISM_INVALID (0x70)

I know, that this means “An invalid mechanism was specified to the cryptographic
operation”. But I thought the Nitrokey is able to decrypt “RSA-PKCS-OAEP” or am I wrong here?

Thank you!


which system and which version of OpenSC(?) are we talking about?
Besides that right now I am not sure if this is supported by HSM 2 or not.

Kind regards


Please check out Nitrokey HSM 2 - RSA Keys >2048 as well. What is your OpenSC version?

I’m using windows with OpenSC 0.19.0. Decryption with the same RSA keys without OAEP padding works, but not with OAEP. We use BearSSL in an embedded system and it no longer supports RSA encryption without OAEP, so I’m in a dilemma :wink:

I see. If pkcs11-tool --list-mechanism is not showing it, than perhaps it is not supported. Alternatively you can ask at OpenSC issues site. It looks like it is already registered at OpenSC: #1678.

If am not mistaken, It looks like the support for it was merged in the February: #1600. It should be then handled by the Nightly build, and available from v0.20.

Last night build artefacts are available at:
I just want to stress out this is a development build.

Ah, okay, thank you I’ll try this next week and give you feedback. :slight_smile:

RSA OEAP is supported by our own PKCS#11 module. You find it at

1 Like

Thank you!

With the sc-hsm-embedded he seems to encrypt and decrypt a message with RSA-OAEP. Had no luck yet to decrypt a message encrypted with BearSSL, but it is looking good so far…

Edit: No luck :frowning: The test program (sc-hsm-pkcs11-test) seems to encrypt and decrypt OAEP, but if I encrypt the message with openssl or bearSSL I get “CKR_GENERAL_ERROR” from C_Decrypt when I try to decrypt it. (while RSA without OAEP works…)

sc-hsm-pkcs11-test uses the RSA primitives from the PKCS#11 module, which uses openssl for public key operations. Maybe the hash function used for MGF is different from SHA-256 ?

The public key operation can be found in encryptRSA() in src/pkcs11/crypto-libcrypto.c

1 Like

Yes, you are right! openSSL and BearSSL are using SHA1 for default, while the PKCS#11 module uses SHA256. Changing in file src/pkcs11/crypto-libcrypto.c in the functions stripOAEPPadding and encryptRSA from EVP_sha256() to EVP_sha1() works!

Thank you very much!