Yes, you are right! openSSL and BearSSL are using SHA1 for default, while the PKCS#11 module uses SHA256. Changing in file src/pkcs11/crypto-libcrypto.c in the functions stripOAEPPadding and encryptRSA from EVP_sha256() to EVP_sha1() works!
Thank you very much!