SSH with valid x509 fails

Hello! I have an issue with logging in to SSH server using Nitrokey Storage 2 with Actalis x.509 S/MIME (RSA/2048) certificate on it. I have no problems signing mail or documents in LibreOffice, logging in with pam_pkcs11, everything works as it should, only SSH has some problems.
I’ve extracted pubkey with pkcs15-tool --read-ssh-key 3, placed it on remote server, it even gets detected while I try to log in with ssh -I but then it fails with:

C_SignInit failed: 99
sign_and_send_pubkey: signing failed: error in libcrypto