With a siging request it worked for me.
Create a certificate with the private key, stored within the Nitrokey at ID 11:
$ openssl req -engine pkcs11 -keyform engine -new -key 11 -nodes -days 35600 -x509 -sha256 -out "id11-cert.pem" -subj "/CN=Test Cert 1" -key "pkcs11:pin-value=648219" engine "pkcs11" set.
The certificate contains the RSA public key (Modulus, Exponent):
$ cat id11-cert.pem |openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
70:fe:41:f8:1f:a1:36:bd:b1:65:15:cf:41:ca:2f:c9:82:b8:0d:68
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN = Test Cert 1
Validity
Not Before: Nov 6 12:13:33 2020 GMT
Not After : Apr 27 12:13:33 2118 GMT
Subject: CN = Test Cert 1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (1024 bit)
Modulus:
00:b3:88:f1:d6:9d:87:50:94:d3:52:9d:8a:97:e6:
3b:35:01:a4:ee:3b:65:4b:e6:14:b7:6d:74:57:ee:
29:dc:c8:7f:5d:a7:b5:c8:ea:54:75:97:ea:31:d6:
2e:38:07:23:5e:de:04:5a:59:05:03:af:01:67:a6:
74:9a:8b:ef:5c:6a:95:13:f9:7a:08:fe:91:5d:f8:
95:ca:96:40:14:0f:84:7a:b8:89:e8:a6:4e:3c:a9:
e4:24:7b:9e:a3:1a:0b:5f:6b:2a:8e:fe:c5:ac:e4:
01:79:00:4a:f9:e0:14:3f:2b:39:31:b2:1b:c5:c7:
d8:d9:f7:e9:1d:65:4b:a3:41
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
97:D2:F0:EF:25:44:D2:CE:D6:B8:9D:A5:7C:5C:ED:39:A0:34:C0:61
X509v3 Authority Key Identifier:
keyid:97:D2:F0:EF:25:44:D2:CE:D6:B8:9D:A5:7C:5C:ED:39:A0:34:C0:61
X509v3 Basic Constraints: critical
CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
98:1e:e3:7e:14:f8:b8:4f:0f:4f:b9:57:b3:be:2e:3a:09:d7:
04:0c:df:bd:53:28:03:39:1c:b1:35:bc:b7:cc:1f:a2:78:e9:
c9:ac:c3:e9:4b:ef:82:bd:f3:c2:ff:59:d8:37:82:7b:64:be:
f4:c8:6d:37:43:c6:a3:a6:62:0b:05:89:8d:eb:4a:6b:ac:80:
fa:8f:5b:4f:59:83:b2:57:15:d8:44:6b:f5:9f:48:11:52:03:
9f:48:fe:c0:70:a1:71:59:b9:30:50:fe:c5:bf:80:9c:27:06:
51:40:87:25:04:ea:28:44:cc:19:01:9b:c8:3f:37:b6:52:f5:
cd:82
Now you can extract the public key from the certificate:
$ openssl x509 -pubkey -noout -in id11-cert.pem > id11-pubkey.pem
$ cat id11-pubkey.pem
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCziPHWnYdQlNNSnYqX5js1AaTu
O2VL5hS3bXRX7incyH9dp7XI6lR1l+ox1i44ByNe3gRaWQUDrwFnpnSai+9capUT
+XoI/pFd+JXKlkAUD4R6uInopk48qeQke56jGgtfayqO/sWs5AF5AEr54BQ/Kzkx
shvFx9jZ9+kdZUujQQIDAQAB
-----END PUBLIC KEY-----