Yes, this is a slow device - you can try using sc-hsm-embedded PKCS#11 driver Performance of Nitrokey HSM2 - can it be improved? What am I doing wrong? - #2 by jan but that is only a bit faster.
PKCS#11 drivers read a whole device at once and this seem to be slow.
Newer Nitrokey HSM with Smart-Card HSM 4.0 on board Cannot Initialize new Nitrokey HSM - #11 by daringer should be faster.