Yes, Debian splits it into many packages. In general - what you need is to either have pcsc-lite without polkit support or have polkit support enabled properly, as you did with your configuration file.
If you think this is getting slower because of this and you can measure it somehow (old version works better than the new one) - this might be something to report to the distribution.
Depending on the Nitrokey HSM2 version - older cards can be slower than the newer ones (Cannot Initialize new Nitrokey HSM - #11 by daringer).
It can depend on the number of keys/certificates/files stored on the smartcard.
Speed may also may depend on caching being enabled in OpenSC or not. (I would recommend not to use caching there).