Nitrokey HSM, n-of-m scheme and XCA

I’m trying to use a Nitrokey HSM2 with n-of-m public key auth in XCA.

It works the first time when I unlock the HSM with scsh3gui and generate the keys / certificates. But the next time I XCA does not find the private keys on the HSM anymore. I made sure the HSM is unlocked / authenticated, but I just get the error “could not find private key” when trying to use the key / certificate for anything.
I also tried to setup a new XCA database and reimport the keys from the HSM. Results in the same issue. Even if XCA just found and imported the key, I get the same error.

Any Ideas? It works fine If I use a PIN only for the private key. But thats not what I need.