KeePassXC doesnt use FIDO. YK chalresp equivalent and HMAC arent available on 3C as far as I can tell.
You can put a keyfile in a USB (or Nitro storage) and have a security key dongle that way. Or buy a different key for chalresp (standard Yubikey for instance).
---->Ok, I see you have already tried this. NK 3C can HMAC? Were you successful with this yet?
Command line tool to interact with Nitrokey devices 0.4.39
Usage: nitropy nk3 secrets add-challenge-response [OPTIONS] {1|2} SECRET
hexinput.txt requires (which?) Yubikey. Can you generate a new hex just for a Nitrokey?