Yay, after hours of fernagling, I finally got past the frustrating sign_and_send_pubkey: signing failed: agent refused operation errors for SSH operations.
It turns out what helped me was doing echo UPDATESTARTUPTTY | gpg-connect-agent as mentioned in this post. I assume the problem was related to pinentry never firing, so I was never prompted to enter the PIN for my Nitrokey.
I’ve no idea where the bug came from, but I’m glad I got through it 