Nitrokey 3A mini spontaneous reset?

Nitrokeys
1

My NitroKey 3A mini stopped working recently. It spew some USB errors from the Linux kernel:

[Mo Jan 29 07:15:08 2024] usb 1-1: device descriptor read/64, error -71
[Mo Jan 29 07:15:08 2024] usb 1-1: device descriptor read/64, error -71
[Mo Jan 29 07:15:08 2024] usb 1-1: new full-speed USB device number 123 using xhci_hcd
[Mo Jan 29 07:15:09 2024] usb 1-1: device descriptor read/64, error -71
[Mo Jan 29 07:15:09 2024] usb 1-1: device descriptor read/64, error -71
[Mo Jan 29 07:15:09 2024] usb usb1-port1: attempt power cycle
[Mo Jan 29 07:15:10 2024] usb 1-1: new full-speed USB device number 124 using xhci_hcd
[Mo Jan 29 07:15:10 2024] usb 1-1: Device not responding to setup address.
[Mo Jan 29 07:15:10 2024] usb 1-1: Device not responding to setup address.
[Mo Jan 29 07:15:10 2024] usb 1-1: device not accepting address 124, error -71
[Mo Jan 29 07:15:10 2024] usb 1-1: new full-speed USB device number 125 using xhci_hcd
[Mo Jan 29 07:15:10 2024] usb 1-1: Device not responding to setup address.
[Mo Jan 29 07:15:10 2024] usb 1-1: Device not responding to setup address.
[Mo Jan 29 07:15:11 2024] usb 1-1: device not accepting address 125, error -71
[Mo Jan 29 07:15:11 2024] usb usb1-port1: unable to enumerate USB device
[Mo Jan 29 07:15:18 2024] usb 1-1: new full-speed USB device number 126 using xhci_hcd
[Mo Jan 29 07:15:18 2024] usb 1-1: New USB device found, idVendor=20a0, idProduct=42b2, bcdDevice= 1.03
[Mo Jan 29 07:15:18 2024] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[Mo Jan 29 07:15:18 2024] usb 1-1: Product: Nitrokey 3
[Mo Jan 29 07:15:18 2024] usb 1-1: Manufacturer: Nitrokey
[Mo Jan 29 07:15:18 2024] usb 1-1: Device is not authorized for usage
[Mo Jan 29 07:15:18 2024] hid-generic 0003:20A0:42B2.0013: hiddev0,hidraw0: USB HID v1.11 Device [Nitrokey Nitrokey 3] on usb-0000:00:14.0-1/input1
[Mo Jan 29 07:15:18 2024] cdc_acm 1-1:1.2: ttyACM0: USB ACM device
[Mo Jan 29 07:15:18 2024] usb 1-1: authorized to connect
[Mo Jan 29 07:15:19 2024] usb 1-1: USB disconnect, device number 126

I was not able to get the thing do work again with ssh. It seems, though, that the USB side got a grip on itself after some time:

[Mo Jan 29 07:16:22 2024] usb 1-6: new full-speed USB device number 5 using xhci_hcd
[Mo Jan 29 07:16:22 2024] usb 1-6: New USB device found, idVendor=20a0, idProduct=42b2, bcdDevice= 1.03
[Mo Jan 29 07:16:22 2024] usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[Mo Jan 29 07:16:22 2024] usb 1-6: Product: Nitrokey 3
[Mo Jan 29 07:16:22 2024] usb 1-6: Manufacturer: Nitrokey
[Mo Jan 29 07:16:22 2024] usb 1-6: Device is not authorized for usage
[Mo Jan 29 07:16:22 2024] hid-generic 0003:20A0:42B2.0017: hiddev0,hidraw0: USB HID v1.11 Device [Nitrokey Nitrokey 3] on usb-0000:00:14.0-6/input1
[Mo Jan 29 07:16:22 2024] cdc_acm 1-6:1.2: ttyACM0: USB ACM device
[Mo Jan 29 07:16:22 2024] usb 1-6: authorized to connect
[Mo Jan 29 07:16:52 2024] usb 1-6: USB disconnect, device number 5

This was me disconnecting it after testing ssh connections that failed to work.

What is permanent: I cannot authenticate with the device. I always get

sign_and_send_pubkey: signing failed for ED25519-SK "/home/user/.ssh/id_ed25519_nitro1": device not found

I realized that this just seems to mean that the device doesn’t offer the correct keys anymore … it seems to have done a reset, nuking all my keys that relied on it (all non-resident ed25519-sk, plus some FIDO2 2FA use for websites)!

I can create new keys and re-register the thing with 2FA services … but when will this happen again? Current info:

$ nitropy nk3 status
Command line tool to interact with Nitrokey devices 0.4.45
UUID:  xxxxx
Firmware version:   v1.3.1
Init status:        ok
Free blocks (int):  253
Free blocks (ext):  478

Is this a known firmware issue? A hardware glitch? My trust is rather shaken.

2

Hey @securityconscious

generally, it would be grat if you would use code blocks for such log-pastes in the future, thank you.

The next generic item to state is: Your device has a quite old firmware, I do not entirely remember if this was a firmware release which had a fixed bug around Fido2/RKs, so please update your device using nitropy nk3 update.

Further:

...
[Mo Jan 29 07:15:08 2024] usb 1-1: device descriptor read/64, error -71
[Mo Jan 29 07:15:08 2024] usb 1-1: device descriptor read/64, error -71
[Mo Jan 29 07:15:08 2024] usb 1-1: new full-speed USB device number 123 using xhci_hcd
...

this has usually one of two reasons:

  • The USB Port “doesn’t like” the Nitrokey 3 (to rule out, please try to reproduce on other ports)
  • There is some mechanical/electrical damage within the Nitrokey 3

Under the line, if the behavior persists please write support (at) nitrokey (dot) com together with your sales order number (SOxxxxxxx) and we can look at the option of a replacement or other options.

best

3

Hi, thanks for the response. I used the device for several months, rather often on that very USB port. As the issues started, I switched it around the different ports on the host.

I now figure that it had an initial hickup, probably during an attempted use via ssh, causing one event of unable to enumerate USB device. This apparently triggered a reset of its state, losing my keys (the private key that would reproduce my non-resident SSH keys). The subsequent attempts to use it got the device not found message, which does not refer to the physical device, but to the fact that the matching private key is gone.

Does that make sense?

This happened once in half a year or so. I’d rather not collect statistics of this happening multiple times. Do you have information that suggests that the bug you refer to could have caused this kind of misbehaviour? Crash and full reset of state?

4

Ok, re-checking on some older firmware versions (please update! there is already v1.6 out), there might be different reasons here which might play a role:

Before v1.4 there were situations in which the filesystem on the NK3 Mini corrupted itself if power-cycled in the wrong moment - if this happened, you might see this with nitropy nk3 test, the FIDO2 test should then show “x5c not found”.

Your dmesg logs indicate that there might be mechanical issues with the connection, especially you report “device not found” after some attempts, which likely also means touching the device. If there is a non-optimal connection within the USB port paired with the bug for firmwares < v1.4 this might easily end up in a situation you described.

5

To be clear: device not found is the error I get from ssh (without agent, which would confuse the error diagnostic) with the Nitrokey in the current state, also with fresh firmware. I can create new keys and use them. So this message just means „I don’t have the correct FIDO2/U2F key. You inserted a different one.”

There might be mechanical issues … as I noticed somewhat recently that the plastic frame of the key broke and it’s prone to disassemble when trying to pull it out.* Maybe there was a moment of glitchy connection.

So I’ll go with that hypothesis … glitchy power cycle, firmware bug, corruption, automatic self-reset (???) … I did not find an issue with the device in the old state, though. The nitropy nk3 test went through fine, just se050 testing is not supported.

I really hope such thing never happens again and the current firmware is robust against also unconsiderate handling.

(*) PS: I cannot pull it out via the lanyard, as the metal ring is no ring anymore, but two horns. I soon ditched the little lanyard thingie with its connection, as that one is prone to setting the Nitrokey free when pulling out the keyring out of a pocket. That happened just about the first day I went to work with it. A colleague found it next to the bike stands. What luck! So I fixed it with a little keyring onto the big keyring along with the other keys (and, at that time, a Yubikey among them … which has a flimsy USB connection, but is itself a lot more robust, apparently). But that went through the too thin stretch of soft metal on the Nitrokey all to quickly. I got some small USB memory keys that serve me well since 10 years now. They are shorter than (just about 2 cm, which is fine), but of the same type of build like USB Stick Mini Innovativ mit geschlossenem USB Port … I’d wish the Nitrokey used that sort of packaging. Maybe needs a small opening for the LED …

6

It happened again. I got some kind of reset now, also with v1.6 firmware. I’ll write to support.

$ nitropy nk3 status
Command line tool to interact with Nitrokey devices 0.4.50
UUID:               xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Firmware version:   v1.6.0
Init status:        ok
Free blocks (int):  250
Free blocks (ext):  478
Variant:            NRF52

The device seems to be susceptible to some kind of glitching. I can imaginge a loosened USB port connecting the power pins in just the wrong manner with some chance.

The NitroKey is working again just fine … just that it has had a reset, apparently,
and my previously generated SSH keys are rubbish now. I cannot trust this, sadly.

7

Could this be the USB autosuspend?

Then this might help: echo -1 >/sys/module/usbcore/parameters/autosuspend

8

Well, even if USB autosuspend would interfere … should that ever cause
the device to loose its state and reset?

So far the Yubikey wasn’t fuzzy about the USB connection in the same
port. Definitely it seems to have to do something with non-ideal USB
contact … or about the laptop entering and leaving S3 suspend mode.

Actually, yes … maybe it isn’t even the USB port contact but
suspending/reactivating. I remember having to replug the Nitrokey after
resume from time to time to make it work again.

All in all: Anything like that should never cause a reset of the data
on the Nitrokey.

9

Definitely this should be handled correctly.

If it is the autosuspend, it should affect more people. And at least the reconnect seems to happen with my Nitrokey.

Maybe it can be recreated:
One could prepare a script to provoke the issue and do some endless test while autosuspend cuts the connection or the system is put in S3 and a mousejiggler wakes it up again.

I could also imagine that this issue may happen when there is a USB harddisk and the Nitrokey connected to the same port or that there is an internal USB Hub. Then there could be a brownout happen when e.g. the disk starts spinning and draws more power.