Nitropad T430 Qubes OS kernel freeze

NitroPad, NitroPC
1

I have Qubes 4.0 and Heads 1.4. Installed kernels in /boot directory are xen-4.8.5-36, vmlinuz-5.4.98 and vmlinuz-5.4.156.

Kernel 5.4.156 freezes sometimes at load time or initramfs stage. The last message in console is kexec_core: Starting new kernel and there everything freezes.

With kernel 5.4.98 the system doesn’t freeze, at least I haven’t seen it so far.
(Edit: The kernel loading freezes sometimes with kernel 5.4.98 too, so the kernel version doesn’t explain the whole issue.)

Any explanations, references, similar experiences to this issue?

2

Mh :face_with_monocle:
I’m using the same setup with kernel 5.4.98 and did not run in the issue so far. Can you say how frequent this is (from 10 boots how often does this happen) Does it happen with the power supply connected? Do you have a SSD ore HDD? What CPU you have (I5 or i7)?

3

I have SSD and i5-3320M CPU. I haven’t found any patterns yet. The freezes are so random. They happend with and without power supply, with both kernel versions (see the original message). The freezing frequency is maybe 1–3 times in 10 boots. journalctl in dom0 doesn’t tell anything about those boots.

4

ok, i think this will be hard to track down just hardreseted and booted my device 10 times and could not reproduce it. I will try later with a i5 version, if I can reproduce it there

5

Ok did the same 10 times booting with the i5 Version and could not reproduce it. That journalctl does not show anything is expected, since the freeze seems to happen before the kernel is relay started. So to find out more we would need to add some debugging parameter to the kernel like in this post Linux Debugging using a Bootloader with Kernel Parameters and hope that you could trigger the failure. And then get same clues or meaning full error messages

6

How do I boot with debugging parameters? I guess that I need to enter “emergency shell” (or what is it called) at Heads menu and then enter a long command line that maybe starts with kexec command and has probably some debug keyword somewhere. But that is not nearly enough and I would need help with forming the complete boot command.

7

The only pattern that there seems to be is that cold boots are more likely to freeze. First one or two boots of the day tend to go like this:

  1. Power on. Freeze at kernel loading. Power off.
  2. Power on. Freeze at kernel loading. Power off.
  3. The third boot succeeds.

“Warm” reboots will succeed more likely.

8

The best way to do is is via the grub configuration (your way in adding it on the fly would also work seems a little complicated to me). Please do a backup before since this can disable your Nitropad and you would need to reinstall it

  • open a terminal in dom0
  • sudo nano /etc/default/grub
  • there add the parameter you want to GRUB_CMDLINE_XEN_DEFAULT
  • remove the quit parameter in GRUB_CMDLINE_LINUX
  • also make sure GRUB_DISABLE_RECOVERY="false"
  • save
  • sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  • reboot
  • under Options → Boot Options → OS you can see the different entry’s if something goes wrong you can choose there the recovery entry
  • you will be ask to sign the /boot partion again

What options are fitting you have to try out see the last link. a simble debug for the start might be enough

9

Thanks. Done. I added debug to Xen command line and removed quiet from Linux command line, created new Grub config etc. We’ll see…

10

Unfortunately debug ignore_loglevel options in Xen and removed quiet in Linux command line do not seem to help. A frozen kernel load doesn’t write anything to systemd journal. I tried command journalctl -b -1 in the first succesful session after freeze.

11

I strongly suspect a power related issue. Could you try booting without battery and see if the issue persists

12

Without battery and with power supply connected this morning’s first boot froze but second boot succeeded. This seems to be the pattern: first one or two boots after a long power off period tend to freeze. After that it is hard to reproduce, although it still happens occasionally in warm reboots too.

Positive thing is that this issue does not prevent normal computer usage. It’s a minor annoyance. Computer’s used battery functions well in practice. It lasts over four hours if I don’t do much. With heavier load it’s obviously less. I think that is quite normal for Qubes OS and used batteries.

13

The freeze problem is really serious with Qubes 4.1 installer: the kernel loading always freezes and I can’t start Qubes 4.1 installer at all. What I did and see:

  1. Boot my Nitropad T430 with Nitrokey plugged in. Boot files verified successfully.
  2. Remove Nitrokey and insert Qubes 4.1 installer in USB drive.
  3. In Heads menu select “Options / Boot options / USB boot”.
  4. Two boot partition options are shown: “Qubes 4.1 [something]” and “ANACONDA”. I chose “Qubes 4.1”.
  5. Nine installer options are shown. I usually chose “Qubes 4.1 basic graphical install” but have tried others as well.
  6. Kernel is loaded. Message “starting new kernel” appears and then freeze. Always. The USB drive connected to different USB ports. The computer with only battery, without battery (only power supply), and with both battery and power supply.

So, it seems that I can’t install Qubes 4.1 with Qubes upstream installer image. I have not tried online system upgrade from Qubes 4.0 to 4.1. Maybe 4.1’s kernel will freeze too. Any ideas? I will probably contact Nitrokey shop people soon.

See the details about installer image verification.

Someone always asks about verifying the installer image file and USB drive. I verified them. I verified the OpenPGP signature of Qubes 4.1 upstream installer image. I checked the sha256sum of the image file. I wrote the image file to USB drive’s device file (/dev/sdc). I read back image’s size number of bytes from the device file and redirected it to sha256sum. It printed exactly the same hash value as the original image file in my file system. I booted two different computers with the USB drive: in my desktop computer the Qubes 4.1 installer starts. I also used its “test media” boot option which tests the installer. It verified correctly.

14

I tried online upgrade from Qubes 4.0 to 4.1 (upgrade guide). The system froze very early (STAGE1) and hard reset was needed. It left a broken system: no network, Qubes manager doesn’t start anymore. Next I will reinstall 4.0.

Things aren’t going well with this certified Qubes OS laptop.

15

This is probably a Hardware fail of the T430 mainboard . Since this is refurbished Hardware this can happen (and even to new hardware this happens)

16

I can confirm, that installing Qubes 4.1 from USB is not possible with Nitropad T430. (Tested with newest minimal Heads 1.4)

The installer early goes into freeze. Right after saying kexec_core: Starting new kernel.

This is a serious issue, since many important Qubes-4.0-templates will not receive any updates soon.

What can we do?

17

I think some Thinkpads have problems but in general the reinstallation is working. If your Nitropad has such a problem please contact support@nitrokey.com

18

I did some testing. Here are the results:

Qubes 4.0 on Kernel 4.19 => Works OK. Freezes are very rare.
Qubes 4.0 on Kernel 5.4 => Freezes regularly.
Qubes 4.0 on Kernel 5.15 => No problems at all.

Qubes 4.1 is another story. They for whatever reason did not compile ath11k driver into the Kernel of the installer. So, in order to boot the 4.1 installer on T430 with Atheros Wifi Chip, the Wifi switch next to the SD card slot must be turned off.

Conclusion: T430 hardware is not the problem. Qubes Kernel is.

3 Likes
19

Very likely it was a hardware issue. Warranty allowed me a another Nitropad T430, and the problem now is gone. Qubes OS 4.1 works nicely.

20

I was having a similar issue with my X230 and indeed I managed to install Qubes with WiFi turned off. Thank you for posting your findings.