Nextbox start loop

NextBox
1

Hello,

Yesterday, my nextbox have restarded (maybe update). Since the green LED is flashing and I have not access to the web page.

I have checked the nextbox.log

2025-04-24 16:21:27,584 [i] jobs      Hello World - I am NextBox - call me: v1.25.2-1 - let'sa gooo
2025-04-24 16:21:27,587 [D] worker    finished worker job: SelfUpdate
2025-04-24 16:21:27,588 [D] worker    starting worker job: AutomaticDebianUpdate
2025-04-24 16:21:27,591 [D] worker    finished worker job: AutomaticDebianUpdate
2025-04-24 16:21:28,591 [D] worker    starting worker job: EnableNextBoxApp
2025-04-24 16:21:28,601 [D] cmd_run   started ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'app:enable', 'nextbox') (blocking)
2025-04-24 16:21:28,602 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:21:28,717 [D] cmd_run   blocking call finished
2025-04-24 16:21:28,717 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'app:enable', 'nextbox')
2025-04-24 16:21:28,718 [i] cmd_run   Error response from daemon: Container 0606a03628ea158f63920e5de1ce11d1a2490c5d23e0df1a185a251d2bfa5fb6 is not running
2025-04-24 16:21:28,718 [i] cmd_run   
2025-04-24 16:21:28,718 [i] cmd_run   #### log output end
2025-04-24 16:21:28,718 [D] worker    finished worker job: EnableNextBoxApp
2025-04-24 16:21:29,719 [D] worker    starting worker job: LED
2025-04-24 16:21:29,737 [D] worker    finished worker job: LED
2025-04-24 16:21:33,739 [D] worker    starting worker job: EnableNextBoxApp
2025-04-24 16:21:33,748 [D] cmd_run   started ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'app:enable', 'nextbox') (blocking)
2025-04-24 16:21:33,749 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:21:33,865 [D] cmd_run   blocking call finished
2025-04-24 16:21:33,865 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'app:enable', 'nextbox')
2025-04-24 16:21:33,866 [i] cmd_run   Error response from daemon: Container 0606a03628ea158f63920e5de1ce11d1a2490c5d23e0df1a185a251d2bfa5fb6 is not running
2025-04-24 16:21:33,866 [i] cmd_run
...
2025-04-24 16:21:50,282 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:21:50,449 [D] cmd_run   blocking call finished
2025-04-24 16:21:50,450 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '
--ansi', '--no-warnings', 'app:enable', 'nextbox')
2025-04-24 16:21:50,450 [i] cmd_run   Error: No such container: nextbox-compose_app_1
2025-04-24 16:21:50,450 [i] cmd_run   
2025-04-24 16:21:50,450 [i] cmd_run   #### log output end
2025-04-24 16:21:50,451 [D] worker    finished worker job: EnableNextBoxApp
2025-04-24 16:21:51,452 [D] worker    starting worker job: LED
2025-04-24 16:21:51,461 [D] worker    finished worker job: LED
2025-04-24 16:21:55,468 [D] worker    starting worker job: EnableNextBoxApp
2025-04-24 16:21:55,477 [D] cmd_run   started ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'app:enable', 'nextbox') (blocking)
2025-04-24 16:21:55,478 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:21:55,594 [D] cmd_run   blocking call finished
2025-04-24 16:21:55,594 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'app:enable', 'nextbox')
2025-04-24 16:21:55,595 [i] cmd_run   Error: No such container: nextbox-compose_app_1

The liste of the running docker containers.

# docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED              STATUS                         PORTS      NAMES
9a41310bcf96   mariadb:10.11.11      "docker-entrypoint.s…"   About a minute ago   Restarting (1) 2 seconds ago              nextbox-compose_db_1
84fd74871840   redis:5.0.11-alpine   "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)    6379/tcp   nextbox-compose_redis_1

Try to reboot properly, same issue

Do you have an idea to solve this issue ?

Thank you

2

Yersterday I did a similar observation: Nextbox access was blocked due to mantainance mode. Log checking made obvious that an upgrade is running. After hours the upgrade was still running, which is quite unusual.

A quick view in the nextcloud.log showed cyclically messages like

Unable to generate a URL for the named route \"workspace.page.index\
Where the named route and its corresponding request did vary.

I decied to intervene, stopped the docker composition and rebooted the box.
Login was possible afterwards, but several applications weren’t available any more.

Today i had a look to the admin view and realized, that a number of previously “tested” applications are now flagged as “untested” ones:

  • Calendar
  • Contacts
  • External sites
  • File access control
  • Files automated tagging
  • Full text search
  • Full text search - Elasticsearch platform
  • Full text search - Files
  • GpxPod
  • Group folders
  • Guests
  • Maps
  • MediaDC
  • Nextcloud Office
  • Talk

Something weird happened during the update, and did not (completely) cured by my intervention.

I decided to update all those flagged apps.
Finally, 4 of them weren’t updated at all:

  • Full text search
  • Full text search - Elasticsearch platform
  • Full text search - Files
  • GpxPod

From experience and guts, I assume one of thema as potentially conflicting candidate.

Maybe there will be time to further investigate the logfiles within the next two weeks.

3

My Nextbox was also entering maintenance mode yesterday evening. Will investigate…

2025-04-25 09:24:24,586 [i] cmd_run   #### log output end
2025-04-25 09:24:24,586 [W] jobs      cannot re-index files
2025-04-25 09:24:24,587 [D] worker    finished worker job: ReIndexAllFiles
2025-04-25 09:24:27,089 [D] worker    starting worker job: LED
2025-04-25 09:24:27,091 [D] worker    finished worker job: LED
2025-04-25 09:24:39,600 [D] worker    starting worker job: ReIndexAllFiles
2025-04-25 09:24:39,616 [D] cmd_run   started ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'files:scan', '--shallow', '--all') (blocking)
2025-04-25 09:24:39,618 [D] cmd_run   blocking call, waiting now...
2025-04-25 09:24:40,699 [D] cmd_run   blocking call finished
2025-04-25 09:24:40,699 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'files:scan', '--shallow', '--all')
2025-04-25 09:24:40,700 [i] cmd_run   
2025-04-25 09:24:40,700 [i] cmd_run                                                              
2025-04-25 09:24:40,700 [i] cmd_run     There are no commands defined in the "files" namespace.  
2025-04-25 09:24:40,701 [i] cmd_run    (last 2 messages repeated 2.0 times)
2025-04-25 09:24:40,701 [i] cmd_run   #### log output end

nextcloud.log above.

root@nextbox:~# docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED        STATUS                  PORTS                                      NAMES
1a7707eda9d9   nextbox               "/cron.sh"               17 hours ago   Up 17 hours             80/tcp                                     nextbox-compose_cron_1
ff0ffecf2ced   nextbox               "/entrypoint.sh apac…"   17 hours ago   Up 17 hours             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nextbox-compose_app_1
8b6c0aa25319   mariadb:10.11.11      "docker-entrypoint.s…"   17 hours ago   Up 17 hours (healthy)   3306/tcp                                   nextbox-compose_db_1
55f46e70c20b   redis:5.0.11-alpine   "docker-entrypoint.s…"   17 hours ago   Up 17 hours (healthy)   6379/tcp                                   nextbox-compose_redis_1

docker ps shows 4 running containers, but I don’t know why not all of them are healthy?

I wonder if I should just reboot, or try to disable maintenance mode?

4

Maybe the root cause, the failed upgrade (broken database migrations), from yesterday afternoon:

2025-04-24 16:36:21,581 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'upgrade')
2025-04-24 16:36:21,581 [i] cmd_run   Setting log level to debug
2025-04-24 16:36:21,582 [i] cmd_run   Updating database schema
2025-04-24 16:36:21,582 [i] cmd_run   Exception: Database error when running migration 30000Date20240708160048 for app core
2025-04-24 16:36:21,582 [i] cmd_run   An exception occurred while executing a query: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'scheduled_at'
2025-04-24 16:36:21,582 [i] cmd_run   Update failed
2025-04-24 16:36:21,583 [i] cmd_run   Maintenance mode is kept active
2025-04-24 16:36:21,583 [i] cmd_run   Resetting log level
2025-04-24 16:36:21,583 [i] cmd_run   
2025-04-24 16:36:21,583 [i] cmd_run   #### log output end
2025-04-24 16:36:21,584 [W] jobs      cannot occ upgrade
2025-04-24 16:36:21,584 [D] worker    finished worker job: OccUpgrade
2025-04-24 16:36:22,585 [D] worker    starting worker job: TrustedDomains
2025-04-24 16:36:22,594 [D] cmd_run   started ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'config:system:get', 'trusted_domains') (blocking)
2025-04-24 16:36:22,595 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:36:23,564 [D] cmd_run   blocking call finished
2025-04-24 16:36:23,564 [D] worker    finished worker job: TrustedDomains
2025-04-24 16:36:23,565 [D] worker    starting worker job: DynDNSUpdate
2025-04-24 16:36:23,575 [D] cmd_run   started ['/usr/bin/systemctl', 'is-active', 'ddclient.service'] (blocking)
2025-04-24 16:36:23,576 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:36:23,585 [D] cmd_run   blocking call finished
2025-04-24 16:36:23,586 [D] worker    finished worker job: DynDNSUpdate
2025-04-24 16:36:24,587 [D] worker    starting worker job: ReIndexAllFiles
2025-04-24 16:36:24,598 [D] cmd_run   started ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'files:scan', '--shallow', '--all') (blocking)
2025-04-24 16:36:24,599 [D] cmd_run   blocking call, waiting now...
2025-04-24 16:36:25,720 [D] cmd_run   blocking call finished
2025-04-24 16:36:25,720 [i] cmd_run   #### log output for command: ('docker', 'exec', '-u', 'www-data', 'nextbox-compose_app_1', '/var/www/html/occ', '-n', '--ansi', '--no-warnings', 'files:scan', '--shallow', '--all')
2025-04-24 16:36:25,720 [i] cmd_run   
2025-04-24 16:36:25,721 [i] cmd_run                                                              
2025-04-24 16:36:25,721 [i] cmd_run     There are no commands defined in the "files" namespace.  
2025-04-24 16:36:25,721 [i] cmd_run   
2025-04-24 16:36:25,721 [i] cmd_run   #### log output end
2025-04-24 16:36:25,721 [W] jobs      cannot re-index files
2025-04-24 16:36:25,722 [D] worker    finished worker job: ReIndexAllFiles

@nielsson4711 @gaby do you find something similar, in your logs (for me it was in nextbox.log.1)? This looks like it could possibly affect a few users? I am no expert with Nextcloud management commands, so I would appreciate a helping hand from @daringer ore someone else from Nitrokey, to fix those db issues?

5

Time for log analysis will not be before next week.

What I found is:
The app gpxpod isn’t compatible with nextcloud V 30.x, as stated in https://mybox/settings/apps/discover/gpxpod:

Server version 29 or lower is required.

This might have caused havoc and loops during the upgrade. I’m not shure if there are other apps contributing to havoc in this manner.

In general: I’m not happy with this upgrades integration and communication quality.

6

To stop und get up again the system using brute force i apply the following steps:

  1. CLI log in
  2. Ensure you’re able to use “root” access rights
  3. Stop all containers:
    cd /usr/lib/nextbox-compose/
    /usr/bin/nextbox-stop-compose.sh
  4. Have a look at the output
  5. check if no containers are running
    docker ps
  6. reboot
    init 6
  7. Log in again
  8. With “root” access rights run the maintenance command sequence, I illustrate below in form of a shell script. It’s not perfect, but works
#!/bin/bash

theuser=snoopy

echo "starting long maintenance ,,,"
echo " 1 of 13 maintenance:repair ..."
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:repair
echo " 2 of 13 maintenance:mode --on"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:mode --on
echo " 3 of 13 db:add-missing-indices"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ db:add-missing-indices
echo " 4 of 13 db:add-missing-columns"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ db:add-missing-columns
echo " 5 of 13 db:add-missing-primary-keys"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ db:add-missing-primary-keys
echo " 6 of 13 maintenance:mimetype:update-db"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:mimetype:update-db
echo " 7 of 13 maintenance:mimetype:update-js"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:mimetype:update-js
echo " 8 of 13 maintenance:theme:update"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:theme:update
echo " 9 of 13 maintenance:update:htaccess"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:update:htaccess
echo "10 of 13 maintenance:data-fingerprint"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:data-fingerprint
echo "11 of 13 maintenance:repair --include-expensive"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:repair --include-expensive
echo "12 of 13 maintenance:mode --off"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ maintenance:mode --off
echo "13 of 13 files:scan  nisse"
docker exec -u 33 -it nextbox-compose_app_1 /var/www/html/occ files:scan  $theuser
echo "finished"
  1. check the nextcloud function and adjust with admin work accordingly
7

thanks alot @nielsson4711 ! that looks pretty solid, will try this!

8

I’ve run my nextbox for four years almost without problems…so in general I’m very happy how upgrades are applied. I encouraged nitrokey to communicate better (i.e. post upcoming upgrades here in the forum), that would be my only thing to do better?

9

Works! I don’t know if it would have worked with just “leaving maintenance mode”, but that’s another story :slight_smile: thx again! @nielsson4711

I only have some basic apps enabled, as calendar and notes. no “untested” here either (where do you see “tested/untested”?).

10

My Nextbox is also in permanent maintenance mode.
Unfortunately, I am not able to access the terminal. (How does that work?) I don’t have any exotic software running either.
A little help would be good.
I am also not happy with the communication. I have had maintenance errors quite often in the past and then had to search for solutions myself for a long time. Unfortunately, this makes the Nextbox very unreliable.
Since we want to move further away from dependence on large corporations, that would be a nice alternative. But unfortunately I can’t recommend it to anyone who doesn’t have their own skills.

11

@raro you better write to support. I don’t know how to get out of maintenance mode without SSH access…

12

I would even be willing to pay a yearly fee, to have a more reliable Nextbox, with upgrade announcements, etc.

13

I don’t see Database error in the log of my nextbox.

14

I have solved my issue, my solution (as root):

1- check the docker process

# docker ps    
CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS                                     PORTS      NAMES
c2d9bb95b648   mariadb:10.11.11      "docker-entrypoint.s…"   7 seconds ago   Up Less than a second (health: starting)   3306/tcp   nextbox-compose_db_1
6ef510f07732   redis:5.0.11-alpine   "docker-entrypoint.s…"   7 seconds ago   Up Less than a second (health: starting)   6379/tcp   nextbox-compose_redis_1

2- check the log for mariadb

# docker logs c2d9bb95b648
2025-04-25 14:20:24+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.11+maria~ubu2204 started.
2025-04-25 14:20:25+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2025-04-25 14:20:25+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2025-04-25 14:20:25+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.11+maria~ubu2204 started.
2025-04-25 14:20:25+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2025-04-25 14:20:25+00:00 [Note] [Entrypoint]: Starting temporary server
2025-04-25 14:20:25+00:00 [Note] [Entrypoint]: Waiting for server startup
2025-04-25 14:20:25 0 [Note] Starting MariaDB 10.11.11-MariaDB-ubu2204 source revision e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid 5BqD24+tOteoHr51odDq103BvFw= as process 58
2025-04-25 14:20:25 0 [Warning] You need to use --log-bin to make --binlog-format work.
...
2025-04-25 14:20:25 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.9. You must start up and shut down MariaDB 10.7 or earlier.

there are error with mariadb version

3- stop docker

systemctl stop nextbox-daemon
systemctl stop nextbox-compose

4- check there are no docker process

# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

5-edit yml

cd /usr/lib/nextbox-compose/
vim docker-compose.yml

change mariadb:10:11:11 to mariadb:10:7

services:
  db:
    image: mariadb:10.7

6- restart nextbox docker

systemctl start nextbox-daemon
systemctl start nextbox-compose

After few minutes the green led is fix and the web interface is ok with the new version of nextcloud

1 Like
15

glad you could fix it!

16

Oh :confused: - sorry for the efforts and thanks to the helping hands :raised_hands:

yeah the NC 30 update was also mostly delayed due to app incompatibility - usually you should be able to return from maintainance mode with a single click onto the hardware button while the LED is purple.

Nevertheless NC 30 seems to break an awful lot of apps - sorry for that - obviously it’s not possible to foresee all of them - that’s why getting out of maintenance mode was implemented through the hardware button…

I’ll try to comment on some things to hopefully clear things up:

this is correct, only the direct dependencies for the nextcloud image are health-checked.

For me this doesn’t make sense that this leads to reboot-loop, if you could check the logs and give us some insights we can maybe improve the error handling here to not reboot-loop.

The repair command is quite excessive, but no discussion good - we do not run these automatically as we have observed that these can also cause issues - we currently favor a “repair all” button inside the nextbox app.

Furthermore my expectation is that disabling the maintainance mode should allow you to log in and then check which app has caused these issues - overall the upgrade process for NC is from my point of view not deterministic enough, especially there is this issue which might improve the situation here - the nextcloud docs also propose to disable all 3rd party apps before updating (but lol - e.g., backup is from nextcloud directly and even didn’t get an update to 29) …

best

17

I pressed the hardware button twice within 2 seconds for the software reset as the documentary says. All data have been deleted as I would have done the hardware reset. A mess!!! How can I restructure the access to the data?