Maintenance Mode

NextBox
1

For the past two years I have monitored the HTTP status code responses of my Nextbox.

When not logged in, the homepage redirects to /login and thus, the homepage returns a 302. When the mainteance mode is turned on, Nextcloud returns a 503.

So I wrote a small script that did a curl -w "%{http_code}" just on the homepage, and puts the returned code together with a date into a JSON, that can then be visualized.

Here is the chart that turns out of this:

image
image1874×360 20.4 KB

Itn the beginning of my monitoring it kept going on and off pretty often (that’s why I started the experiment). I did, however, also experiment with the Nextcloud Backup extension during that time (which I had also proposed for Nextbox here), so I am not sure whether that may have caused it back then. I uninstalled it because I was not at all satisfied with its performance. (Thinking about that today, I must question whether the backup app caused the maintenance mode or whether it was the other way around.)

Then there was a long time with everything being ok (no clue what happened that one night of HTTP 200s there) but recently the annoyance of random times of maintenance modes returned.

Here is the chart of just the last three months:

image
image1874×360 40.8 KB

That one week in mid-September the maintenance mode won’t ever go away again. I had to SSH¹ into the Nextbox and manually turn it off doing sudo docker exec -u www-data nextbox-compose_app_1 php occ maintenance:mode --off to make the Nextcloud instance any usable again.

There are only two regular users on that Nextcloud instance and they don’t do much more than having their calendar and their phone book synced via Cal/CardDAV plus every once in a while they upload some pictures with their Nextcloud mobile app. No power using at all.

The amount of times this thing is going to maintenance is unacceptable imho. I have no idea why it is happening, and why it suddenly started doing it (again).

Here is a link to the chart in HTML. You can zoom in and out to see the graph jump up and down.

So I am very curious, what causes this and whether that is expected behaviour of the Nextbox. How can I find out about it?

–

¹ Since we are not allowed to tell that we sneakily do have SSH connectivity or else we are denied of any support for our 350€ product, please ignore for now, how exactly I managed to turn off the maintenance mode myself.

2

I don’t really understand your long message.
It seems that you added a specific backup add-on into the Nextcloud instance of your Nextbox, an add-on that you developed yourself, and that addition resulted in extra switches to maintenance (presumably during the backup).
It seems you are dissatisfied, including on NextBox ‘lack of support’ (for me, I got support both here and directly from NextBox each time I needed it, and I’ve been an user from day 1, having pledged on the very initial box even before it was built)
OK. So what?
Do you expect the Nextbox guys will spontaneously recover your system cleaning up what your extension did?
The only thing I can suggest is restarting with a factory-clean system, something you can obtain through ssh and simple apt commands.
ssh is not sneaky, but very clear and open from the system panel -you just need being aware that connecting this way gives access to the full Linux command set, which can bring huge changes to the system.
FWIW, my old NB doesn’t switch to maintenance the way you describe…

3

Sorry @Herve5 , but you really got most of my post wrong. Let me explain:

  1. Nextcloud Backup is an officially advertised Nextcloud product, and I have nothing to do with it, besides that I have proposed to include it into the stack shipped for Nextboxes behind that link to another forum post up there.
  2. The coherence between the Backup app experiments with my back then dated maintenance problems were just a guess, and I don’t even remember when exactly I stopped using it.
  3. I did express clearly though, that I no longer have the Backup app in use, but the maintenance problems persist, or rather… re-appeared three months ago.
  4. My dissatisfaction with the support is caused by this reply to my bug report, which was first ignored for a month, and then only answered “if you use SSH, there is no support”.
  5. Since the Nextbox was advertised with a “zero administration overhead” proposal (“we do all the updates for you”, “please ignore any warnings in the admin overview”, “never mind the software being almost EOL, we are on it”, and so on), I certainly do expect the Nextbox people to take serious care of the well-being the product I bought. Unlike just installing some FOSS product on some hardware, I have spend quite a lot of money for a raspberry pi and a hard drive in a nice metal box, so I think it is ok to expect that it works at least as well as if I would have assembled this on my own.
  6. High fives for also being crowdfunding supporter and day one user :+1:

Back on topic, I am here to get help with how to find out, what causes the maintenance modes. Who can help?

4

Did you try checking your version, specially if like me you got one of the earliest releases? Did you then update it?
Something like the following (that I received from support@nitrokey.com last year) :

sudo systemctl stop nextbox-compose
sudo systemctl stop nextbox-daemon
sudo dpkg --configure -a
sudo apt-get update
sudo apt-get install -yq --fix-missing --fix-broken --autoremove nextbox
sudo apt-get -yq autoclean

then sudo reboot

5

please don’t say that - that’s just provoking. I guess you can very well understand why we have to draw this line - otherwise we would open the door for generic linux administration support all the way - on the other side I cannot remember one person who was left behind with no help at all with or without ssh…

generally maintenance happens during updates, but this is far too often - we recently added automated scanning of the files - maybe this is too excessive for your system and takes too long. Do you have a hdd setup with many files maybe?

Ok, we could try switching this off, as you’re on ssh (without support - lol) anyways:
This is where this happens: nextbox/daemon/nextbox_daemon/jobs.py at master ¡ Nitrokey/nextbox ¡ GitHub
please comment out this line and make sure it still runs which means we need a pass there, you should find the file here: /usr/lib/python3/dist-packages/nextbox_daemon/jobs.py … the part should then look like this:

        try:
            # re-index all files
            #self.nc.run_cmd("files:scan", "--all")
            pass
        except NextcloudError:

and then just do a reboot … make sure the nextbox-daemon service has started correctly and wait for the 24h to pass …

fun fact: this might be really the thing we’re hunting here (together with the certs) as both run every 24h … but maybe you could also try --shallow instead of --all … maybe the latter is simply to heavy for bigger file amounts - anyways it might be a good idea not to run them at the same time …

6

According to my experience, this might actually be related to the Nextcloud Backup app.
I never had the issue of Nextbox going unexpectedly into maintenance mode (and remaining there).

Only after I had installed the Nextcloud Backup app (which was recommended by Nextbox staff - see this German blog post, English comment “Submitted by meissner on 29. Juli 2024 - 14:13”), the same thing occurred: the Nextbox went into Maintenance mode for the backup and didn’t return to online mode again. I had to switch off maintenance mode manually.
SoI deactivated the scheduled backup, which, however, didn’t help - the Nextbox went back to maintenenance mode the next day and I had to bring it up to online mode manually again.

After that, I uninstalled the Nextcloud Backup app again and don’t experience the issue anymore.

1 Like
7

FWIW, while I never installed a backup at Nextcloud level, I do have an external USB drive on the Nextbox, and I (manually) configured two backup sets on it; in the NB management panel.
One of these two backups contains “even” in the name, and the other “odd”.
Now, each time I do large file movements on the Nextbox, I go to the html management interface and trigger (manually) a backup on the ext. HD, selecting the backup set whose name corresponds to the parity of the day.
This way I always keep two recent backups, while avoiding propagatig undetected failures, at least before my next triggered backup…

8

Thank you very much for that hint. As reported, I was no longer using the app for quite a while. However, I also had only turned off doing the Backups, but left the app enabled. After reading your post, I had disabled and uninstalled it immediately. Since then, about three days ago, there were no more maintenance modes. So I would consider the Nextcloud Backup app being the source of that behaviour.

I still wonder, why it started (again) at the end of July 2024 after not happening for over a year. Will keep an eye on this and get back here in case of new findings. Thanks again for the hint :+1:

The certs issue must have been a different thing after all. More about it over there.

1 Like