No automatic renewal of Let's Encrypt certificate

Hi there,
I configured my nextbox with Guided Dynamic DNS and a deSEC URL - which worked quite fine until now. The Let’s Encrypt certificate expired a couple of days ago, but the Nextbox did not automatically renewed it - which I assumed should be the case?

So currently my Nextbox can’t be reached via a secured connection and I just can’t find a possibilty to manually do the renewal…did I miss something in the docs? The running unsecured connection still works well but is blocked by some browser / applications.
In addition, now the Guided Dynamic DNS configuration fails for the reachability test, which also worked fine before. But an unsecured connection from outside is still possible, though.

For my settings: the deSEC URL xyz.dedyn.io can be resolved properly for IPv4 and IPv6, the Nextbox HTTPS/TLS config says it is activated…with the failed reachability test, though.
My FritzBox settings are set with all the needed port forwarding (IPv4 & IPv6), DNS-Rebind, DNS prefix IA_PD with DNS address IA_NA)

Any hints on this? How can I get my Nextbox to renew the Let’s Encrypt certificate?

Regards, Matthias

1 Like

Hey @foerster,

this should not happen, the NextBox takes care of the renewal.
In “System Settings” you can download the logs, there you’ll find “nextbox.log” inside, once a day the log should contain some line containing “RenewCertificates”, that roughly look like this:

2021-09-27 15:56:07,523 [D] worker    starting worker job: RenewCertificates (last 2 messages repeated 65.0 times)
2021-09-27 15:56:07,541 [D] cmd_run   started ['certbot', '--config-dir', '/srv/letsencrypt', 'renew'] (blocking)
2021-09-27 15:56:08,962 [D] worker    finished worker job: RenewCertificates

This command should ensure that certificate renewal is done once a day …
Can you please check if these lines occur during the last days?

It should be possible to skip the certificate check, if you use https, the plan that it’s not needed to manually do the renewal, but you could do so using the command above.

As the certificate is not valid currently, the reachability test also fails, so this is expected behavior for your case.

So the quick fix is to run certbot --config-dir /srv/letsencrypt renew, but can I ask you to not do this yet? Please check the log first, would be important to know if there is something that keeps the mechanism from working.

Oh and there should also be a “letsencrypt.log” inside the log-zip you can download, you might also want to check it for errors…

best

1 Like

Oh, just got an idea, maybe apache is not reloaded: could you just reboot the nextbox using the reboot button inside “System Settings”, afterwards it might be ok.

released a version which takes care of the apache reload, chances are good that it is fixed in <24h automatically

Hi @daringer
thx for the fast reply. I just checked the logs. So the RenewCertrificates job seems to run (Nextbox.log):

2021-09-27 16:21:41,914 [D] worker starting worker job: RenewCertificates (last 2 messages repeated 42.0 times)
2021-09-27 16:21:41,937 [D] cmd_run started [‘certbot’, ‘–config-dir’, ‘/srv/letsencrypt’, ‘renew’] (blocking)
2021-09-27 16:21:41,939 [D] cmd_run blocking call, waiting now…
2021-09-27 16:21:51,940 [D] cmd_run blocking-cmd timeout (10secs)
2021-09-27 16:25:42,267 [D] cmd_run blocking call finished (last 2 messages repeated 11.5 times)
2021-09-27 16:25:42,268 [D] worker finished worker job: RenewCertificates

But there seem to be other issues with the resolving of the IP addresses or whatever. The letsencrypt.log says:

2021-09-27 16:25:42,052:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: xyz.dedyn.io
Type: connection
Detail: Fetching https://xyz.dedyn.io/.well-known/acme-challenge/…: Error getting validation data

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you’re using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.

Which confuses me a bit, as I think I set all IP/DNS and Port Forwarding settings correctly. Even the deSEC infos show the right IPv4 and IPv6 address for my Fritz.Box, so the DNS update works at least for this part.

A reboot of the Nextbox did not fix the problem.

Regards

mmmh, any chance you are using an underscore inside your subdomain name ? like foo_bar.dedyn.io ? guess not, or you wouldn’t have received a TLS cert…

mmh, really weird, do you have ssh access on the NextBox, can you try pinging the domain from within the NextBox?
This actually tells us that likely something is weird with your access to your NextBox from the “outside”, i.e., the internet is wrong.

  1. could you try if you can access your NextBox using e.g., your smartphone via the mobile network ?
  2. can you check if your NextBox’s local IP maybe changed, then your port-forwarding would not work anymore?

The DynDNS URL uses a “-”, but since everything worked fine till the expiration of the certificate that does not seem to be a problem.

The Nextbox is reachable from “outside”, i.e. my smartphone can connect via mobile network but with an unsecure connection, telling me the cert is expired since 5 days. The nextcloud app is connecting, too.

The Nextbox IP is fixed within my local LAN and hasn’t changed since I set everything up the first time.

So all in all, the Nextbox can be accessed from outside - but with an unsecured connection only (the cert is not renewed). This makes it not really usable in some cases, e.g. some browsers just refuse to connect unsecured. I’m running out of ideas and tend to set everything back to factory defaults and start from scratch.

This is really weird, just to be sure, the port-forwarding is active for port 80 and 443, right ?

You could also send the log-package to support@nitrokey.com so I can check the logs in more detail. In particular there should be a reason why the connection failed.

Yep, PortForwarding to the NextBox is set for 80 and 443, both for IPv4 and IPv6.

I’ll send the current log package, sure.

Hello guys,

I have the same problem as foerster describes.

> 2021-10-01 12:27:56,175:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:
> 
> Dohonsmain: abc.dedyn.io
> Type:   connection
> Detail: Fetching https://abc.dedyn.io/.well-known/acme-challenge/Mqk_b_JThF-8ObJEA0jZPkQ8CNVtD3FMAicfCrPlk0s: Error getting validation data
> 
> To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.

Would be great if you could tell me the solution. :slight_smile:

Thank you very much
Christian

Inside the log, please check what did not work exactly, ipv4? ipv6?
generally it applies what’s written there: the server is not reachable …

Well, well, well…

Looks like we have made some pretty good (although painful :face_with_raised_eyebrow:) progress on this topic.

tl;dr: ddclient is lacking the needed flexibility to support NextBox’s use-cases, therefore we decided to drop ddclient for the “guided dynamic dns” IP-Updates. A new release is already in the testing pipeline and is to be expected within the next few days, which makes the impression this issue and the “multiple-ipv6” addresses-issue will be resolved. *holding-thumbs*

some more details:

  • ddclient can simply not update the same domain with two different IPs, e.g., both IPv6 and IPv4.
  • currently desec.io (our guided dynamic dns-provider)“guesses” the IPv6 address, based on the request, which is sent by ddclient to update the IPv4 address, this “guess” is sometimes wrong (nothing special to have multiple IPv6 addresses for a single device)
  • Let’s encrypt is quite picky when it comes to resolving IPs to domains (well, obviously :smiley: ), so a wrong “guessed” IPv6 address, does not lead to your NextBox, thus Let’s Encrypt won’t grant you a certificate.
  • Workaround: you can login to desec.io and simply delete your IPv6 address (or correct it), then the certificate renewal will work without issues.
  • although we decided for ddclient to avoid re-inventing the wheel, we here hit the limit of what is possible with ddclient is reached (sure, various hacks/workarounds would somehow solve it, but nothing sustainable)
  • so from now on the guided-dynamic-dns updates are done by the nextbox-daemon

long story short, this issue should be resolved with the next release, the next days…

best

2 Likes

Hey thanks for the positive update. I am curious. For me it is the Ipv6. The IPV6 address that is transmitted to desec.io is different than the nextbox expects. But in the Fritzbox both IPV6 addresses are assigned to the nextbox. I would then wait for your update and only report if there are still problems.

Thanks

Hello,
I am still facing the same problem. Was the update with the fix already distributed?
Kind regards
Wolfram

Hey @Rhaban,

yes, the update was rolled out already, check for version 1.0.13 on your overview page.
Although, it seems, for some configurations it still does not lead to a properly renewed certificate.
We are already working on the next release (with even more safety on this end), but I am afraid this will take 1-2 additional weeks.

best

@daringer Thanks for your fast and positive response. I received the version you mention, but still experienced problems. For whatever reason it is working since yesterday, after I reset my desec.io password. Since I do not see any technical explanation, this might be pure coincidence, but I am fine now. Thanks again for your help.

1 Like

Till now I have a working TLS certificate which is still valid till end of february. However I’m interested in the point of time when the certificate will be renewed. As the IPv6 reachability is still causing problems in my case (no idea why) I don’t want to run into not having a valid certificate again, so I tend to ask this question early to start troubleshooting with patience if necessary :slight_smile:

so far I remember less than 30 days until expiry is when the renewal actually will be accepted, so if you see less than 29 days you/we should look closer.

edit: also starting from 1.0.16 there is no need for reachability (neither IPv4 nor IPv6) for certificate renewal (and acquire) using “guided DNS”, as the NextBox will use a DNS-driven verification mechanism. So I have some hopes, that generally the whole process should be far more robust and reliable.

best

1 Like

Halo Markus,
Thank you for this fast clarification!
What happens in the case of an ‘old’ Nextbox (from the initial crowdfunding) that has been steadily updated to version 1.0.16-1?
Should I understand ‘because of 1.0.16’ it’s OK, or on the contrary, my fossil certificate has to be updated?
TIA!
Hervé

Hey @Herve5,

there are no old NextBoxes, every NextBox out in the wild is equal, means nothing to worry, your certificate will update independent of how old it is.

best

1 Like