Forum outage 2020-01-14(ish)

[Martin Blank]

Not quite sure what happened, but the Apache2 service on the forums stopped. I got logged in and restarted it, and now, of course, it's up (at least while I'm writing this).

[Doc Giggles]

I was wondering what had happened. Glad you got it taken care of.

[nosystemd]

Thanks for fixing this.

[Deacon]

Thank you!

[Jamie Bond]

I appreciate your time on this

[BtEO]

Thanks.

[Rorschach]

Yay! We're back on!

Thank you.

[FirebirdNC]

I am as always both pleased and regretful to be here hanging out with my fellow nerdlings.

[Martin Blank]

I think I may have figured out what is causing it, but not why it's happening.

I use Let's Encrypt for all HTTPS certificates. The certbot process that it currently uses requires that it stop the HTTP service, does its thing, then restarts the service. What seems to be happening is that the service is getting stopped, but not necessarily restarted. The major reasons for this are the graceful stop of the HTTP service, and also that at least this morning's outage was roughly coincident with a failed cert update attempt.

There are a couple of possibilities for this, one of them being that the legacy "service apache2 stop" and "service apache2 start" commands are for some reason not properly executing (the modern commands are "systemctl stop apache2" and "systemctl start apache2").

Another is that certbot may have changed somehow in the last few weeks in a breaking way. This wouldn't be the first time certbot made a breaking change, especially regarding HTTP renewals. It does give me the impetus to make a move to the more stable DNS update method. I kind of know how that is supposed to work, but not entirely.

In the meantime, I have disabled the certbot service until I can get whatever it is figured out. Hopefully, that will at least get the server stable.

[Martin Blank]

Well, that didn't go as planned. Trying to make some things work, I broke some networking things, and then the server was completely unavailable. The provider's link to the VMWare remote console was broken, and it took them two days to fix. Frustrating as hell. Things are mostly working but not entirely. Trying to fix all the issues.

[Deacon]

Things are mostly working but not entirely. Trying to fix all the issues.

As frustrating as it is, it’s working now! Thank you