A cautionary tale on old settings
It's never good to prematurely declare victory, but...I think I fixed it.
We recently launched a launch page for a domain we've been building a platform for, and right off the bat, despite launching the exact same code on another site, there's been issues - primarily with the SSL certificate.
The sites are simple landing pages running on Blazor 8 apps (a bit overkill for what it is, but it's good practice) for a page with some analytics and a link to a newsletter sign up. The second domain is brand new, no history behind it. Deploys with no problem using an Azure App Service Certificate. When trying to do the same thing with the other one, however, was a complete mess.
The Problem
First it was 404 errors. A page shows up saying it can't find the site. Do some troubleshooting in the portal, open a support ticket, buy a premium SSL certificate, all to no avail. Go back to using an Azure App Service certificate again, and it seems to work, mostly, so stick with it and see what happens.
Next, it's time to look at the code and its relationship with the App Service. When running a website in Azure App Service, the connections are secure right up until they hit the datacenter - from there, it's unencrypted and delivered to the web app. Thinking that's the problem, we start to tinker with the startup settings. Disable HSTS, comment out this line:
//app.UseHttpsRedirection();
This idea came from a random Azure GitHub repo regarding Linux App Service architecture, but alas - for a moment it seemed to be the fix, until it wasn't.
The bottom line is that there's an old Let's Encrypt certificate that keeps rearing its ugly head from 2020 - but why?
When testing on mobile phones, on the mobile networks (no wifi), we're always getting ERR_CERT_AUTHORITY_INVALID errors with the old domain showing up (and failing).
Wat mean?
The Fix
Being an old domain, and something we've tried a number of things to use on it, there's a good bit of entries in the DNS. Thinking something is pointing to Let's Encrypt somewhere, we decided to take yet another look at the entries to see if we could match them up to something happening over there.
Sure enough - some AAAA records with IPv6 addresses seemed to be the culprit.
Thinking these were Let's Encrypt addresses, we removed them, and after a few minutes...the problem went away for our test case.
The Theory
Here's where it gets a little peculiar - those IP addresses are for Google Workspace.
The only thing that makes sense here is this theory: The server that received the Let's Encrypt certificate was hosted in Google Cloud. By having these records pointing towards Google, something in DNS/SSL world seemed to reach out in that direction, only to somehow retrieve this expired certificate.
The Conclusion
If you're working with a domain you've had for a while, it's best to take a close - I mean, really close - look at your DNS configuration to see if there's any possible overlap. This was by no means an obvious solution; in fact, you'd think it wouldn't be the fix that it was, since (presumably) those are Google Workspace addresses. Time will tell, of course, but it would make some(?) sense for there to be some (in our case) 'legacy datacenter overlap' of some kind.
This was a nightmare to troubleshoot, and an embarrassment for something so simple, but there was a lot of interesting things learned here, and I hope this is of some help to anyone else experiencing this kind of App Service certificate troubleshooting.
Enjoy!
Looking for help upgrading your web apps to Blazor? Contact us and let's see what we can do to help!