The mystery of the disappearing certificate

Wow – so after a couple of very frustrating days, we finally seem to understand a really annoying problem in our web application.

We first noticed a problem when a wildcard certificate (i.e. one with a name like “*.au.mycompanyname.net”) disappeared from ALL four of our UAT environments – 8 servers in total. We use this certificate for a number of critical components in both tiers of our app, and we had UAT in progress in 2 of those environments.

We quickly identified and fixed the issue, but we had no idea what the root cause of the issue was. A few hours later it happened again. After a lot of investigation (and some discussion with our infrastructure support teams) we finally identified a scheduled task that seemed to relate to the issue. The first hint was the history of the job (which lined up with the disappearing certificate), and after some quick testing we established that we could replicate the issue.

The task was a Microsoft task, and was simply named “SystemTask”. It was located in the Microsoft\Windows\CertificateServicesClient folder. From what we can tell, it relates to the certificate autoenrollment process. Now, at this stage it’s worth mentioning two things that seem to be very specific to the issue:

1. The wildcard certificate was 6 weeks away from expiry
2. There were also server specific certs also installed on the server (e.g. servername.au.mycompanyname.net).

From what we could tell if both of these conditions were present during the autoenrollment process then it would fail for that specific certificate, which would then be dropped from the store. However, if we resolved either of the above two issues then the certificate was left untouched, as expected.

This ended up being a fairly simple fix – we simply issued a new cert from our internal Certificate Authority (expiry ~2020). It’s something we needed to action anyway within the next month. But this was a very frustrating issue for us, especially as we focus on building trust with our (many) stakeholders!

I’m posting this here on the off chance this might help someone else in future!

Photo Credit: jointcracker