F5 ASM shenanigans

So we had a really obscure issue the other day. We’d just pushed from our UAT to our staging environment, and users started reporting a really odd bug.

The symptom was in a list of values we pushed to the browser – a specific value (0.430000000000000) had been replaced with stars (0.***************). There were about 100 items in the list, but this was the only one affected. The issue was intermittent, and we could find no reason for this behaviour.

After studying the issue for a day we were able to isolate it to a single data centre, which had two servers behind an F5 load balancer. After tossing around a number of ideas we threw out the idea that maybe something outside our application was modifying the page. We’d only recently been talking to our infra team and had been discussing the F5 security module (ASM). We’d specifically noted it had a feature to obfuscate customer data on pages, such as credit cards.

Sure enough, a quick check of the Luhn algorithm showed that this number did in fact pass the test. We talked to the infra team, who confirmed the ASM was turned on only for the affected data centre. Turning it off solved the problem.

Interestingly I’m told the ASM was in passive mode (i.e. watching and recording only), and was not even configured to look for potentially confidential information. Further, there were other items on the page that logically should also have been obfuscated.

We’ve still to identify why exactly this happened, but I’m noting this here in case some other poor soul has the same issue.

Image Credit: Ethan Lofton