On Sunday, April 8th, around 9 PM GMT, Genius Scan suddenly stopped working—not just for a few users or because of a faulty update, but for all our iOS users. Every version of the app, across all devices, started displaying the same cryptic and frustrating error: “Expired key. Please ensure that the license key is valid.” Scanning functionality was completely broken.

An old, rusty key

Luckily, we quickly figured out what went wrong. Genius Scan relies internally on our own Genius Scan SDK, and to enable it in the production app, we use an internal “free” license key. Back in 2015, we generated a key valid for ten years and hardcoded it into the app. At the time, we were a five-year-old company, and ten years felt like forever. But ten years later, right on schedule, that key expired.

The fix itself was easy: generate a new key, update the SDK, and release a new version of the app. But because Genius Scan is a native iOS application, deploying that fix wasn’t immediate. Apple needs to approve all updates through their App Store review process. Fortunately, by around 10 PM, we had submitted the fix and requested an expedited review. Apple approved it quickly, and the update was live not long after.

However, an update being available in the App Store doesn’t mean users immediately install it—or even see it. As the night went on, our support inbox exploded with messages from frustrated users. We’re talking multiple thousands of emails. We immediately started triaging the flood, while setting up workflows to automatically respond to common questions. Our initial responses recommended updating the app, but it quickly became clear that even that wasn’t straightforward. Some users didn’t know how to update apps. Others couldn’t see the update yet, since it takes time to propagate across App Store regions. We kept improving our automated replies to better guide users through the process, iterating based on feedback and support patterns.

Beyond that, three major complications made things even harder.

First, the error message led some users to believe they had to pay to unlock the app. We will never trick a customer into buying something as we don’t rely on deceptive design. So we had to explain the situation and help them request refunds from Apple—something developers can’t do directly.

Second, many users tried the old trick of deleting and reinstalling the app. That sometimes solved the issue because they got the update when reinstalling the app, but this also deleted their locally stored documents (Genius Scan doesn’t force you to use an account, for privacy reasons, but nowadays, people are so used to having accounts for everything they don’t realize data may be stored locally.) So, we had to spend a lot of time helping users recover their data when possible.

And finally, about 30,000 of our 4 million monthly active iOS users are still on iOS 15 due to older devices. Since Genius Scan requires iOS 16+, these users had no way to get the update and were stuck. We decided it was only fair to release a version compatible with iOS 15 so those users could keep using the app, and we have been working on retro-compatibility this week.

This was one of the more stressful events we’ve experienced at The Grizzly Labs. We’re still catching up with support and helping users who are just now encountering the issue. At times like this, the millions of active monthly users become very real. We’re very sorry for the disruption—Genius Scan is a tool many of you rely on daily, and we hated to see it break in such a fundamental way. At the same time, we were blown away by the kindness and understanding of our users. Amidst the chaos, we received so many thoughtful messages thanking us for the app and encouraging us as we worked through the problem. That was the unexpected upside to a rough situation, and it meant the world to us.