Service Availability Issues (Now Resolved)

There have been intermittent issues with accessing KashFlow over the past 24-48 hours.
This affected about 20% of our customers and made the system too slow for them to use.

This is of course totally unacceptable for an application that you rely on for your accounting needs.
I feel personally responsible for the disruption to your business and I am very sorry this happened.

I think I’m right in saying that in our 6 year history this is the first time we’ve had an issue so severe that it warrants a blog post to explain and apologise.

What Happened?

On Monday we rolled out a big new feature, PDF Themes. With any big change comes a big risk. So we tested the hell out of it.

What we simply hadn’t done adequately enough was to load test it. In simple terms that means seeing what happens when thousands of people try to use it simultaneously.

As the feature was gradually extended to more and more users between Monday and today, we saw exactly what happens when thousands of people use it: it eats up memory and kills the service.

We made two mistakes here. First, the problem could have been detected before release by giving the new feature a thorough load test. We just weren’t thorough enough.

Secondly, we should have identified the issue and restored normal service much quicker than we actually did.

The speed of resolution was further impacted by the fact that my entire technical team and I are out of the country this week.

We’ve now removed the new feature and the system is back to its usual stable self. We will re-release the feature once we’re happy it isn’t going to cause further problems.

Again, I apologise for the impact this has had on your ability to run your business. We’re here to simplify and automate running a small business and we failed in that mission today.