One of the challenges that face a team when they start thinking in terms of distributed systems – such as Microservices – is eventual consistency. We’re so used to writing software with transactions and ACID style consistency, that this can be un-nerving at first. Things that used to happen immediately, in the same action, are now in the background – and this creates some questions. Quite simply, eventual consistency is what you get when there are multiple components in a system, and it takes time for information to be spread between these components; given enough time, the information will spread, and become consistent, but before this happens different components may be out of sync with each other. When people experience this for the first time, it can seem weird and a bit un-nerving – and it can cause worries over the user experience.

Concurrency is a similar type of problem in software, where we are so used to developing on quiet systems with very few users and low risk of a conflict that we take concurrency for granted. Again, there’s a user experience element to this, where people are reluctant to tell the end user there’s been a concurrency conflict, or not sure how to handle it.

However, when we take a step back, these types of situation are not unique to computing – and most people have encountered these things in day to day life. Knowing this, we can make better informed decisions about how we handle the user experience around them in our software products.

Eventual consistency and Past-posting

Have you seen the 1973 Robert Redford/Paul Newman classic The Sting? It’s well worth watching if you haven’t, it’s a very enjoyable and highly acclaimed film. The main story is of a pair of con-artists, who plan to cheat their nemesis using an old trick known as past-posting. In the period the film is set in, before satellites and real-time communications, things like sports results would be relayed across the country to betting shops by a network of manual exchanges and telegraphs. This took long much longer, such that it was possible to get the information across the country by telephone fast enough that an accomplice could place a bet on a horse race having already been told the result by someone who was at the track. In this case, deliberately creating eventual consistency could be very profitable, albeit illegal!

Concurrency and today’s specials

In restaurants, it’s quite common for a waiter to bring a menu, and to explain to you what today’s special (non-menu) dishes are. Let’s say it’s something you want, so you ask for it – only to be disappointed a couple of minutes later, when the waiter returns after taking your order to inform you someone else just ordered the last portion before you. This is a classic concurrency problem; the information the waiter had when he started telling you the specials became inaccurate, so that by the time your transaction went through it was no longer valid.

Dealing with Consistency and Concurrency

Hopefully we’ve now seen that consistency and concurrency aren’t purely technical concerns – and as such the question of how we deal with them isn’t purely technical either. The first key lesson here is that in a system operating at any kind of scale, neither problem can be avoided. There are things you can do to minimize the exposure to timing problems, but obsessing over this too much is futile. It’s very likely there are real-world analogies to the situation you’re dealing with, and you can work with the business to use these to guide the system’s models.

An obvious example of this that we see is when booking hotel rooms or flights; Expedia now shows you how many rooms are left, and how many people are also looking at the same hotel. It’s entirely possible that by the time you go to complete your purchase the availability or price has changed. Meanwhile on many retail websites you may be buying an item that has limited stock – the concurrency problem here is normally resolved by cancelling your order and issuing a refund. Another valid approach if an item goes out of stock could be to send an email with a revised delivery date, if you’re expecting more stock soon.

Whenever we’re building a system, it’s always helpful to be mindful of the real world thing we’re modelling – and whenever there’s a decision to be made, the real world context can be a great guide.