Retrying after the inevitable failure: Idempotent HTTP Requests

Idempotence, while it may sound like something men get in their fifties, is a common property among web service resources. Many times it’s a given, but sometimes you need to explicitly facilitate it yourself due to specific business and reliability requirements. This post aims to clarify what is idempotence and how the property provides certain safety guarantees which help us develop more reliable and predictable web services. A subsequent post will detail how one can implement idempotence for endpoints which don’t implicitly provide it.

A request is idempotent if the intended effect of multiple identical requests is the same as the effect of one such request.

From the previous definition read-only requests are clearly idempotent. Additionally, DELETE and PUT requests should be implemented as such according to RFC7231. Lets break it down.

Idempotent requests

A read-only request like HEAD, GET or OPTIONS does not - in general - modify the applications state. They might produce some side-effects like writing access logs, but all in all they’re intended for transfering, not changing, state. No effect is the same effect, so they are idempotent.

PUT and DELETE on the other hand, are intended for state modification. Intuitively:

PUT should upsert a completely defined object. Calling PUT on the same resource with the same payload several times should result in just one object with the given data existing. The first PUT creating the object and the following ones updating it - but given that the payload is the same, the state effectively remains the same.
Calling DELETE on a uniquely identifiable object results in the deletion of just that object. While sending a DELETE request on an endpoint several times might end up with a different response - the state remains the same. You can’t really delete the same object twice. Bulk delete semantics sometimes break this rule depending on the implementation.

What are non-idempotent requests?

A request which increments a counter. Every subsequent call will increment the number, resulting in a different state each time.
A request which exclusively adds to a collection. E.g. adding a comment to a BBS or adding a post to Facebook. POSTing a comment twice results in duplicate comments.

Generalizing - a non-idempotent request, when performed multiple times with the same payload, results in multiple application state modifications.

Why idempotence matters?

Idempotence gives the client the ability to repeat a request without fear of causing trouble. This creates a more predictable, smoother and user friendly experience by reducing the problems user or network errors might introduce.

Lets examine a few scenarios.

Adding a post to a website

A user is attempting to write a post to their website via a CMS. The admin panel does so by sending a POST request to a specific resource on the server. The POST semantics are idempotent - specifically, the app sends the title of the post, a description, the post itself, and a unique slug. The slug uniquely identifies the post on the website. An impatient user clicks the post button again, which sends another request to the server - this request is rejected because a post with this slug already exists. After a while, just one post appears.

If the service didn’t provide an idempotent resource for this action, this would result in two posts being added to the website, with the user being forced to fix the problem manually themselves. E.g. if instead of having an explicit, user set identifier - the slug - the server itself generated a slug or other unique ID for the post.

Sending a money transfer request to a payment API

Lets say your server is the back-end of a shopping website. You manage user accounts, listings and shopping carts. Without idempotent semantics, a user can order the same stuff multiple times by accident. Or even worse your service might end up double or triple billing the same order, if not written very carefully to mitigate any networking issues in communication to the third party service. Retrying requests to the payment processor becomes non-trivial, prone to error and provider specific. On the other hand, with idempotent semantics you can rest assured the payment is guaranteed to occur only once with many requests.

Compared to the previous example, double billing or double ordering is a far more significant problem - in a lot of cases it’s very undesirable, needs a lot of steps to mitigate, and depending on the transaction, can be very damaging to the reputation of a company.

Wrap-up

Idempotence can prevent user errors and mitigate networking issues by ensuring that a given action results in the same state, no matter how many times we retry it.
Some errors can be costlier that others, with the benefits of idempotence easily outweighing the costs of implementation.
With that in mind, when choosing third-party providers, it may be wise to consider idempotence - especially with payment APIs.

In one of my next posts I will provide an example how you can make a non-idempotent resource idempotent by adding an IdempotenceID header to the interface. Stay tuned!