REST API Error Handling — Problem Details Response

I have been a REST API developer for many years and helped many companies to create APIs. One of the areas that are key to building a successful API is the error handling/response. Some time ago, I wrote about the "REST API Error Handling Best Practices," which became a very popular post. At that time, I wish that I knew what I know now about rfc7807, the standard that actually defines the interface of a correct and well-known error response.

Background

Each company and even sectors are trying to define their own error response philosophy, but the common structure that defines a "good error code" consists of three basic criteria in order to be truly helpful:

• An HTTP Status Code, so that the source and realm of the problem can be ascertained with ease. This code takes the form of a standardized status code in the HTTP Status Code scheme. By noting the status using this very specific standardization, you not only communicate the type of error, you communicate where that error has occurred (5xx means that it is a server issue, whereas 4xx means that the client did something wrong).
• An Internal Reference ID for documentation-specific notation of errors. In some cases, this can replace the HTTP Status Code, as long as the internal reference sheet includes the HTTP Status Code scheme or similar reference material.
• Human readable messages that summarize the context, cause, and general solution for the error at hand.

FHIR

If we take for example the healthcare FHIR standard, the error is defined there as an FHIR resource called operation outcome, here is an example of the error response:

{
 "resourceType": "OperationOutcome",
 "id": "exception",
 "text": {
  "status": "additional",
  "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\">\n<p>SQL Link Communication Error (dbx = 34234)</p>\n</div>"
 },
 "issue": [{
  "severity": "error",
  "code": "exception",
  "details": {
   "text": "SQL Link Communication Error (dbx = 34234)"
  }
 }]
}

The concept is very good and the interface (https://www.hl7.org/fhir/operationoutcome.html) is very generic and can be used for many responses, even successful ones. However, in order to use it, you need to know the FHIR standard from A-Z and for some, the healthcare concepts are "a bit" overkill.

Google

Some companies, like Google, are using the following error response for their APIs: 

{
 "error": {
  "errors": [{
   "domain": "global",
   "reason": "invalidParameter",
   "message": "Invalid string value: 'asdf'. Allowed values: [mostpopular]",
   "locationType": "parameter",
   "location": "chart"
  }],
  "code": 400,
  "message": "Invalid string value: 'asdf'. Allowed values: [mostpopular]"
 }
}

https://developers.google.com/doubleclick-search/v2/standard-error-responses

I just think that the JSON structure is a bit strange — errors array inside an error object?

Facebook

For Facebook, the error response is different as well: 

{
 "error": {
  "message": "Message describing the error",
  "type": "OAuthException",
  "code": 190,
  "error_subcode": 460,
  "error_user_title": "A title",
  "error_user_msg": "A message",
  "fbtrace_id": "EJplcsCHuLu"
 }
}

Spotify

{
 "error": {
  "status": 502,
  "message": "Bad gateway."
 }
}

There are other examples out there, but I think that everyone will agree that most of the error code structures are different. How you reference links, what error code you generate, and how to display those codes is subject to change from company to company.

However, there has been headway to standardize these approaches; the IETF recently published RFC 7807, which outlines how to use a JSON object as a way to model problem details within HTTP response.

RFC 7807 to the Rescue

This document defines a "problem detail" as a way to carry machine-readable details of errors in an HTTP response to avoid the need to define new error response formats for HTTP APIs.

By providing more specific machine-readable messages with an error response, the API clients can react to errors more effectively, and eventually, it makes the API services much more reliable from the REST API testing perspective and the clients as well.

In general, the goal of error responses is to create a source of information to not only inform the user of a problem but of the solution to that problem as well. Simply stating a problem does nothing to fix it — and the same is true of API failures.

RFC 7807 provides a standard format for returning problem details from HTTP APIs. In particular, it specifies the following:

• Error responses MUST use standard HTTP status codes in the 400 or 500 range to detail the general category of error.
• Error responses will be of the Content-Type application/problem, appending a serialization format of either json or xml: application/problem+json, application/problem+xml.
• Error responses will have each of the following keys:
  • detail (string) — A human-readable description of the specific error.
  • type (string) — a URL to a document describing the error condition (optional, and "about:blank" is assumed if none is provided; should resolve to a human-readable document).
  • title (string) — A short, human-readable title for the general error type; the title should not change for given types.
  • status (number) — Conveying the HTTP status code; this is so that all information is in one place, but also to correct for changes in the status code due to the usage of proxy servers. The status member, if present, is only advisory as generators MUST use the same status code in the actual HTTP response to assure that generic HTTP software that does not understand this format still behaves correctly.
  • instance (string) — This optional key may be present, with a unique URI for the specific error; this will often point to an error log for that specific response.

{
 "type": "https://example.net/validation-error",
 "title": "Your request parameters didn't validate.",
 "invalid-params": [{
  "name": "age",
  "reason": "must be a positive integer"
 }, {
  "name": "color",
  "reason": "must be 'green', 'red' or 'blue'"
 }]
}

You may provide additional keys that give the consumer more information about the error, also the Problem Details is extensible if a way that new problem details can be defined.

New problem type definitions MUST document:

1. A type URI (typically, with the "http" or "https" scheme)
2. A title that appropriately describes it (think short)
3. The HTTP status code for it to be used with.

For example, if you are publishing an HTTP API to your online shopping cart, you might need to indicate that the user is out of credit (our example from above), and therefore cannot make the purchase.

If you already have an application-specific format that can accommodate this information, it's probably best to do that. However, if you don't, you might consider using one of the problem details formats - JSON if your API is JSON-based, or XML if it uses that format.

To do so, you might look for an already-defined type URI that suits your purposes. If one is available, you can reuse that URI.

If one isn't available, you could mint and document a new type URI (which ought to be under your control and stable over time), an appropriate title and the HTTP status code that it will be used with, along with what it means and how it should be handled.

Example:

{
 "type": "https://example.com/problems/request-parameters-missing",
 "title": "required parameters are missing",
 "detail": "The parameters: limit, date were not provided",
 "limit_parameter_format": "number",
 "date_parameter_format": "YYYY-mm-ddThh:mm-ss"
}

Summary

An instance URI will always identify a specific occurrence of a problem. On the other hand, type URIs can be reused if an appropriate description of a problem type is already available someplace else, or they can be created for new problem types.

Implementations

Java : https://github.com/zalando/problem-spring-web

Node.js : https://www.npmjs.com/package/problem-json

Conclusion

I think that using it should be more and more common to use the proposed RFC-7807 — Problem Details specification, especially for its flexibility and simplicity. You can have your own custom error types, so long as you have a description of them to link to. You can provide as little or as much detail as you want, and even decide what information to expose based on environment (e.g., production vs development).

The advantages of using this can be a unification of the interfaces, making the APIs easier to build, test and maintain. Also, I think that more advantages will come in the future when more and more API providers will adjust to this standard. I believe that client and server libraries will spawn and find many usages and even AI that deals with common error resolution is not farfetched.