Convert API Log Data Into Actionable Information

You’ve built an API to solve technical problems, but you know that’s just the beginning. In addition to helping developers use it, you need to understand how they use it. You want to measure its performance and popularity and make adjustments based on what you discover.

Maybe some developers are seeing a lot of errors when making API calls or it’s taking too long for them to get to the first “Hello World.” Perhaps the number of developers converting to paying customers is below your expectations. You want to understand the usage of your API and ensure that customers use your API in the long term.

At this point, you might reasonably look to your API logs. You’re seeking deep insights into your API and its users, but by default what you’ll find is raw usage. How can you visualize not just the usage, but also insights that help you take action? This post will cover how you might approach this on your own.

Your API Logs Contain a Wealth of Actionable Information

The amount of valuable information you can obtain from your API logs may amaze you. For example, you can use API log data to answer a multitude of questions about your API, such as:

• What are the request and response payloads being sent?
• Which API version is used the most?
• Where is the API experiencing higher latency?
• What is the sequence of transactions leading to errors on POST /*/count/events?
• Which SDKs are used to access the API the most?

You can also use API log data to answer questions about the users of your API, like:

• What are the top endpoints used by each of my customers?
• Who are my top customers by API usage?
• Which users are scraping large amounts of data from the API?
• Which customers are running into 401 unauthorized errors?
• What is the API activity for a specific customer?

Your API logs contain valuable information that you can use to the benefit of your customers and your business. But how do you access and analyze that log data?

You Probably Pipe Your Logs Into a Data Visualizer

If you want to gain insights from your API log data, then you’re probably piping those logs into a data visualizer like Kibana. If you’re not familiar, Kibana is one of the most popular, if not the most popular, tools for log visualization. It’s part of the Elastic Stack (formerly ELK Stack), which consists of three open-source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine built on Apache Lucene. Logstash is a server-side data processing pipeline. And Kibana lets you explore and visualize Elasticsearch data.

Let's continue with the example of Kibana. Like any stack, you’ll need to put the pieces together. While they work well together, these are three separate tools also used apart. You’ll need to stitch together the customer journey data, from UI to API. These traces, as they’re called in Kibana terms, are important for turning data into something actionable. It’s not as simple as connecting logs and being done—you need to determine what’s needed, handle collisions, and assume the ongoing cost of growing indices.

Kibana is a log visualization tool, which makes it an appropriate place to start when thinking about API analytics. However, the primary use case for Kibana is infrastructure monitoring and metrics. It’s not designed specifically for API products, so there are some drawbacks to using Kibana for API logs and metrics. This is often true of homebrew solutions as well.

Pros and Cons of Visualization Only for API Logs and Metrics

If you’ve built an API product, you may already use a tool like Kibana or a custom-built solution to gain insights into its performance or how developers use it. Perhaps you’re considering using an open-source tool Kibana because of its popularity when it comes to log visualization. If you have a home-built solution, perhaps you've tailored it to your specific needs. Either way, it helps to know some of the pros and cons of sticking only to visualization for API logs and analytics.

Pros

• Can be free to use, or suited uniquely to your project
• Great at visualizing API logs.
• You can explore massive volumes of log data.
• In the case of Kibana, it's layered on top of Elasticsearch (also a con), making it ideal for use on high-cardinality, high-dimension log data- a must-have feature for API logs.

And now onto the cons.

Cons

• Compatible with Elasticsearch and Logstash only. If you want to use Kibana with other databases, you’re out of luck
• It may be designed for infrastructure metrics and not specifically for API products. For example, you have to customize Kibana for API log use cases.
• To connect user behavior and API activity into a single journey is often incredibly manual and prone to errors over time
• In the case of Kibana, maintaining the Elastic Stack takes a lot of effort. For example, you must periodically upgrade each part of the stack and make sure upgrades won’t break any plugins you’re using or require that you rewrite any of your visualizations.

Visualization is only a starting point when it comes to gaining insights from your API logs. And while a tool like Kibana is great at visualizing logs, it often leaves you wondering, “what’s next?” What action should I take based on this data?

API Analytics Yields Greater Insight Into Your API

A dedicated API Analytics tool is more than fragmented data visualization. It can extract actionable information from your API logs and guide you as to what to do with it. Ideally, an API analytics tool also provides a monitoring service that features a fast query engine and structured event-based data. Additionally, it should support a wide variety of API protocols including REST, GraphQL, JSON-RPC, Hypermedia (HATEOAS), and SOAP.

With this type of tool, you can sometimes filter and aggregate billions of API calls and user actions on pretty much any field, even high cardinality fields like a session token or user id.

A dedicated API Analytics solution helps you quickly understand what your data is telling you. When available, it is a great growth accelerant for any API-first project.

Understand What Your API Log Data Is Telling You

Extracting data from your API logs won’t help much if you don’t understand what that data is telling you. Both log visualization and API analytics can provide custom dashboards that help you make sense of the data obtained from API logs. However, unlike a simple visualizer, an API analytics tool can support dashboards designed specifically for API products, as well as embed templates and public share links. 

For example, let’s say a fintech company creates a dashboard that lets them track key API and user metrics, which include:

• Recent API errors
• HTTP status requests
• Product usage
• Daily active users (DAU)
• Most active users

Through the dashboard, the fintech API product team sees who their customers are, how they integrated the API with applications, and what users of their API are experiencing. The team also gains insights into how well their API is performing and how many developers moved from a trial or sandbox version of the API to a production version.

Ultimately, it is best to move beyond log visualizations and towards user behavior API analytics so that you can gain actionable insights from your API log data.