Vanity Metrics for APIs vs Tracking Business Value From API Transactions

As an API product manager, you want your API to have a great developer experience. This means that developers can get up and running quickly, they get consistent behavior from your API, it’s easy for them to troubleshoot any errors they encounter, and your API makes it easy for them to address their business needs.

Tracking your APIs is an important part of understanding how well they perform, which leads most organizations to build out their own internal API tracking systems. While this approach can work out for some organizations, it is much more common for organizations to track metrics incorrectly. Tracking the wrong metrics leads to an incorrect understanding of your API.

In this blog post, we’ll look at some of the common pitfalls companies face when they start defining and collecting data for how developers are using their API, and how you and your team can focus on real, actionable metrics and not just vanity metrics.

Why Internal API Analytics Tools Miss the Mark

Some organizations prefer to build their own API analytics tools, and track metrics internally. When organizations build homegrown analytics tools for API metrics, they encounter certain pitfalls that lead to inaccurate metrics:

• Rebuilding website analytics for APIs
• Trade-offs between building products and building analytics tools
• Pressure to report the most successful-looking metrics
• Failing to appropriately measure API transactions

It’s common for product teams to take the tools they use for website analytics and repurpose them for API analytics. This approach often involves considering API calls equivalent to site views, and popular endpoints as a similar metric to popular pages and doesn’t have a method of analyzing developer experience.

For example, traditional web analytics will lead to reporting at the endpoint or call volume level when that’s rarely what matters. Sometimes more granular information within the response tells a better story. Or, perhaps aggregating additional request and response meta-data is more important than the usage of a single API endpoint.

When you try to fit your API into existing analytics tools, you may misinterpret the results. An organization may falsely believe that they’re receiving many API calls because their API is popular, for example. The real story may be that the developers accessing it are experiencing frequent errors, or that certain endpoints are unpopular when developers may be having fewer, but longer conversations with your endpoints. If your internal development team doesn’t have experience building API metrics tools, they may not be the right team to build your API analytics software.

Unless you have unlimited engineering resources, when the question of building API analytics tools comes up internally, it always means diverting developers from developing the product your business is focused on to working on API analytics tools. We’ve already discussed how this isn’t ideal if your development team doesn’t have experience building API analytics tools. There is also the additional downside of less time spent building your product because developers are focused on other projects. Your developers would also most likely prefer to focus on your product instead of building out analytics tools. Building internal analytics tools puts a different type of pressure on the teams that develop them.

Don’t Trust Vanity Metrics

When internal teams have to build API analytics tools, they are also often burdened with the pressure to report good metrics for the API. Nobody wants to be the bearer of bad news, and many organizations will ask their internal metrics teams for good metrics to report to motivate their co-workers, validate their success for investors and share data for marketing purposes. This is what leads highly-skilled developer teams to report vanity metrics. They are faced with unintentional, but quickly compounding pressure to report metrics that look good, which leads them to focus on metrics that look impressive but don’t reflect how successful their API actually is.

As tempting as it may be to share a meaningless hockey stick chart, there are more useful metrics than API requests over time unless you can tie the calls directly to revenue. Instead, consider the overall health and success of the API, such as how your latency improvements have impacted the developer experience.

Many organizations define the wrong success metrics, which leaves them with the wrong idea of how successfully their API achieves their goals. Vanity metrics are easy to create and they are especially common in products built for developers. Products and services built for developers often focus on the number of hours spent in development or the number of first-time users and API keys provisioned. Metrics like these don’t tell you anything about your API’s developer experience, how much value your API creates for developers, or how well your API aligns with your company’s broader goals.

Connect API Transactions to Business Value

Now that we’ve discussed the challenges of internal API analytics tools and how they can lead to vanity metrics, let’s have a look at what metrics to focus on so that you can understand your API’s performance and how well it achieves your business goals. Let’s start by looking at the developer experience for your API users.

Your API’s developer experience starts when they sign up to use it. You should track each developer’s journey starting from their initial sign-up to the first API key they generate followed by their first API calls and the length and content of their earliest API transactions. This data gives you a lot of information about your API’s developer experience, including:

• How easy it is for developers to start using your API
• How quickly they integrate it into their projects
• What endpoint looked most useful at first glance
• Where developers are most likely to encounter errors
• Which developers need support

This information makes it easy for you to further develop your API and improve your developer experience by providing quick support when they encounter errors while using your API. You can also see whether users from particular channels are more likely to reach a relevant level of API usage.

In the medium term, this lets you track how well your API retains developers, which is an important part of understanding how much value your API provides. You will also learn which endpoints have the highest number of transactions and how long a typical transaction lasts which signals where your resources should be focused.

The API metrics you track should include unique identifiers to make it easier to track usage data on an individual customer basis. This makes it easy to understand your API performance and makes your customer support lightning-fast. Customer usage is a crucial metric when it comes to tracking business value, it’s important to continuously track API transactions so that your organization clearly understands how well your API addresses your customer’s needs.

Your customers are also interested in metrics about their usage over time. You should provide each customer with their usage information so that they understand when they’re approaching usage quotas and don’t hit quota limits unexpectedly. They may also want to know how many network resources they typically use while accessing your API. This makes it easy for them to allocate enough resources to access your API. The endpoints they use most frequently or use to transfer the most data are also important metrics for your customers because this information tells them which endpoints provide the most business value to them. This makes for a great customer experience, which makes them more likely to use your API.

An important aspect of API analytics that you’ll need to consider in the long term is how you’ll scale your analytics. As your customers grow, the number of API calls will grow and the length and content of your API calls will grow as well. While it is already quite challenging to perform API analytics accurately, it becomes much more difficult as time passes and the sheer number of API transactions grows. Your API analytics implementation needs to accurately track business value and handle your API’s growth.

It’s difficult to build accurate API analytics tools on your own. Even experienced teams need a lot of resources to measure their first analytics, and you’ll need to constantly develop your metrics to get more detailed information for your organization, but this is no reason to give up. Great API metrics will bring huge value to your business, they make it easier for your product and support teams to improve your customer experience and your metrics show you what aspects of your API you should focus on.

Is Tracking API Transactions Effective?

So far we’ve covered the most common pitfalls organizations fall into when analyzing APIs, why they occur, and what metrics you should focus on instead. Now we’re going to look at a direct comparison between common vanity metrics and API transaction metrics for different use cases.

API adoption is a metric that is valuable to many organizations, so measuring it correctly is important. The popular approach is to look at the number of API keys that have been provisioned for an API. This suggests that an API is popular and a lot of developers find it useful, but relying on this metric is dangerous. It only tells you that many developers have considered using your API. This won’t tell you if they successfully implemented their first API call, what stage of their onboarding journey they reached, or how many errors they encountered. On the other hand, if you use unique identifiers to look at how well you retain users and track how long it takes to go from creating an API key to “Hello World!” you’ll know if you need to make improvements to your onboarding, documentation, and support.

Understanding your API can be challenging, but with the right analytics tools, you can maximize value for your customers and track how successfully you achieve your organization’s goals.