Minimizing Latency in Kafka Streaming Applications That Use External API or Database Calls

Kafka is widely adopted for building real-time streaming applications due to its fault tolerance, scalability, and ability to process large volumes of data. However, in general, Kafka streaming consumers work best only in an environment where they do not have to call external APIs or databases. In a situation when a Kafka consumer must make a synchronous database or API call, the latency introduced by network hops or I/O operations adds up and accumulates easily (especially when the streaming pipeline is performing an initial load of a large volume of data before starting CDC). This can significantly slow down the streaming pipeline and result in the blowing of system resources impacting the throughput of the pipeline. In extreme situations, this may even become unsustainable as Kafka consumers may not be able to commit offsets due to increased latency before the next polling call and get continuously rebalanced by the broker, practically not processing anything yet incrementally consuming more system resources as time passes.

This is a real problem faced by many streaming applications. In this article, we’ll explore some effective strategies to minimize latency in Kafka streaming applications where external API or database calls are inevitable. We’ll also compare these strategies with the alternative approach of separating out the parts of the pipeline that require these external interactions into a separate publish/subscribe-based consumer.

Challenges of API/Database Calls in Kafka Consumers

In a typical Kafka streaming application, the main source of latency arises when a consumer must wait for external systems like databases or third-party APIs to respond. Since Kafka Streams processes data in real-time, any blocking or delay in the response from these systems directly impacts the pipeline’s performance.

Traditional synchronous calls in consumers can be a significant source of bottlenecks due to:

• Network latency: Every request to an external API or database involves network communication, which adds latency.
• Response time variability: Some APIs may have unpredictable response times due to factors like rate limiting, server load, or other performance issues.
• Database I/O: Database operations, especially those that require complex queries or updates, may take longer than anticipated.

To address these issues, we need to adopt several key strategies that reduce the latency impact without compromising the real-time nature of Kafka Streams. 

Async Operations and Non-Blocking I/O

Overview

One of the most effective ways to reduce latency in Kafka consumers is to adopt asynchronous operations for API and database interactions. By using non-blocking I/O, the consumer can continue processing other records while waiting for the external system to respond, thereby minimizing idle time.

How to Implement

For API calls, you can use non-blocking HTTP clients such as AsyncHttpClient or frameworks like Spring WebFlux that are built on top of Reactor and provide reactive, non-blocking capabilities. You can also use Spring framework's @Async annotation to mark your method that contains API/database calling code as asynchronous and use CompletableFuture to orchestrate the response asynchronously. For database operations, consider using reactive database drivers like R2DBC for relational databases or reactive MongoDB drivers.

Benefits

• Prevents the consumer from being blocked during external calls, allowing for higher throughput
• More efficient resource utilization, as Kafka consumers can process other records while waiting

Trade-Offs

• Increased complexity, as you now need to manage asynchronous processing and handle responses properly
• Potential challenges with record ordering if not handled correctly

Pitfalls

If you are using Java and the Spring framework, you need to be careful and make sure you do not call Future.get() after your asynchronous call, because Future.get() is a blocking operation and will negate the benefits of using the asynchronous approach. Use one of the more advanced asynchronous abstractions like the CompletableFuture or Spring WebFlux. Below is an example of a code template that you can use for orchestrating responses from an asynchronous call using CompletableFuture.

CompletableFuture.supplyAsync(() -> {
    // Simulate API call or database operation
    return performApiCall();
}).thenApply(response -> {
    // Process the response asynchronously
    return processResponse(response);
}).thenAccept(result -> {
    // Handle the result (e.g., writing to Kafka)
    writeResultToKafka(result);
}).exceptionally(ex -> {
    // Handle any exception that occurred during the async operation
    handleException(ex);
    return null;
});

Batching of External Calls

Overview

Instead of making an API or database call for each individual record, you can batch records and send them in bulk. This reduces the number of external requests and leads to significant improvements in performance, particularly when dealing with high-throughput systems.

How to Implement

Kafka Streams provides the capability to batch records using its grouped operations. Once the records are grouped into batches, you can process them together and send one API request with multiple payloads or make a single database transaction.

Benefits

• Reduces the number of API or database calls, cutting down network overhead
• Lower API/database response time due to fewer, larger requests

Trade-Offs

• Batching introduces a slight delay as you accumulate records before processing, so you may need to fine-tune batch sizes and timeouts to avoid excessive delay.
• Memory overhead increases as you accumulate records in batches.

Using Kafka’s Punctuator or Scheduling

Overview

Another strategy to reduce the frequency of external calls is to leverage Kafka's Punctuator mechanism. Instead of performing API or database operations for every record, you can accumulate records in a state store and perform the external operations periodically.

How to Implement

The Punctuator allows you to schedule operations at regular intervals. For example, you could accumulate records and, every minute, perform a batch API call or database write operation.

Benefits

• Reduces load on external systems by limiting the frequency of API or database calls
• Helps maintain higher throughput by keeping the pipeline focused on real-time processing, while external operations happen at regular intervals

Trade-Offs

• Additional complexity in managing the accumulation of records and the timing of punctuations
• Careful consideration is needed to balance the trade-off between processing delay and throughput.

Caching API/Database Results

Overview

For consumers who frequently query external data (e.g., fetching user information from an API), caching can significantly reduce the number of external requests, leading to improved performance. By storing the results of API or database queries in memory, you can minimize external calls and serve subsequent requests from the cache.

How to Implement

You can use an in-memory cache like Redis, Hazelcast, or even Kafka’s local state stores to cache data. When a consumer needs data, it first checks the cache before making an API or database call.

Benefits

• Drastically reduces the number of external API or database calls, lowering latency
• Caching is ideal for frequently queried data that doesn’t change frequently.

Trade-Offs

• Cache invalidation becomes crucial — stale data can lead to inconsistent or incorrect results.
• Managing cache size and eviction policies is important to avoid memory bloat.

Comparing With a Publish/Subscribe Consumer Approach

An alternative approach to dealing with API/database latency in Kafka is to separate out the parts of the pipeline that require these calls into a separate publish/subscribe-based consumer. This strategy involves splitting the pipeline into two stages:

1. Stage 1: Main Kafka Stream - The first stage is a regular Kafka Streams application, where records are processed quickly without calling external systems.
2. Stage 2: Separate Consumer Group - In the second stage, a separate consumer group subscribes to a topic where records requiring external API/database calls are forwarded. This consumer group is optimized for interacting with external systems asynchronously.

Benefits of Publish/Subscribe Separation

• Isolation of latency: External interactions are handled separately, minimizing their impact on the main real-time pipeline.
• Fault isolation: Failures in API calls or database operations do not affect the core stream processing.
• Scalability: The separate consumer group can scale independently, allowing for better resource utilization when interacting with external systems.

Trade-Offs

• Increased operational complexity: Managing two separate stages introduces more complexity, especially in coordinating between the main pipeline and the consumer group.
• Higher infrastructure overhead: This approach may require additional infrastructure (e.g., another Kafka cluster or topic), increasing operational costs.

Conclusion

Latency in Kafka streaming applications that involve external API or database calls can be managed effectively by adopting strategies such as async operations, batching, caching, and circuit breakers. These approaches help to keep the core pipeline running efficiently while minimizing the impact of external system interactions.

However, for systems that experience frequent or unpredictable latency from APIs or databases, separating these parts of the pipeline into a separate publish/subscribe consumer group can be a more robust solution. This isolates the latency, making the core Kafka stream more resilient and allowing external operations to be handled asynchronously.

By carefully evaluating the trade-offs and requirements of your specific use case, you can choose the strategy that best fits your needs, ensuring both high throughput and low latency in your Kafka streaming applications.