Maintaining Database Connectivity in Serverless Infrastructure With TiProxy

Businesses often struggle to ensure smooth database connections during scaling and upgrades. Frustrations can surface from underused instances after scaling out, disruptions when scaling in, and potential disconnections during upgrades.

While connection pools and long-lived connections offer stability, they can cause issues in fluid environments where databases need to adapt quickly. This leads to increased overhead, wasted resources, and dissatisfied customers.

TiDB Serverless is a fully managed cloud DBaaS for auto-scaling workloads built with TiDB, an advanced open-source, distributed SQL database. While TiDB Serverless enables auto-scaling and database upgrades without downtime and manual intervention, TiProxy — a powerful connection management tool built into TiDB Serverless — works its magic and tackles the above-mentioned challenges.

In this post, we’ll explore how TiProxy maintains database connectivity and optimizes resource allocation without impacting business.

Understanding the Multi-Tenancy Architecture of TiDB Serverless

Before diving into the connection challenges in serverless databases, let’s look at the underlying architecture of TiDB Serverless. In a nutshell, this DBaaS uses a multi-tenancy model with a shared gateway. This gateway directs client connections to TiDB server instances for each tenant. Additionally, to achieve quick response times, TiDB Serverless has a pool of idle TiDB server instances provisioned and standing by for instant allocation when needed.

Figure 1. A diagram of TiDB Serverless’ architecture. 

Connection Challenges With TiDB Serverless

One of the platform’s key features is its auto-scaling capability, which adjusts the TiDB server instances based on the current workload. On the other hand, many clients utilize connection pools to establish long-lived connections with a specific TiDB server instance. While this approach generally works well, it can pose several connection-related challenges in a serverless context:

• Underused instances after scale-out: When TiDB Serverless scales out a TiDB instance to accommodate the increased load, the newly created instance may remain idle. This is because existing clients continue to connect to the original instances, resulting in underused instances and sustained high response times. In some cases, clients have to reset their connection pools to address this issue.
• Scale-in disruptions: During a scale-in operation that involves reducing the number of TiDB instances, clients connected to the instance being removed will experience disconnections. While some clients may automatically reconnect, this disruption can lead to operational challenges. Hence, it requires careful scheduling to minimize its impact.
• Upgrade-related disruptions: In case of database upgrades, the process recycles old TiDB instances and allocates new ones. Consequently, clients connected to those old instances will also experience disconnections.

Manage Connectivity Efficiently With TiProxy

TiProxy is a dedicated component in the gateway layer. Residing between the load balancer and TiDB server instances, TiProxy efficiently manages connections and ensures the system remains agile and responsive. In this section, we’ll explore the mechanisms and features of TiProxy:

• Dynamic load balancing for TiDB servers
• Enhanced auto-scaling for TiDB servers
• Facilitating online upgrade of TiDB servers with zero-downtime

Dynamic Load Balancing for TiDB Servers

TiProxy can seamlessly migrate existing backend connections from one TiDB server instance to another on the fly. This migration process involves establishing a new connection on the target TiDB server instance and gracefully dropping the previous one. As a result, TiProxy can rebalance the SQL workloads across TiDB servers, ensuring optimized performance and resource utilization.

Figure 2. TiProxy migrating existing backend connections

Database sessions are inherently stateful, encompassing various critical aspects like session variables, user information, prepared statements, and more. TiProxy is responsible for managing these session states and ensuring a seamless experience for clients. During the connection migration, TiProxy adeptly handles the transfer of session states, making it appear as though the backend hasn’t undergone any changes.

Enhanced Auto-Scaling for TiDB Servers

TiDB Serverless can scale TiDB servers automatically and facilitate optimal resource allocation based on workloads. TiProxy enhances the auto-scaling feature in the following ways:

Effective Scaling-Out

During the scaling-out process, TiProxy continuously monitors the backend connections on the TiDB server instances to maintain a balanced state. If it detects any connection skews, such as newly-added TiDB instances sitting idle, TiProxy will migrate existing backend connections from the old instances to the new ones.

Coordinated Scaling-In

The process of scaling in TiDB servers within TiDB Serverless involves close coordination between the Gateway and TiProxy. When a scale-in is scheduled, the gateway notifies TiProxy about the specific TiDB server instance for recycling. In response, TiProxy orchestrates the migration of backend connections away from that instance. This ensures uninterrupted client connectivity and service continuity.

Recycling All TiDB Server Instances for Idle Workloads

TiDB Serverless will recycle all TiDB server instances when a tenant’s workload becomes idle. When a new connection request arrives, TiProxy alerts the Gateway and waits for a new TiDB server instance to be assigned to the tenant. This normally takes only half a second, as the TiDB server instance is assigned from the pool. Once TiProxy gets the new instance, it re-establishes the backend connections. This ensures that query processing resumes smoothly, offering a seamless user experience even during idle periods. 

Preserve Connections While Upgrading

TiDB Serverless consistently provides users with the latest features every few months. Thanks to TiProxy’s ability to maintain connections, these upgrades are rolled out to users seamlessly and without disruption.  

Proactive Upgrade Preparations

Before initiating an upgrade, TiDB Serverless scales out by temporarily adding a new TiDB instance. This strategic action mitigates potential performance degradation during the upgrade process. The actual upgrade is then executed in a rolling fashion, ensuring that a sufficient number of healthy TiDB instances are always available.

Streamlined Upgrade Workflow With TiProxy

When it’s time to phase out an older TiDB version, TiProxy efficiently shifts backend connections away from the outdated instance. As new versions are introduced, TiProxy seamlessly migrates backend connections to the upgraded instance. This means that client connections remain intact throughout the entire upgrade process, so there is no need for any manual intervention on the user’s part.

After the upgrade, TiProxy continues to monitor and balance the TiDB instances. This ensures optimal resource utilization and performance. 

Real-World Impact: How OSS Insight Benefits from TiProxy

TiDB Serverless is trusted by a wide array of applications, including OSS Insight — a comprehensive open-source software tool that analyzes massive events from GitHub. TiDB Serverless allows OSS Insight to effortlessly manage fluctuating traffic without requiring manual scaling — a capability put to the test when the tool achieved a Top 10 ranking on Hacker News.

On the day OSS Insight gained this spotlight, its website traffic surged to seven times the usual volume. While such a spike could have overwhelmed many databases, TiDB Serverless effortlessly adapted. It autonomously scaled out to meet the sudden increase in demand. Once the peak traffic subsided, TiDB Serverless accurately scaled back in, ensuring efficient resource allocation and cost-effectiveness.

Throughout this volatile period, TiProxy played an invaluable role. It deftly managed all the backend connections, allowing the engineering team at OSS Insight to remain focused on their primary responsibilities rather than worrying about database stability.

Conclusion

TiProxy is more than just a feature. It represents a solid solution that eliminates manual intervention or client adaptation when using TiDB Serverless. With TiProxy working behind the scenes, users don’t have to worry about scaling or upgrading and can stay focused on their core objectives.  

It’s worth noting that TiProxy’s benefits extend beyond TiDB Serverless. In environments like TiDB Dedicated or TiDB Self-Hosted clusters, connection-related challenges can still arise when users manually initiate scaling or upgrades. In these scenarios, TiProxy serves as a versatile tool that effectively addresses these issues, ensuring a seamless experience for users.

In the meantime, as a dynamic database proxy, TiProxy continually evolves to address a wide array of requirements. Its future enhancements aim to further support tenant isolation, optimize traffic management, and facilitate post-upgrade validation. For more details, please refer to the project’s GitHub repo.