How Jenkins Can Stay Relevant in the Next Decade

Continuing from my previous article, I want to discuss how Jenkins remains relevant in 2024 and will continue to be crucial in the microservices and container world over the next decade by adopting principles from other open-source projects.

Why Jenkins Still Has a Place in 2024

Let us start with some stats.

Despite the rise of modern, cloud-native CI/CD tools like GitHub Actions and CircleCI, Jenkins remains a heavyweight in the continuous integration and delivery space. Holding an estimated 44-46% of the global CI/CD market in 2023, Jenkins continues to be widely adopted, with more than 11 million developers and over 200,000 active installations across various industries (CD Foundation, CloudBees). 

This widespread usage reflects Jenkins' strong position in enterprise environments, where its robust plugin ecosystem and extensive customization options continue to deliver value.

One of Jenkins' major strengths is its extensibility. With over 1,800 plugins, Jenkins can integrate deeply with legacy systems, internal workflows, and various third-party tools, making it an essential part of many large-scale and complex projects (CloudBees). In industries where infrastructure and application delivery rely on specialized or customized workflows — such as finance, healthcare, and manufacturing — Jenkins' ability to adapt to unique requirements remains unmatched. This flexibility is a key reason why Jenkins is still preferred in enterprises that have heavily invested in their CI/CD pipelines.

Moreover, Jenkins continues to see substantial growth in its usage. Between 2021 and 2023, Jenkins Pipeline usage increased by 79%, while overall job workloads grew by 45% (CD Foundation, CloudBees). These numbers indicate that, even in the face of newer competition, Jenkins is being used more frequently to automate complex software delivery processes.

Another factor contributing to Jenkins' staying power is its open-source nature and community support. With thousands of active contributors and corporate backing from major players like AWS, IBM, and CloudBees, Jenkins benefits from a large knowledge base and ongoing development (CD Foundation, CloudBees). This ensures that Jenkins remains relevant and adaptable to emerging trends, even if its architecture is not as cloud-native as some of its newer competitors.

While Jenkins may not be the go-to for modern Kubernetes or GitOps-focused workflows, it plays a critical role in on-premise and hybrid environments where companies require greater control, customization, and integration flexibility. Its deep entrenchment in enterprise systems and ongoing improvements ensure that Jenkins still has a crucial place in the CI/CD ecosystem in 2024 and beyond.

Jenkins to a Prometheus-like architecture and strengthening the plugin system is insightful and could address several of Jenkins' current challenges, especially regarding scalability, plugin complexity, and reliability. Let’s break it down and explore how this shift could help Jenkins remain competitive in the future.

1. Prometheus-Like Architecture: Moving to Statelessness

Currently, Jenkins operates in a stateful architecture, which complicates horizontal scaling and cloud-native operations. Adopting a Prometheus-like architecture — which is stateless and relies on external storage — could drastically improve Jenkins’ scalability. Prometheus, a popular monitoring tool, uses an exporter model, gathering data without retaining state internally. Jenkins could benefit from this model by offloading state management to external databases or services.

Benefits

• Improved horizontal scaling: By decoupling Jenkins' state from its core operations, it would become easier to spin up and scale multiple Jenkins instances without worrying about syncing the state across nodes.
• Resilience: In a stateless system, individual failures (e.g., job failures or plugin crashes) wouldn’t affect the entire server. This makes the system more resilient to downtime.
• Kubernetes-native: A Prometheus-like architecture would align better with Kubernetes, where stateless microservices are the norm, allowing Jenkins to thrive in modern cloud-native environments.

2. Strengthen Plugins to Work With Web Servers

One of Jenkins' key strengths is its extensive plugin ecosystem, but as you rightly point out, plugin management can be a nightmare. Strengthening plugins to work more autonomously — perhaps interfacing directly with web servers or microservices — could alleviate many of Jenkins' pain points.

Proposal for Plugin Architecture

Jenkins Proposed Architecture 

• Segregated plugin operations: Plugins should be run in isolated, self-contained environments or sandboxes, much like how Docker containers operate. This would ensure that plugin failures do not bring down the main server, enhancing reliability.
• Web server-driven: Plugins could act as services that register with a central web server (Jenkins core). If a plugin is needed for a specific build step, the server could query the plugin service, much like how microservices communicate through APIs. This would allow each plugin to manage its own state and dependencies independently of Jenkins’ core.
• Clearer API  contracts: By having plugins act like web-based microservices, Jenkins could enforce clear API contracts that streamline interactions between the core and plugins, reducing the risk of misconfigurations.

3. Segregation of Duties: Isolate Plugin Failures

One of the most frequent complaints about Jenkins is that plugin failures can cause major disruptions. Your proposal to segregate duties so that plugins do not affect the overall server is crucial.

How This Could Work

• Plugin sandboxing: Each plugin could be run in an isolated process or container. If a plugin fails, the failure is contained to that specific process, and Jenkins’ core server remains unaffected. This is akin to how web browsers handle tabs: if one tab crashes, the browser stays up.
• Service mesh: Jenkins could adopt a service mesh architecture similar to tools like Istio. Each plugin operates as a service, with a central Jenkins server acting as the orchestrator. This would allow the core to handle orchestration and state management without being tied down by plugin-related failures.

4. CI/CD as a Cloud-Native Service

With Jenkins' existing adoption in enterprise settings, transitioning to a cloud-native CI/CD service with this new architecture would allow it to better compete with tools like GitHub Actions and Argo CD, both of which are tailored for cloud-native environments.

Additional Considerations

Native Support for Containers and Microservices

With plugins functioning independently, Jenkins could be fully optimized for containerized environments. For example, each build step could be executed as a separate container, making it easier to manage resources and scale as needed.

Enhanced Monitoring and Observability

Borrowing from Prometheus’ architecture, Jenkins could adopt a more observability-focused model, with built-in metrics and monitoring services. This would give DevOps teams more insight into the health of their pipelines, helping to reduce the complexity of managing large installations.

In Conclusion

Transitioning Jenkins towards a Prometheus-like, stateless architecture and restructuring plugins to operate independently would provide significant improvements in scalability, resilience, and deployment flexibility. This approach would enable seamless scaling by decoupling state from individual Jenkins nodes, reducing dependency on centralized databases and simplifying failure recovery. Additionally, independent plugins would facilitate faster development cycles, enhance fault isolation, and improve overall system performance by allowing each plugin to function autonomously without impacting others.