Shared Platforms Explained: Navigating the Enterprise Ecosystem

In the world of enterprise technology, shared platforms like Kafka, RabbitMQ, Apache Flink clusters, data warehouses, and monitoring platforms are essential components that support the robust infrastructure leading to modern microservices architectures. We see shared platforms acting as mediators between microservices, aggregating logs from them, providing cross-domain analytics, and many other cross-cutting functionalities.

In this blog, I will explore shared platforms from the perspectives of both platform owners and platform users, revealing some best practices and strategies vital for a healthy technological ecosystem.

Shared Platforms: The Rationale Behind Setting Them Up

The existence of shared platforms in enterprise environments is born out of both necessity and strategic choice. Not every application can or should rely on its own isolated platform. Here's why shared platforms are a calculated decision for modern enterprises:

1. Specialized skillset requirements: The operation and maintenance of advanced platforms such as Kafka or Flink require a high level of expertise. Acquiring such specialized skills for every separate application is not practical or economical. Shared platforms allow for a pool of Subject Matter Experts (SMEs) who maintain and optimize these resources efficiently.
2. Centralized data sharing and management: In a microservices architecture, different applications often need to exchange data. Shared platforms facilitate this data exchange, acting as a central hub, which is more efficient than managing multiple points of integration between isolated platforms.
3. Cost optimization: The financial implications of licensing, infrastructure, and operational costs are significant when duplicated across multiple platforms. Shared platforms consolidate these costs, enabling businesses to benefit from economies of scale. By sharing resources, enterprises can optimize their investments and reduce overall expenses.
4. Resource utilization: Dedicated platforms can lead to underutilization of capacity, where servers and services might idle when the demand from a single application does not match the maximum capacity. Shared platforms ensure that resources are utilized more evenly and effectively, reducing waste of resources and improving overall return on investment (ROI).
5. Agility and scalability: Shared platforms offer a flexible foundation that can quickly adapt to changing needs. As new applications come online or existing ones grow, shared platforms can scale to accommodate these demands without the lead time and costs associated with setting up new infrastructure.
6. Consistency and compliance: Ensuring compliance with industry regulations and standards can be complex and resource-intensive. Shared platforms can be designed to meet these requirements universally, providing a consistent and compliant environment for all applications.
7. Innovation and collaboration: Shared platforms can foster an environment of innovation and collaboration. They provide a common ground where different teams can work together, share insights, and develop new solutions that benefit from the shared platform's capabilities.
8. Disaster recovery and business continuity: Centralized shared platforms can be more easily managed for disaster recovery purposes. They allow for streamlined backup processes and quicker restoration of services in the event of a system outage or other disruptive incidents.
9. Ready integrations with other platforms: Shared platforms commonly come with a suite of integrations and connectors that facilitate interaction with other systems, including monitoring and alerting tools. This interconnectedness means that new applications can plug into a rich ecosystem of services without the need for additional integration work. The ease of connecting to established monitoring systems, for example, simplifies the monitoring of applications significantly. 

Now that we have explored some advantages of shared platforms, it's important to talk about considerations for platform owners and users. This understanding will ensure that the benefits of shared platforms are fully realized while minimizing the potential for bottlenecks. The considerations for each group play a vital role in maintaining a healthy, efficient, and scalable shared platform ecosystem.

Protecting the Platform: The Owner's Perspective

Platform owners bear the crucial responsibility of safeguarding their platforms against potential abuses, such as a single application going rogue or the side effects of overprovisioning. Platform owners should also make sure they follow FinOps practices to make their systems maintainable and feasible from a cost point of view.

To maintain a healthy, efficient, and cost-efficient platform, several practices are essential:

1. Implementing quotas: Quotas are vital in preventing any single application from monopolizing resources. By setting limits on usage, platform owners ensure fair resource distribution among all consumers. RabbitMQ maximum queue sizes and Kafka throughput quotas are common examples we see in the enterprise.
2. Monitoring metrics: Continuous monitoring helps in identifying unusual patterns or potential issues before they escalate. Metrics provide insights into the platform's health and guide decision-making processes. Metrics also reveal the usage and changes in demand information, which is crucial for platform planning practices. 
3. Retention period policies: The retention period determines how long data is stored before being discarded.  Implementing data retention policies is crucial to prevent disk space from filling up, which can lead to performance degradation or even system failures.
4. Emitting important updates: When there are updates or changes to the Flink cluster, such as version upgrades or maintenance schedules, it's important to inform all users. This ensures that they can prepare for any potential impact on their data processing tasks. For instance, a version upgrade might introduce new features or deprecate old ones, and users need to be aware of these changes to adapt their applications accordingly. To implement this, platform owners can use a variety of tools and methods, such as email alerts, messaging services like Slack or Microsoft Teams, or even custom webhooks that integrate with users' own monitoring systems. The key is to ensure that the communication is timely, relevant, and actionable, thereby maintaining transparency and trust between the platform owners and the users.
5. The platform team: The Platform Team plays a critical role that extends beyond merely keeping the lights on; they are the custodians of the platform's integrity and efficiency. Governance is a key part of their responsibilities, encompassing tasks such as approving new publishers or consumers to ensure that those who use the platform can contribute positively to the shared ecosystem.

Maintaining an optimal team size is crucial to ensure agility and responsiveness. It's important that the team is not bloated, which can slow down processes, nor too lean, which can lead to burnout and oversight issues.

Investing in continuous education for the team is equally important. A well-informed team stays ahead of the curve, adopting the latest best practices in technology and governance to streamline operations.

Streamlining governance practices is also essential to avoid becoming a bottleneck for new applications that are looking to onboard. The goal is to establish clear, efficient processes that facilitate rather than slow down progress. This might involve automating certain approval processes or setting up self-service portals for routine requests, thereby freeing up the team to focus on more complex tasks that require their expertise. By optimizing governance procedures, the Platform Team not only enhances its own productivity but also drives the overall velocity of the enterprise's innovation and growth.

1. Cost concerns: In managing shared platforms, financial operations, or FinOps, are critical disciplines that platform owners must embrace. The essence of FinOps is to drive cost efficiency without compromising on performance or capability. This involves a continuous cycle of monitoring, optimization, and negotiation. Here's how platform owners can apply FinOps practices to their shared platforms:

• Tiered storage: Implementing tiered storage solutions can lead to significant cost savings. By storing older, less frequently accessed data on cheaper storage while keeping hot data on higher-performance (but more expensive) storage, platform owners can optimize for both cost and performance.
• Demand-driven scaling: Scale-in initiatives during periods of low demand help in reducing costs. Conversely, scaling out when demand spikes ensures performance isn't compromised. This elastic approach to infrastructure management is essential to maintain a balance between cost and capability.
• Negotiating cloud costs: Regular discussions with cloud account managers can reveal discount opportunities. Platform owners should be proactive in seeking out cost-saving measures through commitments or custom pricing packages.
• Reserved instances: For predictable workloads with consistent demand, purchasing reserved instances can provide substantial savings over on-demand pricing. This guarantees a base level of resource availability and can be significantly more cost-effective.
• Spot instances: Utilizing spot instances for non-critical or flexible workloads can further reduce costs. These instances are available at a fraction of the cost of on-demand resources but require the ability to handle possible interruptions.
• Budgeting and reporting: Implementing a transparent budgeting and reporting process helps in tracking cloud spending. This should be done in real-time, where possible, to allow for immediate adjustments and prevent budget overruns.

Protecting the Application: The User's Perspective

Platform users, including developers and consumer teams, must align their practices with platform constraints and capabilities. Writing software that can handle throttling effects, understanding resource limits, and maintaining open communication with platform owners are key. This approach ensures users can maximize the benefits of shared platforms without disrupting the overall system balance.

As a developer or architect, while you engage with your platform teams and collaborate with their experts, it's crucial to study not only the strengths but also the limitations of the platform. Ensure that there's a seamless integration of your applications by thoroughly understanding the platform's SLAs and assessing whether they align with your business requirements. This due diligence is key to confirming that the platform can support your operational goals and drive your enterprise toward success.

Conclusion

In this blog, I've dived into the strategic details of shared platforms, highlighting their critical role in enterprise technology. From cost-saving efficiencies to fostering innovation, shared platforms are more than a choice; they are a necessity for agile and scalable growth.

The next step is being more proactive! Actively reach out to your platform teams and establish a dialogue. Take the initiative to understand the service level agreements (SLAs) and complexities of the platforms you rely on. Assess how they match up with your business needs and where you might need to plan for adjustments. Together, as a unified “one team” of application and platform teams, you have the power to drive your enterprise to a new maturity level of efficiency and innovation.