Developers Get Real-Time Analytics for Observability and Anomaly Detection

In today's fast-paced, data-driven world, developers, engineers, and architects face the challenge of building resilient and performant applications while ensuring optimal user experiences. Real-time analytics plays a crucial role in addressing these challenges, enabling businesses to make informed decisions based on up-to-the-minute data. StarTree, the company behind Apache Pinot, has recently announced significant advancements in its real-time analytics capabilities, focusing on observability and anomaly detection use cases.

Streamlining Observability With StarTree Cloud

Observability is a critical aspect of modern software development, allowing teams to gain insights into the health and performance of their applications. StarTree's new offering, StarTree Cloud Observability, provides a managed service that streamlines the process of storing and querying metrics, logs, and traces in real time. By leveraging StarTree's disaggregated approach, developers can seamlessly integrate their existing collection agents and pipelines with StarTree's infrastructure, ensuring flexibility and avoiding vendor lock-in.

Chinmay Soman, Head of Product at StarTree, emphasizes the company's focus on real-time and external use cases: "The focus of StarTree is, as I mentioned, user-facing, what that means, in more technical terms, right? So if you have data that is real-time, data and queries that are more real-time in nature, right, so you want to generate insights quickly, and you want to generate insights on data arriving right now."

Efficient Handling of Large-Scale Data

One of the key benefits of StarTree Cloud Observability is its ability to handle large-scale data efficiently. With support for terabytes to petabytes of logs, StarTree's optimized storage and indexing capabilities enable fast query processing while maintaining cost-effectiveness. This empowers developers to analyze and troubleshoot issues in real time, reducing mean time to detection (MTTD) and mean time to resolution (MTTR).

Soman highlights the scalability and cost-efficiency of StarTree's offering: "The reason StarTree stands out is that the storage offering of StarTree. It's very scalable. It provides very fast queries. We have seen that in our previous user-facing applications. And it's very efficient in terms of cost of hosting all this data."

Unleashing the Power of Vector Indexing

StarTree's commitment to open-source innovation is evident in its recent contributions to Apache Pinot, particularly the addition of vector indexing support. This feature opens up new possibilities for developers working on AI and machine learning applications. By storing embeddings of words, images, or videos directly in Pinot, developers can perform similarity searches and build powerful applications such as chatbots, recommendation systems, and content discovery platforms. The combination of Pinot's distributed architecture and optimized query processing ensures scalability and performance, even with large-scale datasets.

Soman explains the significance of vector indexing in Apache Pinot: "We recently added vector index capability to Apache Pinot. And this is really where you could store embeddings of words, images, or videos in your cluster, be able to do similarity search, for example, directly in Pinot so you can imagine with these two capabilities, you can build all kinds of interesting Gen AI applications."

Automated Anomaly Detection With StarTree ThirdEye

Another exciting development from StarTree is the general availability of StarTree ThirdEye, a real-time anomaly detection and root cause analysis solution. ThirdEye addresses the challenge of identifying anomalies in time-series data, specifically for business metrics. With advanced statistical algorithms that account for seasonality and variations, ThirdEye detects anomalies in real time, eliminating the need for manual threshold setting.

Soman elaborates on the capabilities of StarTree ThirdEye: "This is a product that we shipped with our cloud offering to all our customers. And it's really focusing on identifying anomalies or outliers in your time series data sets, and specifically for business metrics."

Revolutionizing Root Cause Analysis

Moreover, ThirdEye's automated root cause analysis (RCA) capabilities revolutionize the way developers investigate and resolve issues. By analyzing hundreds of dimensions and their combinations, ThirdEye pinpoints the root cause of anomalies in seconds, saving valuable time and effort. This enables developers to quickly identify and address issues, minimizing the impact on user experience and business operations.

Soman explains how ThirdEye automates root cause analysis: "Once you identify an anomaly, the other big thing it does is automated RCA root cause analysis so where humans spend hours or days this does it in seconds."

Simplifying Data Ingestion With Write API

To further empower developers, StarTree has introduced a write API for Apache Pinot, simplifying data ingestion processes. With the ability to write data directly to Pinot via HTTP calls, developers can streamline their data pipelines, eliminating the need for intermediate systems like Kafka. This not only reduces complexity but also lowers costs associated with maintaining additional infrastructure.

Soman highlights the benefits of the write API: "We are now announcing a private preview of write API, where you can simply make an HTTP call and add a record into Pinot very easily. This is important for cases where you're using an ETL or ELT tool like Fivetran or if you have a data set already in StarTree. And you want to create a derived data set using DBT. Now it's easier with write API."

Seamless Integration With Existing Workflows

StarTree's growing ecosystem of integrations, including support for popular data systems like Debezium, Fivetran, and dbt, as well as visualization tools like Grafana and Tableau, ensures that developers can easily incorporate real-time analytics into their existing workflows. The seamless integration capabilities enable developers to focus on building innovative applications while leveraging the power of real-time data.

Soman mentions the expanding integration support: "Today, we, you know, we're extending our support for Tableau and Grafana. So you could actually do real-time bi, you StarTree cloud and Tableau which is actually a big pain point. For many, many customers, right."

Forever-Free Tier for Exploration and Collaboration

For developers eager to explore StarTree's capabilities, the company now offers a forever-free tier of StarTree Cloud. This allows developers to experience the full suite of functionalities, including low-code data ingestion and enhanced query capabilities, without time limitations. The forever-free tier provides a risk-free environment for developers to prototype, experiment, and collaborate with their peers, fostering innovation and accelerating time-to-market.

Peter Corless, Director of Product Marketing at StarTree, highlights a unique feature of the forever-free tier: "The one thing I wanted to point out, is that you can even invite your friends to work on your own private little workspace. It's not like you're downloading it to Docker and nobody can see the work that you're doing. You can actually share this so you can do rapid prototyping and bounce ideas off your fellow developers."

Conclusion

In conclusion, StarTree's advancements in real-time analytics for observability and anomaly detection empower developers, engineers, and architects to build robust, data-driven applications. With scalable infrastructure, automated anomaly detection, and seamless integrations, StarTree simplifies the process of leveraging real-time insights to drive business success. As the demand for real-time analytics continues to grow, StarTree's innovative solutions position developers at the forefront of this transformative technology.