Big Data Resources

Using Datafold to Enhance DBT for Data Observability

We’ll look at how Datafold and dbt work in tandem to bring observability and repeatability into your data quality process.

December 7, 2021

by Michael Bogan

CORE

· 8,420 Views · 1 Like

What Is LoRa IoT?

Choosing a manufacturer that can manufacture Internet of things devices with multiple protocols can make your Internet of things project progress more smoothly.

November 29, 2021

by herry petty

· 23,402 Views · 16 Likes

Building an ETL Pipeline With Airflow and ECS

Get your data flowing.

November 27, 2021

by Nikita Vasilev

· 6,067 Views · 4 Likes

How To Use SingleStore With Spark ML for Fraud Detection

In this final part of our Fraud Detection series, we’ll use Spark to build a Logistic Regression model from data stored in SingleStore.

November 25, 2021

by Akmal Chaudhri

CORE

· 7,953 Views · 2 Likes

What KPIs Measure the Success of an AI Project?

The performance of a machine learning model is first assessed based on its success rate. Then about the compatibility of this rate with business objectives.

November 17, 2021

by Ketan Parmar

· 8,153 Views · 3 Likes

Monitor Apache Kafka with Telegraf and Grafana in Azure

This article talks about monitoring Apache Kafka on Azure using Telegraf and Grafana. I will guide you on installation, setup, and running monitoring solutions.

November 16, 2021

by Praveen KG

· 6,130 Views · 2 Likes

Reporting in Microservices: How To Optimize Performance

In this article, learn about a new reporting microservice developed by Freschode to overcome reporting problems they faced.

November 15, 2021

by Karina Skrinnik

· 5,111 Views · 2 Likes

What is Data Lineage and How Can It Ensure Data Quality?

By providing a clear representation of the flow of data, a data lineage system essentially allows you to have your cake and eat it too.

November 10, 2021

by Michael Bogan

CORE

· 3,941 Views · 2 Likes

How to Use SingleStore With Spark ML for Fraud Detection

In this 3-part series of articles, we’ll see how easy it is to use the SingleStore Spark Connector and the performance benefits it provides.

November 9, 2021

by Akmal Chaudhri

CORE

· 8,048 Views · 2 Likes

Lambda Architecture is one of the most useful frameworks to design big data applications and distributed data processing systems like social network platforms.

November 5, 2021

by Phani Krishna Kollapur Gandla

· 5,439 Views · 3 Likes

Kappa Architecture is Mainstream Replacing Lambda

This blog post explores why a single real-time pipeline, called Kappa architecture, is the better fit and the role of batch processing in it without requiring Lambda.

November 3, 2021

by Kai Wähner

CORE

· 7,420 Views · 8 Likes

Confluent Kafka Installation and Demo

Learn how to insert rows to SQLite-DB and show those rows on an auto-created topic on Kafka via the JDBC-source connector.

October 26, 2021

by Ersin Gulbahar

· 5,226 Views · 2 Likes

Benchmarking AWS Graviton2 and gp3 Support for Apache Kafka

Using AWS's ACCP alongside Apache Kafka on Graviton2 instances, Kafka users can get a particularly highly performant Kafka experience

October 22, 2021

by Hendra Gunadi

· 7,788 Views · 3 Likes

Distributed Tracing With Spring-cloud-Sleuth Kafka and Zipkin

In this post, you will learn to use Spring Cloud Sleuth for distributed tracing between Spring Boot microservices and Kafka with results displayed on the Zipkin server.

October 22, 2021

by Ming Jiang

· 9,380 Views · 5 Likes

Data Fabric: What Is It and Why Do You Need It?

Data fabric is a viable strategy to help overcome the barriers that previously made it hard to access data and process it in a distributed data environment.

October 21, 2021

by John Kutay

· 5,254 Views · 2 Likes

How to Accelerate Hyper-Automation With Industrial IoT

This article will walk you through specific areas where industrial IoT integration can help accelerate your pace of hyper-automation.

October 21, 2021

by Edward Ossawa

CORE

· 15,634 Views · 3 Likes

Data Fabric vs. Data Lake: Operational Comparison

In this article, we will focus on which is the most appropriate big data store for high-scale, real-time, operational use cases – data fabric vs data lake.

October 21, 2021

by Ian Tick

· 9,208 Views · 3 Likes

How To Get Closer to Consistency in Microservice Architecture

Some scenarios in microservice-based architecture involve multiple microservices. Read how completion of the scenario depends on the task completion of each microservice.

October 20, 2021

by Reza Ganji

CORE

· 8,858 Views · 10 Likes

Data Versioning 101

Data versioning makes it easy for data consumers to repeat a process or research, compare outputs as a result of using different data versions.

October 20, 2021

by Ogundiran Ayobami

· 7,454 Views · 4 Likes

Building a Real-Time Data Warehouse With TiDB and Pravega

After researching the databases and storage systems on the market, we found a more efficient and accurate real-time data warehouse solution: Pravega + TiDB.

October 20, 2021

by Tianyi Wang

· 8,102 Views · 4 Likes

The Latest Big Data Topics