Preetdeep Kumar

My working with large messages (GBs, TBs) have been mostly solving problem of volume and variety. Using a scheduled ETL job which collects and transforms large payloads in chunks (< 10 MBs) after partitioning. Then sending these chunks into kafka for further processing is better alternative which allowed us to not worry about the varying size payload in kafka.

There are multiple options, most of them rely on creating a Java POJO from your JSON and then serialize/deserialize the Java object. You can also use AVRO is you want versioning & strict type in your JSON schema

Thanks for feedback

Glad you liked it

Glad you liked it

There is a nice blog in dzone which has compared Pulsar and Kafka. But with Flink, I don't think it should be compared with Pulsar.

Nice, we are also using lombok in our project to reduce lot of repetitive code specially setters/getters

Thanks Hammad for your feedback. You are correct and I have too pointed out in the blog. This application is more than decade old and has many limitations when it comes to deployment and integration. My team will be refactoring this application in next phase.