When you only pay for the queries that you run, or resources like CPU and storage, it is important to look at optimizing the data those systems rely on.
If you aren't convinced that having your build definition as code rather than on a server is better, IntelliSense and Azure DevOps might change your mind.
Look at an overview of streaming concepts and learn how to ingest Kafka Events with Spark Structured Streaming and enrich events with a machine learning model.