KEDA is an event-driven autoscaler that horizontally scales a container by adding additional pods based on the number of events needing to be processed.

The article discusses the architectures currently popular for achieving this Serverless Architecture for Scale use cases and how and when we can use them.

Getting Docker working as a local network host on Windows requires quite a lot of kludgy hodgepodge of hacks to work, but it can be done.

In this second part, learn how to centralize IAM for multiple AWS accounts, create and use EC2 instance profiles, and implement just-in-time access with Vault.

Covering the basics of managing AWS IAM with Terraform and learn how to delete the Root/User Access key, enforce MFA, customize password policy, and more.

This article will show how to expose an application using the service type load balancer.

A few weeks ago I wrote a blog on synchronizing multithreaded integration tests, which was republished on DZone Javalobby from where it received a comment from Robert Saulnier who quite rightly pointed out that you can also use join() to synchronize a worker thread and its unit tests. This got me thinking, just how many ways can you synchronise multi-threaded integration tests? So, I started counting... and came up with: Using a random delay. Adding a CountDownLatch Thread.join() Acquiring a Semaphore With a Future and ExecutorService Now, I’m not going to explain all the following in great detail, I’ll let the code speak for itself, except to say that all the code samples do roughly the same thing: the unit test creates a ThreadWrapper instance and then calls its doWork() method (or call() in the case of the Future). The unit test’s main thread then waits for the worker thread to complete before asserting that the test has passed. For the sample code demonstrating points 1 and 2 take a look at my original blog on Synchronizing Multithreaded Integration Tests, though I wouldn’t recommend point 1: using a random delay. Thread.join() public class ThreadWrapper { private Thread thread; /** * Start the thread running so that it does some work. */ public void doWork() { thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; thread.start(); System.out.println("Off and running..."); } /** * Synchronization method. */ public void join() { try { thread.join(); } catch (InterruptedException ex) { Thread.currentThread().interrupt(); } } } public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); instance.doWork(); instance.join(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } Acquiring a Semaphore public class ThreadWrapper { /** * Start the thread running so that it does some work. */ public void doWork() { doWork(null); } @VisibleForTesting void doWork(final Semaphore semaphore) { Thread thread = new Thread() { /** * Run method adding data to a fictitious database */ @Override public void run() { System.out.println("Start of the thread"); addDataToDB(); System.out.println("End of the thread method"); semaphore.release(); } private void addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } } }; aquire(semaphore); thread.start(); System.out.println("Off and running..."); } private void aquire(Semaphore semaphore) { try { semaphore.acquire(); } catch (InterruptedException e) { e.printStackTrace(); } } } public class ThreadWrapperTest { @Test public void testDoWork() throws InterruptedException { ThreadWrapper instance = new ThreadWrapper(); Semaphore semaphore = new Semaphore(1); instance.doWork(semaphore); semaphore.acquire(); boolean result = getResultFromDatabase(); assertTrue(result); } /** * Dummy database method - just return true */ private boolean getResultFromDatabase() { return true; } } With a Future public class ThreadWrapper implements Callable { @Override public Boolean call() throws Exception { System.out.println("Start of the thread"); Boolean added = addDataToDB(); System.out.println("End of the thread method"); return added; } /** * Add to the DB and return true if added okay */ private Boolean addDataToDB() { try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } return Boolean.valueOf(true); } } public class ThreadWrapperTest { @Test public void testCall() throws ExecutionException, InterruptedException { ThreadWrapper instance = new ThreadWrapper(); ExecutorService executorService = Executors.newFixedThreadPool(1); Future future = executorService.submit(instance); Boolean result = future.get(); assertTrue(result); } } Having listed all these methods, the next thing to consider is which one is the best? In asking that question you have to define the word “best” in terms of best for what? Best for simplicity? Maintainability? Speed or code size? After all Programming the Art of Making the Right Decision. You may have guessed that I don’t like the random delay idea and prefer the use of a CountDownLatch. Thread.join() is a bit old school; remember that the likes of Semaphore and CountDownLatch were written to improve on the original Java threading techniques. ExecutorService seems a little heavy weight for what we need here. At the end of the day the choice of technique really comes down to personal preference. Finally, I’ve listed five ways of synchronizing multi-threaded integration tests; however, if you can think of any others please let me know... The source code for this blog is available on Github.

Separate out analysis to preserve commit stage processing time The entry point of a Continuous Delivery pipeline is its Commit Stage, and as such manages the compilation, unit testing, analysis, and packaging of source code whenever a change is committed to version control. As the commit stage is responsible for identifying defective code it represents a vital feedback loop for developers, and for that reason Dave Farley and Jez Humble recommend a commit stage that is “ideally less than five minutes and no more than ten” – if the build process is too slow or non-deterministic, the pace of development can soon grind to a halt. Both compilation and unit testing tasks can be optimized for performance, particularly when the commit stage is hosted on a multi-processor Continuous Integration server. Modern compilers require only a few seconds for compilation, and a unit test suite that follows the Michael Feathers strategy of no database/filesystem/network/user interface access should run in parallel in seconds. However, it is more difficult to optimize analysis tasks as they tend to involve third-party tooling reliant upon byte code manipulation. When a significant percentage of commit stage time is consumed by static analysis tooling, it may become necessary to trade-off unit test feedback against static analysis feedback and move the static analysis tooling into a separate Analysis Stage. The analysis stage is triggered by a successful run of the commit stage, and analyses the uploaded artifact(s) and source code in parallel to the acceptance testing stage. If a failure is detected the relevant pipeline metadata is updated and Stop The Line applies. That binary cannot be used elsewhere in the pipeline and further development efforts should cease until the issue is resolved. For example, consider an organisation that has implemented a standard Continuous Delivery pipeline. The commit stage has an average processing time of 5 minutes, of which 1 minute is spent upon static analysis. Over time the codebase grows to the extent that commit stage time increases to 6 minutes, of which 1 minute 30 seconds is spent upon static analysis. With static analysis time growing from 20% to 25% the decision is made to create a separate Analysis stage, which reduces commit time to 4 minutes 30 seconds and improves the developer feedback loop. Static analysis is the definitive example of an automated task that periodically needs human intervention. Regardless of tool choice there will always be a percentage of false positives and false negatives, and therefore a pipeline that implements an Analysis Stage must also offer a capability for an authenticated human user to override prior results for one or more application versions.

Continuous Delivery and DevOps are interdependent, not equivalent Since the publication of Dave Farley and Jez Humble’s seminal book on Continuous Delivery in 2010, its rise within the IT industry has been paralleled by the growth of the DevOps movement. While Continuous Delivery has an explicit goal of optimising for cycle time and an established set of principles and practices, DevOps is a more organic philosophy that is defined as “aligning development and operations roles and processes in the context of shared business objectives“, and gradually codifying into principles and practices. Continuous Delivery and DevOps possess a shared background in agile methods and Lean Thinking, and a shared desire to eliminate Waterscrumfall silos – but what is the nature of their relationship? In Continuous Delivery, practitioners such as Jez Humble have warned that organisations require “a culture that enables collaboration and understanding between the functional groups that deliver IT services“, which refers to the culture-centric principles – Continuous Improvement, Done Means Released, and Everybody Is Responsible – that reduce handover delays between siloed teams. DevOps provides an implementation strategy for these principles – its emphasis upon “the integration of Agile principles with Operations practices” aligns Development and Operations working practices and encourages cooperation. However, these principles can be also implemented independently of DevOps – for example, an organisation might forego a QA team in favour of mandatory Development support for production releases, as at Facebook. In DevOps, one of the four key areas described by Patrick Debois is Extend Delivery To Production. The intention is for the delivery mechanism to act as a focal point for collaboration between Development and Operations, resulting in improved speed/reliability of releases and a sense of shared responsibility for production systems. Continuous Delivery offers an implementation strategy for this key area – a deployment pipeline provides a shared one-button workflow, encourages the emergence of a shared codebase and toolchain, and facilitates a release cadence that minimises change sets and the risk of failure. However, it should be noted that Extend Delivery To Production could be accomplished without Continuous Delivery – for example, a push-based Continuous Deployment mechanism might underpin the value stream instead of a pull-based pipeline, as at IMVU. From the above we can surmise that Continuous Delivery and DevOps are interdependent, but the inherent fuzziness of the DevOps philosophy allows different interpretations of the relationship. For example, Jeff Sussna recently contended that “delivering software as service makes operations an explicit part of the customer value proposition… customers view functionality and operability as inseparable aspects of service” and that by defining DevOps “not in terms of how IT structures itself, but rather in terms of what customers expect” we can say “DevOps IS Continuous Delivery“. While it is an interesting approach to couple DevOps to customer expectations, the commonly accepted definitions focus upon internal organisational change in order to meet business objectives, which may or may not include operability as a first-class concept. It is evident that SaaScustomers will have explicit operability requirements, but for many organisations the reality is that customers explicitly expect functionality and timeliness while implicitly expecting operability. For example, Jeff uses a restaurant review metaphor to describe the combined value of functionality and operability (“the food was great but the service was terrible“), but restaurant customers cannot observe back-of-house operability and will likely only comment upon front-of-house operability if it impacts upon functionality and/or timeliness. Jeff also makes a comparison of nomenclature, suggesting that for agile development and Continuous Delivery the name describes the value… in the case of DevOps, the name describes the implementation, not the desired outcome“. Surely the desired outcome of DevOps is expressed in the portmanteau – Development and Operations teams seamlessly working together to deliver value-adding features to the customer.

a recent study by gartner predicts a very significant increase in cloud usage by consumers in a few years, due in great part to the ever growing use of smartphone cameras by the average household. in this context, it could be useful to have a smartphone application that is able to upload / download digital content from a cloud provider. in this article, we will construct a basic android prototype that will allow us to plug in the windows azure cloud provider, and use the windows azure toolkit for android ( available at github ) to do all of the basic cloud operations : upload content to cloud storage, browse the storage, download or delete files in cloud storage. once those operations are implemented, we will see how to enable our android application to receive server push notifications . first things first, we need to set up a storage account in the azure cloud: a storage account comes with several options as for data management : we can keep data in blob, table or queue storage. in this article, we will use the blob storage to work with images. the storage account has a primary and secondary access key , either one of the two can be used to execute operations on the storage account. any of those keys can be regenerated if compromised. 1. preliminaries first, the prerequisites: eclipse ide for java android plugin for eclipse ( adt ) windows azure toolkit for android windows azure subscription (you can get a 90-day free trial ) a getting-started document on windows azure toolkit’s github page covers the installation procedure of all the the required software in detail. this whole project ( cloid ) is freely available at github . so here we’ll limit ourselves to presenting the most relevant code sections along with the corresponding screens. the user interface is composed of a few basic activity screens, spawned from the main screen (top center): since we use a technology not for its own sake but according to our needs, let’s start by specifying what we want: public abstract class storage { /** all providers will have accesss to context*/ protected context context; /** all providers will have accesss to sharedpreferences */ protected cloudpreferences prefs; /** all downloads from providers will be saved on sd card */ protected string download_path = "/sdcard/dcim/camera/"; /** * @throws operationexception * */ public storage(context ctx) throws operationexception { context = ctx; prefs = new cloudpreferences(ctx); } /** * @throws operationexception * */ public abstract void uploadtostorage(string file_path) throws operationexception; /** * @throws operationexception * */ public abstract void downloadfromstorage(string file_name) throws operationexception; /** * @throws operationexception * */ public abstract void browsestorage() throws operationexception; /** * @throws operationexception * */ public abstract void deleteinstorage(string file_name) throws operationexception; } the above is the contract that our cloud storage provider will satisfy. we’ll provide a mockstorage implementation that will pretend to carry out a command in order to test our ui (i.e. our scrollable items list, progress bar, exception messages, etc.), so that we can later just plug in azure storage operations. note from our activities screen above, that we can switch anytime between azure storage and mock storage with the press of the toggle button “cloud on/off” in the settings screen, saving the preferences afterward. public class mockstorage extends storage { // code here... @override public void uploadtostorage(string file_path) throws operationexception { donothingbutsleep(); //throw new operationexception( "test error message", // new throwable("reason: upload test") ); } // other methods will also do nothing but sleep... /***/ private void donothingbutsleep(){ try{ thread.sleep(5000l); } catch (interruptedexception iex){ return; } } 2. the azure toolkit the toolkit comes with a sample application called “simple”, and two library jars: access control for android.jar in the wa-toolkit-android\library\accesscontrol\bin folder azure storage for android.jar in the wa-toolkit-android\library\storage\bin folder here we will only use the latter, since we will access directly azure’s blob storage. needless to say, this is not the recommended way , since our credentials will be stored on the handset. a better approach security-wise would be to access azure storage through web services hosted on either azure or other public/private clouds. once the toolkit is ready for use, we need to think a bit about settings . using an azure blob storage only requires 3 fields: an account name , an access key , and a container for our images. the access key is quite a long string (88 characters) and is kind of a pain to type, so one way to do the setup is to configure the android res/values/strings.xml file to set the default values: ... cloid insert-access-key-here pictures ... however, because we may want to overwrite the default values above (e.g. create another container), we will also save the values on the settings screen in android’s sharedpreferences . and now, let’s implement the azurestorage class. 3. azure blob storage operations 3.1. storage initialization the azurestorage constructor gets its data from android preferences (from its superclass), then constructs a connection string used to access the storage account, creates a blob client and retrieves a reference to the container of images. if the user changed the default container “pictures” in settings, then a new (empty) one will be created with that new name. a container is any grouping of blobs under a name. no blob exists outside of a container. // package here // other imports import com.windowsazure.samples.android.storageclient.blobproperties; import com.windowsazure.samples.android.storageclient.cloudblob; import com.windowsazure.samples.android.storageclient.cloudblobclient; import com.windowsazure.samples.android.storageclient.cloudblobcontainer; import com.windowsazure.samples.android.storageclient.cloudblockblob; import com.windowsazure.samples.android.storageclient.cloudstorageaccount; public class azurestorage extends storage { private cloudblobcontainer container; / * @throws operationexception * */ public azurestorage(context ctx) throws operationexception { super(ctx); // set from prefs string acct_name = prefs.getaccountname(); string access_key = prefs.getaccesskey(); // get connection string string storageconn = "defaultendpointsprotocol=http;" + "accountname=" + acct_name + ";accountkey=" + access_key; // get cloudblobcontainer try { // retrieve storage account from storageconn cloudstorageaccount storageaccount = cloudstorageaccount.parse(conn); // create the blob client // to get reference objects for containers and blobs cloudblobclient blobclient = storageaccount.createcloudblobclient(); // retrieve reference to a previously created container container = blobclient.getcontainerreference( prefs.getcontainer() ); container.createifnotexist(); } catch (exception e) { throw new operationexception("error from initblob: " + e.getmessage(), e); } } // code... we will use that container reference cloudblobcontainer throughout our upcoming cloud operations. 3.2. uploading images we will upload a file from android’s gallery to the cloud, keeping the same filename. “screener” is just a utilities class (see github repository) that does a number of useful things, e.g. extracting a file name from its path and setting the right mime type (“image/jpeg”, “image/png”, etc.). the two kinds of blobs are page blobs and block blobs . the (very) short story is that page blobs are optimized for read & write operations, while block blobs let us upload large files efficiently. in particular we can upload multiple blocks in parallel to decrease upload time. here we are uploading a blob (gallery image) as a set of blocks. /** * @throws operationexception */ @override public void uploadtostorage(string file_path) throws operationexception { try { // create or overwrite blob with contents from a local file // use same name than in local storage cloudblockblob blob = container.getblockblobreference( screener.getnamefrompath(file_path) ); file source = new file(file_path); blob.upload( new fileinputstream(source), source.length() ); blob.getproperties().contenttype = screener.getimagemimetype(file_path); blob.uploadproperties(); } catch (exception e) { throw new operationexception("error from uploadtostorage: " + e.getmessage(), e); } } bear in mind that we are not checking if the file already exists in cloud storage. therefore we will overwrite any existing file with the same name as the one we are uploading. that is usually not desirable in production code. here’s the screen flow of the upload operation: 3.3. browsing the cloud for browsing, we store all our blobs in our container into a list of items that we will display in android as a scrollable list of image names in a subclass of android.app.listactivity . once one item in the list is clicked (“touched”) by the user, we want to display some image properties such as the image size (important when deciding to download), its mime type, and the date it was last operated upon. /** * @throws operationexception * */ @override public void browsestorage() throws operationexception{ // reset uri list for refresh - no caching item.itemlist.clear(); // loop over blobs within the container try { for (cloudblob blob : container.listblobs()) { blob.downloadattributes(); blobproperties props = blob.getproperties(); long ksize = props.length/1024; string type = props.contenttype; date lastmodified = props.lastmodified; item item = new item(blob.geturi(), blob.getname(), ksize, type, lastmodified); item.itemlist.add(item); } // end loop } catch (exception e) { throw new operationexception("error from browsestorage: " + e.getmessage(), e); } } here’s the screen flow of the browse operation. pressing on an item on the list displays its details and operations on the image, which we will look at next: 3.4. downloading images our download method is pretty straightforward. note that we are downloading to the android handset’s sd card by using download_path from the superclass. /** * @throws operationexception * */ @override public void downloadfromstorage(string file_name) throws operationexception{ try { for (cloudblob blob : container.listblobs()) { // download the item and save it to a file with the same name as arg if(blob.getname().equals(file_name)){ blob.download( new fileoutputstream(download_path + blob.getname()) ); break; } } } catch (exception e) { throw new operationexception("error from downloadfromstorage: " + e.getmessage(), e); } } and the corresponding ui flow. instead of displaying the image right after the download, we chose to include a link to the gallery (bottom of the screen) where the freshly retrieved image appears on top of the gallery’s stack of pictures: 3.5. deleting images the delete operation performed on a blob up in the cloud is also rather simple: /** * @throws operationexception */ @override public void deleteinstorage(string file_name) throws operationexception{ try { // retrieve reference to a blob named file_name cloudblockblob blob = container.getblockblobreference(file_name); // delete the blob blob.delete(); } catch (exception e) { throw new operationexception("error from deleteinstorage: " + e.getmessage(), e); } } and its associated ui screens series. note that after confirming the operation, and when deletion completes, the browsing list of items is automatically refreshed, and we can see that the image is no longer on the list of blobs in our storage container. 3.6. wrapping up the azurestorage methods are called inside a basic work thread, which will take care of all cloud operations: // called inside a thread try { // get storage instance from factory storage store = storagefactory.getstorageinstance(this, storagefactory.provider.azure_storage); // for the progress bar incrementworkcount(); // do ops switch(operation){ case upload : store.uploadtostorage(path); break; case browse : store.browsestorage(); break; case download : store.downloadfromstorage(path); // refresh gallery sendbroadcast( new intent( intent.action_media_mounted, uri.parse("file://"+ environment.getexternalstoragedirectory()) ) ); break; case delete : store.deleteinstorage(path); break; } // end switch } catch (operationexception e) { recorderror(e); } notice how we are telling the android image gallery to refresh by issuing a broadcast once a new file is downloaded from the cloud to the sd card. there are different ways to do this, but without that call, the gallery won’t show the new image before the next system scheduled media scan. again, for the full code, refer to this project on github. we are done with the basic cloud operations. all we had to do was plug in our azurestorage implementation class and get an instance of it through a factory, with minimal impact on preexisting code. 4. push notifications up to this point we have demonstrated device-initiated communication with the cloud. for cloud-initiated or push communication, the android platform uses google cloud messaging (gcm). in a previous article , i wrote about how to integrate gcm into an existing android application. here we will add a second set of settings for server push. our client code will connect with any gcm server and it will set the status on our main activity (last screen shot on the right) once the information in push preferences is correctly set. 5. conclusions the toolkit documentation is kind of sparse (which is why the community needs more articles like this). also, the sample application doesn’t cover much (maybe the reason why it’s called “simple”), and it has room for improvement. however, the library itself is fully functional, and once we figure out the api, it all works quite nicely. of course, this application is itself pretty basic and doesn’t cover lots of other features, like access control, permissions, metadata, and snapshots. but it is a start.

Building Continuous Delivery into an organisation requires radical change While Continuous Delivery has a well-defined value proposition and a seminal bookon how to implement a deployment pipeline, there is a dearth of information on how to transform an organisation for Continuous Delivery. Despite its culture-focussed principles and an adoption process described by Jez Humble as ”organisational-architecture-process not tools-code-infrastructure“, many Continuous Delivery initiatives fail to emphasise an organisational model in which software is always releasable. This contravenes Lean Thinking and the Deming 95/5 Rule – that 95% of problems are attributable to system faults, while only 5% are due to special causes of variation. Building an automated deployment pipeline can eliminate the 5% of special causes of variation in our value stream (e.g. release failures), but it cannot address the remaining 95% of problems caused by our organisation structure (e.g. wait times between silos). From this we can infer that: Continuous Delivery = 95% organisation, 5% automation Establishing a Continuous Delivery culture requires a change management programme more challenging, time-consuming, and valuable than any technology-based efforts. Donella Meadows recommended that to effect change we “arrange the structures and conditions to reduce the probability of destructive behaviours and to encourage the possibility of beneficial ones“, and we can achieve this by using the change patterns of Linda Rising and Mary Lynn Adams within the change management supermodel of Jurgen Appelo: Dance with the System Mind the People Stimulate the Network Change the Environment To dance with the system, we propose a made to order Continuous Delivery programme, with a Tailor Made business case that emphasises reduced transaction costs and/or increased customer value according to the needs of our organisation. We must identify a Local Sponsor to support our efforts and a Corporate Angel to increase awareness, and we should communicate successfulcase studies to our stakeholders as External Validation. To mind the people, we construct a collaborative, bottom-up change programme that encourages participation. We need to Involve Everyone from the outset, and apply a Personal Touch with each individual stakeholder to pitch Continuous Delivery in terms of their incentives. We should use Corridor Politics to promote our change initiative, Just Say Thanks to our contributors, and highlight value stream waste without dispute - as Morgan Wootten said “a lighthouse doesn’t blow a horn, it shines a light“. To stimulate the network, we emulate the Diffusion of Innovations theory of Everett Rogers and exploit the social network that comprises our organisation. We must encourage Innovators to spark an interest in our change initiative, and then form a group of Early Adopters to offer us early feedback. We need to Ask For Help from Connectors to evangelise to their peers on our behalf, and by Staying In Touch with our supporters we can work towards an Early Majority invested in Continuous Delivery. To change the environment, we focus upon changing our organisation structure and processes to instil a culture of Continuous Delivery. We need to radiate our value stream In Your Space to raise awareness of cycle time, lead times, and wait times using Just Enough repackaged Lean terminology (e.g. “average time to market” instead of cycle time). We must work as Bridge Builders between differentsiloed teams to reduce our communications burden, and we should develop our pipeline Step By Step to encourage the good practices and discourage the bad (e.g. enforcing decouple deployment from release in a user interface). Building Continuous Delivery into an organisation can be achieved by automating a deployment pipeline and implementing a change management programme, but we should remember Jurgen Appelo’s advice that changing people “is hard to do without an expensive operating table“. Our change programme must be tailored to business requirements, personalised for each stakeholder, and focussed upon improving the environment – and we should always remember: Building a Continuous Delivery pipeline is easy. Building a Continuous Delivery organisation is hard

A rundown of NodePorts, LoadBalancers, Ingresses, and more in Kubernetes!

Using TLS everywhere is one of the Kubernetes team's recommendations for hardening cluster security and increasing resilience. In this tutorial, you'll learn how to automate TLS certificate issuance to Kubernetes deployments.

Here we will use the Mulesoft Anypoint platform to attach VPC to the AWS transit gateway to form a single network topology.

Learn about nine Sprint Goal principles, from avoiding pseudo and output-driven Sprint Goals to playing safe all the time.

Cloud platforms provide greater flexibility. How do you choose to compute service in Google Cloud?

The serverless offerings of AWS are getting more and more popular. But it remains a challenge to know them well enough to leverage them properly.

This article makes some observations on the advancements in real-time, event-driven messaging with hierarchical topics from the MoM perspective.

Here, we explore the five stages of the work cycle: plan, develop, test, release, and operate, as well as the best tools to use in each stage.

Learn how to build a modern data stack with cloud-native technologies, such as data warehouse, data lake, and data streaming, to solve business problems.