Object Store Connector in Mulesoft
Object Store Connector is a Mule component that allows the storage of key-value. It stores data in the form of Key-Value pair and provides 7 types of operation.
Join the DZone community and get the full member experience.
Join For FreeObject Store Connector is a Mule component that allows the storage of key-value. Although it can serve a wide variety of use cases, it is mainly designed for storing watermarks, access tokens, user information, etc. It stores data in the form of Key-Value pair.
Object store connector provides 7 types of operation:
- Store: Stores the given value using the given key.
- Retrieve: Retrieves the value stored for the given key. If there is no key with the given name then the default value is returned.
- Retrieve all keys: Returns a list of all keys that the object store currently holds values for.
- Retrieve all: Retrieves all the key-value pairs present in the object-store.
- Contains: Checks if there is any value exists to the given key. If no value exists for the key, then false is returned.
- Remove: Removes the value associated with the given key from the object-store.
- Clear: Clears all the contents in the store. It removes all the content (keys and values) from the object-store.
Let’s try to understand all of these operations by a demo.
1. Store:
This operation stores the given value in the object store using the given key.
We will create a mule flow as shown in the below image, here we are using an HTTP listener, a logger that is displaying the input provided, and a Store operation that will store the given Key and value in the object store, and finally, a logger that will show us the success message in logs.
When we drag and drop store operation for object store from mule palette, it would require details like a key, object store, etc. as shown in the below image.
Firstly, we need to create an object store, for this, we will click on the plus (+) icon, and the object store configuration window will open.
In the configuration, we will set the values for Max Entries, Entry TTL, Entry TTL unit, etc. as shown in the below image.
Note: We can see that option Persistent is checked, this is because By default the Object store is persistent and it stores all the data in the disk. So the data would be present in the object store even in case if app restarts or crashes.
If we don’t select the Persistent checkbox then the object store would be non-persistent (transient) and it stores all the data in the in-memory which means the data stored in the object store would be lost in the case of app restarts or crashes.
Max Entries represents max no. of entries allowed in the object store.
- Entry TTL: Represents Entry timeout (after what time an entry times out).
- Entry TTL Unit: Entry time to live unit (Seconds/Minutes/Hours).
- Expiration Interval: How frequently the expiration thread should run.
- Expiration Interval unit: The expiration interval.
So as per the configuration, our object can store a max of 5 entries, and the time to live is of 30 minutes which mean once stored an entry can exist only for 30 minutes. We haven’t set any Expiration interval, but if this is set then all the values inside the object store expire after the configured time duration.
On entering the values for the keys and value that we want to store, our store operation will look like below:
Let's try to send some values to the object store:
Our given value is stored in the object store, we can check same in Mule logs:
INFO 2021-03-18 02:49:13,754 [[MuleRuntime].uber.04: [object-store-example].object-store-exampleFlow.CPU_LITE @4075527c] [processor: object-store-exampleFlow/processors/0; event: c88b73b0-7165-11eb-b6f6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: "Input sent is " { "key" : "Batman", "value" : "Bruce Wayne" } INFO 2021-03-18 02:49:14,017 [[MuleRuntime].uber.01: [object-store-example].object-store-exampleFlow.BLOCKING @5dad5d18] [processor: object-store-exampleFlow/processors/2; event: c88b73b0-7165-11eb-b6f6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Value stored in the OS |
In this way, we can store the value in the object store.
2. Retrieve:
As the name suggests, retrieve operation is used to retrieve/fetch the value of a given key from the object store.
For a demonstration of this, we create a flow as shown in the below image. It is the same as store flow, the only difference is instead of Store operation we are using retrieve operation and in the end, we added a transform component that will convert the payload in application/json, so we can see the response on Postman.
Here, inside the key tab, we are passing the key on the basis of which we want to retrieve the value from the object store.
In the Default value tab, we will set a default value that would be provided/returned if the given key is not present in the object store.
Let's try to retrieve value from the object store using Postman as shown in the below image:
We can see the same in the Mule logs also:
INFO 2021-03-18 03:31:51,683 [[MuleRuntime].uber.10: [object-store-example].object-store-retrieveFlow.BLOCKING @336b297c] [processor: object-store-retrieveFlow/processors/1; event: bd52c650-716b-11eb-b6f6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Retrieved Value is ' "Bruce Wayne" |
Let's try to retrieve a value that is not present:
We can see as the given Key is not present in the object store default value is returned and we can see the same in Mule logs.
INFO 2021-02-18 03:32:57,182 [[MuleRuntime].uber.10: [object-store-example].object-store-retrieveFlow.BLOCKING @336b297c] [processor: object-store-retrieveFlow/processors/1; event: e45e3270-716b-11eb-b6f6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: Retrieved Value is ' Default Value |
3. Retrieve All Keys:
As the name suggests, this operation is used to find the keys which are present inside the object store. This operation returns us an array that contains all the keys.
For understanding this operation, let us create a flow as shown in the below image:
For retrieving all keys, we just need to add the object store config details.
Let's try to trigger a request from Postman for this.
From the output, we can see that there are 3 keys present inside the object store.
INFO 2021-03-18 03:41:26,074 [[MuleRuntime].uber.10: [object-store-example].object-store-RetrieveAllKeysFlow.BLOCKING @669c1d1a] [processor: object-store-RetrieveAllKeysFlow/processors/1; event: 13aa1660-716d-11eb-b6f6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: "Retrieved keys are " [Batman, Ironman, Spiderman] |
In this way, we can use Retrieve all operation of object store connector.
4. Retrieve All:
This operation is used to find all the key-value pairs which are present inside the object store.
For testing this operation, we will create a flow as shown in the below image, we only need to provide the object store configuration details in the Retrieve all operation. We are using a transform message component to convert the output of the Retrieve all operation into the required format.
The below DataWeave code is written inside the Transform component:
We will trigger the below request from Postman and we will get all the key and values from the object store in response:
We can see the same details in our Mule logs:
INFO 2021-03-18 18:47:42,420 [[MuleRuntime].uber.07: [object-store-example].object-store-RetrieveAllFlow.CPU_LITE @6cf83761] [processor: object-store-RetrieveAllFlow/processors/0; event: ae855900-71eb-11eb-a4bd-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Lets see what are the values in the Object Store' INFO 2021-03-18 18:47:42,673 [[MuleRuntime].uber.07: [object-store-example].object-store-RetrieveAllFlow.CPU_INTENSIVE @54aeb997] [processor: object-store-RetrieveAllFlow/processors/3; event: ae855900-71eb-11eb-a4bd-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Values are ' { "Batman": "Bruce Wayne", "Ironman": "Tony Stark", "Spiderman": "Peter Parker", "Superman": "Clark Kent" } |
In this way, we can use the object store Retrieve all operation.
5. Contains:
This operation checks if there is any value associated with the given key.
If a value is present then true is returned, and if no value exists for the key, then false is returned.
For demonstration, we will create a Mule flow as shown in the below image.
Below is the request that we triggered from Postman and we can see the response received.
Same is also visible in below Mule logs:
INFO 2021-03-20 02:31:32,500 [[MuleRuntime].uber.08: [object-store-example].object-store-containsFlow.CPU_LITE @edff8fe] [processor: object-store-containsFlow/processors/0; event: a4f362a0-72f5-11eb-87e4-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Value passed to check :' { "key" : "Hulk" } INFO 2021-03-20 02:31:32,572 [[MuleRuntime].uber.08: [object-store-example].object-store-containsFlow.CPU_INTENSIVE @279f54e2] [processor: object-store-containsFlow/processors/3; event: a4f362a0-72f5-11eb-87e4-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Result after Cotain operation is ' false |
Let's send another request from Postman with the key which is present in the object store. We can see that true is returned this time.
Below are the Mule logs for the above request:
INFO 2021-03-20 02:33:30,517 [[MuleRuntime].uber.08: [object-store-example].object-store-containsFlow.CPU_LITE @edff8fe] [processor: object-store-containsFlow/processors/0; event: eb4bf5f0-72f5-11eb-87e4-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Value passed to check :' { "key" : "Ironman" } INFO 2021-03-20 02:33:30,531 [[MuleRuntime].uber.08: [object-store-example].object-store-containsFlow.CPU_INTENSIVE @279f54e2] [processor: object-store-containsFlow/processors/3; event: eb4bf5f0-72f5-11eb-87e4-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Result after Contain operation is ' true |
6. Remove:
This operation is used to remove the values associate to the given key. If there is no value for the key, then OS:KEY_NOT_FOUND
error is thrown.
For demonstration, we will use the below mule flow where after receiving the data from the HTTP listener, we are storing the key to be removed in a variable, and then we are using retrieve all operation to see all the key-value pairs present inside the object store before performing the remove operation. Then we are performing the remove operation, after that, we are again using retrieve all operations to check what are the values present inside the object store.
Let's try to make a call from Postman to remove the key shown in below image:
Let's have a look at the Mule logs, which show us the values which were present in the object store before and after the Remove operation.
INFO 2021-03-20 03:04:29,121 [[MuleRuntime].uber.07: [object-store-example].object-store-clearFlow.CPU_LITE @511d5d47] [processor: object-store-clearFlow/processors/0; event: 3f1d08f0-72fa-11eb-b685-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: "Values inside Object Store before Remove Operation :: " INFO 2021-03-20 03:04:29,325 [[MuleRuntime].uber.07: [object-store-example].object-store-clearFlow.CPU_INTENSIVE @3b8cec1a] [processor: object-store-clearFlow/processors/4; event: 3f1d08f0-72fa-11eb-b685-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: { "Batman": "Bruce Wayne", "Ironman": "Tony Stark", "Spiderman": "Peter Parker", "Superman": "Clark Kent" } INFO 2021-03-20 03:04:29,334 [[MuleRuntime].uber.08: [object-store-example].object-store-clearFlow.BLOCKING @18029f8e] [processor: object-store-clearFlow/processors/6; event: 3f1d08f0-72fa-11eb-b685-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Values inside object store after Remove Operation ' INFO 2021-03-20 03:04:29,354 [[MuleRuntime].uber.08: [object-store-example].object-store-clearFlow.CPU_INTENSIVE @3b8cec1a] [processor: object-store-clearFlow/processors/9; event: 3f1d08f0-72fa-11eb-b685-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: { "Batman": "Bruce Wayne", "Spiderman": "Peter Parker", "Superman": "Clark Kent" } |
So, in this way, we can use the object store Remove operation.
7. Clear:
This operation is used to clear all the key-value pairs present in the object store.
For demonstration, we will be using mule flow as shown in the below image. It is almost similar to the flow we used for the Remove operation, the only change is here we are using the Clear operation.
With the Clear operation, we need to only provide details of the object store configuration.
Lets try to make a call from Postman as shown in below image:
INFO 2021-03-20 03:22:11,519 [[MuleRuntime].uber.02: [object-store-example].object-store-clearFlow.CPU_LITE @10698c5a] [processor: object-store-clearFlow/processors/0; event: b850f310-72fc-11eb-90a6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: "Values inside object store before clear Operation :: " INFO 2021-03-20 03:22:11,659 [[MuleRuntime].uber.02: [object-store-example].object-store-clearFlow.CPU_INTENSIVE @fed11f8] [processor: object-store-clearFlow/processors/3; event: b850f310-72fc-11eb-90a6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: { "Batman": "Bruce Wayne", "Spiderman": "Peter Parker", "Superman": "Clark Kent" } INFO 2021-03-20 03:22:11,663 [[MuleRuntime].uber.06: [object-store-example].object-store-clearFlow.BLOCKING @226d5b4a] [processor: object-store-clearFlow/processors/5; event: b850f310-72fc-11eb-90a6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: 'Values inside object store after clear Operation ' INFO 2021-03-20 03:22:11,668 [[MuleRuntime].uber.06: [object-store-example].object-store-clearFlow.CPU_INTENSIVE @fed11f8] [processor: object-store-clearFlow/processors/8; event: b850f310-72fc-11eb-90a6-c8b29becab99] org.mule.runtime.core.internal.processor.LoggerMessageProcessor: { } |
From Mule logs we can see that there were 3 key-value pairs were present in the object store and after performing the Clear operation there is nothing in the object store.
In this way, we can use object store and its different operations.
Below is the code used for the demonstration:
x
<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core" xmlns:os="http://www.mulesoft.org/schema/mule/os"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/os http://www.mulesoft.org/schema/mule/os/current/mule-os.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="849d6777-391e-4913-b563-ae19186b7b78" >
<http:listener-connection host="0.0.0.0" port="8081" />
</http:listener-config>
<os:object-store name="Object_store" doc:name="Object store" doc:id="bf262f61-4834-4d63-88e2-672f053ead0f" maxEntries="5" entryTtl="30" entryTtlUnit="MINUTES"/>
<flow name="object-store-exampleFlow" doc:id="3640bff1-1fd3-497a-b936-0314bbc987ed" >
<http:listener doc:name="Listener" doc:id="edefa0c1-9837-4425-bd40-76a410ac9424" config-ref="HTTP_Listener_config" path="store"/>
<logger level="INFO" doc:name="Logger" doc:id="3ab473b2-c053-49bd-8fa6-2815e7f7b0bd" message='"Input sent is " #[payload]'/>
<os:store doc:id="febc714c-cf8b-432d-9904-8d9ac4d9f667" key="#[payload.key]" objectStore="Object_store">
<os:value ><![CDATA[#[payload.value]]]></os:value>
</os:store>
<logger level="INFO" doc:name="Logger" doc:id="0c53dbb2-24c9-4d17-989c-7856d97b09a1" message="Value stored in the OS"/>
</flow>
<flow name="object-store-retrieveFlow" doc:id="55e0312f-e3b5-452c-82b8-d1f51aaa49e5" >
<http:listener doc:name="Listener" doc:id="a713b77e-5563-4538-bc78-ba6c8d704e86" config-ref="HTTP_Listener_config" path="retrieve"/>
<os:retrieve doc:name="Retrieve" doc:id="1741330a-4b70-41ff-b9bd-6341c2751c58" key="#[payload.key]" objectStore="Object_store">
<os:default-value ><![CDATA[Default Value]]></os:default-value>
</os:retrieve>
<logger level="INFO" doc:name="Logger" doc:id="35e2e497-f5a5-40f5-a212-5b1797648abd" message="Retrieved Value is ' #[payload]" />
<ee:transform doc:name="Transform Message" doc:id="1c65b096-d9ba-45b6-8c43-cac299287388">
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/json
---
payload as String]]></ee:set-payload>
</ee:message>
</ee:transform>
</flow>
<flow name="object-store-containsFlow" doc:id="1b3be95b-a709-46f7-aea1-f3f16475e0c6" >
<http:listener doc:name="Listener" doc:id="11f72211-c9ac-47a6-89f5-0ba5077031bb" config-ref="HTTP_Listener_config" path="contain"/>
<logger level="INFO" doc:name="Logger" doc:id="769f4add-3540-48e7-b124-16dc13b39a3b" message="'Value passed to check :' + #[payload]"/>
<os:contains doc:name="Contains" doc:id="67e3ae88-3ea6-430f-8b31-813442749b69" key="#[payload.key]" objectStore="Object_store"/>
<ee:transform doc:name="Transform Message" doc:id="304f1475-0919-43aa-b97b-f50d228c172f" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="5cf98bf7-09b2-4a64-949b-0eff5d23fc1b" message="'Result after Cotain operation is ' #[payload]"/>
</flow>
<flow name="object-store-RetrieveAllKeysFlow" doc:id="30752b24-d4ae-4f4e-9163-25c7bbc1ab04" >
<http:listener doc:name="Listener" doc:id="322f7414-40d8-4eb8-9a53-7bb6e70e65bf" config-ref="HTTP_Listener_config" path="Retrieveallkeys"/>
<os:retrieve-all-keys doc:name="Retrieve all keys" doc:id="de663920-a000-440f-b5d3-170fbc5f1ea3" objectStore="Object_store"/>
<logger level="INFO" doc:name="Logger" doc:id="405f5eec-96d0-41bb-8974-0f9f0118a6b8" message='"Retreived keys are " #[payload]'/>
<ee:transform doc:name="Transform Message" doc:id="4fe7b555-5513-49e5-b54b-1d6f7f012ded" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload ]]></ee:set-payload>
</ee:message>
</ee:transform>
</flow>
<flow name="object-store-RetrieveAllFlow" doc:id="3676406d-1c09-42c2-8e4f-a657520c0fb9" >
<http:listener doc:name="Listener" doc:id="73cf4d47-817a-433d-8031-c4657006f6d7" config-ref="HTTP_Listener_config" path="retrieveall"/>
<logger level="INFO" doc:name="Logger" doc:id="81d3f72c-567d-4b68-98a0-fab12c61be2c" message="'Lets see what are the values in the Object Store'"/>
<os:retrieve-all doc:name="Retrieve all" doc:id="bcd09c82-89df-41df-a4a0-63a453076241" objectStore="Object_store"/>
<ee:transform doc:name="Transform Message" doc:id="cc5aa678-3c15-4298-a56a-eef42581677a" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload mapObject (value, key ) ->
{
"$(key)" : read(value,'application/json')
}
]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="baffe60e-8064-48c7-b5ad-390b5a454947" message="'Values are ' #[payload]"/>
</flow>
<flow name="object-store-removeFlow" doc:id="1ba387cf-0de7-484b-b55c-d754419aed3d" >
<http:listener doc:name="Listener" doc:id="fa9b0e93-acee-40c3-a55e-df356e697063" config-ref="HTTP_Listener_config" path="remove"/>
<logger level="INFO" doc:name="Logger" doc:id="c7aaf08b-610d-42a6-916d-6b7a7da2aabf" message='"Values inside object store before Remove Operation :: "'/>
<set-variable value="#[payload.key]" doc:name="Set Variable" doc:id="75b28a7b-12a1-4a3e-a857-56d64c081f07" variableName="KeytoRemove"/>
<os:retrieve-all doc:name="Retrieve all" doc:id="52ccf50f-a795-4d9b-b368-acc4106791de" objectStore="Object_store"/>
<ee:transform doc:name="Transform Message" doc:id="668174ad-3932-4414-a16c-9da72fa154ee">
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/json
---
payload mapObject (value, key ) ->
{
"$(key)" : read(value,'application/json')
}
]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="04f6c468-0358-47e7-8dfd-b1ace05b17d4" message="#[payload]"/>
<os:remove doc:name="Remove" doc:id="c94e9f21-8caf-44d3-afcc-661a5c08a6ec" key="#[vars.KeytoRemove]" objectStore="Object_store"/>
<logger level="INFO" doc:name="Logger" doc:id="7e509b8c-81ea-4cac-bb60-d7c8dddce2cd" message="'Values inside object store after Remove Operation '"/>
<os:retrieve-all doc:name="Retrieve all" doc:id="1e8d3998-e67a-4e77-add9-9c6d55da4b7d" objectStore="Object_store"/>
<ee:transform doc:name="Transform Message" doc:id="6b6ba504-9388-4e38-a3f1-d71a8a205ea0">
<ee:message>
<ee:set-payload><![CDATA[%dw 2.0
output application/json
---
payload mapObject (value, key ) ->
{
"$(key)" : read(value,'application/json')
}
]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="e84023c9-3d38-4299-a8de-9117a61c4ed8" message="#[payload]"/>
</flow>
<flow name="object-store-clearFlow" doc:id="70491db9-47cb-4070-9b45-462bf22f4cb4" >
<http:listener doc:name="Listener" doc:id="7d67c6f1-41d9-4c87-aeca-4b7e7a796165" config-ref="HTTP_Listener_config" path="clear"/>
<logger level="INFO" doc:name="Logger" doc:id="03c1e2ff-0f86-4ce9-b014-cae74f708b94" message='"Values inside object store before clear Operation :: "' />
<os:retrieve-all doc:name="Retrieve all" doc:id="0f7c1dbd-6eeb-4ca4-aca5-af0069a3e62b" objectStore="Object_store" />
<ee:transform doc:name="Transform Message" doc:id="08da344c-2445-456c-9ed5-ec9b535b7008" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload mapObject (value, key ) ->
{
"$(key)" : read(value,'application/json')
}
]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="c0f473f5-09a9-4cb9-b6c8-823a75714504" message="#[payload]" />
<os:clear doc:name="Clear" doc:id="bf89bf71-2974-4ff5-8880-0e9f09665ce8" objectStore="Object_store"/>
<logger level="INFO" doc:name="Logger" doc:id="361dc482-0757-434b-ba5a-5513d345d739" message="'Values inside object store after Contain Operation '" />
<os:retrieve-all doc:name="Retrieve all" doc:id="d28f0141-2d72-4b68-83f3-c888ae2ca995" objectStore="Object_store" />
<logger level="INFO" doc:name="Logger" doc:id="47111001-be54-49e3-bcb8-8f12006988fb" message="#[payload]" />
</flow>
</mule>
Opinions expressed by DZone contributors are their own.
Comments