Unit Testing Mule DataWeave Scripts with MUnit

dataweave is a powerful transformation language introduced with mule enterprise edition 3.7. it allows you to transform data from one format to another and supports csv, xml, json, flat/fixed width (v3.8+) & java. you can look at these dataweave examples to see it in action.

like any other code of programming world, it is always a good idea to unit test the dataweave script you write. in this post, we will see how we can unit test the dataweave code.

writing dataweave script

dataweave script can be included in two ways into mule flow -

1. add an inline script -

<dw:transform-message doc:name="transform message">
    <dw:set-payload><![cdata[%dw 1.0
    %output application/java
    ---
    {
        employees: payload.root.*employee map {

            name: $.fname ++ ' ' ++ $.lname,
            dob: $.dob,
            age: (now as :string {format: "yyyy"}) - 
            (($.dob as :date {format:"mm-dd-yyyy"}) as :string {format:"yyyy"})

        }
    }]]></dw:set-payload>
</dw:transform-message>

2. add script to file (.dwl) and refer with resource attribute -

 <dw:transform-message doc:name="transform message">
     <dw:set-payload resource="classpath:dwl/employees.dwl"/>
 </dw:transform-message>

i prefer using resource option for writing my dataweave scripts. this has few advantages over inline option -

1. script (.dwl) is reusable by other transform messages components by referring to the same file.
2. mule configuration xml file remains clean and readable.
3. most important for us, that makes it possible to test the script as a unit of code.
4. hmm, there may be more but i just don't know them yet.

flow with dataweave script

to keep demonstration simple, we will use below flow that consumes an employee's .xml and transforms it to a java map. during transformation it also calculates the employee's age.

<?xml version="1.0" encoding="utf-8"?>

    <mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:spring="http://www.springframework.org/schema/beans" 
    xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"
    xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
    http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
    http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
    http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd">

        <flow name="dataweave-testingflow">
            <file:inbound-endpoint path="input" movetodirectory="output" responsetimeout="10000" doc:name="file"/>
                <dw:transform-message doc:name="transform message">
                    <dw:set-payload resource="classpath:dwl/employees.dwl"/>
                </dw:transform-message>
            <logger level="info" message="#[message.payloadas(java.lang.string)]" doc:name="logger"/>
        </flow>
    </mule>

dataweave script - employees.dwl

%dw 1.0
%output application/java
---
employees: payload.root.*employee map {

    name: $.fname ++ ' ' ++ $.lname,
    dob: $.dob,
    age: (now as :string {format: "yyyy"}) - 
    (($.dob as :date {format:"mm-dd-yyyy"}) as :string {format:"yyyy"})

}

first munit test

we will use munit for writing our unit test cases. if you haven't written any munit test cases before then you can take a look at munit tutorial .

to write our first unit test, we will create a new munit test suite src/test/munit/dataweave-testing-test-suite.xml with the code below:

<?xml version="1.0" encoding="utf-8"?>

    <mule xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns="http://www.mulesoft.org/schema/mule/core"
        xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
        xmlns:munit="http://www.mulesoft.org/schema/mule/munit" xmlns:spring="http://www.springframework.org/schema/beans"
        xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"
        xsi:schemalocation="http://www.mulesoft.org/schema/mule/munit http://www.mulesoft.org/schema/mule/munit/current/mule-munit.xsd
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
        http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
        http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd">
            <munit:config name="munit" doc:name="munit configuration" />

            <munit:test name="dataweave-testing-test-suite-dataweave-testingflowtest"
                description="test">
                    <munit:set
                        payload="#[getresource('sample_data/employees.xml').asstream()]"
                        doc:name="set message" mimetype="application/xml" />
                    <dw:transform-message doc:name="transform message">
                        <dw:set-payload resource="classpath:dwl/employees.dwl" />
                    </dw:transform-message>
                    <munit:assert-on-equals expectedvalue="#[2]"
                        actualvalue="#[payload.employees.size()]" doc:name="assert equals"
                        message="missing some employees" />
                    <munit:assert-on-equals expectedvalue="#[36]"
                        actualvalue="#[payload.employees[0].age]" doc:name="assert equals" />
            </munit:test>
    </mule>

this code has an munit test dataweave-testing-test-suite-dataweave-testingflowtest . you can see that we are not importing our actual flow config and that is because, we will add a transform-message component and refer to the same dataweave script resource dwl/employees.dwl that main flow uses (reuse and unit testability of script!!). here is what this test is doing:

1. create a test message using sample xml file as payload. we can use mel expression to read file as stream. we will set the mimetype of message as "application/xml".
2. transform the input xml to csv using employees.dwl script. as we are transforming it into java object, output of dw will be a hashmap with employee list.
3. assert the number of employees we except in dataweave output.
4. for first employee record, assert the expected value of age. this will ensure that our age calculation is working as expected.

important: not setting mimetype on test message will cause dataweave to throw below exception because dataweave will recieve the input as binary input stream and wouldn't know how to interpret content of it.

message               : exception while executing: 
employees: payload.root.*employee map {
           ^
    type mismatch for 'value selector' operator
    found :binary, :name
    required :datetime, :name or
    required :localdatetime, :name or
    required :object, :name or
    required :time, :name or
    required :array, :name or
    required :date, :name or
    required :localtime, :name or
    required :period, :name
    element               : /dataweave-testing-test-suite-dataweave-testingflowtest/processors/1 @ 22f3f850-4d52-11e6-b92d-1a0124cf99a6:dataweave-testing-test-suite.xml:17 (transform message)

now run this as munit test case and you should see it running successfully:

writing java unit test case

for those who prefer to write java instead of xml, functionalmunitsuite , class can be used to write the test case.

let's create an src/test/munit/dataweave-testing-munit.xml mule config (not a munit xml suite) and add a test subflow with target dataweave component. we are using the same dwl resource file.

<?xml version="1.0" encoding="utf-8"?>

    <mule xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
    xmlns:spring="http://www.springframework.org/schema/beans" 
    xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"
    xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
    http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
    http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd">

        <sub-flow name="dataweave-testing-suitesub_flow">
            <dw:transform-message doc:name="transform message">
                <dw:set-payload resource="classpath:dwl/employees.dwl"></dw:set-payload>
            </dw:transform-message>
        </sub-flow>

    </mule>

here is our java test case equivalent to earlier xml test -

package com.mms.mule.explore;

import java.io.file;
import java.util.hashmap;
import java.util.list;
import java.util.map;

import org.hamcrest.matcherassert;
import org.hamcrest.matchers;
import org.junit.test;
import org.mule.defaultmulemessage;
import org.mule.api.muleevent;
import org.mule.munit.runner.functional.functionalmunitsuite;
import org.mule.transformer.types.mimetypes;
import org.mule.util.fileutils;

public class dataweavetests extends functionalmunitsuite {

    @override
    protected string getconfigresources() {
        return "dataweave-testing-suite.xml";
    }

    @test
    public void testdw() throws exception{

        string payload = fileutils.readfiletostring(new file(dataweavetests.class.getclassloader().getresource("sample_data/employees.xml").getpath()));

        muleevent event = testevent(payload);
        //setting mimetype is critical.
        ((defaultmulemessage)event.getmessage()).setmimetype(mimetypes.application_xml);

        //call our test flow
        muleevent reply = runflow("dataweave-testing-suitesub_flow", event);

        hashmap obj = reply.getmessage().getpayload(hashmap.class);
        list<map> lst = (list<map>) obj.get("employees");
        //put some asserts
        matcherassert.assertthat(2, matchers.equalto(lst.size()));
        matcherassert.assertthat(36, matchers.equalto(lst.get(0).get("age")));

    }
}

verifying csv content output

in the previous example, dataweave output was java map which is easy to verify. how about verifying csv ouput? there are two ways to verify csv output:

verify as strings:

• after dataweave, use object-to-string transformer to convert output to string.
• split the content with new line (you may want to replace '\r\n' with '\n' before splitting)
• verify values in array.

use another dataweave to convert csv to map and then verify the map data:

below munit flow adds another dataweave which outputs application/java and script is as simple as (payload) which converts the csv as is to java map. you can see in second debug screenshot below:

<munit:test name="dataweave-testing-test-suite-dataweave-csv-testingflowtest"
    description="test">
        <munit:set
            payload="#[getresource('sample_data/employees.xml').asstream()]"
            doc:name="set message" mimetype="application/xml" />
        <dw:transform-message doc:name="transform message">
            <dw:set-payload resource="classpath:dwl/employees2.dwl" />
        </dw:transform-message>
            <dw:transform-message doc:name="transform message">
                <dw:set-payload><![cdata[%dw 1.0
                    %output application/java
                    ---
                    (payload)]]></dw:set-payload>
            </dw:transform-message>
            <munit:assert-on-equals expectedvalue="#[2]"
                actualvalue="#[payload.size()]" doc:name="assert equals"
                message="missing some employees" />
            <munit:assert-on-equals expectedvalue="#['36']"
            actualvalue="#[payload[0].age]" doc:name="assert equals" />
        </munit:test>

if you look at the second assert that verifies age and compare with that of earlier java testing, you will notice that expected value is defined as string literal #['36'] vs. number #[36] . this is because, all values from csv are transformed as string data type in map and #[36] would cause test case to fail. to have strongly typed values, we can write a full mapping in second dataweave but i am skipping that step for now.

with this minimal setup, you can verify the data in your munit.

troubleshooting

org.threeten.bp.zone.tzdbzonerulesprovider could not be instantiated while running java test case

if you are manipulating dates in your dataweave script and writing test cases in java, then you may see tests failing with below error -

org.mule.api.messagingexception: org.threeten.bp.zone.zonerulesprovider: provider org.threeten.bp.zone.tzdbzonerulesprovider could not be instantiated (java.util.serviceconfigurationerror).
at org.mule.execution.exceptiontomessagingexceptionexecutioninterceptor.execute(exceptiontomessagingexceptionexecutioninterceptor.java:42)
at org.mule.execution.messageprocessornotificationexecutioninterceptor.execute(messageprocessornotificationexecutioninterceptor.java:108)
  ...
caused by: java.util.serviceconfigurationerror: org.threeten.bp.zone.zonerulesprovider: provider org.threeten.bp.zone.tzdbzonerulesprovider could not be instantiated
at java.util.serviceloader.fail(serviceloader.java:232)
at java.util.serviceloader.access$100(serviceloader.java:185)
  ...
caused by: org.threeten.bp.zone.zonerulesexception: unable to load tzdb time-zone rules: jar:file:/users/manik/.m2/repository/org/threeten/threetenbp/1.2/threetenbp-1.2.jar!/org/threeten/bp/tzdb.dat
at org.threeten.bp.zone.tzdbzonerulesprovider.load(tzdbzonerulesprovider.java:146)
at org.threeten.bp.zone.tzdbzonerulesprovider.<init>(tzdbzonerulesprovider.java:87)
  ...
caused by: org.threeten.bp.zone.zonerulesexception: data already loaded for tzdb time-zone rules version: 2014i
at org.threeten.bp.zone.tzdbzonerulesprovider.load(tzdbzonerulesprovider.java:139)
... 87 more

resolution: this error is thrown when threetenbp library gets loaded twice. tzdbzonerulesprovider is already available in java runtime and dataweave pulls this jar as its dependency. simple resolution is to exclude this from maven dependency, modify dataweave plugin dependency in your pom:

<dependency>
    <groupid>com.mulesoft.weave</groupid>
    <artifactid>mule-plugin-weave_2.11</artifactid>
    <version>${mule.version}</version>
    <scope>provided</scope>
    <exclusions>
        <exclusion>
            <groupid>org.threeten</groupid>
                <artifactid>threetenbp</artifactid>
        </exclusion>
    </exclusions>
</dependency>

arrayindexoutofbound exception when setting csv payload

if you are testing csv input payload with dataweave on windows, then you may get arrayindexoutofbound exception. i think this is a bug due windows formatted eol (\r\n) characters.

resolution: you can use tools like notepad++ or dos2unix to convert your file to unix (\n) eol format. i usually like to do it like below which makes test compatible with both formats -

string payload = fileutils.readfiletostring(new file(dataweavetests.class.getclassloader().getresource("sample_data/employees.csv").getpath()));
payload = payload.replace("\r\n", "\n");

conclusion

unit testing is a crucial part of any software development. mule esb provides numerous components for system integrations and data transformation. in this post, we saw how we can write unit test cases for dataweave (transform message) component and ensure the transformed data is as per expecations. i hope this will help you to write (almost) bug-free scripts.

feel free to comment and let me know your thoughts or questions.