Use MTOM to Efficiently Transmit Binary Content in SOAP

JSON-based REST services are en vogue, but when it comes to integrating enterprise services, SOAP is still widely used. In a recent project I had to write a spring boot based microservice, which was a kind of gateway for a third party SOAP based web service built on the Microsoft WCF. When called, the microservice was collecting some data from other microservices, loaded a PDF document from a storage system, and transferred both data and PDF to the SOAP service in one SOAP call. When the first implementation had been finished and deployed, I checked the access logs: wow, the request size was enormous on some requests. OK, the PDFs varied in size from some kB to several MB, but the request seemed to be much, much larger. This is because SOAP is using Base64 to encode binary content. And Base64 encodes 6 Bits to a character, means 3 Bytes are encoded into 4 characters. That's 33% more content than the raw data. For small content this is not an issue, but if you have to transfer 4MB instead of 3MB... that is an issue. Here is an example of a simple SOAP request with binary content. The content is just 30 Bytes which are encoded into 40 characters (the XML has been formatted for readability):

POST /ws/documents HTTP/1.1
...
Content-Type: text/xml; charset=utf-8
Content-Length: 400

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Header />
  <SOAP-ENV:Body>
    <ns2:storeDocumentRequest xmlns:ns2="https://github.com/ralfstuckert/mtom">
      <ns2:document>
        <ns2:name>30</ns2:name>
        <ns2:author>Herbert</ns2:author>
        <ns2:content>zenpS60mUBfHXln3tAbCds82IK1NKhAjAMoI9Czw</ns2:content>
      </ns2:document>
    </ns2:storeDocumentRequest>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

MTOM

To overcome this issue, MTOM - Message Transmission Optimization Mechanism - has been invented and standardized by the W3C. Instead of providing the binary content as Base64-encoded characters - contained as element text in the SOAP message - a multipart mime format is used, to devide the binary content from the SOAP message. Means: the SOAP message itself is contained in one part, and the binary content is in a separate part. The content element in the SOAP message just has a reference to the binary part. Sounds weird? Just have a look at it (again, the XML is formatted for readability):

POST /ws/documents HTTP/1.1
...
Content-Type: Multipart/Related; start-info="text/xml"; type="application/xop+xml"; boundary="----=_Part_0_2494886.1441553075493"
Content-Length: 842

------=_Part_0_2494886.1441553075493
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
  <SOAP-ENV:Header />
  <SOAP-ENV:Body>
    <ns2:storeDocumentRequest xmlns:ns2="https://github.com/ralfstuckert/mtom">
      <ns2:document>
        <ns2:name>30</ns2:name>
        <ns2:author>Herbert</ns2:author>
        <ns2:content>
          <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include"
            href="cid:ecc6eb7a-ce73-4ab7-8266-3bc2869cb0ae%40github.com" />
        </ns2:content>
      </ns2:document>
    </ns2:storeDocumentRequest>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>
------=_Part_0_2494886.1441553075493
Content-Type: application/octet-stream
Content-ID: <ecc6eb7a-ce73-4ab7-8266-3bc2869cb0ae@github.com>
Content-Transfer-Encoding: binary

!�öâ6[ê�ĨÅ
·Î½ªÖÓ$+yò'½ni
------=_Part_0_2494886.1441553075493--

As you can see, the part containing the binary data just contains the 30 bytes, not more. But there is for sure some overhead for the multipart metadata you have to pay. As a rule of thumb, MTOM only makes sense for content > 1 kB. If you have a look at the content element you will notice the xop element:

<ns2:content>
  <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include"
    href="cid:ecc6eb7a-ce73-4ab7-8266-3bc2869cb0ae%40github.com" />
</ns2:content>

While MTOM describes the abstract feature of optimizing the transmission in SOAP, the concrete implementation using MIME multiparts is kept in a separate specification: XOP, the XML-binary Optimized Packaging. By that it can be used indepently from SOAP for any binary content in XML documents. Therefore you will ofter find the wording MTOM/XOP.

Example Application

To give you an example to fool around with, I prepared a simple SOAP server and client on github implemented with spring boot. They are built with the STS, but you can build and run them with just plain Java; just check the Readme. Clone the repository and switch to the branch base64, it provides the initial setup with two spring projects mtom-server and mtom-client. Just start the server and the client as described in the Readme. The client asks you for a document size to upload. If you enter a size, the client will generate a document of that size (containing some random binary data), and upload it to the server. Both client and server also trace the request and response, so you can inspect them.

Client console output:

enter size of document to upload, or just press enter to exit: 30

Storing document of size 30
2015-09-07 17:27:22.645 TRACE 13988 --- [main] o.s.ws.client.MessageTracing.received: Received response [<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><ns2:storeDocumentResponse xmlns:ns2="https://github.com/ralfstuckert/mtom"><ns2:success>true</ns2:success></ns2:storeDocumentResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>] for request [<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><ns2:storeDocumentRequest xmlns:ns2="https://github.com/ralfstuckert/mtom"><ns2:document><ns2:name>30</ns2:name><ns2:author>Bert</ns2:author><ns2:content>UZrGmb6QfI78BHRezp2VvzCvtyzRkTYwXhP0FmM/</ns2:content></ns2:document></ns2:storeDocumentRequest></SOAP-ENV:Body></SOAP-ENV:Envelope>]
success: true

Server console output:

received 30 bytes
[2015-09-07 17:27:22.592] boot - 13712 TRACE [http-nio-9090-exec-1] --- sent: Sent response [<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><ns2:storeDocumentResponse xmlns:ns2="https://github.com/ralfstuckert/mtom"><ns2:success>true</ns2:success></ns2:storeDocumentResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>] for request [<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><ns2:storeDocumentRequest xmlns:ns2="https://github.com/ralfstuckert/mtom"><ns2:document><ns2:name>30</ns2:name><ns2:author>Bert</ns2:author><ns2:content>UZrGmb6QfI78BHRezp2VvzCvtyzRkTYwXhP0FmM/</ns2:content></ns2:document></ns2:storeDocumentRequest></SOAP-ENV:Body></SOAP-ENV:Envelope>]

The base of a SOAP service is usually a WSDL that defines the types and the operations. In our case we just provide a schema documents.xsd in the server project describing the types and use JAXB to create the Java classes. Spring than creates the WSDL than on the fly for us. Just start the server and browse to http://localhost:9090/ws/documents.wsdl. The WSDL is also contained in the client in the resources under wsdl/documents.wsdl. Here is an excerpt from the WSDL:

<xs:element name="storeDocumentRequest">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="document" type="tns:document" />
    </xs:sequence>
  </xs:complexType>
</xs:element>

<xs:element name="storeDocumentResponse">
  <xs:complexType>
    <xs:sequence>
      <xs:element name="success" type="xs:boolean" />
    </xs:sequence>
  </xs:complexType>
</xs:element>

<xs:complexType name="document">
  <xs:sequence>
    <xs:element name="name" type="xs:string" />
    <xs:element name="author" type="xs:string" />
    <xs:element name="content" type="xs:base64Binary" />
  </xs:sequence>
</xs:complexType>

<wsdl:portType name="DocumentsPort">
  <wsdl:operation name="storeDocument">
    <wsdl:input message="tns:storeDocumentRequest" name="storeDocumentRequest"></wsdl:input>
    <wsdl:output message="tns:storeDocumentResponse" name="storeDocumentResponse"></wsdl:output>
  </wsdl:operation>
</wsdl:portType>

So we just have on operation storeDocument with a document in the storeDocumentRequest, and a boolean in the storeDocumentResponse. The document itself contains some metadata like name and author, and - at last - the binary content.

Using MTOM

In order to use MTOM we have to apply some changes to our example. (You can either perform these steps on your example or directly check out the prepared solution in branch mtom). At first let's enable MTOM on the client, which is just enabling it on the JAXB marshaller:

@Bean
public Jaxb2Marshaller marshaller() {
  Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
  ...
  marshaller.setMtomEnabled(true);
  return marshaller;
}

Basically, that's the same we would have to do on server also. But we also have to tell spring to use that marshaller for the endpoints:

@Bean
public Jaxb2Marshaller marshaller() {
  Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
  marshaller.setContextPath("rst.sample.mtom.jaxb");
  marshaller.setMtomEnabled(true);
  return marshaller;
}

@Bean
@Override
public DefaultMethodEndpointAdapter defaultMethodEndpointAdapter() {
  List<MethodArgumentResolver> argumentResolvers =
  new ArrayList<MethodArgumentResolver>();
  argumentResolvers.add(methodProcessor());

  List<MethodReturnValueHandler> returnValueHandlers =
  new ArrayList<MethodReturnValueHandler>();
  returnValueHandlers.add(methodProcessor());

  DefaultMethodEndpointAdapter adapter = new DefaultMethodEndpointAdapter();
  adapter.setMethodArgumentResolvers(argumentResolvers);
  adapter.setMethodReturnValueHandlers(returnValueHandlers);

  return adapter;
}

@Bean
public MarshallingPayloadMethodProcessor methodProcessor() {
  return new MarshallingPayloadMethodProcessor(marshaller());
}

Also we have to add support for multiparts by providing an appropriate multipart resolver:

@Configuration
public class MultipartResolverConfig {

  @Bean
  public CommonsMultipartResolver multipartResolver() {
    return new CommonsMultipartResolver();
  }

  @Bean
  public CommonsMultipartResolver filterMultipartResolver() {
    final CommonsMultipartResolver resolver = new CommonsMultipartResolver();
    return resolver;
  }
}

That's it, if you run it, you will the following output on the client console:

enter size of document to upload, or just press enter to exit: 30

Storing document of size 30
2015-09-07 17:36:37.740 TRACE 11644 --- [main] o.s.ws.client.MessageTracing.received : Received response [------=_Part_2_8618207.1441640197738
Content-Type: application/xop+xml; charset=UTF-8; type="text/xml"

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><ns2:storeDocumentResponse xmlns:ns2="https://github.com/ralfstuckert/mtom"><ns2:success>true</ns2:success></ns2:storeDocumentResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
------=_Part_2_8618207.1441640197738--] for request [------=_Part_1_12274722.1441640197737
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Header/><SOAP-ENV:Body><ns2:storeDocumentRequest xmlns:ns2="https://github.com/ralfstuckert/mtom"><ns2:document><ns2:name>30</ns2:name><ns2:author>Bibo</ns2:author><ns2:content><xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:3f9e1eef-fc65-4bda-bcee-c2764a3cbf3a%40github.com"/></ns2:content></ns2:document></ns2:storeDocumentRequest></SOAP-ENV:Body></SOAP-ENV:Envelope>
------=_Part_1_12274722.1441640197737
Content-Type: application/octet-stream
Content-ID: <3f9e1eef-fc65-4bda-bcee-c2764a3cbf3a@github.com>
Content-Transfer-Encoding: binary

&.}p
R�4Ѿ��m��B�C�T�yK}>}
------=_Part_1_12274722.1441640197737--]

MTOM and Streaming

If you look at the Java classes generated from the model, you will see that binary content is hold in a byte array:

public class Document {

   @XmlElement(required = true)
   protected String name;

   @XmlElement(required = true)
   protected String author;

   @XmlElement(required = true)
   protected byte[] content;
...

When you are dealing with large binary data, this is a problem, since the complete data has to be kept in memory. The OutOfMemoryException is awaiting you. The solution to this problem is streaming: instead of keeping the data in memory, you provide the data in a stream, resp. read the data from a stream. This is quite natural, since most datastores like e.g. filesystem, database etc. provide streaming interfaces. Even if the MTOM spec does not say a single word about streaming, the XOP spec says that - even if it is not mandatory - most implementations provide the possibility to stream data. Let's do this now in our example. Again, you can follow the steps and convert your MTOM application into streaming, or checkout the master branch of project. The master provides the final version for you. At first we need a way to provide a streaming interface in our Java classes. The way to do this, is to change the schema a little bit. Just add the attribute xmime:expectedContentTypes="application/octet-stream" to the content element:

<xs:complexType name="document">
  <xs:sequence>
    <xs:element name="name" type="xs:string" />
    <xs:element name="author" type="xs:string" />
    <xs:element name="content" type="xs:base64Binary" xmime:expectedContentTypes="application/octet-stream" />
  </xs:sequence>
</xs:complexType>

Now JAXB gerenates a DataHandler instead of byte[] for the field content:

public class Document {
  ...
  @XmlElement(required = true)
  @XmlMimeType("application/octet-stream")
  protected DataHandler content;

A DataHandler is based on streams, so you may use Input- and OutputStreams to pass data. That's what we do; instead of reading our content into a byte array first, we directly pass our input stream to the DataHandler on the client:

public class DocumentsClient extends WebServiceGatewaySupport {

  public boolean storeDocument(int size) {
    Document document = new Document();
    document.setContent(getContentAsDataHandler(size));
    ...
  }

  private DataHandler getContentAsDataHandler(final int size) {
    InputStream input = getContentAsStream(size);
    DataSource source = new InputStreamDataSource(input, ...
    return new DataHandler(source);
  }

On the server side we also get a DataHandler which we use to directly read the data from the InputStream. That's it? Just try it... hmm, still out of memory. The client still seems to read the content into memory first. The answer to this problem is the HTTP protocol, let's recap our MTOM request:

POST /ws/documents HTTP/1.1
...
Content-Type: Multipart/Related; start-info="text/xml"; type="application/xop+xml"; boundary="----=_Part_0_2494886.1441553075493"
Content-Length: 842

HTTP wants the content-length in advance, so the content is read completely into memory in order to calculate the content-length. To avoid this, we have to use the chunked transfer encoding:

  public DocumentsClient() {
    setMessageSender(new ChunkedEncodingMessageSender());
  }
  ...

public class ChunkedEncodingMessageSender extends HttpUrlConnectionMessageSender {
  protected void prepareConnection(final HttpURLConnection connection) throws IOException {
    super.prepareConnection(connection);
    connection.setChunkedStreamingMode(-1);
  }
}

Now we have got it? Let's retry... holy chihuahua, now the server moans. But it should work right away now?!? That's a bug in the SAAJ implementation, see SAAJ-31. As you can read over there, we have to set a switch to force SAAJ to use mimepull, so we do so:

@SpringBootApplication
public class Application {

  public static void main(String[] args) {
    // needed for streaming, see https://java.net/jira/browse/SAAJ-31
    System.setProperty("saaj.use.mimepull", "true");

    SpringApplication.run(Application.class, args);
  }
}

Now it works... really. Try it with 1.000.000.000 bytes (the SOAP tracing is disbaled in master, so don't worry). It will take some time, since our random data InputStream generates every single byte:

enter size of document to upload, or just press enter to exit: 1000000000

Storing document of size 1000000000
success: true

MTOM and Streaming in WCF

In the beginning I told you about my project where I had to communicate with a third party server product build on .NET and the WCF. This product was not using MTOM at first, so I had to make changes to that software also. Luckily, in WCF this just some configuration you have to do... and I had access to that configuration file ;-) In the HTTP binding, you have to set the attribute messageEncoding to Mtom. To use streaming, just set the attribute transferMode to Streamed:

<bindings>
  <basicHttpBinding>
    <binding name="...." messageEncoding="Mtom" transferMode="Streamed" >

That was easy, eh? Yep, the WCF handles all those nifty little details for you.

Conclusion

MTOM allows you to efficiently transmit large binary data, and even allows to stream it in order to avoid memory problems. Yep, there other streaming mechansims available; but if you want your SOAP service to interoperate with others, the MTOM standard is your choice.

Confused? You won't be after this episode of Soap!
Announcer of Soap, a late 70s sitcom I loved to watch :-)