Integration Testing with MongoDB & Spring Data

Integration Testing is an often overlooked area in enterprise development. This is primarily due to the associated complexities in setting up the necessary infrastructure for an integration test. For applications backed by databases, it’s fairly complicated and time-consuming to setup databases for integration tests, and also to clean those up once test is complete (ex. data files, schemas etc.), to ensure repeatability of tests. While there have been many tools (ex. DBUnit) and mechanisms (ex. rollback after test) to assist in this, the inherent complexity and issues have been there always.

But if you are working with MongoDB, there’s a cool and easy way to do your unit tests, with almost the simplicity of writing a unit test with mocks. With ‘EmbedMongo’, we can easily setup an embedded MongoDB instance for testing, with in-built clean up support once tests are complete. In this article, we will walkthrough an example where EmbedMongo is used with JUnit for integration testing a Repository Implementation.

Here’s the technology stack that we will be using.

• MongoDB 2.2.0
• EmbedMongo 1.26
• Spring Data – Mongo 1.0.3
• Spring Framework 3.1

The Maven POM for the above setup looks like this.

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.yohanliyanage.blog.mongoit</groupId>
  <artifactId>mongo-it</artifactId>
  <version>1.0</version>
  <dependencies>
    <dependency>
      <groupId>org.springframework.data</groupId>
      <artifactId>spring-data-mongodb</artifactId>
      <version>1.0.3.RELEASE</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.10</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-context</artifactId>
      <version>3.1.3.RELEASE</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>de.flapdoodle.embed</groupId>
      <artifactId>de.flapdoodle.embed.mongo</artifactId>
      <version>1.26</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

Or if you prefer Gradle (by the way, Gradle is an awesome build tool which you should check out if you haven’t done so already).

apply plugin: 'java'
apply plugin: 'eclipse'

sourceCompatibility = 1.6
group = "com.yohanliyanage.blog.mongoit"
version = '1.0'

ext.springVersion = '3.1.3.RELEASE'
ext.junitVersion = '4.10'
ext.springMongoVersion = '1.0.3.RELEASE'
ext.embedMongoVersion = '1.26'

repositories {
    mavenCentral()
    maven { url 'http://repo.springsource.org/release' }
}

dependencies {
    compile "org.springframework:spring-context:${springVersion}"
    compile "org.springframework.data:spring-data-mongodb:${springMongoVersion}"
    testCompile "junit:junit:${junitVersion}"
    testCompile "de.flapdoodle.embed:de.flapdoodle.embed.mongo:${embedMongoVersion}"
}

To begin with, here’s the document that we will be storing in Mongo.

package com.yohanliyanage.blog.mongoit.model;

import org.springframework.data.mongodb.core.index.Indexed;
import org.springframework.data.mongodb.core.mapping.Document;

/**
* A Sample Document.
*
* @author Yohan Liyanage
*
*/
@Document
public class Sample {

    @Indexed
    private String key;

    private String value;

    public Sample(String key, String value) {
        super();
        this.key = key;
        this.value = value;
    }

    public String getKey() {
        return key;
    }

    public void setKey(String key) {
        this.key = key;
    }

    public String getValue() {
        return value;
    }

    public void setValue(String value) {
        this.value = value;
    }

}

To assist with storing and managing this document, let’s write up a simple Repository implementation. The Repository Interface is as follows.

package com.yohanliyanage.blog.mongoit.repository;

import java.util.List;

import com.yohanliyanage.blog.mongoit.model.Sample;

/**
* Sample Repository API.
*
* @author Yohan Liyanage
*
*/
public interface SampleRepository {

    /**
* Persists the given Sample.
* @param sample
*/
    void save(Sample sample);
    
    /**
* Returns the list of samples with given key.
* @param sample
* @return
*/
    List<Sample> findByKey(String key);
}

And the implementation…

package com.yohanliyanage.blog.mongoit.repository;

import java.util.List;

import static org.springframework.data.mongodb.core.query.Query.query;
import static org.springframework.data.mongodb.core.query.Criteria.*;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.stereotype.Repository;

import com.yohanliyanage.blog.mongoit.model.Sample;

/**
* Sample Repository MongoDB Implementation.
*
* @author Yohan Liyanage
*
*/
@Repository
public class SampleRepositoryMongoImpl implements SampleRepository {

    @Autowired
    private MongoOperations mongoOps;
    
    /**
* {@inheritDoc}
*/
    public void save(Sample sample) {
        mongoOps.save(sample);
    }

    /**
* {@inheritDoc}
*/
    public List<Sample> findByKey(String key) {
        return mongoOps.find(query(where("key").is(key)), Sample.class);
    }

    /**
* Sets the MongoOps implementation.
*
* @param mongoOps the mongoOps to set
*/
    public void setMongoOps(MongoOperations mongoOps) {
        this.mongoOps = mongoOps;
    }

}

To wire this up, we need a Spring Bean Configuration. Note that we do not need this for testing. But for the sake of completion, I have included this. The XML configuration is as follows.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:mongo="http://www.springframework.org/schema/data/mongo"
	xmlns:context="http://www.springframework.org/schema/context"
	xsi:schemaLocation="http://www.springframework.org/schema/data/mongo http://www.springframework.org/schema/data/mongo/spring-mongo-1.0.xsd
		http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.1.xsd">

	<!-- Enable Annotation Driven Configuration -->
	<context:annotation-config />
	
	<!-- Component Scan Packages for Annotation Driven Configuration -->
	<context:component-scan base-package="com.yohanliyanage.blog.mongoit.repository" />

	<!-- Mongo DB -->
	<mongo:mongo host="127.0.0.1" port="27017" />
	
	<!-- Mongo DB Factory -->
	<mongo:db-factory dbname="mongoit" mongo-ref="mongo"/>
	
	<!-- Mongo Template -->
	<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
		<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
	</bean>
	
</beans>

And now we are ready to write the Integration Test for our Repository Implementation using Embed Mongo.

Ideally, the integration tests should be placed in a separate source directory, just like we place our unit tests (ex. src/test/java => src/integration-test/java). However, neither Maven nor Gradle supports this out of the box (yet – v1.2. For Gradle, there’s an on going discussion for this facility).

Nevertheless, both Maven and Gradle are flexible, so you can configure the POM / build.gradle to handle this. However, to keep this discussion simple and focused, I will be placing the Integration Tests in the ‘src/test/java’, but I do not recommend this for a real application.

Let’s start writing up the Integration Test. First, let’s begin with a simple JUnit based Test for the methods.

package com.yohanliyanage.blog.mongoit.repository;

import static org.junit.Assert.fail;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

/**
 * Integration Test for {@link SampleRepositoryMongoImpl}.
 * 
 * @author Yohan Liyanage
 */
public class SampleRepositoryMongoImplIntegrationTest {

    private SampleRepositoryMongoImpl repoImpl;

    @Before
    public void setUp() throws Exception {
        repoImpl = new SampleRepositoryMongoImpl();
    }

    @After
    public void tearDown() throws Exception {
    }
    

    @Test
    public void testSave() {
        fail("Not yet implemented");
    }

    @Test
    public void testFindByKey() {
        fail("Not yet implemented");
    }

}

When this JUnit Test Case initializes, we need to fire up EmbedMongo to start an embedded Mongo server. Also, when the Test Case ends, we need to cleanup the DB. The below code snippet does this.

package com.yohanliyanage.blog.mongoit.repository;

import static org.junit.Assert.fail;

import java.io.IOException;

import org.junit.*;
import org.springframework.data.mongodb.core.MongoTemplate;

import com.mongodb.Mongo;
import com.yohanliyanage.blog.mongoit.model.Sample;

import de.flapdoodle.embed.mongo.MongodExecutable;
import de.flapdoodle.embed.mongo.MongodProcess;
import de.flapdoodle.embed.mongo.MongodStarter;
import de.flapdoodle.embed.mongo.config.MongodConfig;
import de.flapdoodle.embed.mongo.config.RuntimeConfig;
import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.process.extract.UserTempNaming;

/**
* Integration Test for {@link SampleRepositoryMongoImpl}.
*
* @author Yohan Liyanage
*/
public class SampleRepositoryMongoImplIntegrationTest {

    private static final String LOCALHOST = "127.0.0.1";
    private static final String DB_NAME = "itest";
    private static final int MONGO_TEST_PORT = 27028;
    
    private SampleRepositoryMongoImpl repoImpl;

    private static MongodProcess mongoProcess;
    private static Mongo mongo;
    
    private MongoTemplate template;
    

    @BeforeClass
    public static void initializeDB() throws IOException {

        RuntimeConfig config = new RuntimeConfig();
        config.setExecutableNaming(new UserTempNaming());

        MongodStarter starter = MongodStarter.getInstance(config);

        MongodExecutable mongoExecutable = starter.prepare(new MongodConfig(Version.V2_2_0, MONGO_TEST_PORT, false));
        mongoProcess = mongoExecutable.start();

        mongo = new Mongo(LOCALHOST, MONGO_TEST_PORT);
        mongo.getDB(DB_NAME);
    }

    @AfterClass
    public static void shutdownDB() throws InterruptedException {
        mongo.close();
        mongoProcess.stop();
    }

    
    @Before
    public void setUp() throws Exception {
        repoImpl = new SampleRepositoryMongoImpl();
        template = new MongoTemplate(mongo, DB_NAME);
        repoImpl.setMongoOps(template);
    }

    @After
    public void tearDown() throws Exception {
        template.dropCollection(Sample.class);
    }

    @Test
    public void testSave() {
        fail("Not yet implemented");
    }

    @Test
    public void testFindByKey() {
        fail("Not yet implemented");
    }

}

The initializeDB() method is annotated with @BeforeClass to start this before test case beings. This method fires up an embedded MongoDB instance which is bound to the given port, and exposes a Mongo object which is set to use the given database. Internally, EmbedMongo creates the necessary data files in temporary directories.

When this method executes for the first time, EmbedMongo will download the necessary Mongo implementation (denoted by Version.V2_2_0 in above code) if it does not exist already. This is a nice facility specially when it comes to Continuous Integration servers. You don’t have to manually setup Mongo in each of the CI servers. That’s one less external dependency for the tests.

In the shutdownDB() method, which is annotated with @AfterClass, we stop the EmbedMongo process. This triggers the necessary cleanups in EmbedMongo to remove the temporary data files, restoring the state to where it was before Test Case was executed.

We have now updated setUp() method to build a Spring MongoTemplate object which is backed by the Mongo instance exposed by EmbedMongo, and to setup our RepoImpl with that template. The tearDown() method is updated to drop the ‘Sample’ collection to ensure that each of our test methods start with a clean state.

Now it’s just a matter of writing the actual test methods.

Let’s start with the save method test.

@Test
public void testSave() {
    Sample sample = new Sample("TEST", "2");
    repoImpl.save(sample);
    
    int samplesInCollection = template.findAll(Sample.class).size();
    
    assertEquals("Only 1 Sample should exist collection, but there are " 
            + samplesInCollection, 1, samplesInCollection);
}

We create a Sample object, pass it to repoImpl.save(), and assert to make sure that there’s only one Sample in the Sample collection. Simple, straight-forward stuff.

And here’s the test method for findByKey method.

@Test
public void testFindByKey() {
    
    // Setup Test Data
    List<Sample> samples = Arrays.asList(
            new Sample("TEST", "1"), new Sample("TEST", "25"),
            new Sample("TEST2", "66"), new Sample("TEST2", "99"));
    
    for (Sample sample : samples) {
        template.save(sample);
    }
    
    // Execute Test
    List<Sample> matches = repoImpl.findByKey("TEST");
    
    // Note: Since our test data (populateDummies) have only 2
    // records with key "TEST", this should be 2
    assertEquals("Expected only two samples with key TEST, but there are "
            + matches.size(), 2, matches.size());
}

Initially, we setup the data by adding a set of Sample objects into the data store. It’s important that we directly use template.save() here, because repoImpl.save() is a method under-test. We are not testing that here, so we use the underlying “trusted” template.save() during data setup. This is a basic concept in Unit / Integration testing. Then we execute the method under test ‘findByKey’, and assert to ensure that only two Samples matched our query.

Likewise, we can continue to write more tests for each of the repository methods, including negative tests. And here’s the final Integration Test file.

package com.yohanliyanage.blog.mongoit.repository;

import static org.junit.Assert.*;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

import org.junit.*;
import org.springframework.data.mongodb.core.MongoTemplate;

import com.mongodb.Mongo;
import com.yohanliyanage.blog.mongoit.model.Sample;

import de.flapdoodle.embed.mongo.MongodExecutable;
import de.flapdoodle.embed.mongo.MongodProcess;
import de.flapdoodle.embed.mongo.MongodStarter;
import de.flapdoodle.embed.mongo.config.MongodConfig;
import de.flapdoodle.embed.mongo.config.RuntimeConfig;
import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.process.extract.UserTempNaming;

/**
 * Integration Test for {@link SampleRepositoryMongoImpl}.
 * 
 * @author Yohan Liyanage
 */
public class SampleRepositoryMongoImplIntegrationTest {

    private static final String LOCALHOST = "127.0.0.1";
    private static final String DB_NAME = "itest";
    private static final int MONGO_TEST_PORT = 27028;
    
    private SampleRepositoryMongoImpl repoImpl;

    private static MongodProcess mongoProcess;
    private static Mongo mongo;
    
    private MongoTemplate template;
    

    @BeforeClass
    public static void initializeDB() throws IOException {

        RuntimeConfig config = new RuntimeConfig();
        config.setExecutableNaming(new UserTempNaming());

        MongodStarter starter = MongodStarter.getInstance(config);

        MongodExecutable mongoExecutable = starter.prepare(new MongodConfig(Version.V2_2_0, MONGO_TEST_PORT, false));
        mongoProcess = mongoExecutable.start();

        mongo = new Mongo(LOCALHOST, MONGO_TEST_PORT);
        mongo.getDB(DB_NAME);
    }

    @AfterClass
    public static void shutdownDB() throws InterruptedException {
        mongo.close();
        mongoProcess.stop();
    }

    
    @Before
    public void setUp() throws Exception {
        repoImpl = new SampleRepositoryMongoImpl();
        template = new MongoTemplate(mongo, DB_NAME);
        repoImpl.setMongoOps(template);
    }

    @After
    public void tearDown() throws Exception {
        template.dropCollection(Sample.class);
    }
    

    @Test
    public void testSave() {
        Sample sample = new Sample("TEST", "2");
        repoImpl.save(sample);
        
        int samplesInCollection = template.findAll(Sample.class).size();
        
        assertEquals("Only 1 Sample should exist in collection, but there are " 
                + samplesInCollection, 1, samplesInCollection);
    }

    @Test
    public void testFindByKey() {
        
        // Setup Test Data
        List<Sample> samples = Arrays.asList(
                new Sample("TEST", "1"), new Sample("TEST", "25"), 
                new Sample("TEST2", "66"),  new Sample("TEST2", "99"));
        
        for (Sample sample : samples) {
            template.save(sample);
        }        
        
        // Execute Test
        List<Sample> matches = repoImpl.findByKey("TEST");
        
        // Note: Since our test data (populateDummies) have only 2 
        // records with key "TEST", this should be 2
        assertEquals("Expected only two samples with key TEST, but there are " 
                + matches.size(), 2, matches.size());
    }
    
}

On a side note, one of the key concerns with Integration Tests is the execution time. We all want to keep our test execution times as low as possible, ideally a couple of seconds to make sure that we can run all the tests during CI, with minimal build and verification times. However, since Integration Tests rely on underlying infrastructure, usually Integration Tests take time to run. But with EmbedMongo, this is not the case. In my machine, above test suite runs in 1.8 seconds, and each test method takes only .166 seconds max. See the screenshot below.

I have uploaded the code for above project into GitHub. You can download / clone it from here: https://github.com/yohanliyanage/blog-mongo-integration-tests. For more information regarding EmbedMongo, refer to their site at GitHub https://github.com/flapdoodle-oss/embedmongo.flapdoodle.de.