Testcontainers With Kotlin and Spring Data R2DBC

In this article, we will discuss testcontainers library and how to use it to simplify our life when it comes to integration testing our code.

For the purpose of this example, I am going to use a simple application with its business centered around reviews for some courses. Basically, the app is a service that exposes some GraphQL endpoints for review creation, querying, and deletion from a PostgreSQL database via Spring Data R2DBC. The app is written in Kotlin using Spring Boot 2.7.3. 

I decided to write this article, especially for Spring Data R2DBC, as with Spring Data JPA, integration testing with testcontainers is straightforward. Still, when it comes to R2DBC, there are some challenges that need to be addressed.

Review the App

So let’s cut to the chase. Let’s examine our domain.

@Table("reviews")
data class Review(
    @Id
    var id: Int? = null,
    var text: String,
    var author: String,
    @Column("created_at")
    @CreatedDate
    var createdAt: LocalDateTime? = null,
    @LastModifiedDate
    @Column("last_modified_at")
    var lastModifiedAt: LocalDateTime? = null,
    @Column("course_id")
    var courseId: Int
)

And here is its repository:

@Repository
interface ReviewRepository : R2dbcRepository<Review, Int> {
    @Query("select * from reviews r where date(r.created_at) = :date")
    fun findAllByCreatedAt(date: LocalDate): Flux<Review>
    fun findAllByAuthor(author: String): Flux<Review>
    fun findAllByCreatedAtBetween(startDateTime: LocalDateTime, endDateTime: LocalDateTime): Flux<Review>
}

And here are the connection properties:

spring:
  data:
    r2dbc:
      repositories:
        enabled: true
  r2dbc:
    url: r2dbc:postgresql://localhost:5436/reviews-db
    username: postgres
    password: 123456

When it comes to testing, Spring offers a fairly easy way to set up an in-memory H2 database for this purpose, but… There is always a but. H2 comes with some disadvantages:

• First, usually, H2 is not the production DB; in our case, we use PostgreSQL, and it is hard to maintain two DB schemas, one for production use and one for integration testing, especially if you depend on some provider features (queries, functions, constraints and so on). To have as much confidence as possible in tests, it is always advisable to replicate the production environment as much as possible, which is not the case with H2.
• Another disadvantage might be the possibility of production migration to another database – in this case, H2 having another schema won’t catch any issues, therefore, is unreliable.

Others might argue that you can have separate testing dedicated DB on a server, machine, or even in a different schema. Still, again, the hassle of maintaining schemas and the possibility of collision with some other developer`s changes/migrations is way too big.

Testcontainers to the Rescue

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

With Testcontainers, all the disadvantages mentioned above are gone: because you are working with the production-like database, you don’t need to maintain two or more different separate schemas, you don’t need to worry about migration scenarios since you’ll have failing tests, and last, but not least you don’t need to worry about another server/VM’s maintenance since Testcontainers/Docker will take care of that.

Here are the dependencies that we are going to use:

testImplementation("org.testcontainers:testcontainers:1.17.3")
testImplementation("org.testcontainers:postgresql:1.17.3")

There are different ways of working with Testcontainers in Spring Boot. Still, I will show you the `singleton-instance pattern` (one database for all tests) since it is much faster to fire up on a database instance once and let all your tests communicate with it. For this to work, I am going to create an abstract class holding all the Testcontainers instances/start-up creation logic.

@Tag("integration-test")
abstract class AbstractTestcontainersIntegrationTest {

    companion object {

        private val postgres: PostgreSQLContainer<*> = PostgreSQLContainer(DockerImageName.parse("postgres:13.3"))
            .apply {
                this.withDatabaseName("testDb").withUsername("root").withPassword("123456")
            }

        @JvmStatic
        @DynamicPropertySource
        fun properties(registry: DynamicPropertyRegistry) {
            registry.add("spring.r2dbc.url", Companion::r2dbcUrl)
            registry.add("spring.r2dbc.username", postgres::getUsername)
            registry.add("spring.r2dbc.password", postgres::getPassword)
        }

        fun r2dbcUrl(): String {
            return "r2dbc:postgresql://${postgres.host}:${postgres.getMappedPort(PostgreSQLContainer.POSTGRESQL_PORT)}/${postgres.databaseName}"
        }

        @JvmStatic
        @BeforeAll
        internal fun setUp(): Unit {
            postgres.start()
        }
    }

}

Let’s take a close look at what we have here. Here we make use of testcontainers ability to pull Docker images and therefore instantiate or database container.

private val postgres: PostgreSQLContainer<*> = PostgreSQLContainer(DockerImageName.parse("postgres:13.3"))

Here we make sure to override our R2DBC connection properties in runtime with the ones pointing to our newly created container.

 @JvmStatic
 @DynamicPropertySource
 fun properties(registry: DynamicPropertyRegistry) {
     registry.add("spring.r2dbc.url", Companion::r2dbcUrl)
     registry.add("spring.r2dbc.username", postgres::getUsername)
     registry.add("spring.r2dbc.password", postgres::getPassword)
 }

Since we are using Spring Data R2DBC, the spring.r2dbc.url needs a bit more care in order to be properly constructed as, at the moment, PostgreSQLContainer provides only the getJdbcUrl() method.

fun r2dbcUrl(): String {
     return "r2dbc:postgresql://${postgres.host}:${postgres.getMappedPort(PostgreSQLContainer.POSTGRESQL_PORT)}/${postgres.databaseName}"
 }

And here, we make sure to start our container before all of our tests.

@JvmStatic
@BeforeAll
internal fun setUp(): Unit {
    postgres.start()
}

Having this in place, we are ready to write some tests.

@DataR2dbcTest
@Tag("integration")
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
class ReviewRepositoryIntegrationTest : AbstractTestcontainersIntegrationTest() {

    @Autowired
    lateinit var reviewRepository: ReviewRepository

    @Test
    fun findAllByAuthor() {
        StepVerifier.create(reviewRepository.findAllByAuthor("Anonymous"))
            .expectNextCount(3)
            .verifyComplete()
    }

    @Test
    fun findAllByCreatedAt() {
        StepVerifier.create(reviewRepository.findAllByCreatedAt(LocalDate.parse("2022-11-14")))
            .expectNextCount(1)
            .verifyComplete()
    }


    @Test
    fun findAllByCreatedAtBetween() {
        StepVerifier.create(
            reviewRepository.findAllByCreatedAtBetween(
                LocalDateTime.parse("2022-11-14T00:08:54.266024"),
                LocalDateTime.parse("2022-11-17T00:08:56.902252")
            )
        )
            .expectNextCount(4)
            .verifyComplete()
    }
}

What do we have here:

• @DataR2dbcTest is a spring boot starter test slice annotation that can be used for an R2DBC test that focuses only on Data R2DBC components. 

• @AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE) - here, we tell spring not to worry about configuring a test database since we are going to do it ourselves.

• And since we have an R2dbcRepository that delivers data in a reactive way via Flux/Mono, we use StepVerifier to create a verifiable script for our async Publisher sequences by expressing expectations about the events that will happen upon subscription.

Cool, right? Let’s run it.

io.r2dbc.postgresql.ExceptionFactory$PostgresqlBadGrammarException: [42P01] relation "reviews" does not exist

Extensions in Action

Bummer! We forgot to take care of our schema and data/records. But how do we do that? In Spring Data JPA, this is nicely taken care of by using the following:

spring.sql.init.mode=always # Spring Boot >=v2.5.0
spring.datasource.initialization-mode=always # Spring Boot <v2.5.0

And by placing a schema.sql with DDLs and data.sql with DMLs in the src/main/resources folder, everything works automagically. Or if you have Flyway/Liquibase, there are other techniques to do it. And even more, Spring Data JPA has @Sql, which allows us to run various .sql files before a test method, which would have been sufficient for our case.

But that’s not the case here, we are using Spring Data R2DBC, which at the moment doesn’t support these kinds of features, and we have no migration framework. 

So, the responsibility falls on our shoulders to write something similar to what Spring Data JPA offers that will be sufficient and customizable enough for easy integration test writing. Let’s try to replicate the @Sql annotation by creating a similar annotation @RunSql

@Target(AnnotationTarget.FUNCTION)
annotation class RunSql(val scripts: Array<String>)

Now we need to extend our test’s functionality with the ability to read this annotation and run the provided scripts. How lucky of us that Spring already has something exactly for our case, and it is called BeforeTestExecutionCallback. Let’s read the documentation:

BeforeTestExecutionCallback defines the API for Extensions that wish to provide additional behavior to tests immediately before an individual test is executed but after any user-defined setup methods (e.g., @BeforeEach methods) have been executed for that test.

That sounds right; let’s extend it and override the beforeTestExecution method.

class RunSqlExtension : BeforeTestExecutionCallback {
    override fun beforeTestExecution(extensionContext: ExtensionContext?) {
        val annotation = extensionContext?.testMethod?.map { it.getAnnotation(RunSql::class.java) }?.orElse(null)
        annotation?.let {
            val testInstance = extensionContext.testInstance
                .orElseThrow { RuntimeException("Test instance not found. ${javaClass.simpleName} is supposed to be used in junit 5 only!") }
            val connectionFactory = getConnectionFactory(testInstance)
            if (connectionFactory != null)
                it.scripts.forEach { script ->
                    Mono.from(connectionFactory.create())
                        .flatMap<Any> { connection -> ScriptUtils.executeSqlScript(connection, ClassPathResource(script)) }.block()
                }
        }
    }

    private fun getConnectionFactory(testInstance: Any?): ConnectionFactory? {
        testInstance?.let {
            return it.javaClass.superclass.declaredFields
                .find { it.name.equals("connectionFactory") }
                .also { it?.isAccessible = true }?.get(it) as ConnectionFactory
        }
        return null
    }
}

Okay, so what we’ve done here: 

1. We take the current test method from the extension context and check for the @RunSql annotation.
2. We take the current test instance – the instance of our running test.
3. We pass the current test instance to getConnectionFactory , so it can take the @Autowired ConnectionFactory from our parent, in our case, the abstract class. AbstractTestcontainersIntegrationTest 
4. Using the obtained connectionFactory and ScriptUtils, we execute the scripts found on @RunSql annotation in a blocking-manner.

As I mentioned, our AbstractTestcontainersIntegrationTest needs a tiny change; we need to add @Autowired lateinit var connectionFactory: ConnectionFactory so it can be picked by our extension. 

Having everything in place, it is a matter of using this extension with @ExtendWith(RunSqlExtension::class) and our new annotation @RunSql. Here’s how our test looks now.

@DataR2dbcTest
@Tag("integration")
@ExtendWith(RunSqlExtension::class)
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
class ReviewRepositoryIntegrationTest : AbstractTestcontainersIntegrationTest() {

    @Autowired
    lateinit var reviewRepository: ReviewRepository

    @Test
    @RunSql(["schema.sql", "/data/reviews.sql"])
    fun findAllByAuthor() {
        StepVerifier.create(reviewRepository.findAllByAuthor("Anonymous"))
            .expectNextCount(3)
            .verifyComplete()
    }

    @Test
    @RunSql(["schema.sql", "/data/reviews.sql"])
    fun findAllByCreatedAt() {
        StepVerifier.create(reviewRepository.findAllByCreatedAt(LocalDate.parse("2022-11-14")))
            .expectNextCount(1)
            .verifyComplete()
    }

    @Test
    @RunSql(["schema.sql", "/data/reviews.sql"])
    fun findAllByCreatedAtBetween() {
        StepVerifier.create(
            reviewRepository.findAllByCreatedAtBetween(
                LocalDateTime.parse("2022-11-14T00:08:54.266024"),
                LocalDateTime.parse("2022-11-17T00:08:56.902252")
            )
        )
            .expectNextCount(4)
            .verifyComplete()
    }
}

And here is the content of schema.sql from resources.

create table if not exists reviews
(
    id               integer generated by default as identity
        constraint pk_reviews
            primary key,
    text             varchar(3000),
    author           varchar(255),
    created_at       timestamp,
    last_modified_at timestamp,
    course_id        integer
);

And here is the content of reviews.sql from resources/data.

truncate reviews cascade;

INSERT INTO reviews (id, text, author, created_at, last_modified_at, course_id) VALUES (-1, 'Amazing, loved it!', 'Anonymous', '2022-11-14 00:08:54.266024', '2022-11-14 00:08:54.266024', 3);
INSERT INTO reviews (id, text, author, created_at, last_modified_at, course_id) VALUES (-2, 'Great, loved it!', 'Anonymous', '2022-11-15 00:08:56.468410', '2022-11-15 00:08:56.468410', 3);
INSERT INTO reviews (id, text, author, created_at, last_modified_at, course_id) VALUES (-3, 'Good, loved it!', 'Sponge Bob', '2022-11-16 00:08:56.711163', '2022-11-16 00:08:56.711163', 3);
INSERT INTO reviews (id, text, author, created_at, last_modified_at, course_id) VALUES (-4, 'Nice, loved it!', 'Anonymous', '2022-11-17 00:08:56.902252', '2022-11-17 00:08:56.902252', 3);

Please pay attention to create table if not exists from schema.sql and truncate reviews cascade from reviews.sql – this is to ensure a clean state of the database for each test.

Now, if we run our tests, all is green.

We can go ahead and do a little more to follow the DRY principle in regard to the annotations used on this test which can be collapsed under one annotation and then reused, like this.

@DataR2dbcTest
@Tag("integration-test")
@Target(AnnotationTarget.CLASS)
@ExtendWith(RunSqlExtension::class)
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
annotation class RepositoryIntegrationTest()

And basically, that’s it, folks. With the common abstract class, test extension, and custom annotation in place, from now on, every new database integration test suite should be a piece of cake.

Bonus

But wait, why stop here? Having our setup, we can play around with component tests (broad integration tests), for example, to test our endpoints - starting with a simple HTTP call, surfing through all business layers and services all the way to the database layer. Here is an example of how you might wanna do it for GraphQL endpoints.

@ActiveProfiles("integration-test")
@AutoConfigureGraphQlTester
@ExtendWith(RunSqlExtension::class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
internal class ReviewGraphQLControllerComponentTest : AbstractTestcontainersIntegrationTest() {

    @Autowired
    private lateinit var graphQlTester: GraphQlTester

    @Test
    @RunSql(["schema.sql", "/data/reviews.sql"])
    fun getReviewById() {
        graphQlTester.documentName("getReviewById")
            .variable("id", -1)
            .execute()
            .path("getReviewById")
            .entity(ReviewResponse::class.java)
            .isEqualTo(ReviewResponseFixture.of())
    }

    @Test
    @RunSql(["schema.sql", "/data/reviews.sql"])
    fun getAllReviews() {
        graphQlTester.documentName("getAllReviews")
            .execute()
            .path("getAllReviews")
            .entityList(ReviewResponse::class.java)
            .hasSize(4)
            .contains(ReviewResponseFixture.of())
    }
}

And again, if you want to reuse all the annotations there, simply create a new one.

@ActiveProfiles("integration-test")
@AutoConfigureGraphQlTester
@ExtendWith(RunSqlExtension::class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
@Target(AnnotationTarget.CLASS)
annotation class ComponentTest()

You can find the code on GitHub.

Happy coding and writing tests!