Upgrade Guide To Spring Data Elasticsearch 5.0

Recently, I wrote the article, "Pagination With Spring Data Elasticsearch 4.4," but we already have new Spring Data Elasticsearch 5.0 and Spring Boot 3.0 releases. There are several changes and issues we should consider when doing the upgrade.

The goal of this article is to highlight changes when upgrading the sat-elk project with these technologies:

1. Spring Data Elasticsearch 5.0.1
2. Spring Boot 3.0.2
3. Elasticsearch 8.5.3

Key Changes

• Elasticsearch 8.5 has enabled SSL by default.
• Spring Data Elasticsearch 5.0 uses a new Elasticsearch client library.
• Spring Boot 3.0 moved from javax to jakarta package (dependencies).
• SearchHits behaves a little bit differently.

In This Article, You Will Learn

• How to set up Elasticsearch 8.5.3
• How to configure Spring Data Elasticsearch 5.0 in a project
• How to deal with upgrading to the latest technologies

Elasticsearch

Our goal is to have an application to manage data via the Spring Data Elasticsearch in Elasticsearch. You can find more details in my initial article "Introduction to Spring Data Elasticsearch 4.1." Here, you can find only the basic steps with the highlighted differences or changes.

Let's check the changes first.

Changes

The last article used Spring Data Elasticsearch in version 4.4, but the latest version (at the time of writing this article) is version 5.0.1. You can find all the details here.

We should keep in mind the compatibility matrix that contains the compatible versions of the main technologies -> Spring framework, Spring Boot, Spring Data Release Train, Spring Data Elasticsearch, and of course, Elasticsearch itself.

Our versions are driven by Spring Boot 3.0.2 and Spring Data Release Train 2022.0.1.

Docker Setup

The first significant change in the used  elasticsearch Docker image is the switch to HTTPS by default instead of the previous HTTP. The differences in Elasticsearch configuration (located in /usr/share/elasticsearch/config/elasticsearch.yml file) in these versions are:

Elasticsearch 7.17.8:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

Elasticsearch 8.5.3:

cluster.name: "docker-cluster"
network.host: 0.0.0.0

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

The previous tip (see the section"Disable XPack in Elasticsearch" in my previous article) doesn't work now, probably because it was a kind of workaround. Therefore, we should disable X-Pack security properly.

Let's follow these steps to set up Elasticsearch correctly.

Custom Network

docker network create sat-elk-net

Elasticsearch

docker run -d --name sat-elasticsearch --net sat-elk-net -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "xpack.security.enabled=false" elasticsearch:8.5.3

Note: See the -e "xpack.security.enabled=false" extra parameter for disabling HTTPS.

ElasticHQ

docker run -d --name sat-elastichq --net sat-elk-net -p 5000:5000 elastichq/elasticsearch-hq

Spring Data Elasticsearch

All upgrade notes/hints for version 5.0.x can be found here.

Maven Dependency

We use the spring-boot-starter-data-elasticsearch dependency in our Maven project (pom.xml) as demonstrated below. We can find the latest available version in the Maven Central repository.

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
	<version>3.0.2</version>
</dependency>

Additionally, our code also depends on the Spring MVC (for exposing REST endpoints) and the jackson.dataformat CSV module (for reading CSV files). We should add them to our project as: 

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-web</artifactId>
	<version>3.0.2</version>
</dependency>
<dependency>
	<groupId>com.fasterxml.jackson.dataformat</groupId>
	<artifactId>jackson-dataformat-csv</artifactId>
	<version>2.14.1</version>
</dependency>

Elasticsearch client

A second key change (after the Elasticsearch Docker image configuration) represents a new Elasticsearch client used by Spring Data Elasticsearch. So far we used configuration spring.elasticsearch.uris in application.properties file, but it doesn't work now.

We can still re-use it with a small adjustment as:

spring:
  elasticsearch:
    rest:
      uris: oxygen-arnost.ifs.dev.fra.aws.dbgcloud.io:9200

Note: Only the leading protocol was removed.

For the new Elasticsearch client, we need to add a ElasticsearchClientConfig class where we use this configuration property as:

@Configuration
public class ElasticsearchClientConfig extends ElasticsearchConfiguration {

	@Value("${spring.elasticsearch.rest.uris}")
	String connetionUrl;

	@Override
	public ClientConfiguration clientConfiguration() {
		return ClientConfiguration.builder()
				.connectedTo(connetionUrl)
				.build();
	}

}

We also need to add a ElasticsearchClientConfigTest test for this class in order to satisfy Sonar.

@SpringBootTest
class ElasticsearchClientConfigTest {

	@MockBean
	CityRepository cityRepository;

	@Autowired
	ElasticsearchClientConfig elasticsearchClientConfig;

	@Test
	void clientConfiguration() {
		assertThat(elasticsearchClientConfig.connetionUrl).contains("oxygen-arnost");
	}

}

SearchHitsImpl Changes

There were two minor issues related to our usage of SearchHits interface in the previous article.

SearchHitsImpl Changed Interface

Since version 5.0, the constructor of the SearchHitsImpl class was extended with a new pointInTimeId argument. It has no impact on our implementation, as it was used only in tests that were removed after all.

The change is not documented well, but the definition can be found in SearchHits class:

When doing a search with a point in time, the response contains a new point in time id value.

Not Working Jackson Conversion for Aggregations

Since Spring Data Elasticsearch 5.0.1, the SearchHit instance contains aggregations attribute filled (it was null in the previous version). 

See the result from Spring Data Elasticsearch 4.4.6:

And also from Spring Data Elasticsearch 5.0.1.

The search action itself is working correctly. The issue happens when we try to send the SearchHits result directly to the client as JSON. Spring doesn't contain any converter for the AggregationsContainer class and produces this error:

2023-01-05T08:28:44.707+01:00 ERROR 6656 --- [nio-8080-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.http.converter.HttpMessageConversionException: Type definition error: [simple type, class org.springframework.data.elasticsearch.client.elc.ElasticsearchAggregations]] with root cause

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class org.springframework.data.elasticsearch.client.elc.ElasticsearchAggregations and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: org.springframework.data.elasticsearch.core.SearchHitSupport$SearchPageImpl["searchHits"]->org.springframework.data.elasticsearch.core.SearchHitsImpl["aggregations"])
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77) ~[jackson-databind-2.14.1.jar:2.14.1]
	at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1306) ~[jackson-databind-2.14.1.jar:2.14.1]
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:408) ~[jackson-databind-2.14.1.jar:2.14.1]
	at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:53) ~[jackson-databind-2.14.1.jar:2.14.1]
	at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:30) ~[jackson-databind-2.14.1.jar:2.14.1]
	...
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Therefore, we can simplify our search feature with pagination as:

• Remove both previous searchPage and searchHits methods in CityController and CityService.
• Modify our search method in the CityService as:

@SuppressWarnings("unchecked")
public Page<City> search(String name, String country, String subcountry, Pageable pageable) {
	return (Page<City>) unwrapSearchHits(searchPageFor(searchHits(name, country, subcountry, pageable), pageable));
}

This way we achieve dynamic search with pagination without the need to tackle the SearchHits related issues.

The new dynamic search implementation can be verified on http://localhost:8080/api/cities?name=be&country=Czech&subcountry=bohemia&size=5&sort=name,asc with this output: 

{
  "content": [
    {
      "id": "ePHmOIUBcEaiCL6qmck4",
      "name": "Benešov",
      "country": "Czech Republic",
      "subcountry": "Central Bohemia",
      "geonameid": 3079508
    },
    ...
  ],
  "pageable": {
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 0,
    "pageSize": 5,
    "pageNumber": 0,
    "paged": true,
    "unpaged": false
  },
  "last": true,
  "totalElements": 3,
  "totalPages": 1,
  "size": 5,
  "number": 0,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "first": true,
  "numberOfElements": 3,
  "empty": false
}

Conclusion

This article has covered the upgrade to the latest Spring Data Elasticsearch 5.0.1 with Elasticsearch 8.5.3 (at the time of the article). We started with the proper configuration of the Elasticsearch Docker image. Next, we demonstrated the changed configuration due to the new Elasticsearch client library. In the end, we summarized all the needed fixes in our application in order to make all our features work again as before.

The complete source code demonstrated above is available in my GitHub repository. All the changes (related to this upgrade) are visible in PR #58.