How to Use Elasticsearch With a Spring Data Elasticsearch Project

Overview 

Elasticsearch is a real-time distributed and open source full-text search and analytics engine. It is document-based search platform with fast searching capabilities. It’s optimized for needle-in-haystack problems rather than consistency or atomicity.

Elasticserach is a huge topic so in this blog I am going to keep it very simple and at the 101 level. In this blog, I will cover how to download Elasticsearch and set it up. Also, how you can use Spring Boot and Spring Data ElasticSearch projects to integrate with Elasticsearch engine.

How to Setup Elasticsearch

Elasticsearch's latest version is 6.5.x as of today. However, I am going to download the 2.4.4 version only. The reason is that the spring-data-elasticsearch project is compatible only with 2.4.4 version as of now. If you want to know more details on the compatibility, please check this link: Spring-Data-Elasticsearch---Spring-Boot---version-matrix.

If you still want to use the latest Elasticsearch 5.x or 6.x, you would need to use Spring with Transport Client or Node client APIs directly instead of spring-data-elasticsearch. Follow the below steps to download and setup your Elasticsearch engine.

Step 1 - Go to Elastic's official website: https://www.elastic.co/downloads/past-releases.

Step 2 - Select Elasticsearch in drop down and then version as 2.4.4 and click on Download button.

Step 3 - It will give you options if you want to download as Zip, TAR, RPM. You can choose as per your convenience. I selected the Zip format as using it on windows.

Step 4 - Unzip the downloaded content and go to bin folder. There will be a file named elasticsearch.bat.

Step 5 - Run this file on windows OS through command prompt and it will start the elasticsearch engine for you. Once started, it will start listening to port 9200. So the URL will be http://localhost:9200/ Also, port 9300 is exposed as a cluster node. Port 9200 is for REST communication and can be used by Java or any other language and 9300 is for Elasticsearch's Cluster Nodes to communicatewith each other. Java can also connect to this cluster node using the Transport protocol.

Step 6 - Test to verify if it started properly by launching the URL using the curl command. You can use PowerShell on Windows.

curl http://localhost:9200/ 

curl http://localhost:9200/


StatusCode        : 200
StatusDescription : OK
Content           : {
                      "name" : "Rl'nnd",
                      "cluster_name" : "elasticsearch",
                      "cluster_uuid" : "9m9VQLE9SIqcmbn9nb19Eg",
                      "version" : {
                        "number" : "2.4.4",
                        "build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6...
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 360
                    Content-Type: application/json; charset=UTF-8

                    {
                      "name" : "Rl'nnd",
                      "cluster_name" : "elasticsearch",
                      "cluster_uuid" : "9m9VQLE9SIqcmbn9nb19Eg",
                      "versio...
Forms             : {}
Headers           : {[Content-Length, 360], [Content-Type, application/json; charset=UTF-8]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 360

How to Develop Our Spring Boot Application

Now, we will develop a Spring Boot application which will showcase how ElasticsearchTemplate and ElasticsearchRepository accesss the Elasticsearch engine and do CRUD operations.

Before we develop the application, let's first understand how ElasticsearchTemplate and ElasticsearchRepository works.

ElasticsearchTemplate - It is a Template class which implements the ElasticsearchOperations. It is more powerful than ElasticsearchRepository as it can do more than CRUD operations. It has operations to create, delete indexes, bulk upload. It can do aggregated search as well.

ElasticsearchRepository - If we define an interface which extends the ElasticsearchRepository,which is provided by Spring data Elasticsearch, it will provide the CRUD operations automatically for that Document. For example, the UserRepository interface is defined below with the "User" document by extending ElasticsearchRepository. Now all the find, save, delete, and update default operations can be done via the User document. If you don't know what a document is in Elasticsearch, keep reading for a more in-depth explanation. 

@Repository
public interface UserRepository extends ElasticsearchRepository<User, String> {
}

The above code extends the ElasticsearchCrudRepository which eventually extends the Repository interface. This repository interface is a standard feature of the Spring Data framework. There's no need to provide implementation details for this interface. You can write customized queries as well by using the @Query annotation.

Both of these use Transport client or Node Client

Prerequisites

•   Java 1.8 installed

•   Eclipse or Visual Studio Code IDE

•   Maven

 Maven Dependencies   

<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

Configuration 

application.properties needs to have Spring Data properties which are used by ElasticsearchTemplate and  ElasticsearchRepository to connect the engine. I have used Transport client properties like cluster-nodes and index name to connect the Elasticsearch engine.

#application.properties

# Local Elasticsearch config
spring.data.elasticsearch.repositories.enabled=true
spring.data.elasticsearch.cluster-nodes=localhost:9300
spring.data.elasticsearch.cluster-name=elasticsearch

elasticsearch.index.name=my_index
elasticsearch.user.type=user

# App config
server.port=8102
spring.application.name=BootElastic

Mappings

In Elasticsearch, Index is like a DB in RDBMS and Mappings/Type is similar to a table in RDBMS. Document is a collection of fields belonging to a type and resides in the index. Spring Data provides annotation slike @Document to create documents. Here, we define User as a document which has an index of "my_index" and type of "user".

@Document(indexName = "my_index", type = "user")
public class User {

    @Id
    private String userId;
    private String name;
    private Date creationDate = new Date();
    private Map<String, String> userSettings = new HashMap<>();

 //getter and setters 
}

Develop Controllers

Th first controller is UserController. It will use UserDAOImpl to allow ElasticserachTemplate to interact with the ElasticsearchEngine. 

@RestController
public class UserController {

    @Autowired
    private UserDAO userDAO;

    @RequestMapping("/all")
    public List<User> getAllUsers() {
        return userDAO.getAllUsers();
    }
     @RequestMapping(value = "/new", method = RequestMethod.POST)
    public User addUsers(@RequestBody User user) {
        userDAO.addNewUser(user);
        return user;
    }
  --- Other methods
}

UserDAOImpl - This class initializes ElasticsearchTemplate and uses the queryForList method to retrieve data.

@Repository
public class UserDAOImpl implements UserDAO {

    private final Logger LOG = LoggerFactory.getLogger(getClass());

    @Value("${elasticsearch.index.name}")
    private String indexName;

    @Value("${elasticsearch.user.type}")
    private String userTypeName;

    @Autowired
    private ElasticsearchTemplate esTemplate;

    @Override
    public List<User> getAllUsers() {
        SearchQuery getAllQuery = new NativeSearchQueryBuilder()
                .withQuery(matchAllQuery()).build();
        return esTemplate.queryForList(getAllQuery, User.class);
    }
  // Other methods
}

Another controller is UserRepositoryConroller. This uses UserRepository to interact with the Elasticsearch engine.

@RestController
@RequestMapping("/repo")
public class UserRepositoryController {

    @Autowired
    private UserRepository userRepository;

    @RequestMapping("/all")
    public List<User> getAllUsers() {
        List<User> users = new ArrayList<>();
        userRepository.findAll().forEach(users::add);
        return users;
    }
  //Other methods
}

This repository class extends the ElasticsearchRepository class which internally extends ElasticsearchCrudRepository -> PagingAndSortingRepository.

@Repository
public interface UserRepository extends ElasticsearchRepository<User, String> {
}

You can find the full code on GitHub.

Build the Application

The application can be built using Maven commands.

mvn clean install will build the code and create the elasticsearch-0.0.1-SNAPSHOT.jar file.

Run the Application

java -jar  target/elasticsearch-0.0.1-SNAPSHOT.jar will start the application. And the application will listen to port 8102 as defined in the application.properties file.

Test the Application

Test the UserController flow which uses ElasticsearchTemplate.

Step 1 - Add new Users. Use this REST API URL to add new users: http://localhost:8102/new 

Add JSON data in the Request body.

{
  "name": "Sumit",
   "userSettings": {
   "gender" : "male",
   "occupation" : "CA",
   "hobby" : "chess"
   }
}

Step 2 - Check the response. You will see a new user is created with a userId generated which is unique tp this document. The output will be as below:

{
    "userId": "AWdj-3KcTJbZRlQtLZfO",
    "name": "Sumit",
    "creationDate": 1543570682521,
    "userSettings": {
        "gender": "male",
        "occupation": "CA",
        "hobby": "chess"
    }
}

Step 3 - To retrieve all users, use http://localhost:8102/all 

{
        "userId": "AWdj-3KcTJbZRlQtLZfO",
        "name": "Sumit",
        "creationDate": 1543570682521,
        "userSettings": {
            "gender": "male",
            "occupation": "CA",
            "hobby": "chess"
        }
    },
    {
        "userId": "AWdZuKFRgzULDLBu_Y0c",
        "name": "Suresh",
        "creationDate": 1543398531296,
        "userSettings": {}
    }

Test the UserRepositoryController flow which uses ElasticsearchRepository.

Step 1 - Add new users. Use this REST API URL to add a new user: http://localhost:8102/repo/new 

Add JSON data in the request body as we did with earlier test case.

Step 2 - Check the response. You will see new a user is created with a userId which is unique for this document.

There are many other APIs I have put in the code which you can play around with. It will showcase how ElasticsearchTemplate can work with indexes, How you can search by field name. With ElasticsearchRepository, you can search by userId as well.

There are many other ways we can connect to Elasticsearch Engine. I will just provide brief introduction about them here as it would require whole new blog to explain in detail.

Spring Boot with Transport Client - As explained above, if you want to use the latest Elasticsearch engine 5.x or 6.x, Spring Data Elasticsearch will not support that as of now. But Elasticsearch provides the RestHighLevelClient class which works well with the latest version and uses the RestClient class to configure the host and port for the cluster. It uses the index() method with IndexRequest as input to create a document. It uses theget() method with GetRequest as input to search a document. It does have the update() and delete() methods as well.

Node.js - If you are not a Java/Spring fan, then you have the option to use Node.js as well. What you need to do is download the elasticsearchand get-json modules using npm. Elasticsearchhas a client module which can configure the host and port of the Elasticsearch server cluster. This client has methods like create() to create the index, and index() to put documents in that index.

That's all for Elasticsearch integration with this Spring Data project. Please do let me know your views through comments. Happy to address your queries if I can.