Advanced Filtering and Full-Text Search Using Hibernate Search With Angular/Spring Boot

The JPA Criteria API can provide support in the implementation of the optional filter clauses. For the long text columns, Hibernate Search can provide a full-text search.

The combination of Hibernate Search and the JPA Criteria API is shown in the MovieManager project. The APIs are used to create search user interfaces for movies and actors.

Using Hibernate Search and JPA Criteria API

The MovieManager project stores text data for movie overviews and actor biographies. The new filter functions for movies and actors include a full-text search for the text data. The JPA Criteria API is used to implement the additional filter functions so it can help with the optional query components like age or release date.

The Backend

The MovieController has a new rest interface:

@RequestMapping(value = "/filter-criteria", method = RequestMethod.POST, produces = MediaType.APPLICATION_JSON_VALUE, consumes = MediaType.APPLICATION_JSON_VALUE)
public List<MovieDto> getMoviesByCriteria(@RequestHeader(value =  
   HttpHeaders.AUTHORIZATION) String bearerStr,
   @RequestBody MovieFilterCriteriaDto filterCriteria) {
   return this.service.findMoviesByFilterCriteria(bearerStr,  
      filterCriteria).stream().map(m -> this.mapper.convert(m)).toList();
}

The ActorController has a similar rest interface:

@RequestMapping(value = "/filter-criteria", method = RequestMethod.POST,  
   produces = MediaType.APPLICATION_JSON_VALUE, consumes =   
   MediaType.APPLICATION_JSON_VALUE)
public List<ActorDto> getActorsByCriteria(@RequestHeader(value = 
   HttpHeaders.AUTHORIZATION) String bearerStr,
   @RequestBody ActorFilterCriteriaDto filterCriteria) {
      return this.service.findActorsByFilterCriteria(bearerStr,  
         filterCriteria).stream().map(m -> this.mapper.convert(m)).toList();
}

These are rest endpoints that require a posted JSON, that is mapped with the @RequestBody annotation, to the FilterCriteria DTO. The DTO is used to call the service for filtering.

Search Services

The ActorService and the MovieService implement the services for filtering. The MovieService is shown here:

public List<Movie> findMoviesByFilterCriteria(String bearerStr, 
   MovieFilterCriteriaDto filterCriteriaDto) {
   List<Movie> jpaMovies = 
      this.movieRep.findByFilterCriteria(filterCriteriaDto,
         this.auds.getCurrentUser(bearerStr).getId());
   SearchTermDto searchTermDto = new SearchTermDto();	  
   searchTermDto.setSearchPhraseDto(filterCriteriaDto.getSearchPhraseDto());
   List<Movie> ftMovies = this.findMoviesBySearchTerm(bearerStr, 
      searchTermDto);
   List<Movie> results = jpaMovies;
   if (filterCriteriaDto.getSearchPhraseDto() != null && 
      !Objects.isNull(filterCriteriaDto.getSearchPhraseDto().getPhrase()) && 
      filterCriteriaDto.getSearchPhraseDto().getPhrase().length() > 2) {
      Collection<Long> dublicates =  CommonUtils.
         findDublicates(Stream.of(jpaMovies,ftMovies)
         .flatMap(List::stream).toList());
      results = Stream.of(jpaMovies, ftMovies).flatMap(List::stream)
         .filter(myMovie -> CommonUtils.
            filterForDublicates(myMovie, dublicates)).toList();
      // remove dublicates
      results = results.isEmpty() ? ftMovies : 
      List.copyOf(CommonUtils.filterDublicates(results));
   }
   return results.subList(0, results.size() > 50 ? 50 : results.size());
}

public List<Movie> findMoviesBySearchTerm(String bearerStr, 
   SearchTermDto searchTermDto) {
   List<Movie> movies = searchTermDto.getSearchPhraseDto() != null ?   
      this.movieRep.findMoviesByPhrase(searchTermDto.getSearchPhraseDto()) :  
      this.movieRep.
         findMoviesBySearchStrings(searchTermDto.getSearchStringDtos());
   List<Movie> filteredMovies = movies.stream().filter(myMovie -> 
      myMovie.getUsers().stream().anyMatch(myUser -> myUser.getId()
      .equals(this.auds.getCurrentUser(bearerStr).getId()))).toList();
   return filteredMovies;
}

The findMoviesByFilterCriteria(...) method first calls the JPA repository to select the movies. The method getCurrentUser(...) finds the user entity that the JWT Token was issued for and returns the id. The movies have a database related to the user. Due to this relation, a movie is stored only once in the table and is used by all users that have imported it.

Then the SearchTermDto is created to call the findMoviesBySearchTerm(...) method for the full-text search. The method uses the MovieRep to execute the search in the 'overview' index of the movies and filters the results for the movies of the current user. 

Then the results of the JPA query and the full-text search are merged in 3 steps:

1. The findDublicates(...) returns the ids that are found in both search results.
2. The filterForDublicates(...) returns the entities of the ids.
3. The filterDublicates(...) removes the objects with duplicate ids and returns them. If no common results are found, the full-text search results are returned.

The combined results are limited to 50 entities and returned.

Data Repositories

The MovieRepositoryBean and the ActorRepositoryBean implement the JPA Criteria searches and the Hibernate search searches. The JPA search of the MovieRepositoryBean is shown here:

public List<Movie> findByFilterCriteria(MovieFilterCriteriaDto 
   filterCriteriaDto, Long userId) {
   CriteriaQuery<Movie> cq = this.entityManager.getCriteriaBuilder()
      .createQuery(Movie.class);
   Root<Movie> cMovie = cq.from(Movie.class);
   List<Predicate> predicates = new ArrayList<>();
  ...
   if (filterCriteriaDto.getMovieTitle() != null &&   
      filterCriteriaDto.getMovieTitle().trim().length() > 2) {
      predicates.add(this.entityManager.getCriteriaBuilder().like(
         this.entityManager.getCriteriaBuilder()
            .lower(cMovie.get("title")), 
               String.format("%%%s%%", filterCriteriaDto
                  .getMovieTitle().toLowerCase())));
   }
   if (filterCriteriaDto.getMovieActor() != null && 
      filterCriteriaDto.getMovieActor().trim().length() > 2) {
      Metamodel m = this.entityManager.getMetamodel();
      EntityType<Movie> movie_ = m.entity(Movie.class);
      predicates.add(this.entityManager.getCriteriaBuilder()
         .like(this.entityManager.getCriteriaBuilder().lower(
            cMovie.join(movie_.getDeclaredList("cast", Cast.class))
	       .get("characterName")),
               String.format("%%%s%%", filterCriteriaDto.
                  getMovieActor().toLowerCase())));
   }
   ...
   // user check
   Metamodel m = this.entityManager.getMetamodel();
   EntityType<Movie> movie_ = m.entity(Movie.class);
   predicates.add(this.entityManager.getCriteriaBuilder()
      .equal(cMovie.join(movie_.getDeclaredSet("users", User.class))
      .get("id"), userId));
   cq.where(predicates.toArray(new Predicate[0])).distinct(true);
   return this.entityManager.createQuery(cq)
      .setMaxResults(1000).getResultList();
}

This part of the method filterByCriteria(...) shows the where criteria for the movie title search and the actor name search with the join. 

First, the criteria query and the root movie objects are created. The predicate list is created to contain the query criteria for the search. 

The movie title is checked for its existence and for its minimum length. The EntityManager is used to create a 'like' criteria that contains a 'lower' function for the 'title' property. The title string is converted to lowercase and surrounded with "%%" to find all titles that contain the case-insensitive string. Then the criteria is added to the predicate list.

The actor name is checked for its existence and for its minimum length. Then the JPA Metamodel is created to get the EntityType for the cast entity to join. The EntityManager is used to create the 'like' and the 'lower' criteria. The root entity ('cMove') is used to join the cast entity to the query. The 'characterName' of the cast entity is used in the 'like' criteria. The actor name string for the search is converted to lowercase and surrounded with "%%" to find all actor names that contain the search string. The complete actor criteria is finally added to the predicate list.

Then the user check criteria is created and added to the predicate list in the same manner as the actor name search criteria.

The criteria predicates are added to CriteriaQuery.where(...) and a 'distinct(true)' call is added to remove duplicates. 

The query result is limited to 1000 entities to protect the server from I/O and memory overload.

Full-Text Search

The full-text search is implemented in the findMoviesBySearchPhrase(...) method of the MovieRepository:

@SuppressWarnings("unchecked")
public List<Movie> findMoviesByPhrase(SearchPhraseDto searchPhraseDto) {
   List<Movie> resultList = List.of();
   if (searchPhraseDto.getPhrase() != null && 
      searchPhraseDto.getPhrase().trim().length() > 2) {
      FullTextEntityManager fullTextEntityManager = 
         Search.getFullTextEntityManager(entityManager);
      QueryBuilder movieQueryBuilder = 
         fullTextEntityManager.getSearchFactory().buildQueryBuilder()
         .forEntity(Movie.class).get();
      Query phraseQuery = movieQueryBuilder.phrase()
         .withSlop(searchPhraseDto.getOtherWordsInPhrase())
         .onField("overview")
         .sentence(searchPhraseDto.getPhrase()).createQuery();
      resultList = fullTextEntityManager
         .createFullTextQuery(phraseQuery, Movie.class)
         .setMaxResults(1000).getResultList();
   }
   return resultList;
}

The method findMoviesByPhrase(...) has the SearchPhraseDto as a parameter. That contains the properties:

• otherWordsInPhrase that has the default value of 0.
• 'phrase' that contains the search string for the movie overviews in the Hibernate Search indexes.

The existence and the length of the 'phrase' are checked. Then the FullTextEntityManager and the QueryBuilder are created. The 'QueryBuilder' is used to create a full-text query on the movie entity field 'overview' with the search parameter 'phrase'. The otherWordsInPhrase are added to the FullTextEntityManager with the withSlop(…) parameter.

The FullTextEntityManager is used to execute the full-text query on the movie 'overview' index with a limit of 1000 results. The limit is set to protect the server from I/O and memory overload.

Keeping the Hibernate Search Indices up to Date

The Hibernate Search indices are checked on the application start for needed updates in the CronJobs class:

@Async
@EventListener(ApplicationReadyEvent.class)
public void checkHibernateSearchIndexes() throws InterruptedException {
   int movieCount = this.entityManager.createNamedQuery("Movie.count", 
      Long.class).getSingleResult().intValue();
   int actorCount = this.entityManager.createNamedQuery("Actor.count", 
      Long.class).getSingleResult().intValue();
   FullTextEntityManager fullTextEntityManager = 
   Search.getFullTextEntityManager(entityManager);
   int actorResults = checkForActorIndex(fullTextEntityManager);
   int movieResults = checkForMovieIndex(fullTextEntityManager);
   LOG.info(String.format("DbMovies: %d, DbActors: %d, FtMovies: %d, 
      FtActors: %d", movieCount, actorCount, movieResults, actorResults));
   if (actorResults == 0 || movieResults == 0 
      || actorResults != actorCount || movieResults != movieCount) {
      fullTextEntityManager.createIndexer().startAndWait();
      this.indexDone = true;
      LOG.info("Hibernate Search Index ready.");
   } else {
      this.indexDone = true;
      LOG.info("Hibernate Search Index ready.");
   }
}

private int checkForMovieIndex(FullTextEntityManager fullTextEntityManager) {
   org.apache.lucene.search.Query movieQuery = fullTextEntityManager
      .getSearchFactory().buildQueryBuilder()				  
      .forEntity(Movie.class).get().all().createQuery();
   int movieResults = fullTextEntityManager.createFullTextQuery(movieQuery, 
      Movie.class).getResultSize();
   return movieResults;
}

The @Async and the @EventListener(ApplicationReadyEvent.class) annotations of Spring execute the checkHibernateSearchIndexes() method on application startup on its own background thread. 

First, the number of movie and actor entities in the database is queried with named queries. 

Second, the number of movie and actor entities in the Hibernate Search indexes is queried with the checkForMovieIndex(...) and checkForActorIndex(...) methods.  

Then, the results are compared and the Hibernate Search indices are recreated with the FullTextEntityManager if a difference is found. To have normal startup times, the method has to be executed on its own background thread. The Hibernate Search indices are files on the file system. They need to be created or checked on the first startup. By having local indices on each instance of the application, conflicts and inconsistencies are avoided.

Conclusion Backend

The queries have several optional criteria, and JPA Criteria queries support this use case. The code is verbose and needs some getting used to. The alternatives are to either create the query string yourself or to add a library (with possible code generation) for more support. This project tries to add only libraries that are needed and the code is maintainable enough. I have not found support to execute Hibernate Search and JPA criteria queries together. Because of that, the results have to be combined in code. That requires limits on the result sizes to protect the I/O and memory resources of the server, which can cause missed matches in large result sets.  

Hibernate Search was easy to use and the indices can be created/updated on application start. 

Angular Frontend

The movie/actor filters are displayed in the lazy loaded Angular modules filter-actors and filter-movies. The modules are lazy loaded to make application startup faster and because they are the only users of the components of the Ng-Bootstrap library.  The template filter-movies.component.html uses the Offcanvas component and the Datepicker component for the filter criteria:

<ng-template #content let-offcanvas>
  <div class="offcanvas-header">
    <h4 class="offcanvas-title" id="offcanvas-basic-title" 
       i18n="@@filtersAvailiable">Filters availiable</h4>
    <button type="button" class="btn-close" aria-label="Close" 
       (click)="offcanvas.dismiss('Cross click')"></button>
  </div>
  <div class="offcanvas-body">
    <form>
      <div class="select-range">
      <div class="mb-3 me-1">
      <label for="releasedAfter" i18n="@@filterMoviesReleasedAfter">
         Released after</label>
        <div class="input-group">
          <input id="releasedBefore"  class="form-control" 
             [(ngModel)]="ngbReleaseFrom" placeholder="yyyy-mm-dd" 
             name="dpFrom" ngbDatepicker #dpFrom="ngbDatepicker">
          <button class="btn btn-outline-secondary calendar" 
             (click)="dpFrom.toggle()" type="button"></button>
        </div>
      </div>
...
</ng-template>
<div class="container-fluid">
   <div>
      <div class="row">
         <div class="col">
            <button class="btn btn-primary filter-change-btn"
               (click)="showFilterActors()"  
               i18n="@@filterMoviesFilterActors">Filter<br/>Actors</button>
            <button class="btn btn-primary open-filters" 
               (click)="open(content)"  
               i18n="@@filterMoviesOpenFilters">Open<br/>Filters</button>
...

• The <ng-template ...> has the template variable '#content' to identify it in the component. 
• The <div class="offcanvas-header"> contains the label and its close button. 
• The <div class="offcanvas-body"> contains the datepicker component with the input for the value and the button to open the datepicker. The template variable #dpFrom="ngbDatepicker" gets the datepicker object. The button uses it in the 'click' action to toggle the datepicker. 
• The <div class="container-fluid"> contains the buttons and the result table. 
• The button Filter<br/>Actors executes the showFilterActors() method to navigate to the filter-actors module. 
• The button Open<br/>Filters executes the 'open(content)' method to open the Offcanvas component with the <ng-template> that contains the template variable '#content'. 

The filter-movies.component.ts shows the Offcanvas component, calls the filters, and shows the results:

@Component({
  selector: 'app-filter-movies',
  templateUrl: './filter-movies.component.html',
  styleUrls: ['./filter-movies.component.scss']
})
export class FilterMoviesComponent implements OnInit {
  protected filteredMovies: Movie[] = [];
  protected filtering = false;
  protected selectedGeneresStr = '';
  protected generes: Genere[] = [];
  protected closeResult = '';
  protected filterCriteria = new MovieFilterCriteria();
  protected ngbReleaseFrom: NgbDateStruct;
  protected ngbReleaseTo: NgbDateStruct;
  
  constructor(private offcanvasService: NgbOffcanvas, 
     public ngbRatingConfig: NgbRatingConfig, 
     private movieService: MoviesService, private router: Router) {}
  
...

  public open(content: unknown) {
    this.offcanvasService.open(content, 
       {ariaLabelledBy: 'offcanvas-basic-title'}).result.then((result) =>   
          { this.closeResult = `Closed with: ${result}`;
    }, (reason) => {
      this.closeResult = `Dismissed ${this.getDismissReason(reason)}`;
    });
  }

  public showFilterActors(): void {
     this.router.navigate(['/filter-actors']);
  }

  private getDismissReason(reason: unknown): void {
    //console.log(this.filterCriteria);
    if (reason === OffcanvasDismissReasons.ESC) {
      return this.resetFilters();
    } else {
       this.filterCriteria.releaseFrom = !this.ngbReleaseFrom ? null : 
          new Date(this.ngbReleaseFrom.year, this.ngbReleaseFrom.month, 
             this.ngbReleaseFrom.day);
       this.filterCriteria.releaseTo = !this.ngbReleaseTo ? null : 
	  new Date(this.ngbReleaseTo.year, this.ngbReleaseTo.month, 
             this.ngbReleaseTo.day);
       this.movieService.findMoviesByCriteria(this.filterCriteria)
          .subscribe({next: result => this.filteredMovies = result, 
             error: failed => {
	        console.log(failed);
	        this.router.navigate(['/']);
             }
       });
    }
  }
}

The FilterMoviesComponent constructor gets the NgbOffcanvas, NgbRatingConfig, MoviesService, router injected. 

The open method of the 'offCanvasService' opens the Offcanvas component and returns a promise to return the 'closeResult'. 

The showFilterActors(..) navigates to the route of the lazy loaded filter-actors module. 

The method getDismissReason(…) also checks for the "Escape" button that resets the filters. The FilterCriteria contains the dates from the ngbDateStruct objects of the datepicker and calls the findMoviesByCriteria(…) of the 'MovieService'. The subscribe(…) method stores the result in the 'filteredMovies' property. 

Conclusion Frontend

The Angular frontend needed some effort and time until the Ng-Bootstrap components were only included in the lazy loaded modules filter-actors and filter-movies. That enables fast initial load times. The Ng-Bootstrap components were easy to use and worked fine. The front end is a UX challenge. For example, how to show the user a full-text search with "and," "or," and "not" operators.