Dynamically Filter JSON With Jackson and Squiggly

At my current company, we built an internal issue tracking application for our service team to better support our customers. The Service Desk, as we call it, is a typical Spring Boot web application with many REST endpoints that serialize objects to JSON using the popular Jackson library. One such endpoint provides a list of issue objects, which is the root level object in Service Desk.

Several months ago, we noticed that our issue search screen was slow. After some digging, we realized that the issue object had grown from a handful of properties to over 50 properties. We were pulling down an enormous payload just to display a few properties on the screen! 

We wondered if there was a library out there that would allow us to dynamically filter our JSON responses on the query string so we could select only the properties we needed. For example, we could make a GET request to /issues?fields=id,summary and only get the issue id property and the issue summary property.  

We did discover a few libraries that provided some level of filtering, but none provided the granularity that we wanted. So, we decided to create an open-source library called Squiggly.

What Is Squiggly?

Squiggly is a library that plugs into Jackson and gives you the ability to dynamically filter properties during object serialization. It is inspired by the Facebook Graph API syntax.

Using Squiggly

The best way to learn Squiggly is to run an example. In this article, we are going to run the Spring Boot example. However, Squiggly does not require Spring Boot and can integrate with any application that has Jackson and the Servlet API.

To run the example, make sure you have the following installed:

• Java 7 or later.

• The git command line client.

•  Maven 3 or later.

•  curl.

Once these are installed, type the following into your terminal:

git clone https://github.com/bohnman/squiggly-filter-jackson.git
cd squiggly-filter-jackson/examples/spring-boot
mvn spring-boot:run

This will start the Spring Boot application.  In another, terminal type the following:

curl -s -g 'http://localhost:8080/issues/ISSUE-1'

You should see unformatted JSON output equivalent to the following:

{
  "actions": [
    {
      "id": null,
      "text": "I'm going to let Daario get this one..",
      "type": "COMMENT",
      "user": {
        "firstName": "Jorah",
        "lastName": "Mormont"
      }
    },
    {
      "id": null,
      "text": "All set.",
      "type": "CLOSE",
      "user": {
        "firstName": "Daario",
        "lastName": "Naharis"
      }
    }
  ],
  "assignee": {
    "firstName": "Jorah",
    "lastName": "Mormont"
  },
  "id": "ISSUE-1",
  "issueDetails": "I need my dragons fed pronto.",
  "issueSummary": "Dragons Need Fed",
  "properties": {
    "email": "motherofdragons@got.com",
    "priority": "1"
  },
  "reporter": {
    "firstName": "Daenerys",
    "lastName": "Targaryen"
  }
}

Simple Filtering

To retrieve just the id and issueSummary properties, simply type:

curl -s -g 'http://localhost:8080/issues/ISSUE-1?fields=id,issueSummary'
# prints {"id":"ISSUE-1","issueSummary":"Dragons Need Fed"}

Nested Filtering

The simple filter was a bit boring. Now, let's select the issue id as well as the text and type properties of the items in the actions array:

curl -s -g 'http://localhost:8080/issues/ISSUE-1?fields=id,actions{text,type}'
# prints {"id":"ISSUE-1","actions":[{"type":"COMMENT","text":"I'm going to let Daario get this one.."},{"type":"CLOSE","text":"All set."}]}

As an alternative, you can also use the dot syntax, but it's a little more verbose:

curl -s -g 'http://localhost:8080/issues/ISSUE-1?fields=id,actions.text,actions.type'

Regex Filtering

You can also filter using a regular expression. Let's select all properties that start with 'issue.'

curl -s -g 'http://localhost:8080/issues/ISSUE-1?fields=~issue.*~'
# prints {"issueSummary":"Dragons Need Fed","issueDetails":"I need my dragons fed pronto."}

Note that the regex is delimited by the tilde (~) character instead of forward slashes. This is because you don't have to escape a tilde in the query string.

Exclusion Filtering

If you wanted to include all properties except the id properties, just prefix the property with a minus sign like so:

curl -s -g 'http://localhost:8080/issues/ISSUE-1?fields=-id'

Other Features

In addition to the filtering capabilities mentioned, Squiggly also provides the following features:

• Ability to annotate properties with a named view. This is convenient if you want to provide a light view of your object and a full view of your object.

• Ability to filter maps and collections.

• API for providing metrics about various internal caches.

• Detailed documentation.

Conclusion

Squiggly Filter provides a robust library for dynamically filtering JSON. In addition to Spring Boot, you can find examples of how to integrate with Dropwizard, a traditional web app, and even a standalone command line application.

 You can find Squiggly at https://github.com/bohnman/squiggly-filter-jackson.