Spring Boot Plus Apache Ignite Data Grid

Spring Boot Integration With Apache Ignite and Its Durable Memory

In this post, we will show how we can do the following :

• Integrate Spring Boot with Apache Ignite

• Enable and use the persistent durable memory feature of Apache Ignite, which can persist your cache data to the file disk, to survive a crash or restart so you can avoid data loss

• Execute SQL queries over Ignite caches

• Unit test and integration test Ignite with Spring Boot

• Make simple Jenkins pipeline references

The code repository in GitHub is here.

What Is Ignite Durable Memory?

The Apache Ignite memory-centric platform is based on the durable memory architecture that allows storing and processing data and indexes both in memory and on disk when the Ignite Native Persistence feature is enabled. The durable memory architecture helps achieve in-memory performance with the durability of disk using all the available resources of the cluster

What Are Ignite Data-Grid SQL Queries?

Ignite supports a very elegant query API with support for Predicate-based Scan Queries, SQL Queries (ANSI-99 compliant), and Text Queries. For SQL queries, Ignite supports in-memory indexing, so all the data lookups are extremely fast. If you are caching your data in off-heap memory, then query indexes will also be cached in off-heap memory as well.

Ignite also provides support for custom indexing via IndexingSpi and the SpiQuery class.

More information about cache queries can be found here.

Integrating Apache Ignite Server Node With Your Spring Boot App

Add the following Maven dependencies to your Spring Boot app pom file:

      <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-core</artifactId>
            <version>${ignite.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-spring</artifactId>
            <version>${ignite.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-indexing</artifactId>
            <version>${ignite.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.ignite</groupId>
            <artifactId>ignite-slf4j</artifactId>
            <version>${ignite.version}</version>
        </dependency>

Define the Ignite configuration via Java DSL for better portability and management as a Spring configuration and the properties values will be loaded from the application.yml file:

@Configuration
public class AlertManagerConfiguration {

    @Value("${mail.service.baseUrl}")
    private String baseUrl;
    @Value("${mail.service.user}")
    private String user;
    @Value("${mail.service.password}")
    private String password;
    @Value("${enableFilePersistence}")
    private boolean enableFilePersistence;
    @Value("${igniteConnectorPort}")
    private int igniteConnectorPort;
    @Value("${igniteServerPortRange}")
    private String igniteServerPortRange;
    @Value("${ignitePersistenceFilePath}")
    private String ignitePersistenceFilePath;

    @Bean
    IgniteConfiguration igniteConfiguration() {
        IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
        igniteConfiguration.setClientMode(false);
        // durable file memory persistence
        if(enableFilePersistence){
            PersistentStoreConfiguration persistentStoreConfiguration = new PersistentStoreConfiguration();
            persistentStoreConfiguration.setPersistentStorePath("./data/store");
            persistentStoreConfiguration.setWalArchivePath("./data/walArchive");
            persistentStoreConfiguration.setWalStorePath("./data/walStore");
            igniteConfiguration.setPersistentStoreConfiguration(persistentStoreConfiguration);
        }
        // connector configuration
        ConnectorConfiguration connectorConfiguration=new ConnectorConfiguration();
        connectorConfiguration.setPort(igniteConnectorPort);
        // common ignite configuration
        igniteConfiguration.setMetricsLogFrequency(0);
        igniteConfiguration.setQueryThreadPoolSize(2);
        igniteConfiguration.setDataStreamerThreadPoolSize(1);
        igniteConfiguration.setManagementThreadPoolSize(2);
        igniteConfiguration.setPublicThreadPoolSize(2);
        igniteConfiguration.setSystemThreadPoolSize(2);
        igniteConfiguration.setRebalanceThreadPoolSize(1);
        igniteConfiguration.setAsyncCallbackPoolSize(2);
        igniteConfiguration.setPeerClassLoadingEnabled(false);
        igniteConfiguration.setIgniteInstanceName("alertsGrid");
        BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
        binaryConfiguration.setCompactFooter(false);
        igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
        // cluster tcp configuration
        TcpDiscoverySpi tcpDiscoverySpi=new TcpDiscoverySpi();
        TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder=new TcpDiscoveryVmIpFinder();
        // need to be changed when it come to real cluster
        tcpDiscoveryVmIpFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
        tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);
        igniteConfiguration.setDiscoverySpi(new TcpDiscoverySpi());
        // cache configuration
        CacheConfiguration alerts=new CacheConfiguration();
        alerts.setCopyOnRead(false);
        // as we have one node for now
        alerts.setBackups(0);
        alerts.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        alerts.setName("Alerts");
        alerts.setIndexedTypes(String.class,AlertEntry.class);

        CacheConfiguration alertsConfig=new CacheConfiguration();
        alertsConfig.setCopyOnRead(false);
        // as we have one node for now
        alertsConfig.setBackups(0);
        alertsConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
        alertsConfig.setName("AlertsConfig");
        alertsConfig.setIndexedTypes(String.class,AlertConfigEntry.class);
        igniteConfiguration.setCacheConfiguration(alerts,alertsConfig);
        return igniteConfiguration;
    }

    @Bean(destroyMethod = "close")
    Ignite ignite(IgniteConfiguration igniteConfiguration) throws IgniteException {
        final Ignite start = Ignition.start(igniteConfiguration);
        start.active(true);
        return start;
    }



}

Then you can just inject the Ignite instance as a Spring bean, which makes unit testing much easier.

@Component
public class IgniteAlertConfigStore implements AlertsConfigStore {

    private static final Logger logger = LoggerFactory.getLogger(IgniteAlertConfigStore.class);
    // here it will be injected as a spring bean
    @Autowired
    private Ignite ignite;

    @Override
    public AlertConfigEntry getConfigForServiceIdCodeId(String serviceId, String codeId) {
        return Optional.ofNullable(getAlertsConfigCache().get(serviceId + "_" + codeId))
                .orElseThrow(() -> new ResourceNotFoundException(String.format("Alert config for %s with %s not found", serviceId,codeId)));
    }

    @Override
    public void update(String serviceId, String codeId, AlertConfigEntry alertConfigEntry) {
        getAlertsConfigCache().put(serviceId + "_" + codeId, alertConfigEntry);
    }

    @Override
    public Optional<AlertConfigEntry> getConfigForServiceIdCodeIdCount(String serviceId, String codeId) {
        return Optional.ofNullable(getAlertsConfigCache().get(serviceId + "_" + codeId));

    }


    public Cache<String, AlertConfigEntry> getAlertsConfigCache() {
        return ignite.getOrCreateCache(CacheNames.AlertsConfig.name());
    }
}

How to enable Ignite durable memory:

  // durable file memory persistence
        if(enableFilePersistence){
            PersistentStoreConfiguration persistentStoreConfiguration = new PersistentStoreConfiguration();
            persistentStoreConfiguration.setPersistentStorePath("./data/store");
            persistentStoreConfiguration.setWalArchivePath("./data/walArchive");
            persistentStoreConfiguration.setWalStorePath("./data/walStore");
            igniteConfiguration.setPersistentStoreConfiguration(persistentStoreConfiguration);
}

How to use Ignite SQL queries in memory storage:

  @Override
    public List<AlertEntry> getAlertForServiceId(String serviceId) {
        final String sql = "serviceId = ?";
      // create the sql query object with entity type of the value part of the key value cache
        SqlQuery<String, AlertEntry> query = new SqlQuery<>(AlertEntry.class, sql);
      // set the query params
        query.setArgs(serviceId);
       //then execute it over the cache
        return Optional.ofNullable(getAlertsCache().query(query).getAll().stream().map(stringAlertEntryEntry -> stringAlertEntryEntry.getValue()).collect(Collectors.toList()))
                .orElseThrow(() -> new ResourceNotFoundException(String.format("Alert for %s not found", serviceId)));
}

How to do atomic thread-safe actions over the same record via cache invoke API:

 @Override
    public void updateAlertEntry(String serviceId, String serviceCode, AlertEntry alertEntry) {
        final IgniteCache<String, AlertEntry> alertsCache = getAlertsCache();
        // update the alert entry via cache invoke for atomicity
        alertsCache.invoke(alertEntry.getAlertId(), (mutableEntry, objects) -> {
            if (mutableEntry.exists() && mutableEntry.getValue() != null) {
                logger.debug("updating alert entry into the cache store invoke: {},{}", serviceId, serviceCode);
                mutableEntry.setValue(alertEntry);
            } else {
                throw new ResourceNotFoundException(String.format("Alert for %s with %s not found", serviceId, serviceCode));
            }
            // by api design nothing needed here
            return null;
        });
    }

How to unit test Apache Ignite usage in the Spring Boot service:

@RunWith(MockitoJUnitRunner.class)
public class IgniteAlertsSoreTest {
    @Mock
    private Ignite ignite;
    @Mock
    Cache<String, List<AlertEntry>>  cache;
    @Mock
    IgniteCache IgniteCache;
    @InjectMocks
    private IgniteAlertsStore igniteAlertsStore;
    //simulate the needed behaviour for the mocked ignite cache
    @Before
    public void setUp() throws Exception {
        when(ignite.getOrCreateCache(anyString())).thenReturn(IgniteCache);
        List<AlertEntry> entries=new ArrayList<>();
        entries.add(AlertEntry.builder().errorCode("errorCode").build());
        when(IgniteCache.get(anyString())).thenReturn(entries);
    }
    @Test
    public void getAllAlerts() throws Exception {
        assertEquals(igniteAlertsStore.getAlertForServiceId("serviceId").get(0).getErrorCode(),"errorCode");
    }


}

How to trigger integration tests with Ignite and check test resources:

@RunWith(SpringRunner.class)
@SpringBootTest(classes = AlertManagerApplication.class,webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ActiveProfiles("INTEGRATION_TEST")
public class AlertManagerApplicationIT {
    @LocalServerPort
    private int port;
    @Autowired
    private TestRestTemplate template;
    private URL base;
    @Before
    public void setUp() throws Exception {
        this.base = new URL("http://localhost:" + port + "/");
    }
    // then add your integration test which will include real started ignite server node whoch will be closed once the integration test is done
    @Test
    public void contextLoads() {
        assertTrue(template.getForEntity(base+"/health",String.class).getStatusCode().is2xxSuccessful());

    }

}

How to run and test the application over Swagger REST API:

• Build the project via Maven with  mvn clean install 
• You can run it from IDEA via AlertManagerApplication.java or via java -jar jarName.

• Swagger, which contains the REST API and REST API model documentation, will be accessible on the URL below, where you can start triggering different REST API calls exposed by the Spring Boot app: http://localhost:8080/swagger-ui.html#/.

• If you STOP the app or restart it and do the query again, you will find all created entities from the last run so it survived the crash plus any possible restart.
• You can build a portable Docker image of the whole app using the Maven Spotify Docker plugin, if you wish.

References:

• Apache Ignite: https://apacheignite.readme.io/docs
• Code repository on GitHub: GitHubRepo