How to Build a Graph Data System Powered by ScyllaDB and JanusGraph
Learn how to combine ScyllaDB with JanusGraph to make graph model analyses out of your data using Gremlin/Tinkerpop.
Join the DZone community and get the full member experience.
Join For FreeThis tutorial will teach you how to deploy a Graph Data System, using JanusGraph and ScyllaDB as the underlying data storage layer. It is part of a broader training course ScyllaDB University course, The Mutant Monitoring System (MMS) A Graph Data System Powered by ScyllaDB and JanusGraph.
A graph data system (or graph database) is a database that uses a graph structure with nodes and edges to represent data. Edges represent relationships between nodes, and these relationships allow the data to be linked and for the graph to be visualized. It’s possible to use different storage mechanisms for the underlying data, and this choice affects the performance, scalability, ease of maintenance, and cost.
Some common use cases for graph databases are knowledge graphs, recommendation applications, social networks, and fraud detection.
JanusGraph is a scalable open-source graph database that’s optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster. It stores graphs in adjacency list format, which means that a graph is stored as a collection of vertices with their edges and properties.
JanusGraph natively supports the graph traversal language Gremlin. Gremlin is a part of Apache TinkerPop and is developed and maintained separately from JanusGraph. Many graph databases support it, and by using it, users avoid vendor lock-in as their applications can be migrated to other graph databases. You can learn more about the architecture on the JanusGraph Documentation page.
JanusGraphs supports ElasticSearch as an indexing backend. ElasticSearch is a search and analytics engine based on Apache Lucene. JanusGraph server (see Running the JanusGraph server) automatically starts a local ElasticSearch instance. It’s also possible to manually start a local ElasticSearch instance, however, this is not in the scope of this lesson. (However, see the documentation here.)
This lesson presents a step-by-step, hands-on example for deploying JanusGraph and performing some basic operations. Here are the main steps in the lesson:
- Spinning up a virtual machine on AWS
- Installing the prerequisites
- Running the JanusGraph server (using Docker)
- Running a Gremlin Console to connect to the new server (also in Docker)
- Spinning up a three-node ScyllaDB Cluster and setting it as the data storage for the JanusGraph server
- Performing some basic graph operations
Since there are many moving parts to setting up JanusGraph and lots of options, the lesson goes over each step in the setup process. It uses AWS, but you can follow through with the example with another cloud provider or with a local machine.
Note that this example is for training purposes only and in production, things should be done differently. For example, each ScyllaDB node should be run on a single server.
Launch an AWS EC2 Instance
Start by spinning up a t3.large Amazon Linux machine. From the EC2 console, select Instances and Launch Instances. In the first step choose the “Amazon Linux 2 AMI (HVM) – Kernel 5.10, SSD Volume Type – ami-008e1e7f1fcbe9b80 (64-bit x86)” AMI. In the second step, select t3.2xlarge. The reason for using a 2xlarge machine is that we’re running a few docker instances on the same server and it requires some resources. Change the storage to 16 GB.
Once the machine is running, connect to it using SSH. In the command below, replace the path to your key pair and the public DNS name of your instance.
ssh -i ~/Downloads/aws/guy-janus.pem ec2-user@ec2-3-21-28-125.us-east-2.compute.amazonaws.com
[guy@fedora ~]$ ssh -i ~/Downloads/aws/guy-janus.pem ec2-user@ec2-3-21-28-125.us-east-2.compute.amazonaws.com | |
The authenticity of host 'ec2-3-21-28-125.us-east-2.compute.amazonaws.com (3.21.28.125)' can't be established. | |
ED25519 key fingerprint is SHA256:91CQ/okRKYs7AOq6xpkbWPJmlstKiySNV4vbWI0VOb4. | |
This key is not known by any other names | |
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes | |
Warning: Permanently added 'ec2-3-21-28-125.us-east-2.compute.amazonaws.com' (ED25519) to the list of known hosts. | |
__| __|_ ) | |
_| ( / Amazon Linux 2 AMI | |
___|\___|___| | |
https://aws.amazon.com/amazon-linux-2/ | |
No packages needed for security; 5 packages available | |
Run "sudo yum update" to apply all updates. | |
[ec2-user@ip-172-31-20-0 ~]$ |
Next, make sure all your packages are updated:
sudo yum update
Install Java
JanusGraph requires Java. To install Java and its prerequisites, run:
sudo yum install java
[ec2-user@ip-172-31-20-0 ~]$ sudo yum install java | |
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd | |
Resolving Dependencies | |
--> Running transaction check | |
---> Package java-17-amazon-corretto.x86_64 1:17.0.2+8-1.amzn2.1 will be installed | |
--> Processing Dependency: java-17-amazon-corretto-headless(x86-64) = 1:17.0.2+8-1.amzn2.1 for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: dejavu-sans-mono-fonts for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: dejavu-serif-fonts for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: dejavu-sans-fonts for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: alsa-lib for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libXtst for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libXrandr for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libXrender for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libXt for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libXinerama for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libXi for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: libX11 for package: 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 | |
--> Running transaction check | |
---> Package alsa-lib.x86_64 0:1.1.4.1-2.amzn2 will be installed | |
---> Package dejavu-sans-fonts.noarch 0:2.33-6.amzn2 will be installed | |
--> Processing Dependency: dejavu-fonts-common = 2.33-6.amzn2 for package: dejavu-sans-fonts-2.33-6.amzn2.noarch | |
---> Package dejavu-sans-mono-fonts.noarch 0:2.33-6.amzn2 will be installed | |
---> Package dejavu-serif-fonts.noarch 0:2.33-6.amzn2 will be installed | |
---> Package java-17-amazon-corretto-headless.x86_64 1:17.0.2+8-1.amzn2.1 will be installed | |
--> Processing Dependency: log4j-cve-2021-44228-cve-mitigations for package: 1:java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: fontconfig for package: 1:java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64 | |
--> Processing Dependency: jpackage-utils for package: 1:java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64 | |
---> Package libX11.x86_64 0:1.6.7-3.amzn2.0.2 will be installed | |
--> Processing Dependency: libX11-common >= 1.6.7-3.amzn2.0.2 for package: libX11-1.6.7-3.amzn2.0.2.x86_64 | |
--> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.6.7-3.amzn2.0.2.x86_64 | |
---> Package libXi.x86_64 0:1.7.9-1.amzn2.0.2 will be installed | |
--> Processing Dependency: libXext.so.6()(64bit) for package: libXi-1.7.9-1.amzn2.0.2.x86_64 | |
---> Package libXinerama.x86_64 0:1.1.3-2.1.amzn2.0.2 will be installed | |
---> Package libXrandr.x86_64 0:1.5.1-2.amzn2.0.3 will be installed | |
---> Package libXrender.x86_64 0:0.9.10-1.amzn2.0.2 will be installed | |
---> Package libXt.x86_64 0:1.1.5-3.amzn2.0.2 will be installed | |
--> Processing Dependency: libSM.so.6()(64bit) for package: libXt-1.1.5-3.amzn2.0.2.x86_64 | |
--> Processing Dependency: libICE.so.6()(64bit) for package: libXt-1.1.5-3.amzn2.0.2.x86_64 | |
---> Package libXtst.x86_64 0:1.2.3-1.amzn2.0.2 will be installed | |
--> Running transaction check | |
---> Package dejavu-fonts-common.noarch 0:2.33-6.amzn2 will be installed | |
--> Processing Dependency: fontpackages-filesystem for package: dejavu-fonts-common-2.33-6.amzn2.noarch | |
---> Package fontconfig.x86_64 0:2.13.0-4.3.amzn2 will be installed | |
---> Package javapackages-tools.noarch 0:3.4.1-11.amzn2 will be installed | |
--> Processing Dependency: python-javapackages = 3.4.1-11.amzn2 for package: javapackages-tools-3.4.1-11.amzn2.noarch | |
--> Processing Dependency: libxslt for package: javapackages-tools-3.4.1-11.amzn2.noarch | |
---> Package libICE.x86_64 0:1.0.9-9.amzn2.0.2 will be installed | |
---> Package libSM.x86_64 0:1.2.2-2.amzn2.0.2 will be installed | |
---> Package libX11-common.noarch 0:1.6.7-3.amzn2.0.2 will be installed | |
---> Package libXext.x86_64 0:1.3.3-3.amzn2.0.2 will be installed | |
---> Package libxcb.x86_64 0:1.12-1.amzn2.0.2 will be installed | |
--> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.12-1.amzn2.0.2.x86_64 | |
---> Package log4j-cve-2021-44228-hotpatch.noarch 0:1.1-13.amzn2 will be installed | |
--> Running transaction check | |
---> Package fontpackages-filesystem.noarch 0:1.44-8.amzn2 will be installed | |
---> Package libXau.x86_64 0:1.0.8-2.1.amzn2.0.2 will be installed | |
---> Package libxslt.x86_64 0:1.1.28-6.amzn2 will be installed | |
---> Package python-javapackages.noarch 0:3.4.1-11.amzn2 will be installed | |
--> Processing Dependency: python-lxml for package: python-javapackages-3.4.1-11.amzn2.noarch | |
--> Running transaction check | |
---> Package python-lxml.x86_64 0:3.2.1-4.amzn2.0.3 will be installed | |
--> Finished Dependency Resolution | |
Dependencies Resolved | |
============================================================================================================================================================================================================================================================== | |
Package Arch Version Repository Size | |
============================================================================================================================================================================================================================================================== | |
Installing: | |
java-17-amazon-corretto x86_64 1:17.0.2+8-1.amzn2.1 amzn2-core 178 k | |
Installing for dependencies: | |
alsa-lib x86_64 1.1.4.1-2.amzn2 amzn2-core 425 k | |
dejavu-fonts-common noarch 2.33-6.amzn2 amzn2-core 64 k | |
dejavu-sans-fonts noarch 2.33-6.amzn2 amzn2-core 1.4 M | |
dejavu-sans-mono-fonts noarch 2.33-6.amzn2 amzn2-core 433 k | |
dejavu-serif-fonts noarch 2.33-6.amzn2 amzn2-core 777 k | |
fontconfig x86_64 2.13.0-4.3.amzn2 amzn2-core 253 k | |
fontpackages-filesystem noarch 1.44-8.amzn2 amzn2-core 10 k | |
java-17-amazon-corretto-headless x86_64 1:17.0.2+8-1.amzn2.1 amzn2-core 94 M | |
javapackages-tools noarch 3.4.1-11.amzn2 amzn2-core 73 k | |
libICE x86_64 1.0.9-9.amzn2.0.2 amzn2-core 67 k | |
libSM x86_64 1.2.2-2.amzn2.0.2 amzn2-core 39 k | |
libX11 x86_64 1.6.7-3.amzn2.0.2 amzn2-core 606 k | |
libX11-common noarch 1.6.7-3.amzn2.0.2 amzn2-core 165 k | |
libXau x86_64 1.0.8-2.1.amzn2.0.2 amzn2-core 29 k | |
libXext x86_64 1.3.3-3.amzn2.0.2 amzn2-core 39 k | |
libXi x86_64 1.7.9-1.amzn2.0.2 amzn2-core 41 k | |
libXinerama x86_64 1.1.3-2.1.amzn2.0.2 amzn2-core 14 k | |
libXrandr x86_64 1.5.1-2.amzn2.0.3 amzn2-core 27 k | |
libXrender x86_64 0.9.10-1.amzn2.0.2 amzn2-core 26 k | |
libXt x86_64 1.1.5-3.amzn2.0.2 amzn2-core 177 k | |
libXtst x86_64 1.2.3-1.amzn2.0.2 amzn2-core 20 k | |
libxcb x86_64 1.12-1.amzn2.0.2 amzn2-core 216 k | |
libxslt x86_64 1.1.28-6.amzn2 amzn2-core 240 k | |
log4j-cve-2021-44228-hotpatch noarch 1.1-13.amzn2 amzn2-core 144 k | |
python-javapackages noarch 3.4.1-11.amzn2 amzn2-core 31 k | |
python-lxml x86_64 3.2.1-4.amzn2.0.3 amzn2-core 1.0 M | |
Transaction Summary | |
============================================================================================================================================================================================================================================================== | |
Install 1 Package (+26 Dependent packages) | |
Total download size: 100 M | |
Installed size: 256 M | |
Is this ok [y/d/N]: y | |
Downloading packages: | |
(1/27): dejavu-fonts-common-2.33-6.amzn2.noarch.rpm | 64 kB 00:00:00 | |
(2/27): alsa-lib-1.1.4.1-2.amzn2.x86_64.rpm | 425 kB 00:00:00 | |
(3/27): dejavu-sans-mono-fonts-2.33-6.amzn2.noarch.rpm | 433 kB 00:00:00 | |
(4/27): dejavu-sans-fonts-2.33-6.amzn2.noarch.rpm | 1.4 MB 00:00:00 | |
(5/27): dejavu-serif-fonts-2.33-6.amzn2.noarch.rpm | 777 kB 00:00:00 | |
(6/27): fontconfig-2.13.0-4.3.amzn2.x86_64.rpm | 253 kB 00:00:00 | |
(7/27): fontpackages-filesystem-1.44-8.amzn2.noarch.rpm | 10 kB 00:00:00 | |
(8/27): java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64.rpm | 178 kB 00:00:00 | |
(9/27): javapackages-tools-3.4.1-11.amzn2.noarch.rpm | 73 kB 00:00:00 | |
(10/27): libICE-1.0.9-9.amzn2.0.2.x86_64.rpm | 67 kB 00:00:00 | |
(11/27): libSM-1.2.2-2.amzn2.0.2.x86_64.rpm | 39 kB 00:00:00 | |
(12/27): libX11-1.6.7-3.amzn2.0.2.x86_64.rpm | 606 kB 00:00:00 | |
(13/27): libX11-common-1.6.7-3.amzn2.0.2.noarch.rpm | 165 kB 00:00:00 | |
(14/27): libXau-1.0.8-2.1.amzn2.0.2.x86_64.rpm | 29 kB 00:00:00 | |
(15/27): libXext-1.3.3-3.amzn2.0.2.x86_64.rpm | 39 kB 00:00:00 | |
(16/27): libXi-1.7.9-1.amzn2.0.2.x86_64.rpm | 41 kB 00:00:00 | |
(17/27): libXinerama-1.1.3-2.1.amzn2.0.2.x86_64.rpm | 14 kB 00:00:00 | |
(18/27): libXrandr-1.5.1-2.amzn2.0.3.x86_64.rpm | 27 kB 00:00:00 | |
(19/27): libXrender-0.9.10-1.amzn2.0.2.x86_64.rpm | 26 kB 00:00:00 | |
(20/27): libXt-1.1.5-3.amzn2.0.2.x86_64.rpm | 177 kB 00:00:00 | |
(21/27): libXtst-1.2.3-1.amzn2.0.2.x86_64.rpm | 20 kB 00:00:00 | |
(22/27): libxcb-1.12-1.amzn2.0.2.x86_64.rpm | 216 kB 00:00:00 | |
(23/27): libxslt-1.1.28-6.amzn2.x86_64.rpm | 240 kB 00:00:00 | |
(24/27): log4j-cve-2021-44228-hotpatch-1.1-13.amzn2.noarch.rpm | 144 kB 00:00:00 | |
(25/27): python-javapackages-3.4.1-11.amzn2.noarch.rpm | 31 kB 00:00:00 | |
(26/27): python-lxml-3.2.1-4.amzn2.0.3.x86_64.rpm | 1.0 MB 00:00:00 | |
(27/27): java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64.rpm | 94 MB 00:00:01 | |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
Total 62 MB/s | 100 MB 00:00:01 | |
Running transaction check | |
Running transaction test | |
Transaction test succeeded | |
Running transaction | |
Installing : libICE-1.0.9-9.amzn2.0.2.x86_64 1/27 | |
Installing : libxslt-1.1.28-6.amzn2.x86_64 2/27 | |
Installing : fontpackages-filesystem-1.44-8.amzn2.noarch 3/27 | |
Installing : dejavu-fonts-common-2.33-6.amzn2.noarch 4/27 | |
Installing : dejavu-sans-fonts-2.33-6.amzn2.noarch 5/27 | |
Installing : fontconfig-2.13.0-4.3.amzn2.x86_64 6/27 | |
Installing : dejavu-serif-fonts-2.33-6.amzn2.noarch 7/27 | |
Installing : dejavu-sans-mono-fonts-2.33-6.amzn2.noarch 8/27 | |
Installing : python-lxml-3.2.1-4.amzn2.0.3.x86_64 9/27 | |
Installing : python-javapackages-3.4.1-11.amzn2.noarch 10/27 | |
Installing : javapackages-tools-3.4.1-11.amzn2.noarch 11/27 | |
Installing : libSM-1.2.2-2.amzn2.0.2.x86_64 12/27 | |
Installing : alsa-lib-1.1.4.1-2.amzn2.x86_64 13/27 | |
Installing : log4j-cve-2021-44228-hotpatch-1.1-13.amzn2.noarch 14/27 | |
Created symlink from /etc/systemd/system/multi-user.target.wants/log4j-cve-2021-44228-hotpatch.service to /usr/lib/systemd/system/log4j-cve-2021-44228-hotpatch.service. | |
Installing : 1:java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64 15/27 | |
Installing : libXau-1.0.8-2.1.amzn2.0.2.x86_64 16/27 | |
Installing : libxcb-1.12-1.amzn2.0.2.x86_64 17/27 | |
Installing : libX11-common-1.6.7-3.amzn2.0.2.noarch 18/27 | |
Installing : libX11-1.6.7-3.amzn2.0.2.x86_64 19/27 | |
Installing : libXext-1.3.3-3.amzn2.0.2.x86_64 20/27 | |
Installing : libXi-1.7.9-1.amzn2.0.2.x86_64 21/27 | |
Installing : libXrender-0.9.10-1.amzn2.0.2.x86_64 22/27 | |
Installing : libXrandr-1.5.1-2.amzn2.0.3.x86_64 23/27 | |
Installing : libXtst-1.2.3-1.amzn2.0.2.x86_64 24/27 | |
Installing : libXinerama-1.1.3-2.1.amzn2.0.2.x86_64 25/27 | |
Installing : libXt-1.1.5-3.amzn2.0.2.x86_64 26/27 | |
Installing : 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 27/27 | |
Verifying : dejavu-serif-fonts-2.33-6.amzn2.noarch 1/27 | |
Verifying : python-lxml-3.2.1-4.amzn2.0.3.x86_64 2/27 | |
Verifying : fontpackages-filesystem-1.44-8.amzn2.noarch 3/27 | |
Verifying : 1:java-17-amazon-corretto-17.0.2+8-1.amzn2.1.x86_64 4/27 | |
Verifying : libxcb-1.12-1.amzn2.0.2.x86_64 5/27 | |
Verifying : libXrandr-1.5.1-2.amzn2.0.3.x86_64 6/27 | |
Verifying : libXext-1.3.3-3.amzn2.0.2.x86_64 7/27 | |
Verifying : libX11-1.6.7-3.amzn2.0.2.x86_64 8/27 | |
Verifying : dejavu-sans-mono-fonts-2.33-6.amzn2.noarch 9/27 | |
Verifying : 1:java-17-amazon-corretto-headless-17.0.2+8-1.amzn2.1.x86_64 10/27 | |
Verifying : libX11-common-1.6.7-3.amzn2.0.2.noarch 11/27 | |
Verifying : dejavu-fonts-common-2.33-6.amzn2.noarch 12/27 | |
Verifying : libXau-1.0.8-2.1.amzn2.0.2.x86_64 13/27 | |
Verifying : libxslt-1.1.28-6.amzn2.x86_64 14/27 | |
Verifying : libSM-1.2.2-2.amzn2.0.2.x86_64 15/27 | |
Verifying : libXrender-0.9.10-1.amzn2.0.2.x86_64 16/27 | |
Verifying : dejavu-sans-fonts-2.33-6.amzn2.noarch 17/27 | |
Verifying : fontconfig-2.13.0-4.3.amzn2.x86_64 18/27 | |
Verifying : libXt-1.1.5-3.amzn2.0.2.x86_64 19/27 | |
Verifying : libXinerama-1.1.3-2.1.amzn2.0.2.x86_64 20/27 | |
Verifying : libXi-1.7.9-1.amzn2.0.2.x86_64 21/27 | |
Verifying : log4j-cve-2021-44228-hotpatch-1.1-13.amzn2.noarch 22/27 | |
Verifying : python-javapackages-3.4.1-11.amzn2.noarch 23/27 | |
Verifying : libXtst-1.2.3-1.amzn2.0.2.x86_64 24/27 | |
Verifying : alsa-lib-1.1.4.1-2.amzn2.x86_64 25/27 | |
Verifying : libICE-1.0.9-9.amzn2.0.2.x86_64 26/27 | |
Verifying : javapackages-tools-3.4.1-11.amzn2.noarch 27/27 | |
Installed: | |
java-17-amazon-corretto.x86_64 1:17.0.2+8-1.amzn2.1 | |
Dependency Installed: | |
alsa-lib.x86_64 0:1.1.4.1-2.amzn2 dejavu-fonts-common.noarch 0:2.33-6.amzn2 dejavu-sans-fonts.noarch 0:2.33-6.amzn2 dejavu-sans-mono-fonts.noarch 0:2.33-6.amzn2 dejavu-serif-fonts.noarch 0:2.33-6.amzn2 | |
fontconfig.x86_64 0:2.13.0-4.3.amzn2 fontpackages-filesystem.noarch 0:1.44-8.amzn2 java-17-amazon-corretto-headless.x86_64 1:17.0.2+8-1.amzn2.1 javapackages-tools.noarch 0:3.4.1-11.amzn2 libICE.x86_64 0:1.0.9-9.amzn2.0.2 | |
libSM.x86_64 0:1.2.2-2.amzn2.0.2 libX11.x86_64 0:1.6.7-3.amzn2.0.2 libX11-common.noarch 0:1.6.7-3.amzn2.0.2 libXau.x86_64 0:1.0.8-2.1.amzn2.0.2 libXext.x86_64 0:1.3.3-3.amzn2.0.2 | |
libXi.x86_64 0:1.7.9-1.amzn2.0.2 libXinerama.x86_64 0:1.1.3-2.1.amzn2.0.2 libXrandr.x86_64 0:1.5.1-2.amzn2.0.3 libXrender.x86_64 0:0.9.10-1.amzn2.0.2 libXt.x86_64 0:1.1.5-3.amzn2.0.2 | |
libXtst.x86_64 0:1.2.3-1.amzn2.0.2 libxcb.x86_64 0:1.12-1.amzn2.0.2 libxslt.x86_64 0:1.1.28-6.amzn2 log4j-cve-2021-44228-hotpatch.noarch 0:1.1-13.amzn2 python-javapackages.noarch 0:3.4.1-11.amzn2 | |
python-lxml.x86_64 0:3.2.1-4.amzn2.0.3 | |
Complete! | |
[ec2-user@ip-172-31-20-0 ~]$ |
Run the following command to make sure the installation succeeded:
java -version
ec2-user@ip-172-31-20-0 ~]$ java -version | |
openjdk version "17.0.2" 2022-01-18 LTS | |
OpenJDK Runtime Environment Corretto-17.0.2.8.1 (build 17.0.2+8-LTS) | |
OpenJDK 64-Bit Server VM Corretto-17.0.2.8.1 (build 17.0.2+8-LTS, mixed mode, sharing) | |
[ec2-user@ip-172-31-20-0 ~]$ |
JanusGraph required the $JAVA_HOME
environment variable to be set, and it also needs to be added to the $PATH
. This isn’t automatically set when installing Java. So you’ll set it manually.
Locate the Java installation location by running:
sudo update-alternatives --config java
[ec2-user@ip-172-31-20-0 ~]$ sudo update-alternatives --config java | |
There is 1 program that provides 'java'. | |
Selection Command | |
----------------------------------------------- | |
*+ 1 /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java | |
Enter to keep the current selection[+], or type selection number: | |
[ec2-user@ip-172-31-20-0 ~]$ |
Copy the path to the Java Installation. Now, edit the .bashrc file found in the home directory of the ec2-user user, and add the two lines below to the file, replacing the path with the one you copied, without the /bin/java at the end.
export JAVA_HOME="/usr/lib/jvm/java-17-amazon-corretto.x86_64"
PATH=$JAVA_HOME/bin:$PATH
Save the file and run the following command:
source .bashrc
Make sure the environment variables were set:
echo $JAVA_HOME
echo $PATH
[ec2-user@ip-172-31-20-0 ~]$ vim ~/.bashrc | |
[ec2-user@ip-172-31-20-0 ~]$ source .bashrc | |
[ec2-user@ip-172-31-20-0 ~]$ echo $JAVA_HOME | |
/usr/lib/jvm/java-17-amazon-corretto.x86_64 | |
[ec2-user@ip-172-31-20-0 ~]$ echo $PATH | |
/usr/lib/jvm/java-17-amazon-corretto.x86_64/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ec2-user/.local/bin:/home/ec2-user/bin | |
[ec2-user@ip-172-31-20-0 ~]$ |
Docker Installation
Next, since you’ll be running JanusGraph and ScyllaDB in Docker containers, install Docker and its prerequisites:
sudo yum install docker
Start the Docker service:
sudo service docker start
and (optionally) to ensure that the Docker daemon starts after each system reboot:
sudo systemctl enable docker
Add the current user (ec2-user) to the docker group so you can execute Docker commands without using sudo:
sudo usermod -a -G docker ec2-user
To activate the changes to the groups, run:
newgrp docker
Finally, make sure that the Docker engine was successfully installed by running the hello-world image:
docker run hello-world
[ec2-user@ip-172-31-20-0 ~]$ sudo yum install docker | |
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd | |
amzn2-core | 3.7 kB 00:00:00 | |
Resolving Dependencies | |
--> Running transaction check | |
---> Package docker.x86_64 0:20.10.7-5.amzn2 will be installed | |
--> Processing Dependency: runc >= 1.0.0 for package: docker-20.10.7-5.amzn2.x86_64 | |
--> Processing Dependency: libcgroup >= 0.40.rc1-5.15 for package: docker-20.10.7-5.amzn2.x86_64 | |
--> Processing Dependency: containerd >= 1.3.2 for package: docker-20.10.7-5.amzn2.x86_64 | |
--> Processing Dependency: pigz for package: docker-20.10.7-5.amzn2.x86_64 | |
--> Running transaction check | |
---> Package containerd.x86_64 0:1.4.6-8.amzn2 will be installed | |
---> Package libcgroup.x86_64 0:0.41-21.amzn2 will be installed | |
---> Package pigz.x86_64 0:2.3.4-1.amzn2.0.1 will be installed | |
---> Package runc.x86_64 0:1.0.0-2.amzn2 will be installed | |
--> Finished Dependency Resolution | |
Dependencies Resolved | |
============================================================================================================================================================================================================================================================== | |
Package Arch Version Repository Size | |
============================================================================================================================================================================================================================================================== | |
Installing: | |
docker x86_64 20.10.7-5.amzn2 amzn2extra-docker 42 M | |
Installing for dependencies: | |
containerd x86_64 1.4.6-8.amzn2 amzn2extra-docker 24 M | |
libcgroup x86_64 0.41-21.amzn2 amzn2-core 66 k | |
pigz x86_64 2.3.4-1.amzn2.0.1 amzn2-core 81 k | |
runc x86_64 1.0.0-2.amzn2 amzn2extra-docker 3.3 M | |
Transaction Summary | |
============================================================================================================================================================================================================================================================== | |
Install 1 Package (+4 Dependent packages) | |
Total download size: 69 M | |
Installed size: 285 M | |
Is this ok [y/d/N]: y | |
Downloading packages: | |
(1/5): libcgroup-0.41-21.amzn2.x86_64.rpm | 66 kB 00:00:00 | |
(2/5): pigz-2.3.4-1.amzn2.0.1.x86_64.rpm | 81 kB 00:00:00 | |
(3/5): containerd-1.4.6-8.amzn2.x86_64.rpm | 24 MB 00:00:00 | |
(4/5): runc-1.0.0-2.amzn2.x86_64.rpm | 3.3 MB 00:00:00 | |
(5/5): docker-20.10.7-5.amzn2.x86_64.rpm | 42 MB 00:00:01 | |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
Total 58 MB/s | 69 MB 00:00:01 | |
Running transaction check | |
Running transaction test | |
Transaction test succeeded | |
Running transaction | |
Installing : runc-1.0.0-2.amzn2.x86_64 1/5 | |
Installing : containerd-1.4.6-8.amzn2.x86_64 2/5 | |
Installing : libcgroup-0.41-21.amzn2.x86_64 3/5 | |
Installing : pigz-2.3.4-1.amzn2.0.1.x86_64 4/5 | |
Installing : docker-20.10.7-5.amzn2.x86_64 5/5 | |
Verifying : docker-20.10.7-5.amzn2.x86_64 1/5 | |
Verifying : containerd-1.4.6-8.amzn2.x86_64 2/5 | |
Verifying : runc-1.0.0-2.amzn2.x86_64 3/5 | |
Verifying : pigz-2.3.4-1.amzn2.0.1.x86_64 4/5 | |
Verifying : libcgroup-0.41-21.amzn2.x86_64 5/5 | |
Installed: | |
docker.x86_64 0:20.10.7-5.amzn2 | |
Dependency Installed: | |
containerd.x86_64 0:1.4.6-8.amzn2 libcgroup.x86_64 0:0.41-21.amzn2 pigz.x86_64 0:2.3.4-1.amzn2.0.1 runc.x86_64 0:1.0.0-2.amzn2 | |
Complete! | |
[ec2-user@ip-172-31-20-0 ~]$ sudo service docker start | |
Redirecting to /bin/systemctl start docker.service | |
[ec2-user@ip-172-31-20-0 ~]$ sudo systemctl enable docker | |
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. | |
[ec2-user@ip-172-31-20-0 ~]$ sudo usermod -a -G docker ec2-user | |
[ec2-user@ip-172-31-20-0 ~]$ newgrp docker | |
[ec2-user@ip-172-31-20-0 ~]$ docker run hello-world | |
Unable to find image 'hello-world:latest' locally | |
latest: Pulling from library/hello-world | |
2db29710123e: Pull complete | |
Digest: sha256:4c5f3db4f8a54eb1e017c385f683a2de6e06f75be442dc32698c9bbe6c861edd | |
Status: Downloaded newer image for hello-world:latest | |
Hello from Docker! | |
This message shows that your installation appears to be working correctly. | |
To generate this message, Docker took the following steps: | |
1. The Docker client contacted the Docker daemon. | |
2. The Docker daemon pulled the "hello-world" image from the Docker Hub. | |
(amd64) | |
3. The Docker daemon created a new container from that image which runs the | |
executable that produces the output you are currently reading. | |
4. The Docker daemon streamed that output to the Docker client, which sent it | |
to your terminal. | |
To try something more ambitious, you can run an Ubuntu container with: | |
$ docker run -it ubuntu bash | |
Share images, automate workflows, and more with a free Docker ID: | |
https://hub.docker.com/ | |
For more examples and ideas, visit: | |
https://docs.docker.com/get-started/ | |
[ec2-user@ip-172-31-20-0 ~]$ |
Deploying JanusGraph
There are a few options for how to deploy JanusGraph. In this lesson, you’ll deploy JanusGraph in a docker container to simplify the process. To learn about other ways of deploying JanusGraph read the Installation Guide.
Start a docker container for the JanusGraph server:
docker run --name janusgraph-default janusgraph/janusgraph:latest
[ec2-user@ip-172-31-20-0 ~]$ docker run --name janusgraph-default janusgraph/janusgraph:latest | |
Unable to find image 'janusgraph/janusgraph:latest' locally | |
latest: Pulling from janusgraph/janusgraph | |
6552179c3509: Pull complete | |
8aa8d0f5eb64: Pull complete | |
5f76ba172123: Pull complete | |
2d0f168fcf75: Pull complete | |
f5487b6eaad2: Pull complete | |
2725f322a41b: Pull complete | |
2f641065ee43: Pull complete | |
67414d803fef: Pull complete | |
f951860f0318: Pull complete | |
a2b811e553e7: Pull complete | |
Digest: sha256:c09d5d9fa0b105171dd57e1a2681ca4077df59cb394c0a08a920b5aa22397b79 | |
Status: Downloaded newer image for janusgraph/janusgraph:latest | |
/etc/opt/janusgraph/janusgraph-server.yaml will be used to start JanusGraph Server in foreground | |
SLF4J: Class path contains multiple SLF4J bindings. | |
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] | |
0 [main] INFO org.janusgraph.graphdb.server.JanusGraphServer - | |
mmm mmm # | |
# mmm m mm m m mmm m" " m mm mmm mmmm # mm | |
# " # #" # # # # " # mm #" " " # #" "# #" # | |
# m"""# # # # # """m # # # m"""# # # # # | |
"mmm" "mm"# # # "mm"# "mmm" "mmm" # "mm"# ##m#" # # | |
# | |
" | |
140 [main] INFO com.jcabi.manifests.Manifests - | |
... | |
... |
ScyllaDB Data Storage Backend, Transactional and Analytical Workloads
In this part, you’ll spin up a three-node ScyllaDB cluster which you’ll then use as a data storage backend for JanusGraph. The data storage layer for JanusGraph is pluggable. Some options for the data storage layer are Apache HBase, Google Cloud Bigtable, Oracle Berkeley DB Java Edition, Apache Cassandra, and ScyllaDB. The storage backend you use for your application is extremely important. By using ScyllaDB, some of the advantages you’ll get are low and consistent latency, high availability, up to x10 throughput, ease of use, and a highly scalable system.
A group at IBM compared using ScyllaDB as the JanusGraph storage backend vs. Apache Cassandra and HBase. They found that ScyllaDB displayed nearly 35% higher throughput when inserting vertices than HBase and almost 3X Cassandra’s throughput. When inserting edges, ScyllaDB’s throughput was 160% better than HBase and more than 4X that of Cassandra. In a query performance test, ScyllaDB performed 72% better than Cassandra and nearly 150% better than HBase.
With graph data systems, just like with any data system, we can separate our workloads into two categories – transactional and analytical. JanusGraph follows the Apache TinkerPop project’s approach to graph computation. The Gremlin graph traversal language allows us to traverse a graph, traveling from vertex to vertex via the connecting edges. We can use the same approach for both OLTP and OLAP workloads.
Transactional workloads begin with a small number of vertices (found with the help of an index) and then traverse across a reasonably small number of edges and vertices to return a result or add a new graph element. We can describe these transactional workloads as graph local traversals. Our goal with these traversals is to minimize latency.
Analytical workloads require traversing a substantial portion of the vertices and edges in the graph to find our answer. Many classic analytical graph algorithms fit into this bucket. We can describe these as graph global traversals. Our goal with these traversals is to maximize throughput.
With our JanusGraph – ScyllaDB graph data system, we can blend both capabilities. Backed by the high-IO performance of ScyllaDB, we can achieve scalable, single-digit millisecond responses for transactional workloads. We can also leverage Spark to handle large-scale analytical workloads.
With the server running in the foreground in your previous terminal, open a new terminal window and connect to your VM like before:
ssh -i ~/Downloads/aws/guy-janus.pem ec2-user@ec2-3-21-28-125.us-east-2.compute.amazonaws.com
Using Docker, you’ll spin up a three-node ScyllaDB cluster:
Start by setting up a Docker container with one node, called Node_X
:
docker run --name Node_X -d scylladb/scylla:4.5.0 --smp 1 --overprovisioned 1 --memory 750M
Create two more nodes, Node_Y
and Node_Z
, and add them to the cluster of Node_X
. The command “$(docker inspect –format='{{ .NetworkSettings.IPAddress }}’ Node_X)”
translates to the IP address of Node_X
:
docker run --name Node_Y -d scylladb/scylla:4.5.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)" --smp 1 --overprovisioned 1 --memory 750M
docker run --name Node_Z -d scylladb/scylla:4.5.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)" --smp 1 --overprovisioned 1 --memory 750M
Wait a minute or so and check the node status:
docker exec -it Node_Z nodetool status
[ec2-user@ip-172-31-20-0 ~]$ docker run --name Node_X -d scylladb/scylla:4.5.0 --smp 1 --overprovisioned 1 | |
442def02c02b0adeb7e526b16c6df92ebcc09ce52d044358edc992782b22fa5e | |
[ec2-user@ip-172-31-20-0 ~]$ docker run --name Node_Y -d scylladb/scylla:4.5.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)" --smp 1 --overprovisioned 1 | |
a53fe80490ecababcfdc3cea5b65d55ab6e062a1d06cc41dfb9ad2cabc3426ee | |
[ec2-user@ip-172-31-20-0 ~]$ docker run --name Node_Z -d scylladb/scylla:4.5.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)" --smp 1 --overprovisioned 1 | |
5be8c5563ea339217aab15443ffaa5f0b25bc2f6d0a5a48b00f3917574de8d36 | |
[ec2-user@ip-172-31-20-0 ~]$ docker exec -it Node_Z nodetool status | |
Datacenter: datacenter1 | |
======================= | |
Status=Up/Down | |
|/ State=Normal/Leaving/Joining/Moving | |
-- Address Load Tokens Owns Host ID Rack | |
UJ 172.17.0.3 ? 256 ? 3cd5533a-0630-40cb-b0f0-d8ae8297f4b6 rack1 | |
UN 172.17.0.2 ? 256 ? 58ba573b-1070-4baa-aacb-d89dec34f318 rack1 | |
UJ 172.17.0.4 0 bytes 256 ? 3544f077-2491-4832-a0d7-5f3f39b2e8a9 rack1 | |
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless | |
[ec2-user@ip-172-31-20-0 ~]$ docker exec -it Node_Z nodetool status | |
Datacenter: datacenter1 | |
======================= | |
Status=Up/Down | |
|/ State=Normal/Leaving/Joining/Moving | |
-- Address Load Tokens Owns Host ID Rack | |
UN 172.17.0.3 ? 256 ? 3cd5533a-0630-40cb-b0f0-d8ae8297f4b6 rack1 | |
UN 172.17.0.2 104.35 KB 256 ? 58ba573b-1070-4baa-aacb-d89dec34f318 rack1 | |
UN 172.17.0.4 182.47 KB 256 ? 3544f077-2491-4832-a0d7-5f3f39b2e8a9 rack1 | |
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless | |
[ec2-user@ip-172-31-20-0 ~]$ |
Once all nodes are in up status (UN), continue to the next step.
Gremlin Console and Testing
Gremlin is a graph traversal query language developed by Apache TinkerPop. It works for OLTP and OLAP type traversals and supports multiple graph systems. JanusGraph, Neo4j, and Hadoop are some examples.
Run the Gremlin console, and link it to the already running server container:
docker run --rm --link janusgraph-default:janusgraph -e GREMLIN_REMOTE_HOSTS=janusgraph -it janusgraph/janusgraph:latest ./bin/gremlin.sh
Now in the gremlin console, connect to the server:
:remote connect tinkerpop.server conf/remote.yaml
Next, set ScyllaDB as the data storage and instantiate our JanusGraph object in the console with a JanusGraphFactory. Make sure to replace the IP in the command below with the IP of one of your ScyllaDB cluster nodes:
JanusGraph graph = JanusGraphFactory.build().set("storage.backend", "cql").set("storage.hostname", "172.17.0.4").open();
g = TinkerGraph.open().traversal()
Now we can perform some basic queries. Make sure the graph is empty to begin with:
g.V().count()
Add two vertices
g.addV('person').property('name', 'guy') g.addV('person').property('name', 'john')
And make sure they were added:
g.V().values('name')
[ec2-user@ip-172-31-24-121 janusgraph]$ docker run --rm --network janusgraph_web --link janusgraph-server:janusgraph -e GREMLIN_REMOTE_HOSTS=janusgraph-server -it janusgraph/janusgraph:latest ./bin/gremlin.sh | |
May 09, 2022 7:56:50 AM java.util.prefs.FileSystemPreferences$1 run | |
INFO: Created user preferences directory. | |
\,,,/ | |
(o o) | |
-----oOOo-(3)-oOOo----- | |
SLF4J: Class path contains multiple SLF4J bindings. | |
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] | |
plugin activated: tinkerpop.server | |
plugin activated: tinkerpop.tinkergraph | |
07:56:57 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
plugin activated: tinkerpop.hadoop | |
plugin activated: tinkerpop.spark | |
plugin activated: tinkerpop.utilities | |
plugin activated: janusgraph.imports | |
gremlin> :remote connect tinkerpop.server conf/remote.yaml | |
==>Configured janusgraph-server/172.19.0.4:8182 | |
gremlin> :remote console | |
==>All scripts will now be sent to Gremlin Server - [janusgraph-server/172.19.0.4:8182] - type ':remote console' to return to local mode | |
gremlin> g | |
==>graphtraversalsource[standardjanusgraph[cql:[scylla-node1]], standard] | |
gremlin> g.V().count() | |
==>0 | |
gremlin> g.addV('person').property('name', 'guy') | |
==>v[4304] | |
gremlin> g.addV('person').property('name', 'john') | |
==>v[4248] | |
gremlin> g.V().values('name') | |
==>guy | |
==>john | |
gremlin> |
Summary
In this lesson, you learned how to deploy JanusGraph with ScyllaDB as the underlying database. There are different ways to deploy JanusGraph and ScyllaDB. To simplify the lesson, it uses Docker. The main steps were: Spinning up a virtual machine on AWS, running the JanusGraph server, running a Gremlin Console to connect to the new server, spinning up a three-node ScyllaDB cluster, and setting it as the data storage for the JanusGraph server, and performing some basic graph operations.
Published at DZone with permission of Guy Shtub. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments