Understanding Dependencies...Visually!

Show of hands, how many of us truly understand how your build automation tool builds its dependency tree? Now, lower your hand if you understand because you work on building automation tools. Thought so!

One frustrating responsibility of software engineers is understanding your project's dependencies: what transitive dependencies were brought in and by whom; why v1.3.1 is used when v1.2.10 was declared; what resulted when the transitive dependencies changed; how did multiple versions of the same artifact occur?

Every software engineer has piped a dependency tree into a text file, searched for specific artifacts, and then worked their way back up to identify its origin. For anything other than trivial projects, creating a mental map of the dependencies is extremely difficult, if not impossible.

I faced this problem when starting a new job with a mature code base, presenting a challenge to assemble the puzzle pieces.  I've previously worked with graph databases and thought a graphical view of the dependency artifacts could be created using Neo4J, which resulted in DependencyLoader.

Note: this is not a tutorial on graph databases, nor does this tutorial require a background in graph databases. If interested, Neo4J has tutorials and white papers to help you get started.

Set Up Environment

Install Java

Java 11 or later is required. If not already available, install your favorite OpenJDK flavor.

Install Neo4J

The tutorial requires a Neo4J database into which the dependency information is loaded, preferably unshared, as the loader purges the database before each run. You have been warned!

Neo4J provides personal sandboxes, ideal for short-term projects like this tutorial.

Alternatively, install Neo4J locally on your desktop or laptop. Homebrew simplifies MacOS installations: 

brew install neo4j && brew services start neo4j

Before continuing, confirm access to your Neo4J database using the browser, using either the link and credentials for the Neo4J sandbox or locally at http://localhost:7474. The default credentials for a local install is neo4j/neo4j; upon successful login, you are forced to change the password.

Clone Repositories

The neo4j-gradle-dependencies repository contains the for loading the dependencies into Neo4J.  This tutorial will generate a dependency graph for spring-boot. You must clone these two repositories.

Scott.Sosna@mymachine src% git clone git@github.com:scsosna99/neo4j-gradle-dependencies.git
Scott.Sosna@mymachine src% git clone git@github.com:spring-projects/spring-boot.git

Note: local Gradle is not required as both repositories use the Gradle Wrapper, which downloads all necessary components the first time the wrapper is used. 

Generate Dependencies

DependencyLoader takes the dependency tree generated by Gradle as input. Though multiple configurations may be loaded together — i.e., compileClasspath, runtimeClasspath, testCompileClasspath, testRuntimeClasspath — starting with a single configuration is simpler to navigate, especially for a tutorial.

To generate dependencies for all configurations: 

• gradle dependencies
• ./gradlew dependencies

To generate dependencies for a single configuration

• gradle dependencies --configuration <configuration>
• ./gradlew dependencies --configuration <configuration>

Generate Spring Boot Dependencies

This tutorial creates a dependency graph in Neo4J using the compileClasspath dependencies of Spring Boot. From the directory where the repositories were cloned, execute the following commands:

Scott.Sosna@mymachine src% cd spring-boot/spring-boot-project/spring-boot
Scott.Sosna@mymachine spring-boot% ./gradlew dependencies --configuration compileClasspath > dependencies.out

The file dependencies.out contains the compile-time classpath dependencies for Spring Boot.

Load Dependencies

First, confirm the connection URL and authentication credentials in DependencyLoader.java and modify them if necessary.

Execute the following commands to load the Spring Boot dependencies into Neo4j:

Scott.Sosna@mymachine spring-boot% cd ../../../neo4j-gradle-dependencies
Scott.Sosna@mymachine neo4j-gradle-dependencies% ./gradlew clean run  --args="../spring-boot/spring-boot-project/spring-boot/dependencies.out"

When successful, the output lines from gradle are:

Scott.Sosna@PVHY32M6KG neo4j-gradle-dependencies % ./gradlew clean run  --args="../spring-boot/spring-boot-project/spring-boot/dependencies.out"

> Task :compileJava
Note: /Users/Scott.Sosna/data/src/github/neo4j-gradle-dependencies/src/main/java/dev/scottsosna/neo4j/gradle/relationship/DependsOn.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :run
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Jun 02, 2023 6:19:22 AM org.neo4j.driver.internal.logging.JULogger info
INFO: Direct driver instance 1606286799 created for server address localhost:7687
dependencies.out completed.
Jun 02, 2023 6:19:23 AM org.neo4j.driver.internal.logging.JULogger info
INFO: Closing driver instance 1606286799
Jun 02, 2023 6:19:23 AM org.neo4j.driver.internal.logging.JULogger info
INFO: Closing connection pool towards localhost:7687

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3s

View Dependencies

Multiple tools are available for displaying Neo4J graphs, but the built-in browser tool is adequate for this tutorial.

Show the Complete Tree

The query MATCH(a) RETURN a is the relational-equivalent of SELECT * FROM <table>

View Details of an Artifact

Each artifact found creates a node whose properties identify the artifact (groupId/artifactId) and its type, shown on the right-side pane.

View Details of a Dependency

Each dependency is created as a relationship whose properties identify the specifics of the dependency: configuration, specified/version, and configuration. The dependency selected below shows spring-security:spring-web depends on io.micormeter:micrometer-observation, but the spring-web specified version 1.10.7 was resolved as version 1.11.0.

Traverse Dependencies

Neo4J allows you to explore the graph node-by-node, allowing you to manually expand the graph node by node, providing a way to explore specific areas of the dependency tree.

Assume that you want to understand the dependencies for the artifact io.projectreactor.netty:reactor-netty-http.  First, we'll query Neo4J for that specific node.

MATCH(a:Artifact {groupId: 'io.projectreactor.netty', artifactId: 'reactor-netty-http'}) RETURN a

Double-clicking on the node shows its neighboring nodes — the artifact(s) depended on it and the artifact(s) it depends on.

This expanded graph shows one artifact that is dependent on it — the root of the project with an artifact type PROJECT and six other dependencies on which it's dependent.

Next, double-click on io.netty:netty-code https://github.com/netty/netty/tree/4.1/codec-httpc-http to show the next level of dependencies.  Note that besides the relationships (dependencies) of the selected node, additional relationships for nodes already on the graph may be shown.

Identify Version Mismatch

Gradle's dependency output indicates where the specified version was not the version resolved by Gradle.  The properties on the dependency (relationship) can be used in a Neo4J query, restricting the relationships shown and the attached artifacts (nodes).

MATCH (a:Artifact)-[d:DEPENDS_ON]->(b:Artifact) WHERE d.specifiedVersion<>d.resolvedVersion RETURN a,b,d

Neo4J can return results in a tabular format for easier review, if necessary.

MATCH (a:Artifact)-[d:DEPENDS_ON]->(b:Artifact) WHERE d.specifiedVersion<>d.resolvedVersion RETURNa.name AS source, b.name AS dependency, d.specifiedVersion AS specified, d.resolvedVersion AS resolved

Additional Information

mappings.out

The mappings.out file allows you to customize the artifact type assigned to a node based on artifacts groupId, most commonly to specifically identify artifacts created by your organization.

Input Directory

The command line argument for DependencyLoader may be a directory containing multiple Gradle dependency trees loaded into the same Neo4J database.  This helps in understanding the dependencies of related projects with separate build.gradle files.

Constrained and Omitted

Gradle identifies certain dependencies as Constrained and Omitted. Currently, those are not loaded but would be easy to include, likely by creating additional properties for the relationships.