Distributing a Java Command-Line Application

As a professional Java developer, I am comfortable with writing code in Java. Also, when I decided to write a command-line application, I asked myself if Java would be a good option in 2021 or if I should look more into alternative stacks like Rust.

So let’s take a closer look at the challenges one has to face to implement a command-line application in Java.

Creating Command-Line Application Jar

To write an application that can be used from the command-line, a simple public static void main(String[] args) is all you need in Java. However, do we really want to stop there?

What about:

• mandatory/optional arguments/options parsing
• default values
• validations
• sub-commands support
• automatic usage documentation
• automatic help
• auto-completion
• man page generation

For all of this, no need to look for long, Picocli has it all, and more. You can find a sample application in Picocli’s documentation.

Some other serious alternatives are JCommander and JLine. JLine is however a bit different as it is made to handle command-line inputs in a similar way to editline/readline. It can however be integrated with Picocli if need be.

From Jar to Application

So now that we have an application that can be run as a jar, what do we need to do to make it a Command-Line Application?

First, a Command-Line Application only deserves such a title if it is executable. A jar is by itself, not executable, it needs to be executed by the Java Virtual Machine by calling the java -jar command and giving the proper options required by our application. As we do not want to have to specify all those arguments when calling our application, we need a launcher script that will do this for us.

If you’re using Gradle, the application plugin comes in handy. It includes tasks to create a distributable version of the software which contains all runtime dependencies of your application (except the JVM) as well as launcher scripts for Unix (Linux, macOS, etc.) and Windows.

On the maven side, the assembly and shade plugins will help you to prepare the application jar and its dependencies, but you will still have to write the launcher script by yourself.

Official Versus Custom Package Repositories

At this point, we have an application as a jar, the runtime dependencies as jars, and a launcher script. It should not be so hard to create a package out of it.

However, before creating a package, it is important to take a look at the policies of the target package managers and their official repositories as they define particular constraints on how packages should be created.

For example, the Debian Java policy states the following:

Since it is common for Java source tarball to ship jar files of third-party libraries, creating a repack script to remove them from the upstream tarball is mandatory.

This implies that integrating your application in the official repositories will not be possible. Such integration would require the libraries you depend on to be available in official repositories with the proper versions, and that your build system relies on those. This is most of the time not possible as very few jar libraries are making it to official repositories or are only available in really old versions.

If you do not want to be subject to those constraints, you will then have to have to find a way to host your own .deb or .rpm repository. Some companies like packagecloud are providing such services.

For Homebrew, the homebrew tap (alternative repositories) relies on git and a simple GitHub repository is enough so this is not a major issue.

They are plenty of package managers out there. In this article, we will only focus on deb/rpm packages for Linux and homebrew for macOS as they are the most common ones.

Runtime Dependency on the JVM

As the launcher relies on the java command to launch the application, the Java Virtual Machine must be installed and be present on the PATH. If you’re part of the lucky ones having a smart launcher like the ones generated by the Gradle application plugin, it will also look for you in the JAVA_HOME environment variable.

If the target public is Java developers, chances are they already have a JVM installed which is not the one from their system’s package manager. I personally use SDKMAN! for managing my JVMs and I don’t have Java installed via apt on Ubuntu or Homebrew on macOS.

This implies that requiring a dependency in the packaging tool on Java will for those users:

• Install a potentially unnecessary but heavy package.
• Have no or little effect as the JAVA_HOME and the PATH environment variable will be set to something different than the one from the JVM installed by the default package manager.

In particular, Java developers do not always have the choice of the version of Java to use. Therefore it can happen their default JVM is older than you think. It might be imposed by the project. In particular, even though the public support of Java 8 stopped in January 2019, commercial support by different organizations is still very much alive.

According to The State of Developer Ecosystem in 2021 by JetBrains, Java 8 is in July 2021, still the most used version. This was already more than two years after the Java 11 release which is also an LTS release and only a few months before the Java 17 release.

As a consequence, the maximum version of Java the application should rely on is Java 8, as otherwise, the out-of-box installation might not work for users having the main JVM installed in their $PATH being older than Java 17 or even 11.

How Not to Rely on java in the User’s PATH?

If like me you would like to use a more recent JDK for developing your application, a more sophisticated solution is required.

To be able to rely on a specific JRE version for running your applications, a possible solution is to force the path to the java binary or JAVA_HOME to a particular one. This can be achieved via the launcher script, but it requires customization of this script depending on the system.

At this stage, two approaches are possible. Specify the JVM dependency to the exact, or the minimum version required. Let’s take a look at both approaches after a quick clarification on rolling vs non-rolling package managers.

Rolling Repositories vs Non-Rolling Ones

In order to provide users with applications, package managers rely on software repositories. Those repositories can be managed in two ways.

Static repositories as for Linux distributions like Debian/Ubuntu/Fedora/RHEL. In this mode, you will have independent software repositories for version X and version X+1 of the same system (i.e. Linux distribution). This allows for better stability and reproducibility as all the dependencies are pretty much fixed. For example, it is still possible to download Ubuntu 14:04 which is seven years old, and install packages in a fully functional way with the versions that existed at that time.

Rolling repositories as for Homebrew and some other Linux distributions like Arch Linux for example. In this mode, the new software will continuously be updated/added to the same repository. This makes it easier for integrating new software as there is only one place to publish the new releases and for users to use as no need to update the repositories when upgrading the OS. However, it is impossible or very hard to go back in time like for static repositories.

If you release your software via your own repositories, you can choose the rolling mode even if it is not the case with the official repositories of the target system. However, it is very likely the JRE you will depend on will come from the official repositories. This will have some impact as we’ll see later on.

Depending on a Fixed JRE Version

Depending on a fixed JRE version is the easiest thing to do. Expressing the dependency is easy in every package manager. The path to the java executable is statically known. And as the JVM version is fixed, there is no risk for the app to stop working if a later JVM removes a feature or add restrictions like the JDK 17 did with JEP 403: Strongly Encapsulate JDK Internals and JEP 407: Remove RMI Activation.

However, this comes at a cost.

Because of the fast release cadence of the JDK, a new JDK will be available every 6 months. If multiple Java-based applications are using this model, it is very likely they won’t all upgrade immediately their dependencies to the latest JRE. This will cause a proliferation of installed JREs which are quite space-consuming.

On top of that, performance improvements and security fixes of newer major versions won’t be available unless your application dependency on the JRE is updated to target the newer JREs.

Last but not least, some package managers might not keep non-LTS versions of the JDK for a long time after the next one is released. For example, in Fedora 35 repositories, the JDK 11 and 17 are available, but the JDK 12 to 16 aren’t anymore. In Ubuntu 21.10 repositories, JDK 16 is still available but JDK 12 to 15 aren’t anymore.

For all those reasons, depending on a fixed JRE version is not recommended unless you also provide software in a static repository that matches the publication rhythm of the target system.

Depending on a Rolling JRE Version

Depending on a rolling JRE version is way harder. Two problems need to be solved:

1. How to express the dependency to a rolling JRE version instead of a fixed version.
2. How to know the path to the java executable of this rolling JRE version.

Unfortunately for us, those two problems are highly system-dependent and therefore require tuning per package manager/system.

Expressing a dependency on a rolling JRE

The dependency part can be addressed in some package managers by depending on the latest JVM package or a virtual package.

The latest package can be found in Fedora (java-latest-openjdk-headless) or using Homebrew (openjdk).

On Debian/Ubuntu, no such latest JRE package exists and therefore a dependency on a javaXX-runtime-headless virtual package will be needed where XX is the minimal java version required for the application to run. Each JRE package provides multiple javaXX-runtime-headless features. For example, the package openjdk-16-jre-headless provides java16-runtime-headless, java15-runtime-headless, … And the package openjdk-17-jre-headless also provides all of them plus the java17-runtime-headless.

However, when used as a dependency, the Debian/Ubuntu package manager (apt) will take the latest version of OpenJDK providing it. Even if that latest version is an Early Access (EA) version.

For example:

$ apt install software-depending-on-java17-runtime-headless
...
The following NEW packages will be installed:
  ca-certificates-java java-common openjdk-18-jre-headless

To work around this, the dependency for the Debian package has to be expressed on the exact desired JRE (i.e. openjdk-17-jre-headless) with an alternative package being the virtual package (i.e. java17-runtime-headless).

This way, if no installed package provides java17-runtime-headless, the openjdk-17-jre-headless will be installed. Note however that if the user installs later a higher JRE, openjdk-18-jre-headless for example, the openjdk-17-jre-headless will not be uninstalled automatically.

Finding the path to a rolling JRE java executable

Here again, depending on the target system, the task will be an easy one or a hard one.

For Homebrew, both for macOS and Linux, the situation is quite easy as the /usr/local/opt/openjdk path points to the latest installed OpenJDK.

$ ls -l /usr/local/opt/openjdk*
lrwxr-xr-x  1 user  admin  24 Nov  5 19:59 /usr/local/opt/openjdk -> ../Cellar/openjdk/17.0.1
lrwxr-xr-x  1 user  admin  28 Nov 15 16:13 /usr/local/opt/openjdk@11 -> ../Cellar/openjdk@11/11.0.12
lrwxr-xr-x  1 user  admin  24 Nov  5 19:59 /usr/local/opt/openjdk@17 -> ../Cellar/openjdk/17.0.1

So JAVA_HOME can be set to /usr/local/opt/openjdk in the application start script.

On Linux systems, the situation is a bit more complicated. The JREs are generally installed by system package managers under the /usr/lib/jvm directory. This directory contains one folder per installed JVM and some possible symlinks aliases.

For example on Ubuntu, the following structure can be found:

$ ls -1 /usr/lib/jvm/
java-1.11.0-openjdk-amd64 -> java-11-openjdk-amd64/
java-1.16.0-openjdk-amd64 -> java-16-openjdk-amd64/
java-1.17.0-openjdk-amd64 -> java-17-openjdk-amd64/
java-1.8.0-openjdk-amd64 -> java-8-openjdk-amd64/
java-11-openjdk-amd64/
java-16-openjdk-amd64/
java-17-openjdk-amd64/
java-8-openjdk-amd64/

And on Fedora:

$ ls -l /usr/lib/jvm
java-1.8.0-openjdk-1.8.0.312.b07-1.fc35.x86_64
java-11-openjdk-11.0.13.0.8-2.fc35.x86_64
java-17-openjdk-17.0.1.0.12-1.rolling.fc35.x86_64
...

In this case, the JAVA_HOME can be set in the application start script to the latest JRE (17) by using the command:

JAVA_HOME=/usr/lib/jvm/$(ls /usr/lib/jvm | grep -E "java-[0-9]+" | sort -V -r | head -n 1)

This command lists all the folders in /usr/lib/jvm.

• The grep -E "java-[0-9]+" filters to only keep the ones starting by java-[0-9]+. This includes java-17-openjdk-amd64 as well as more complicated folder names like java-1.8.0-openjdk-1.8.0.312.b07-1.fc35.x86_64.
• The sort -V -r sorts them in reverse order (-r) using a version-based sorting algorithm (-V). First java-17-..., then java-11-..., then  java-1.8.0-....
• Finally the head -n 1 only keeps the first result and the result is set in the JAVA_HOME variable.

This command works on Ubuntu, Fedora, and Alpine docker images without any other packages installed then a headless JRE.

Bumping the Version of the JRE Dependency

Whether you express the dependency on the JVM via a specific version or only the minimum version required, you’ll probably want to upgrade it one day. This has to be done in a very careful way. Failing to do so will result in your application, even an old version of it, not being installable on old systems.

For example, the JDK 17 is available in the following package managers:

• Alpine: 3.15+
• Fedora: 33+
• Ubuntu: 18.04 and 20.04+

If at one point, you want to upgrade the JRE version to the JDK 18 or later, you can either:

• Upgrade the dependency version (rolling mode) and let down users on systems not having that package available yet. (if ever for static software repositories).
• Create a new repository for the upgraded version (static mode) and inform/request your users to update their repositories configuration.

Summing Up

To conclude I would highlight the following points:

A dependency on a fixed JRE version should be avoided. Instead, one should try to depend on default, latest versions or specify the minimum version.

The java command on the path doesn’t always point to the latest available JRE. This is especially true for Java developers in a corporate environment. Therefore, to improve the out-of-the-box installation for such users, the full path to the JRE should be specified in the start script.

Upgrading the dependency on the JVM is hard. If the package manager’s official repositories are in rolling mode, the upgrade can be done without too much trouble. If it is a static one, a decision has to be made between not supporting old systems or communicating a repository configuration update to your users.

Going Further and Getting Rid of the JVM Runtime Dependency

As we saw, having a runtime dependency on the JVM brings the following problems:

• The size of the JVM package is quite significant
• Which JVM package to depend on?
  • default, latest, >= 17, = 17, = 11, …
• Which javaexecutable to select?
  • Use the one found in the PATH?
  • Reference a tagged one: default, latest
  • Reference a specific one: 16, 11, …
  • Manually lookup for the ones installed and select one at runtime
• Jars dependencies restrictions for integration in official package managers
• Upgrading the version of the JRE dependency

We saw a few techniques addressing those issues but they are still highly system-dependent and require build/packaging/CI integration effort.

Could there be a way to avoid all or some of those problems? There are currently two ways to do so.

The first one consists in shipping the JRE runtime with our application. We can achieve this through the jpackage tool introduced in JDK 14 via JEP 392: Packaging Tool. This tool can generate a native package deb/rpm for Linux, pkg/dmg for macOS, or msi/exe for Windows. The included JRE will only package the required components and therefore be much slimmer than the JRE available through the OS package managers. The problem is that this slim JRE will not be shared with any other potential Java program and will therefore be duplicated for every single Java application packaged this way. This also does not remove the constraints on jar dependencies for official repositories integration.

The other solution is the generation of an OS/architecture native binary using GraalVM native-image technology, but this will be the topic of the next article.