What is JAR Hell?

what is jar hell? (or is it classpath hell? or dependency hell?) and which aspects are still relevant when considering modern development tools like maven or osgi?

interestingly enough there seems to be no structured answer to these questions (i.e. even the second page listed no promising headlines). this post is supposed to fill that gap.

overview

we’ll start with a list of problems that make up jar hell, momentarily ignoring build tools and component systems. we will come back to them for the second part when we assess the current state of affairs.

jar hell

jar hell is an endearing term referring to the problems that arise from the characteristics of java’s class loading mechanism. some of them build on one another; others are independent.

unexpressed dependencies

a jar cannot express which other jars it depends on in a way that the jvm will understand. an external entity is required to identify and fulfill the dependencies. developers would have to do this manually by reading the documentation, finding the correct projects, downloading the jars and adding them to the project. optional dependencies, where a jar might only require another jar if the developer wants to use certain features, further complicate the process.

the runtime will not detect unfulfilled dependencies until it needs to access them. this will lead to a noclassdeffounderror crashing the running application.

transitive dependencies

for an application to work it might only need a handful of libraries. each of those in turn might need a handful of other libraries, and so on. as the problem of unexpressed dependencies is compounded it becomes exponentially more labor-intensive and error-prone.

shadowing

sometimes different jars on the classpath contain classes with the same fully-qualified name. this can happen for different reasons, e.g. when there are two different versions of the same library, when a fat jar contains dependencies that are also pulled in as standalone jars, or when a library is renamed and unknowingly added to the classpath twice.

since classes will be loaded from the first jar on the classpath to contain them, that variant will “shadow” all others and make them unavailable.

if the variants differ semantically, this can lead to anything from too-subtle-to-notice-misbehavior to havoc-wreaking-errors. even worse, the form in which this problem manifests itself can seem non-deterministic. it depends on the order in which the jars are searched. this may well differ across different environments, for example between a developer’s ide and the production machine where the code will eventually run.

version conflicts

this problem arises when two required libraries depend on different, non-compatible versions of a third library.

if both versions are present on the classpath, the behavior will be unpredictable. first, because of shadowing, classes that exist in both versions will only be loaded from one of them. worse, if a class that exists in one but not the other is accessed, that class will be loaded as well. code calling into the library might hence find a mix of both versions.

since non-compatible versions are required, the program will most likely not function correctly if one of them is missing. again, this can manifests itself as unexpected behavior or as noclassdeffounderrors.

complex class loading

by default all application classes are loaded by the same class loader but developers are free to add additional class loaders.

this is typically done by containers like component systems and web servers. ideally this implicit use is completely hidden from application developers but, as we know, all abstractions are leaky . in some circumstances developers might explicitly add class loaders to implement features, for example to allow their users to extend the application by loading new classes, or to be able to use conflicting versions of the same dependency.

regardless of how multiple class loaders enter the picture, they can quickly lead to a complex mechanism that shows unexpected and hard to understand behavior.

classpath hell and dependency hell

classpath hell and jar hell are essentially the same thing, although the latter seems to focus a little more on the problems arising from complex class loader hierarchies. both terms are specific to java and the jvm.

dependency hell , on the other hand, is a more widely used term. it describes general problems with software packages and their dependencies and applies to operating systems as well as to individual development ecosystems. given its universality it does not cover problems specific to single systems.

from the list above it includes transitive and maybe unexpressed dependencies as well as version conflicts. class loading and shadowing are java specific mechanics, which would not be covered by dependency hell.

published by the wellcome library under cc-by 4.0

state of affairs

build tools

looking over the list of problems we see how build tools help with some of them. they excel in making dependencies explicit so that they can hunt down each required jar along the myriad edges of the transitive dependency tree. this largely solves the problems of unexpressed and transitive dependencies.

but maven et al. do nothing much about shadowing. while they generally work towards reducing duplicate classes, they can not prevent them . build tools do also not help with version conflicts except to point them out. and since class loading is a runtime construct they do not touch on it either.

component systems

i’ve never used a component system like osgi or wildfly so i can not testify to how well they work. from what they claim they seem to be able to solve most of the problems of jar hell.

this comes with additional complexity, though, and often requires the developer to take a deeper dive into class loader mechanics. ironically, also a point on the list above.

but regardless of whether or not component systems indeed considerably ease the pain of jar hell, i am under the impression that a vast majority of projects does not employ them. under this assumption said vast majority still suffers from classpath-related problems.

where does this leave us?

because they are not widely used, component systems leave the big picture untouched. but the ubiquity of build tools considerably changed the severity of the different circles of jar hell.

no build tool supported project i partook in or heard of spent a mentionable amount of time dealing with problems from unexpressed or transitive dependencies. shadowing rears its ugly head every now and then and requires a varying amount of time to be solved – but it always eventually is.

version conflicts are the single most problematic aspect of jar hell.

but every project sooner or later fought with dependencies on conflicting versions and had to make some hard decisions to work these problems out. usually some desired update had to be postponed because it would force other updates that could currently not be performed.

i’d venture to say that for most applications, services, and libraries of decent size, version conflicts are one of the main deciding factors for when and how dependencies are updated. i find this intolerable.

i have too little experience with non-trivial class loader hierarchies to asses how much of a recurring problem they are. but given the fact that none of the projects i have worked on so far required them, i’d venture to say that they are not commonplace. searching the net for reasons to use them often turns up what we already discussed: dependencies resulting in conflicting versions.

so based on my experience i’d say that conflicting versions are the single most problematic aspect of jar hell.

reflection

we have discussed the constituents of jar hell:

• unexpressed dependencies
• transitive dependencies
• shadowing
• version conflicts
• complex class loading

based on what build tools and component systems bring to the game and how widely they are used we concluded that unexpressed and transitive dependencies are largely solved, shadowing at least eased and complex class loading not commonplace.

this leaves version conflicts as the most problematic aspect of jar hell, influencing everyday update decisions in most projects.