The use of Java logging technologies in Aleph2 is pretty standard and yet, because of the use of many external JARs, has made the system quite brittle to innocuous changes to its core libraries (eg I found out last week that Storm start/stop no longer worked).

The purpose of this page is to summarize the issues and how logging is set up in Aleph2 (currently: working!)

(Note that user and application logging should soon be handled via our logging platform so developers using Java logging for plugins will be not recommended.)

Java logging technologies are pretty bad. Here's my understand of the lay of the land:

• log4j2 - the latest apache logging infrastructure, nice (for now) in that a) it has a completely different package space to log4j, and b) most 3rd party libraries don't use it. All aleph2 logging currently occurs via log4j2
• log4j - the legacy apache logging infrastructure (v1.2). Most issues with this come from the use of:
• slf4j - this is a "facade", ie the idea is that applications use the slf4j API and then it can be pointed to a number of different underlying implementations ... the 2 relevant ones are
  • slf4j-log4j12 (also requires log4j): the most popular slf4j/log4j library, but there's also:
  • log4j-for-slf4j: this library provides an slf4j implementation, but importantly also acts as a binary replacement for log4j (eg they then recommend pointing maven at their "empty log4j version 99" repository to satisfy the maven dependency with an empty jar)

Here are the incompatibilities I'm aware of:

• log4j and log4j-for-slf4j: obvious because log4j-for-slf4j is a binary replacement for log4j
• slf4j-log4j12 and log4j-for-slf4j: this one is bizarre (since slf4j-log4j12 requires a log4j instance why not let it be log4j-for-slf4j?!), but it generates a specific (fatal) exception
• you see lots of slf4j warnings kicking around in other cases (typically something like "multiple bindings present"), but none of them seem fatal

OK here's a quick summary of our dependencies on slf4j and log4j (note: all our code requires log4j2)

• aleph2_data_model:
  • log4j2
  • log4j (1.2.16) - This isn't used by any of our code, only transitive dependencies, but I put this here to centralize it, so we can switch it to the empty version/change versions etc in one place
• aleph2_core_distributed_services:
  • log4j via KafkaClient
• aleph2_storm_dependencies
  • log4j-for-sl4j via storm-core (EXCLUDED in maven)
• aleph2_analytic_services_storm
  • (EXCLUDED) log4j-for-sl4j via aleph2_storm_dependencies (via storm-core)
• aleph2_hadoop_dependencies
  • sl4j-for-log4j12 via hadoop*
• aleph2_analytic_services_hadoop
  • sl4j-for-log4j12 via aleph2_hadoop_dependencies (via hadoop*)
• aleph2_storage_service_hdfs
  • sl4j-for-log4j12 via aleph2_hadoop_dependencies (via hadoop*)
• aleph2_security_service_v1
  • sl4j-for-log4j12 via shiro

From the above list it can be seen that Storm's use of log4j-for-sl4j is the "only" problem because it breaks vs all the sl4j libs). I've fixed (?) that by simply excluding it from the main build, so we're running with log4j2/log4j/slf4j-log4j12 as our (apparently compatible!) trio.

Of course there are a few different classpath combos in which aleph2 can run:

• Aleph2 services: aleph2_data_model then aleph2* (all in /opt/aleph2-home/lib)
• Inside Hadoop: selected aleph2 libs (everything but *storm) then hadoop libs
• Inside Storm: storm libs then selected aleph2 libs (including hdfs but excluding *hadoop - storm lib set includes enough Hadoop libs for HDFS to run
• "Misc other": uploaded JARs (via JCL) then all aleph2 libs

The "inside storm" mode would be a problem but is addressed by Caleb's storm jar builder excluding all log4j and slf4j classes from the input JARs.

(Clearly it's also possible for uploaded JARs to break (eg if they package log4j-for-slf4j), but the breakage will only be local to the JCL classloder used to load them.)