Hijacking logs on Tomcat 9 - fbacchella/loghublog4j2 GitHub Wiki
Logs on tomcat can be difficult to handle because they come from many sources: the tomcat container, different applications or libraries using different loggers.
The application contener itself uses the very cumbersome conf/logging.properties
. So it uses the JUL logging, that is not used very often outside of the J2EE world. One could prefer the more powerfull log4j2.
The first step is to removed the file $CATALINA_BASE/conf/logging.properties
file, that will not be used and should be avoided.
To activate the use of log4j2 in, both log4j2.xml and some log4j2 jar must be provided to tomcat at boot start. In a folder $LOGJ2_PATH
, the following content sould be provided:
- log4j-api-2.x.x.jar
- log4j-core-2.x.x.jar
- log4j-appserver-2.x.x.jar.
The log4j2 configuration should be defined in the file $LOGJ2_PATH/log4j2-tomcat.xml
. The path to it should declared in $CATALINA_BASE/conf/setenv.sh
by adding the line :
CLASSPATH=${CLASSPATH:+$CLASSPATH:}$LOGJ2_PATH/*:$LOGJ2_PATH
Some applications use SLF4J. For version 1.7.36 and earlier, it uses unreliable automagic tricks to detect what logging implementation to use. The newer SLF4J 2+ use a much more moderne solution, service loader: https://www.slf4j.org/faq.html#changesInVersion200. So the facade api for SL4J should be updated. To do that, one should remove $CATALINA_BASE/lib/slf4j-api-1.7.36.jar
and CATALINA_BASE/lib/slf4j-jdk14-1.7.36.jar
, and provide slf4j-api-2.0.x.jar
and log4j-slf4j2-impl-2.x.y.jar
in $LOGJ2_PATH
.
If the application or an embeded library use the very old common-loggins, add log4j-jcl-2.x.x.jar
in $LOGJ2_PATH
.
If the application or an embeded library use the deprecated log4j-1
, add log4j-1.2-api-2.x.x.jar
in $LOGJ2_PATH
.
It’s also possible to ensure that any web application is also using only log4j2. In the $CATALINA_BASE/conf/Catalina/localhost/$APP.xml
, add the following element:
<?xml version='1.0' encoding='utf-8'?>
<Context ... >
<!-- optional, it can also be auto loaded if it's in the classpath -->
<Parameter name="log4jConfiguration" value=".../log4j2.xml" />
...
<Resources cachingAllowed="false">
<PreResources base=".../lib/log4j2"
className="org.apache.catalina.webresources.DirResourceSet" readOnly="true"
webAppMount="/WEB-INF/lib" />
</Resources>
</Context>
It’s should always be used as soon as some of the log4j2 jar are missing, to avoid missing classloader confusions with different classloader.
The configuration of tomcat containers logging is in the file $LOGJ2_PATH/log4j2-tomcat.xml
. It should be tweeked for expected log details.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration xmlns:xi="http://www.w3.org/2001/XInclude" status="error" packages="fr.loghub.log4j2" verbose="false">
<Properties>
<Property name="log_file">logs/catalina.log</Property>
</Properties>
<xi:include href="log4j2-common.xml" />
<Loggers>
<Root level="error">
<AppenderRef ref="ZMQAppender" />
<AppenderRef ref="logfile" />
</Root>
<Logger name="org.apache.catalina" level="info" additivity="false">
<AppenderRef ref="ZMQAppender" />
<AppenderRef ref="logfile" />
</Logger>
<Logger name="org.apache.tomcat" level="info" additivity="false">
<AppenderRef ref="ZMQAppender" />
<AppenderRef ref="logfile" />
</Logger>
<Logger name="org.apache.jasper" level="info" additivity="false">
<AppenderRef ref="ZMQAppender" />
<AppenderRef ref="logfile" />
</Logger>
<Logger name="org.apache.coyote" level="info" additivity="false">
<AppenderRef ref="ZMQAppender" />
<AppenderRef ref="logfile" />
</Logger>
<Logger name="org.apache.naming" level="info" additivity="false">
<AppenderRef ref="ZMQAppender" />
<AppenderRef ref="logfile" />
</Logger>
<Logger name="javax" level="error"/>
<Logger name="sun" level="error"/>
<!-- Optionnal, to log GC activity -->
<Logger name="gc" level="info" additivity="false">
<AppenderRef ref="ZMQAppender"/>
</Logger>
</Loggers>
</Configuration>
And log4j2-common.xml
should be something like
<?xml version="1.0" encoding="UTF-8"?>
<Appenders>
<RollingRandomAccessFile name="logfile"
fileName="${log_file}"
filePattern="${log_file}.%d{YYYY-ww}.gz">
<PatternLayout>
<Pattern>[%d] %5p %c : %m%n</Pattern>
</PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy />
</Policies>
<DefaultRolloverStrategy />
</RollingRandomAccessFile>
<ZMQ name="ZMQAppender">
<endpoint>tcp://loghub:1512</endpoint>
<type>pub</type>
<method>connect</method>
<hwm>5000</hwm>
<MsgPackLayout>
<AdditionalField name="host.architecture">...</AdditionalField>
<AdditionalField name="host.hostname">...</AdditionalField>
<AdditionalField name="host.name">...</AdditionalField>
<AdditionalField name="host.type">...</AdditionalField>
<AdditionalField name="host.os.family">...</AdditionalField>
<AdditionalField name="host.os.full">...</AdditionalField>
<AdditionalField name="host.os.kernel">...</AdditionalField>
<AdditionalField name="host.os.name">...</AdditionalField>
<AdditionalField name="host.os.type">...</AdditionalField>
<AdditionalField name="host.os.version">...</AdditionalField>
<AdditionalField name="observer.vendor">Apache</AdditionalField>
<AdditionalField name="observer.product">TomCat</AdditionalField>
<AdditionalField name="observer.type">Web</AdditionalField>
<AdditionalField name="service.environment">...</AdditionalField>
<AdditionalField name="service.type">...</AdditionalField>
</MsgPackLayout>
</ZMQ>
<GCAppender name="GCAppender">
<level>INFO</level>
</GCAppender>
</Appenders>
As a generic rule, all additional fields sould be defined using [https://www.elastic.co/guide/en/ecs/current/index.html](http://Elastic common schema).
The application logs should be defined in the context of the application, for example in conf/Catalina/localhost/ROOT.xml
:
<?xml version='1.0' encoding='utf-8'?>
<Context ....>
....
<Parameter name="log4jConfiguration" value="log4j2-ROOT.xml" />
...
<!-- Not caching, ressources are handled by the Apache front end -->
<Resources cachingAllowed="false" >
<!-- Does not found the root classpath, needs a custom one -->
<PreResources base=".../lib/log4j2"
className="org.apache.catalina.webresources.DirResourceSet" readOnly="true"
webAppMount="/WEB-INF/lib" />
</Resources>
</Context>
And in log4j2-ROOT.xml
<?xml version="1.0" encoding="UTF-8" ?>
<!-- Specify the refresh internal in seconds. -->
<Configuration xmlns:xi="http://www.w3.org/2001/XInclude" monitorInterval="5">
<Properties>
<Property name="log_file">.../logs/application.log</Property>
</Properties>
<xi:include href="log4j2-common.xml" />
<Loggers>
<Logger name="gc" level="info" additivity="false">
<AppenderRef ref="ZMQ"/>
</Logger>
<Root level="info">
<AppenderRef ref="logfile"/>
<AppenderRef ref="ZMQ"/>
</Root>
</Loggers>
</Configuration>
This setup is for server 3.0 and latter, as explained at https://logging.apache.org/log4j/2.x/manual/webapp.html, so it needs log4j-web-2.x.x.jar
to be provided in $LOGJ2_PATH
.
For enhanced security, it’s possible to encrypt ZMQ connexion using Curve.
The ZMQ appender can automatically handle creation of the key. The best way is to start once the application with the following system property set:
* `fr.loghub.logging.zmq.curve.privateKeyFile` indicating the ZMQ file path, the extension is optional.
* `fr.loghub.logging.zmq.curve.autoCreate` set to true to allow the creation of the file.
Once the application is started, 3 files are created.
The first one ends with .p8
if no extension is given. It contains the private part of the curve key, encoded as a p8
file. The default OID for the key type is 1.3.6.4.1.2
, but can be overridden by using the system property fr.loghub.nacl.oid
.
The second one ends with .pub
. It encodes the public part of the curve key in Base64, prefixed by the string Curve
.
The third ends with .zpl
. It encodes the public part of the curve key using the ZPL format using the hierarchical name curve.public-key
.
Once it’s done, the path of private key should be defined in the property privateKeyFile
and the base64-encoded public key defined in the property publicKey
.``