Skip to content

jakarta taglibs standard update broke Manager (bsc#1044804)

Jump to bottom
Silvio Moioli edited this page Jun 3, 2022 · 1 revision

Intro

Bug reference: https://bugzilla.suse.com/show_bug.cgi?id=1044804

Description:

On 06/15/2017 09:16 AM, wrote:

not sure if you have seen it, but since yesterday spacewalk-java failed to build in all projects. This includes 3.1 where we did not submit new code.

This means that again a release of a maintenance update caused trouble:-(

But this time I really hope it is only a build time problem.

osc -A https://api.suse.de rbl Devel:Galaxy:Manager:3.1 spacewalk-java SLE_12_SP2 x86_64

With this command you can see the java exception. AFAIK there was a release of the jakarta-taglib library.

Indeed https://www.suse.com/support/update/ shows:

But it was already a problem to prove that this was really the culprit, as symptom did not point directly to this package:

[  138s] jspcompile:
[  138s]     [mkdir] Created dir: /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build/jsp-temp
[  138s]     [mkdir] Created dir: /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build/apidoc-temp
[  139s]   [jasper2] Jun 16, 2017 7:02:23 AM org.apache.jasper.servlet.TldScanner scanJars
[  139s]   [jasper2] INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
[  139s] 
[  139s] BUILD FAILED
[  139s] /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build.xml:857: The following error occurred while executing this line:
[  140s] /home/abuild/rpmbuild/BUILD/spacewalk-java-git-0.e665778/build.xml:645: org.apache.jasper.JasperException: <h3>Validation error messages from TagLibraryValidator for c in /WEB-INF/decorators/layout_c.jsp</h3><p>null: org.xml.sax.SAXNotRecognizedException: Feature 'http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.</p>

Background info

  • SAX and DOM APIs in Java are standardized under the name JAXP (JSR 5), and implementations are pluggable (Apache Xerces being the most common implementation)
  • most JDKs vendor one or more JAXP implementations - IBM Java 8 vendors has three, all based on Xerces
    • I did a brutal search on all jar files in my system:
for jar in `find . -name "*.jar"`; do echo $jar; unzip -l $jar | grep SAXParser; echo; echo ; done > /tmp/xerces_implementation_list.txt

Steps to the diagnosis

  • reproduce the issue
    • osc build of spacewalk-java was sufficient to reproduce the problem (even in Ubuntu), but was cumbersome
  • prove that the issue was due to the update, get an easier reproduction path
    • deploying via manager-build.xml and visiting any JSP page initially worked, because taglibs-core.jar and taglibs-standard.jar came from Ivy, hence our jar cache did not have the patch
    • deploying via manager-build.xml while using the patched version of the jars (extracted from the updated RPM and placed in buildconf/testjars), then visiting a JSP page DID trigger the same problem, as JSPs are not precompiled by that script. A similar stack trace could be found in rhn_web_ui.log
    • note that Tomcat only attempts to compile JSPs once. A tomcat restart was needed to reproduce
  • understand why the update patch lead to that failure
    • looking at the patch source, one can see that the "secure processing" feature is set, and the stack trace is about the JAXP implementation not supporting it:

https://build.suse.de/package/view_file/SUSE:SLE-12:Update/jakarta-taglibs-standard/CVE-2015-0254.patch?expand=1 lines 2090, 1757

  • understanding which JAXP implementation is in use
    • approach 1: look at what the JAXP implementation is doing. We need to add -Djaxp.debug=1 to the java commandline. Inside of our specfile, for example, one could use: export IBM_JAVA_OPTIONS='-Djaxp.debug=1' ant -Dprefix=$RPM_BUILD_ROOT install-tomcat8-suse
    • approach 2: look at what the Class Loader is loading. We need to add -verbose:class to the java commandline
    • with both approaches, it seemed that the JAXP implementation was coming from the JDK
  • try to use a different implementation
    • JAXP implementations can be swapped out in a number of ways (system properties, META-INF...) but none seemed to work - the JAXP implementation that ended up being used was always the same
export IBM_JAVA_OPTIONS='-Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl
  • figuring out why swapping the implementation didn't work
    • exclude that our version of Xerces did not support the feature. Checked xerces source archive directly and the feature that we missed actually existed since version 2.7.0 (we shipped in SLES 2.8.1)
    • try to attach a debugger to analyse behaviour step-by-step
      • use the manager-build.xml approach above to reproduce in Tomcat
      • attach a debugger and step: Tomcat sources were missing
        • download Tomcat source RPMs, unpack them, attach sources to Eclipse
      • jakarta-taglibs-standard sources are missing
        • download the source RPM, unpack it, apply source patches manually (in the order and with the flags given in the specfile), attach sources to Eclipse
      • jakarta-taglibs-standard jar did not have debug symbols, hence stepping is impossible
        • branch the package, add one missing option to the Ant configuration in the specfile, rebuild:
 cat > build.properties <<EOBP
 build.dir=build
 dist.dir=dist
 servlet24.jar=$(build-classpath servletapi5)
 jsp20.jar=$(build-classpath jspapi)
 xalan.jar=$(build-classpath xalan-j2)
+compile.debug=true
 EOBP
 
 %build
 ant \
   -Dfinal.name=%{short_name} \
   -Dj2se.javadoc=%{_javadocdir}/java \
   -f standard/build.xml \
   dist
  • looking at the patch again it is now possible to see that it uses a hacked class loader, which is not sensitive to those options (see the runWithOurClassLoader method)

Reconstructing history

  • Feb 2015, a vulnerability is disclosed in Apache Standard Taglibs. Upstream advice: update to version 1.2.3 or higher

  • few days later, it was decided to postpone the fix, because new version requires Maven to build and backporting the patch is difficult

  • Oct 2015, Debian folks have done the heavy lifting, patch is taken in openSUSE

  • June 2017, patch lands in SLE breaking SUSE Manager

  • reason of the breakage is that the Debian patch is very intrusive - among other things it adds a hack to force the way the class loader works for the SAX parser. Due to this hack, the ability of configuring which SAX parser implementation to use is lost (at least when compiling JSP pages) - the implementation bundled in the JRE is always used. Problem is this implementation does not support a certain option ("secure process") which is crucial in the patch itself to fix the vulnerability, leading to an Exception being thrown

  • effect is that spacewalk-java does not build any more: JSP compilation is broken as it relies on the patched Apache Standard Taglibs. The patch on one hand requires a SAX parser implementation that supports "secure process", and on the other hand blocks the selection to the JDK-provided one - but that does not support it!

  • users should be safe to upgrade the package, as all JSP compilation should happen in OBS. SLE customers using Apache Standard Taglibs for their own webapps might see the same problem we have

Solution

We followed upstream advice and upgraded to 1.2.3. We created a new tetra package, changed our specfiles and ran the testsuites. No regressions were noted, so solution was accepted.

Clone this wiki locally