Hadoop 2.7.3 - anujajakhade/anuja GitHub Wiki
The instructions provided below specify the steps to build Apache Hadoop 2.7.3 on IBM z Systems for following distributions:
- RHEL (6.8, 7.1, 7.2, 7.3)
- SLES (11 SP4, 12, 12 SP1, 12 SP2)
- Ubuntu (16.04, 16.10)
General Notes:
- When following the steps below please use a standard permission user unless otherwise specified.
- A directory
/<source_root>/
will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
Apache Hadoop requires Google Protobuf 2.5.0, please see Building Google Protobuf 2.5.0 for more information.
##Step 1: Building and Installing Apache Hadoop
####1.1) Install dependencies
-
RHEL 6.8
-
With IBM SDK
Download IBM Java 8 sdk binary from IBM Java 8 and follow the instructions as per given in the link.
sudo yum install -y tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib zlib-devel bison flex binutils-devel
-
-
RHEL (7.1, 7.2, 7.3)
-
With IBM SDK
Download IBM Java 8 sdk binary from IBM Java 8 and follow the instructions as per given in the link.
sudo yum install -y tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib zlib-devel bison flex binutils-devel
-
With Open JDK
sudo yum install tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib-devel cmake java-1.8.0-openjdk-devel pkgconfig openssl-devel
-
-
SLES 11 SP4
-
With IBM SDK
Download IBM Java 8 sdk binary from IBM Java 8 and follow the instructions as per given in the link.
sudo zypper install -y tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib zlib-devel
-
-
SLES 12
-
With IBM SDK
Download IBM Java 8 sdk binary from IBM Java 8 and follow the instructions as per given in the link.
sudo zypper install -y tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib zlib-devel
-
-
SLES (12 SP1, 12 SP2)
-
With IBM SDK
``` sudo zypper install -y tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib zlib-devel java-1_8_0-ibm java-1_8_0-ibm-devel ```
-
With Open JDK
sudo zypper install tar wget autoconf libtool automake gcc-c++ make git bzip2 curl unzip zlib-devel cmake java-1_8_0-openjdk-devel pkgconfig libopenssl-devel pkg-config
-
-
Ubuntu (16.04, 16.10)
-
With IBM SDK
Download IBM Java 8 sdk binary from IBM Java 8 and follow the instructions as per given in the link.
```
sudo apt-get update sudo apt-get install -y tar wget autoconf libtool automake g++ make git bzip2 curl unzip zlib1g-dev maven cmake ```
-
With Open JDK
sudo apt-get update sudo apt-get install tar wget autoconf libtool automake g++ make git bzip2 curl unzip zlib1g-dev maven openjdk-8-jdk cmake openssl libssl-dev pkg-config
-
####1.2) Install Maven ( For RHEL and SLES only)
cd /<source_root>/
wget http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
tar -xvzf apache-maven-3.3.9-bin.tar.gz
export PATH=$PATH:/<source_root>/apache-maven-3.3.9/bin/
####1.3) Download source code
cd /<source_root>/
wget https://github.com/apache/hadoop/archive/rel/release-2.7.3.tar.gz
tar zxvf release-2.7.3.tar.gz
cd hadoop-rel-release-2.7.3/
####1.4) Modify the below file as per the diff contents
Edit line 360
of file hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
- System.getProperty("os.arch").contains("64");
+ System.getProperty("os.arch").contains("64") || System.getProperty("os.arch").contains("s390x");
####1.5) Set environment variables
export MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=1024m"
export JVM_ARGS="-Xms1024m -Xmx1024m"
export JAVA_HOME=<path_to_java>
export PATH=$JAVA_HOME/bin:$PATH
####1.6) Run Maven to create binary distribution with native code
mvn clean install -DskipTests -fn
Note: With IBM SDK, compilation errors might be seen for hadoop-yarn-registry project. Apply this patch to resolve the same.
####1.7) Build LevelDB JNI jar Note: Few test failure are seen as the downloaded LevelDB JNI jar is not compatible with s390x. Build LevelDB JNI jar to support s390x using below instructions.
-
Download and configure Snappy
cd /<source_root>/ wget https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz tar -zxvf snappy-1.1.3.tar.gz export SNAPPY_HOME=`pwd`/snappy-1.1.3 cd ${SNAPPY_HOME} ./configure --disable-shared --with-pic make
-
Download the source code for LevelDB and LevelDB JNI
cd /<source_root>/ git clone https://github.com/chirino/leveldb.git git clone https://github.com/fusesource/leveldbjni.git
-
Set the environment variables
export LEVELDB_HOME=`pwd`/leveldb export LEVELDBJNI_HOME=`pwd`/leveldbjni export LIBRARY_PATH=${SNAPPY_HOME} export C_INCLUDE_PATH=${LIBRARY_PATH} export CPLUS_INCLUDE_PATH=${LIBRARY_PATH}
-
Apply the LevelDB patch
cd ${LEVELDB_HOME} git apply ${LEVELDBJNI_HOME}/leveldb.patch
-
Modify the below file as per the diff contents
Edit file
${LEVELDB_HOME}/port/atomic_pointer.h
@@ -36,6 +36,8 @@
#define ARCH_CPU_X86_FAMILY 1
#elif defined(__ARMEL__)
#define ARCH_CPU_ARM_FAMILY 1
+#elif defined(__s390x__) || defined(__s390__)
+#define ARCH_CPU_S390_FAMILY 1
#elif defined(__ppc__) || defined(__powerpc__) || defined(__powerpc64__)
#define ARCH_CPU_PPC_FAMILY 1
#endif
@@ -50,6 +52,14 @@ namespace port {
// http://msdn.microsoft.com/en-us/library/ms684208(v=vs.85).aspx
#define LEVELDB_HAVE_MEMORY_BARRIER
+// S390
+#elif defined(ARCH_CPU_S390_FAMILY)
+
+inline void MemoryBarrier() {
+ asm volatile("bcr 15,0" : : : "memory");
+}
+#define LEVELDB_HAVE_MEMORY_BARRIER
+
// Gcc on x86
#elif defined(ARCH_CPU_X86_FAMILY) && defined(__GNUC__)
inline void MemoryBarrier() {
@@ -217,6 +227,7 @@ class AtomicPointer {
#undef ARCH_CPU_X86_FAMILY
#undef ARCH_CPU_ARM_FAMILY
#undef ARCH_CPU_PPC_FAMILY
+#undef ARCH_CPU_S390_FAMILY
} // namespace port
} // namespace leveldb
-
Configure LevelDB and LevelDB JNI 1.8
make libleveldb.a cd ${LEVELDBJNI_HOME} git checkout leveldbjni-1.8 mkdir leveldbjni-linux64-s390x cd leveldbjni-linux64-s390x
-
Modify the below file as per the diff contents
Edit file
${LEVELDBJNI_HOME}/leveldbjni-all/pom.xml
@@ -84,7 +84,13 @@
<version>1.8</version>
<scope>provided</scope>
</dependency>
-
+ <dependency>
+ <groupId>org.fusesource.leveldbjni</groupId>
+ <artifactId>leveldbjni-linux64-s390x</artifactId>
+ <version>1.8</version>
+ <scope>provided</scope>
+ </dependency>
+
</dependencies>
<build>
@@ -119,7 +125,8 @@
META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86,
META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86-64,
META-INF/native/linux32/libleveldbjni.so;osname=Linux;processor=x86,
- META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=x86-64
+ META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=x86-64,
+ META-INF/native/linux64/s390x/libleveldbjni.so;osname=Linux;processor=s390x
</Bundle-NativeCode>
</instructions>
</configuration>
- Create a new file
${LEVELDBJNI_HOME}/leveldbjni-linux64-s390x/pom.xml
with the below contents
@@ -0,0 +1,113 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Copyright (C) 2011, FuseSource Corp. All rights reserved.
+ http://fusesource.com
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions are
+ met:
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above
+ copyright notice, this list of conditions and the following disclaimer
+ in the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of FuseSource Corp. nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.fusesource.leveldbjni</groupId>
+ <artifactId>leveldbjni-project</artifactId>
+ <version>1.8</version>
+ </parent>
+
+ <groupId>org.fusesource.leveldbjni</groupId>
+ <artifactId>leveldbjni-linux64-s390x</artifactId>
+ <version>1.8</version>
+
+ <name>${project.artifactId}</name>
+ <description>The leveldbjni linux 64 native libraries on s390x machines</description>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.fusesource.leveldbjni</groupId>
+ <artifactId>leveldbjni</artifactId>
+ <version>1.8</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <testSourceDirectory>${basedir}/../leveldbjni/src/test/java</testSourceDirectory>
+
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jar-plugin</artifactId>
+ <version>2.3.1</version>
+ <configuration>
+ <classesDirectory>${basedir}/target/generated-sources/hawtjni/lib</classesDirectory>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.fusesource.hawtjni</groupId>
+ <artifactId>maven-hawtjni-plugin</artifactId>
+ <version>${hawtjni-version}</version>
+ <executions>
+ <execution>
+ <goals>
+ <goal>build</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <platform>linux64/s390x</platform>
+ <name>leveldbjni</name>
+ <classified>false</classified>
+ <nativeSrcDependency>
+ <groupId>org.fusesource.leveldbjni</groupId>
+ <artifactId>leveldbjni</artifactId>
+ <version>${project.version}</version>
+ <classifier>native-src</classifier>
+ <type>zip</type>
+ </nativeSrcDependency>
+ <configureArgs>
+ <arg>--with-leveldb=${env.LEVELDB_HOME}</arg>
+ <arg>--with-snappy=${env.SNAPPY_HOME}</arg>
+ <arg>CFLAGS=-m64</arg>
+ <arg>CXXFLAGS=-m64</arg>
+ </configureArgs>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <version>2.4.3</version>
+ <configuration>
+ <redirectTestOutputToFile>true</redirectTestOutputToFile>
+ <forkMode>once</forkMode>
+ <argLine>-d64 -ea</argLine>
+ <failIfNoTests>false</failIfNoTests>
+ <workingDirectory>${project.build.directory}</workingDirectory>
+ <includes>
+ <include>**/*Test.java</include>
+ </includes>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
-
Modify the below file as per the diff contents
Edit file
${LEVELDBJNI_HOME}/pom.xml
@@ -249,6 +249,7 @@
<module>leveldbjni-win32</module>
<module>leveldbjni-win64</module>
<module>leveldbjni-all</module>
+ <module>leveldbjni-linux64-s390x</module>
</modules>
</profile>
@@ -298,6 +299,13 @@
<module>leveldbjni-win64</module>
</modules>
</profile>
+
+ <profile>
+ <id>linux64-s390x</id>
+ <modules>
+ <module>leveldbjni-linux64-s390x</module>
+ </modules>
+ </profile>
</profiles>
</project>
-
Build the jar file and place it in destination folder
cd ${LEVELDBJNI_HOME} mvn clean install -P download -Plinux64-s390x –DskipTests mvn clean install -P download -Plinux64,all -DskipTests cp ${LEVELDBJNI_HOME}/leveldbjni-all/target/leveldbjni-all-1.8.jar /<source_root>/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar