Building/Compiling Hadoop Native Library

If you see WARN like

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable

Then you have to re-compile hadoop native code manually and then put libhadoop-*.so in classpath

Prerequistes:

  • $ sudo apt-get install build-essential
  • $ sudo apt-get install g++ autoconf automake

And make sure that cmake is installed correctly.

$ cd $HADOOP_HOME/src
$ mvn package -Pdist,native -Dskiptests -Dtar

You should see the newly-built library in:

$ hadoop-dist/target/hadoop-2.3.0-cdh5.1.0/lib/native

Put following lines in hadoop-env.sh file

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/"
export HADOOP_COMMON_LIB_NATIVE_DIR="path/to/native"

OR simply,

$ cp $HADOOP_HOME/src/hadoop-dist/target/hadoop-2.3.0-cdh5.0.1/lib/native/*  $HADOOP_HOME/lib/native/