Big Data Blog: 2018

Prerequisite

Java JDK 1.8+

Maven 3.0.1+

Hadoop 2.6.0+

Pig 0.10.1+

Step1: Download Oozie 5.x

Download oozie-5.0.0.tar.gz form below link

http://www-eu.apache.org/dist/oozie/

Alternatively you can use wget command in OSX and or curl in linux respectively.

wget http://www-eu.apache.org/dist/oozie/5.0.0/oozie-5.0.0.tar.gz

curl -O http://www-eu.apache.org/dist/oozie/5.0.0/oozie-5.0.0.tar.gz

Step 2: Untar & expand the source distribution

tar xvzf oozie-3.3.2.tar.gz

Step 3: Build Oozie

Go inside Oozie directory and run below command.

cd oozie-5.0.0

bin/mkdistro.sh  -Dhadoop.version=2.7.5 -Dpig.version=0.14.0       -Djetty.version=9.4.11.v20180605 –DskipTests

Note that my installed hadoop version is 2.7.5. Provide your respective hadoop & pig version

Step 4: Update Hadoop core-site.xml and restart Hadoop

<property>     <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>     <value>[OOZIE_SERVER_HOSTNAME]</value>   </property>

<property>     <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>     <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>

</property>

My user name is ‘hadoop’ so in my case it is

<property>

<name>hadoop.proxyuser.hadoop.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.hadoop.groups</name>

<value>*</value>

</property>

Step 5: Setup Oozie Server

Oozie 5.0 uses jetty server, previous versions were using tomcat. Instead of bootstrap now it invokes EmbeddedOozieServe.

Before preparing war, extjs and Hadoop jars should be available in libext directory

cd distro/target/oozie-5.0.0-distro/oozie-5.0.0

mkdir libext

Download copy ext js from below link to libext folder

https://drive.google.com/file/d/0ByJLBTmJojjzcDhxQUsyNEFSQm8/view

Copy hadoop and hcatalog libraries into libext

You can use below command to copy.

Before executing just update bash profile with HADOOP_HOME and $OOZIE_HOME values

cp $HADOOP_HOME/share/hadoop/common/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/common/lib/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/mapreduce/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/mapreduce/lib/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/hdfs/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/hdfs/lib/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/yarn/*.jar $OOZIE_HOME/libext/

cp $HADOOP_HOME/share/hadoop/yarn/lib/*.jar $OOZIE_HOME/libext

Now run oozie-setup.sh

bin/oozie-setup.sh

Step 6: Create share lib

Delete any previous share library in HDFS if exist.

hadoop dfs -rmr /user/hadoop/share

Create share lib

bin/oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib oozie-sharelib-5.0.0.tar.gz

Step 7: Create Oozie DB

bin/ooziedb.sh create -sqlfile oozie.sql –run

Step 8: Start Oozie as a daemon process:

bin/oozied.sh start

URL for the Oozie Web Console is http://localhost:11000/oozie

Big Data Blog

Monday, December 17, 2018

DataLake Architecture

Thursday, August 2, 2018

Steps to install Apache Oozie 5.0