Oozie installation instructions: 1. Launch an Amazon EC2 Instance according to http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-instance_linux.html Step 1: Choose an Amazon Machine Image (AMI) Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-5189a661 Step 2: Choose an Instance Type t2.small Memory: 2 GiB [Step 4: Add Storage Size: 30 GiB Uncheck “Delete on Termination”] Step 6: Configure Security Group Type: All traffic Protocol: All Port Range: All Source: 0.0.0.0/0 $ ssh -i tongshu-key-pair.pem ubuntu@ 2. Add user oozie $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.2 LTS Release: 14.04 Codename: trusty $ sudo addgroup hadoop $ sudo adduser --ingroup hadoop oozie $ sudo adduser oozie sudo $ su oozie $ cd 3. Install Java 1.7, Maven, Native libraries and ProtocolBuffer 2.5.0 $ sudo add-apt-repository ppa:webupd8team/java $ sudo apt-get update $ sudo apt-get -y install oracle-java7-installer $ sudo update-alternatives --config java There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-7-oracle/jre/bin/java Nothing to configure. $ java -version java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode) $ sudo apt-get -y install maven $ mvn -v Apache Maven 3.0.5 Maven home: /usr/share/maven Java version: 1.7.0_80, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-7-oracle/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.13.0-48-generic", arch: "amd64", family: "unix" $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev $ sudo apt-get -y install libprotobuf-dev protobuf-compiler $ protoc --version libprotoc 2.5.0 $ sudo apt-get -y install zip unzip 4. Compile Hadoop $ wget http://mirror.olnevhost.net/pub/apache/hadoop/common/hadoop-2.7.1/hadoop-2.7.1-src.tar.gz $ tar -zxvf hadoop-2.7.1-src.tar.gz $ cd hadoop-2.7.1-src $ mvn clean package -Pdist,native -Dmaven.javadoc.skip=true -DskipTests -Dtar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................ SUCCESS [22.530s] [INFO] Apache Hadoop Project POM ......................... SUCCESS [8.334s] [INFO] Apache Hadoop Annotations ......................... SUCCESS [7.723s] [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.218s] [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [5.632s] [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [7.577s] [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [25.069s] [INFO] Apache Hadoop Auth ................................ SUCCESS [13.132s] [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.390s] [INFO] Apache Hadoop Common .............................. SUCCESS [1:09.671s] [INFO] Apache Hadoop NFS ................................. SUCCESS [1.392s] [INFO] Apache Hadoop KMS ................................. SUCCESS [7.110s] [INFO] Apache Hadoop Common Project ...................... SUCCESS [0.022s] [INFO] Apache Hadoop HDFS ................................ SUCCESS [46.987s] [INFO] Apache Hadoop HttpFS .............................. SUCCESS [14.134s] [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [6.166s] [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [0.789s] [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.022s] [INFO] hadoop-yarn ....................................... SUCCESS [0.021s] [INFO] hadoop-yarn-api ................................... SUCCESS [3.926s] [INFO] hadoop-yarn-common ................................ SUCCESS [12.199s] [INFO] hadoop-yarn-server ................................ SUCCESS [0.019s] [INFO] hadoop-yarn-server-common ......................... SUCCESS [1.803s] [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [5.723s] [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [0.336s] [INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [0.680s] [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [3.557s] [INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.900s] [INFO] hadoop-yarn-client ................................ SUCCESS [0.936s] [INFO] hadoop-yarn-server-sharedcachemanager ............. SUCCESS [0.371s] [INFO] hadoop-yarn-applications .......................... SUCCESS [0.038s] [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [0.291s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.290s] [INFO] hadoop-yarn-site .................................. SUCCESS [0.062s] [INFO] hadoop-yarn-registry .............................. SUCCESS [0.499s] [INFO] hadoop-yarn-project ............................... SUCCESS [3.074s] [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.051s] [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [3.971s] [INFO] hadoop-mapreduce-client-common .................... SUCCESS [1.461s] [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [0.244s] [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [1.277s] [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [0.750s] [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [3.451s] [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [0.203s] [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [0.479s] [INFO] hadoop-mapreduce .................................. SUCCESS [1.719s] [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [0.928s] [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [2.989s] [INFO] Apache Hadoop Archives ............................ SUCCESS [0.596s] [INFO] Apache Hadoop Rumen ............................... SUCCESS [1.172s] [INFO] Apache Hadoop Gridmix ............................. SUCCESS [0.971s] [INFO] Apache Hadoop Data Join ........................... SUCCESS [0.184s] [INFO] Apache Hadoop Ant Tasks ........................... SUCCESS [0.081s] [INFO] Apache Hadoop Extras .............................. SUCCESS [0.291s] [INFO] Apache Hadoop Pipes ............................... SUCCESS [6.159s] [INFO] Apache Hadoop OpenStack support ................... SUCCESS [0.221s] [INFO] Apache Hadoop Amazon Web Services support ......... SUCCESS [7.223s] [INFO] Apache Hadoop Azure support ....................... SUCCESS [1.710s] [INFO] Apache Hadoop Client .............................. SUCCESS [5.375s] [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.126s] [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [0.817s] [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [4.267s] [INFO] Apache Hadoop Tools ............................... SUCCESS [0.016s] [INFO] Apache Hadoop Distribution ........................ SUCCESS [13.389s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5:50.750s [INFO] Finished at: Fri Jan 15 23:31:17 UTC 2016 [INFO] Final Memory: 218M/485M [INFO] ------------------------------------------------------------------------ 5. Install Hadoop $ cd $ cp ~/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1.tar.gz ~/ $ tar -zxvf hadoop-2.7.1.tar.gz $ cd ~/hadoop-2.7.1 $ vi ~/.bashrc export HADOOP_PREFIX=/home/oozie/hadoop-2.7.1 export PATH=$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$PATH export HADOOP_CLASSPATH=$JAVA_HOME/lib/tools.jar export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop export HADOOP_LIBEXEC_DIR=$HADOOP_PREFIX/libexec export HADOOP_HOME=$HADOOP_PREFIX export HADOOP_COMMON_HOME=$HADOOP_PREFIX export HADOOP_HDFS_HOME=$HADOOP_PREFIX export HADOOP_MAPRED_HOME=$HADOOP_PREFIX export HADOOP_YARN_HOME=$HADOOP_PREFIX export YARN_HOME=$HADOOP_PREFIX export YARN_CONF_DIR=$HADOOP_PREFIX/etc/hadoop export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_PREFIX/lib/native export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_PREFIX/lib/native" #export HADOOP_ROOT_LOGGER=DEBUG,console export MAVEN_OPTS='-Xmx512m -XX:MaxPermSize=256m' $ source ~/.bashrc $ vi etc/hadoop/hadoop-env.sh line 26: export JAVA_HOME=/usr/lib/jvm/java-7-oracle line 47: export HADOOP_HEAPSIZE=500 export HADOOP_NAMENODE_INIT_HEAPSIZE=500 $ vi etc/hadoop/mapred-env.sh line 19: export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=250 $ vi etc/hadoop/yarn-env.sh line 36: JAVA_HEAP_MAX=-Xmx500m line 40: YARN_HEAPSIZE=500 $ hadoop version Hadoop 2.7.1 Subversion Unknown -r Unknown Compiled by oozie on 2016-01-15T23:27Z Compiled with protoc 2.5.0 From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a This command was run using /home/oozie/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar 6. Configure HDFS $ vi etc/hadoop/core-site.xml fs.defaultFS hdfs://hadoop-master:8020 hadoop.tmp.dir /home/oozie/hadoop-2.7.1/tmp hadoop.http.staticuser.user oozie hadoop.proxyuser.oozie.hosts hadoop-master,localhost hadoop.proxyuser.oozie.groups oozie,hadoop $ vi etc/hadoop/hdfs-site.xml dfs.namenode.name.dir /home/oozie/hadoop-2.7.1/hdfs/name dfs.datanode.data.dir /home/oozie/hadoop-2.7.1/hdfs/data dfs.replication 1 dfs.permissions false fs.checkpoint.dir /home/oozie/hadoop-2.7.1/hdfs/checkpoint fs.checkpoint.edits.dir /home/oozie/hadoop-2.7.1/hdfs/checkpoint dfs.namenode.http-address hadoop-master:50070 dfs.namenode.secondary.http-address hadoop-master:50090 7. Configure YARN $ cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml $ vi etc/hadoop/mapred-site.xml mapreduce.framework.name yarn mapreduce.jobtracker.address hadoop-master:8021 mapreduce.map.cpu.vcores 1 mapreduce.reduce.cpu.vcores 1 mapreduce.map.memory.mb 1536 mapResourceReqt: MAP capability required (no more than maxContainerCapability) mapreduce.map.java.opts -Xmx1280m mapreduce.reduce.memory.mb 1024 mapreduce.reduce.java.opts -Xmx768m yarn.app.mapreduce.am.resource.mb 1024 yarn.app.mapreduce.am.command-opts -Xmx768m mapreduce.jobhistory.address hadoop-master:10020 mapreduce.jobhistory.webapp.address hadoop-master:19888 mapreduce.jobhistory.intermediate-done-dir /home/oozie/hadoop-2.7.1/mr-history/done mapreduce.jobhistory.done-dir /home/oozie/hadoop-2.7.1/mr-history/done $ vi etc/hadoop/yarn-site.xml yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.resourcemanager.hostname hadoop-master yarn.resourcemanager.address hadoop-master:8032 yarn.resourcemanager.scheduler.address hadoop-master:8030 yarn.resourcemanager.resource-tracker.address hadoop-master:8031 yarn.resourcemanager.admin.address hadoop-master:8033 yarn.resourcemanager.webapp.address hadoop-master:8088 yarn.application.classpath $HADOOP_PREFIX/*,$HADOOP_PREFIX/lib/*,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/* yarn.scheduler.minimum-allocation-mb 128 yarn.scheduler.maximum-allocation-mb 2048 maxContainerCapability: supported max container capability (no less than mapResourceReqt) yarn.scheduler.minimum-allocation-vcores 1 yarn.scheduler.maximum-allocation-vcores 1 yarn.nodemanager.resource.memory-mb 4096 yarn.nodemanager.resource.cpu-vcores 1 8. Configure a cluster Note: Install Hadoop with the same configuration in other nodes. $ sudo vi /etc/hosts 172.31.4.187 hadoop-master 172.31.22.25 hadoop-slave1 $ cp etc/hadoop/slaves etc/hadoop/slaves.bak $ vi etc/hadoop/slaves hadoop-master hadoop-slave1 $ vi etc/hadoop/master hadoop-master 9. Run MapReduce jobs $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys $ vi ~/.ssh/authorized_keys [copy this output and paste it in all the other nodes] $ ssh hadoop-master $ ssh hadoop-slave1 $ bin/hdfs namenode -format $ sbin/start-dfs.sh $ sbin/start-yarn.sh [$ sbin/start-all.sh] $ sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver NameNode - http://hadoop-master:50070/ ResourceManager - http://hadoop-master:8088/ $ bin/hdfs dfs -mkdir /user $ bin/hdfs dfs -mkdir /user/oozie $ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar pi 16 1000 $ bin/hdfs dfs -put etc/hadoop input $ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+' $ bin/hdfs dfs -cat output/* $ bin/hdfs dfs -rm -r input $ bin/hdfs dfs -rm -r output $ sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver $ sbin/stop-dfs.sh $ sbin/stop-yarn.sh [$ sbin/stop-all.sh] $ rm -rf hdfs tmp 10. Install Oozie Note: Stop Hadoop first. $ cd $ wget http://apache.claz.org/oozie/4.2.0/oozie-4.2.0.tar.gz $ tar -zxvf oozie-4.2.0.tar.gz $ cd ~/oozie-4.2.0 $ vi pom.xml 1.7 2.7.1 Codehaus repository http://repository.codehaus.org/ false false $ bin/mkdistro.sh -DskipTests -DtargetJavaVersion=1.7 -Puber -Phadoop-2 -Dhadoop.version=2.7.1 -Dhadoop.auth.version=2.7.1 -Ddistcp.version=2.7.1 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Oozie Main ................................. SUCCESS [14.344s] [INFO] Apache Oozie Hadoop Utils hadoop-2-4.2.0 .......... SUCCESS [16.175s] [INFO] Apache Oozie Hadoop Distcp hadoop-2-4.2.0 ......... SUCCESS [0.338s] [INFO] Apache Oozie Hadoop Auth hadoop-2-4.2.0 Test ...... SUCCESS [0.496s] [INFO] Apache Oozie Hadoop Libs .......................... SUCCESS [0.027s] [INFO] Apache Oozie Client ............................... SUCCESS [4:53.293s] [INFO] Apache Oozie Share Lib Oozie ...................... SUCCESS [10.600s] [INFO] Apache Oozie Share Lib HCatalog ................... SUCCESS [5.411s] [INFO] Apache Oozie Share Lib Distcp ..................... SUCCESS [1.096s] [INFO] Apache Oozie Core ................................. SUCCESS [1:42.936s] [INFO] Apache Oozie Share Lib Streaming .................. SUCCESS [7.208s] [INFO] Apache Oozie Share Lib Pig ........................ SUCCESS [5.672s] [INFO] Apache Oozie Share Lib Hive ....................... SUCCESS [4.849s] [INFO] Apache Oozie Share Lib Hive 2 ..................... SUCCESS [5.006s] [INFO] Apache Oozie Share Lib Sqoop ...................... SUCCESS [3.038s] [INFO] Apache Oozie Examples ............................. SUCCESS [5.757s] [INFO] Apache Oozie Share Lib Spark ...................... SUCCESS [5.431s] [INFO] Apache Oozie Share Lib ............................ SUCCESS [11.283s] [INFO] Apache Oozie Docs ................................. SUCCESS [19.290s] [INFO] Apache Oozie WebApp ............................... SUCCESS [1:18.037s] [INFO] Apache Oozie Tools ................................ SUCCESS [4.859s] [INFO] Apache Oozie MiniOozie ............................ SUCCESS [1.272s] [INFO] Apache Oozie Distro ............................... SUCCESS [29.377s] [INFO] Apache Oozie ZooKeeper Security Tests ............. SUCCESS [4.167s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10:46.918s [INFO] Finished at: Sat Jan 16 00:10:39 UTC 2016 [INFO] Final Memory: 248M/494M [INFO] ------------------------------------------------------------------------ Oozie distro created, DATE[2016.01.15-23:59:49GMT] VC-REV[unavailable], available at [/home/oozie/oozie-4.2.0/distro/target] $ cd $ cp -r ~/oozie-4.2.0/distro/target/oozie-4.2.0-distro/oozie-4.2.0 oozie-4.2.0-bin $ vi ~/.bashrc export OOZIE_HOME=/home/oozie/oozie-4.2.0-bin export PATH=$OOZIE_HOME/bin:$PATH export OOZIE_URL=http://hadoop-master:11000/oozie export OOZIE_VERSION=4.2.0 $ source ~/.bashrc $ cd ~/oozie-4.2.0-bin $ mkdir libext $ cd ~/oozie-4.2.0-bin/libext $ wget http://dev.sencha.com/deploy/ext-2.2.zip $ find ~/hadoop-2.7.1/share/hadoop -name *.jar -type f -print0 | xargs -0 cp -t ~/oozie-4.2.0-bin/libext $ rm -f *-tests*.jar *-sources.jar hsqldb-2.0.0.jar $ cd ~/oozie-4.2.0-bin $ tar -zxvf oozie-sharelib-4.2.0.tar.gz $ cp conf/oozie-default.xml.reference conf/oozie-default.xml $ vi conf/oozie-site.xml oozie.service.JPAService.create.db.schema true Creates Oozie DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. oozie.service.HadoopAccessorService.hadoop.configurations *=/home/oozie/hadoop-2.7.1/etc/hadoop Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*' configuration is used when there is no exact match for an authority. The HADOOP_CONF_DIR contains the relevant Hadoop *-site.xml files. If the path is relative is looked within the Oozie configuration directory; though the path can be absolute (i.e. to point to Hadoop client conf/ directories in the local filesystem. oozie.service.WorkflowAppService.system.libpath /home/oozie/oozie-4.2.0-bin/share/lib System library path to use for workflow applications. This path is added to workflow application if their job properties sets the property 'oozie.use.system.libpath' to true. $ vi conf/hadoop-conf/core-site.xml hadoop.proxyuser.oozie.hosts hadoop-master,localhost hadoop.proxyuser.oozie.groups oozie,hadoop $ bin/oozie-setup.sh prepare-war setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/activation-1.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/annotations-api.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/antlr-2.7.7.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/aopalliance-1.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/apacheds-i18n-2.0.0-M15.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/apacheds-kerberos-codec-2.0.0-M15.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/api-asn1-api-1.0.0-M20.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/api-asn1-ber-1.0.0-M20.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/api-i18n-1.0.0-M20.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/api-ldap-model-1.0.0-M20.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/api-util-1.0.0-M20.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/asm-3.2.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/avro-1.7.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/aws-java-sdk-1.7.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/azure-storage-2.0.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/bootstrap.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/catalina-ant.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/catalina-ha.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/catalina.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/catalina-tribes.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-beanutils-1.7.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-beanutils-core-1.8.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-cli-1.2.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-codec-1.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-collections-3.2.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-compress-1.4.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-configuration-1.6.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-daemon-1.0.13.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-daemon.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-digester-1.8.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-httpclient-3.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-io-2.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-lang-2.6.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-lang3-3.3.2.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-logging-1.1.3.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-math3-3.1.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/commons-net-3.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/curator-client-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/curator-framework-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/curator-recipes-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/ecj-4.3.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/ehcache-core-2.4.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/el-api.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/gson-2.2.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/guava-11.0.2.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/guice-3.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/guice-servlet-3.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-annotations-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-ant-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-archives-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-auth-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-aws-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-azure-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-common-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-datajoin-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-distcp-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-extras-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-gridmix-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-hdfs-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-hdfs-nfs-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-kms-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-app-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-common-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-core-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-hs-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-hs-plugins-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-jobclient-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-client-shuffle-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-mapreduce-examples-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-nfs-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-openstack-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-rumen-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-sls-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-streaming-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-api-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-applications-distributedshell-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-client-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-common-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-registry-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-server-common-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-server-nodemanager-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-server-resourcemanager-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-server-sharedcachemanager-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hadoop-yarn-server-web-proxy-2.7.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/hamcrest-core-1.3.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/htrace-core-3.1.0-incubating.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/httpclient-4.2.5.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/httpcore-4.2.5.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-annotations-2.2.3.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-core-2.2.3.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-core-asl-1.9.13.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-databind-2.2.3.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-jaxrs-1.9.13.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-mapper-asl-1.9.13.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jackson-xc-1.9.13.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jasper-el.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jasper.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/javax.inject-1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/java-xmlbuilder-0.4.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jaxb-api-2.2.2.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jaxb-impl-2.2.3-1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jersey-client-1.9.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jersey-core-1.9.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jersey-guice-1.9.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jersey-json-1.9.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jersey-server-1.9.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jets3t-0.9.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jettison-1.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jetty-6.1.26.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jetty-util-6.1.26.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jline-0.9.94.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/joda-time-2.9.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jsch-0.1.42.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/json-simple-1.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jsp-api-2.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jsp-api.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jsr305-3.0.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/jul-to-slf4j-1.7.10.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/junit-4.11.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/leveldbjni-all-1.8.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/log4j-1.2.17.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/metrics-core-3.0.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/mina-core-2.0.0-M5.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/mockito-all-1.8.5.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/netty-3.6.2.Final.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/netty-all-4.0.23.Final.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/paranamer-2.3.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/protobuf-java-2.5.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/servlet-api-2.5.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/servlet-api.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/slf4j-api-1.7.10.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/slf4j-log4j12-1.7.10.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/snappy-java-1.0.4.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/stax-api-1.0-2.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/tomcat-coyote.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/tomcat-dbcp.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/tomcat-i18n-es.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/tomcat-i18n-fr.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/tomcat-i18n-ja.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/tomcat-juli.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/xercesImpl-2.9.1.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/xml-apis-1.3.04.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/xmlenc-0.52.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/xz-1.0.jar INFO: Adding extension: /home/oozie/oozie-4.2.0-bin/libext/zookeeper-3.4.6.jar New Oozie WAR file with added 'ExtJS library, JARs' at /home/oozie/oozie-4.2.0-bin/oozie-server/webapps/oozie.war INFO: Oozie is ready to be started $ cd ~/hadoop-2.7.1 $ bin/hdfs namenode -format $ sbin/start-dfs.sh $ sbin/start-yarn.sh [$ sbin/start-all.sh] $ sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver $ bin/hdfs dfs -mkdir /user $ bin/hdfs dfs -mkdir /user/oozie $ bin/hdfs dfsadmin -safemode leave Safe mode is OFF $ cd ~/oozie-4.2.0-bin $ bin/oozie-setup.sh sharelib create -fs hdfs://hadoop-master:8020 setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/oozie/oozie-4.2.0-bin/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/oozie/oozie-4.2.0-bin/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/oozie/oozie-4.2.0-bin/libext/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory] the destination path for sharelib is: /home/oozie/oozie-4.2.0-bin/share/lib/lib_20160116001806 [$ bin/ooziedb.sh create -sqlfile oozie.sql -run] $ bin/oozied.sh start Setting OOZIE_HOME: /home/oozie/oozie-4.2.0-bin Setting OOZIE_CONFIG: /home/oozie/oozie-4.2.0-bin/conf Sourcing: /home/oozie/oozie-4.2.0-bin/conf/oozie-env.sh setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" Setting OOZIE_CONFIG_FILE: oozie-site.xml Setting OOZIE_DATA: /home/oozie/oozie-4.2.0-bin/data Setting OOZIE_LOG: /home/oozie/oozie-4.2.0-bin/logs Setting OOZIE_LOG4J_FILE: oozie-log4j.properties Setting OOZIE_LOG4J_RELOAD: 10 Setting OOZIE_HTTP_HOSTNAME: ip-172-31-4-187.us-west-2.compute.internal Setting OOZIE_HTTP_PORT: 11000 Setting OOZIE_ADMIN_PORT: 11001 Setting OOZIE_HTTPS_PORT: 11443 Setting OOZIE_BASE_URL: http://ip-172-31-4-187.us-west-2.compute.internal:11000/oozie Setting CATALINA_BASE: /home/oozie/oozie-4.2.0-bin/oozie-server Setting OOZIE_HTTPS_KEYSTORE_FILE: /home/oozie/.keystore Setting OOZIE_HTTPS_KEYSTORE_PASS: password Setting OOZIE_INSTANCE_ID: ip-172-31-4-187.us-west-2.compute.internal Setting CATALINA_OUT: /home/oozie/oozie-4.2.0-bin/logs/catalina.out Setting CATALINA_PID: /home/oozie/oozie-4.2.0-bin/oozie-server/temp/oozie.pid Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/oozie/oozie-4.2.0-bin/logs/derby.log Adding to CATALINA_OPTS: -Doozie.home.dir=/home/oozie/oozie-4.2.0-bin -Doozie.config.dir=/home/oozie/oozie-4.2.0-bin/conf -Doozie.log.dir=/home/oozie/oozie-4.2.0-bin/logs -Doozie.data.dir=/home/oozie/oozie-4.2.0-bin/data -Doozie.instance.id=ip-172-31-4-187.us-west-2.compute.internal -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=ip-172-31-4-187.us-west-2.compute.internal -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://ip-172-31-4-187.us-west-2.compute.internal:11000/oozie -Doozie.https.keystore.file=/home/oozie/.keystore -Doozie.https.keystore.pass=password -Djava.library.path= Setting up oozie DB setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" Validate DB Connection DONE DB schema does not exist Check OOZIE_SYS table does not exist DONE Create SQL schema DONE Create OOZIE_SYS table DONE Oozie DB has been created for Oozie version '4.2.0' The SQL commands have been written to: /tmp/ooziedb-5642755266608385229.sql Using CATALINA_BASE: /home/oozie/oozie-4.2.0-bin/oozie-server Using CATALINA_HOME: /home/oozie/oozie-4.2.0-bin/oozie-server Using CATALINA_TMPDIR: /home/oozie/oozie-4.2.0-bin/oozie-server/temp Using JRE_HOME: /usr Using CLASSPATH: /home/oozie/oozie-4.2.0-bin/oozie-server/bin/bootstrap.jar Using CATALINA_PID: /home/oozie/oozie-4.2.0-bin/oozie-server/temp/oozie.pid [$ bin/oozied.sh run] $ bin/oozie admin -status SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/oozie/oozie-4.2.0-bin/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/oozie/oozie-4.2.0-bin/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/oozie/oozie-4.2.0-bin/libext/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.authentication.client.KerberosAuthenticator). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. System mode: NORMAL $ jps 31574 NameNode 1566 Jps 32246 NodeManager 1463 Bootstrap 31950 SecondaryNameNode 31745 DataNode 32564 JobHistoryServer 32100 ResourceManager 11. Run an example in Oozie $ tar -zxvf oozie-examples.tar.gz $ vi examples/apps/map-reduce/job.properties nameNode=hdfs://hadoop-master:8020 jobTracker=hadoop-master:8032 queueName=default examplesRoot=examples oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce/workflow.xml outputDir=map-reduce $ hdfs dfs -put examples examples $ bin/oozie job -config examples/apps/map-reduce/job.properties -run job: 0000000-160116001913280-oozie-oozi-W $ bin/oozie job -info 0000000-160116001913280-oozie-oozi-W Job ID : 0000000-160116001913280-oozie-oozi-W ------------------------------------------------------------------------------------------------------------------------------------ Workflow Name : map-reduce-wf App Path : hdfs://hadoop-master:8020/user/oozie/examples/apps/map-reduce/workflow.xml Status : RUNNING Run : 0 User : oozie Group : - Created : 2016-01-16 00:23 GMT Started : 2016-01-16 00:23 GMT Last Modified : 2016-01-16 00:23 GMT Ended : - CoordAction ID: - Actions ------------------------------------------------------------------------------------------------------------------------------------ ID Status Ext ID Ext Status Err Code ------------------------------------------------------------------------------------------------------------------------------------ 0000000-160116001913280-oozie-oozi-W@:start: OK - OK - ------------------------------------------------------------------------------------------------------------------------------------ 0000000-160116001913280-oozie-oozi-W@mr-node RUNNING job_1452903423000_0001 RUNNING - ------------------------------------------------------------------------------------------------------------------------------------ Job ID : 0000000-160116001913280-oozie-oozi-W ------------------------------------------------------------------------------------------------------------------------------------ Workflow Name : map-reduce-wf App Path : hdfs://hadoop-master:8020/user/oozie/examples/apps/map-reduce/workflow.xml Status : SUCCEEDED Run : 0 User : oozie Group : - Created : 2016-01-16 00:23 GMT Started : 2016-01-16 00:23 GMT Last Modified : 2016-01-16 00:24 GMT Ended : 2016-01-16 00:24 GMT CoordAction ID: - Actions ------------------------------------------------------------------------------------------------------------------------------------ ID Status Ext ID Ext Status Err Code ------------------------------------------------------------------------------------------------------------------------------------ 0000000-160116001913280-oozie-oozi-W@:start: OK - OK - ------------------------------------------------------------------------------------------------------------------------------------ 0000000-160116001913280-oozie-oozi-W@mr-node OK job_1452903423000_0001 SUCCEEDED - ------------------------------------------------------------------------------------------------------------------------------------ 0000000-160116001913280-oozie-oozi-W@end OK - OK - ------------------------------------------------------------------------------------------------------------------------------------ $ hdfs dfs -cat /user/oozie/examples/output-data/map-reduce/* 0 To be or not to be, that is the question; 42 Whether 'tis nobler in the mind to suffer 84 The slings and arrows of outrageous fortune, 129 Or to take arms against a sea of troubles, 172 And by opposing, end them. To die, to sleep; 217 No more; and by a sleep to say we end 255 The heart-ache and the thousand natural shocks 302 That flesh is heir to ? 'tis a consummation 346 Devoutly to be wish'd. To die, to sleep; 387 To sleep, perchance to dream. Ay, there's the rub, 438 For in that sleep of death what dreams may come, 487 When we have shuffled off this mortal coil, 531 Must give us pause. There's the respect 571 That makes calamity of so long life, 608 For who would bear the whips and scorns of time, 657 Th'oppressor's wrong, the proud man's contumely, 706 The pangs of despised love, the law's delay, 751 The insolence of office, and the spurns 791 That patient merit of th'unworthy takes, 832 When he himself might his quietus make 871 With a bare bodkin? who would fardels bear, 915 To grunt and sweat under a weary life, 954 But that the dread of something after death, 999 The undiscovered country from whose bourn 1041 No traveller returns, puzzles the will, 1081 And makes us rather bear those ills we have 1125 Than fly to others that we know not of? 1165 Thus conscience does make cowards of us all, 1210 And thus the native hue of resolution 1248 Is sicklied o'er with the pale cast of thought, 1296 And enterprises of great pitch and moment 1338 With this regard their currents turn awry, 1381 And lose the name of action. $ bin/oozied.sh stop Setting OOZIE_HOME: /home/oozie/oozie-4.2.0-bin Setting OOZIE_CONFIG: /home/oozie/oozie-4.2.0-bin/conf Sourcing: /home/oozie/oozie-4.2.0-bin/conf/oozie-env.sh setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m" Setting OOZIE_CONFIG_FILE: oozie-site.xml Setting OOZIE_DATA: /home/oozie/oozie-4.2.0-bin/data Setting OOZIE_LOG: /home/oozie/oozie-4.2.0-bin/logs Setting OOZIE_LOG4J_FILE: oozie-log4j.properties Setting OOZIE_LOG4J_RELOAD: 10 Setting OOZIE_HTTP_HOSTNAME: ip-172-31-4-187.us-west-2.compute.internal Setting OOZIE_HTTP_PORT: 11000 Setting OOZIE_ADMIN_PORT: 11001 Setting OOZIE_HTTPS_PORT: 11443 Setting OOZIE_BASE_URL: http://ip-172-31-4-187.us-west-2.compute.internal:11000/oozie Setting CATALINA_BASE: /home/oozie/oozie-4.2.0-bin/oozie-server Setting OOZIE_HTTPS_KEYSTORE_FILE: /home/oozie/.keystore Setting OOZIE_HTTPS_KEYSTORE_PASS: password Setting OOZIE_INSTANCE_ID: ip-172-31-4-187.us-west-2.compute.internal Setting CATALINA_OUT: /home/oozie/oozie-4.2.0-bin/logs/catalina.out Setting CATALINA_PID: /home/oozie/oozie-4.2.0-bin/oozie-server/temp/oozie.pid Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/oozie/oozie-4.2.0-bin/logs/derby.log Adding to CATALINA_OPTS: -Doozie.home.dir=/home/oozie/oozie-4.2.0-bin -Doozie.config.dir=/home/oozie/oozie-4.2.0-bin/conf -Doozie.log.dir=/home/oozie/oozie-4.2.0-bin/logs -Doozie.data.dir=/home/oozie/oozie-4.2.0-bin/data -Doozie.instance.id=ip-172-31-4-187.us-west-2.compute.internal -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=ip-172-31-4-187.us-west-2.compute.internal -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://ip-172-31-4-187.us-west-2.compute.internal:11000/oozie -Doozie.https.keystore.file=/home/oozie/.keystore -Doozie.https.keystore.pass=password -Djava.library.path= Using CATALINA_BASE: /home/oozie/oozie-4.2.0-bin/oozie-server Using CATALINA_HOME: /home/oozie/oozie-4.2.0-bin/oozie-server Using CATALINA_TMPDIR: /home/oozie/oozie-4.2.0-bin/oozie-server/temp Using JRE_HOME: /usr Using CLASSPATH: /home/oozie/oozie-4.2.0-bin/oozie-server/bin/bootstrap.jar Using CATALINA_PID: /home/oozie/oozie-4.2.0-bin/oozie-server/temp/oozie.pid $ cd ~/hadoop-2.7.1 $ sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver $ sbin/stop-yarn.sh $ sbin/stop-dfs.sh Amazon VM Instance ID: i-618508bb Reference: https://www.digitalocean.com/community/tutorials/how-to-install-java-on-ubuntu-with-apt-get http://baazgusht.blogspot.com/2015/08/how-to-install-hadoop-271-on-ubuntu.html http://idroot.net/tutorials/how-to-install-apache-hadoop-on-ubuntu-14-04/ http://mail-archives.apache.org/mod_mbox/hadoop-user/201405.mbox/%3CCACUUc6BB8pBMRQ_vVwXeCMzoBvMWs2WLgteCsLSm3aamQPaV4g@mail.gmail.com%3E http://mfaizmzaki.com/2015/07/21/how-to-install-hadoop-multin-node-cluster-on-amazon-aws-ec2-instance/ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.15/bk_using-apache-hadoop/content/ch_using-apache-hadoop.html http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_installing_manually_book/content/rpm_chap3.html https://code.google.com/p/hadoop-toolkit/ https://chawlasumit.wordpress.com/2015/03/09/install-a-multi-node-hadoop-cluster-on-ubuntu-14-04/ https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html http://www.alexjf.net/blog/distributed-systems/hadoop-yarn-installation-definitive-guide/ http://arturmkrtchyan.com/how-to-setup-multi-node-hadoop-2-yarn-cluster http://gauravkohli.com/2014/08/26/apache-oozie-installation-on-hadoop-2-4-1/ http://ngsda.blogspot.com/2014/10/install-oozie-on-ubuntu-1404-with.html