Starting Hadoop, RStudio and RHadoop

We need to quickly check whether everything is working in our virtual machine and to become familiar with the environment we will be working with.

Open the terminal by clicking on the black icon on the bottom left and type (note that the symbol $ is the prompt in the terminal and not part of the command you type in):

$ start-dfs.sh
$ start-yarn.sh
$ hadoop fs -ls

Then type:

$ rstudio &

to open the RStudio GUI. We should open an R script file and save it (e.g., init.R) to the local or any other folder up to our choice. Next we should set up system variables by copying the following lines into the script file and execute them:

Sys.setenv(HADOOP_OPTS="-Djava.library.path=/usr/local/hadoop/lib/native")
Sys.setenv(HADOOP_HOME="/usr/local/hadoop")
Sys.setenv(HADOOP_CMD="/usr/local/hadoop/bin/hadoop")
Sys.setenv(HADOOP_STREAMING="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.5.jar")
Sys.setenv(JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64") 

Alternatively, we can execute the above lines one by one in the GUI terminal of RStudio.

We load (in the GUI terminal of RStudio) RHadoop by loading the libraries rhdfs and rmr2 and executing hdfs.init():

library(rhdfs)
library(rmr2)
hdfs.init()

You might see the log and some warning messages, but no filesystem errors while executing the above commands. There might be some glyph rendering errors, which we can simply ignore.

Try to quit and stop the Hadoop servers by using counterpart commands stop-yarn.sh and stop-dfs.sh in terminal.