There are several RHadoop packages that we have to use in order to be able to connect R and Hadoop. Here we give a dense presentation of rhdfs, rmr2 and plyrmr together with available commands. Note that within this MOOC we demonstrate only few of those commands.

rhdfs

The library package rhdfs provides commands for file manipulation in terms of reading, writing and moving files. Namely, R is a programming language that offers data processing. The data themselves are stored in files in the Hadoop filesystem. The following commands are part of the rhdfs package:

hdfs.copy, hdfs.move, hdfs.rename, hdfs.delete, hdfs.rm, hdfs.del, hdfs.chown, hdfs.put, hdfs.get

hdfs.file, hdfs.write, hdfs.close, hdfs.flush, hdfs.read, hdfs.seek, hdfs.tell, hdfs.line.reader, hdfs.read.text.file

hdfs.dircreate, hdfs.mkdir

hdfs.ls, hdfs.list.files, hdfs.file.info, hdfs.exists

hdfs.init, hdfs.defaults

rmr2

The rmr2 package allows the Hadoop MapReduce facility to be used inside the R environment. The package documentation lists the following commands:

big.data.object

dfs.empty

equijoin

from.dfs

hadoop.settings

keyval

make.input.format

mapreduce

rmr.options

rmr.sample

rmr.str

scatter

status

to.map

plyrmr

This package aims to provide a wide palette of predefined operations to cover the basic data-manipulation needs. The documentation of the plyrmr package provides the following list of available commands:

bind.cols (add new columns), where(select rows), select(select columns), rbind, transmute (all of the above plus summaries)

transmute, sample, count.cols, quantile.cols, top.k, bottom.k

union, intersect, unique, merge

transmute appears twice because it is a generalization over transform and summarize that allows us to increase or decrease the number of columns or rows, covering the need for multi-row summaries, flattening of data structures, etc.