Map tasks and Reduce tasks

Map tasks produce models and a Reduce task aggregates them. Reduce strategies differ for Naive Bayes and other model types. We saw in the last lesson that Naive Bayes and JRip are treated differently. The reason is that Naive Bayes is easily parallelized by adding up frequency counts from the individual partitions, producing a single model. For JRip (and other classifiers), separate classifiers are learned for each partition (4 in this case), and a “vote” ensemble learner is produced that combines them. Also, for some classifiers (like JRip) it is beneficial to randomize the dataset before splitting it into partitions. Finally, we look at the “Spark: cross-validate two classifiers” template and examine how DIstributed Weka performs cross-validation.