PETABYTE

Mapper

Mapper maps input key/value pairs to a set of intermediate key/value pairs.

Maps are the individual tasks that transform input records into intermediate records. The transformed intermediate records do not need to be of the same type as the input records. A given input pair may map to zero or many output pairs.

The Hadoop MapReduce framework spawns one map task for each InputSplit generated by the InputFormat for the job.

Overall, Mapper implementations are passed the Job for the job via the Job.setMapperClass(Class) method. The framework then calls map(WritableComparable, Writable, Context) for each key/value pair in the InputSplit for that task. Applications can then override the cleanup(Context) method to perform any required cleanup.

Output pairs do not need to be of the same types as input pairs. A given input pair may map to zero or many output pairs. Output pairs are collected with calls to context.write(WritableComparable, Writable).

Applications can use the Counter to report its statistics.

All intermediate values associated with a given output key are subsequently grouped by the framework, and passed to the Reducer(s) to determine the final output. Users can control the grouping by specifying a Comparator via Job.setGroupingComparatorClass(Class).

The Mapper outputs are sorted and then partitioned per Reducer. The total number of partitions is the same as the number of reduce tasks for the job. Users can control which keys (and hence records) go to which Reducer by implementing a custom Partitioner.

Users can optionally specify a combiner, via Job.setCombinerClass(Class), to perform local aggregation of the intermediate outputs, which helps to cut down the amount of data transferred from the Mapper to the Reducer.

The intermediate, sorted outputs are always stored in a simple (key-len, key, value-len, value) format. Applications can control if, and how, the intermediate outputs are to be compressed and the CompressionCodec to be used via the Configuration.

Reducer

Reducer reduces a set of intermediate values which share a key to a smaller set of values.

The number of reduces for the job is set by the user via Job.setNumReduceTasks(int).

Overall, Reducer implementations are passed the Job for the job via the Job.setReducerClass(Class) method and can override it to initialize themselves. The framework then calls reduce(WritableComparable, Iterable<Writable>, Context) method for each <key, (list of values)> pair in the grouped inputs. Applications can then override the cleanup(Context) method to perform any required cleanup.

Reducer has 3 primary phases: shuffle, sort and reduce.

Partitioner

Partitioner partitions the key space.

Partitioner controls the partitioning of the keys of the intermediate map-outputs. The key (or a subset of the key) is used to derive the partition, typically by a hash function. The total number of partitions is the same as the number of reduce tasks for the job. Hence this controls which of the m reduce tasks the intermediate key (and hence the record) is sent to for reduction.

HashPartitioner is the default Partitioner.

Counter

Counter is a facility for MapReduce applications to report its statistics.

Mapper and Reducer implementations can use the Counter to report statistics.

Hadoop MapReduce comes bundled with a library of generally useful mappers, reducers, and partitioners.

Memory Management

Users/admins can also specify the maximum virtual memory of the launched child-task, and any sub-process it launches recursively, using mapreduce.{map|reduce}.memory.mb. Note that the value set here is a per process limit. The value for mapreduce.{map|reduce}.memory.mbshould be specified in mega bytes (MB). And also the value must be greater than or equal to the -Xmx passed to JavaVM, else the VM might not start.

Note: mapreduce.{map|reduce}.java.opts are used only for configuring the launched child tasks from MRAppMaster. Configuring the memory options for daemons is documented in Configuring the Environment of the Hadoop Daemons.

The memory available to some parts of the framework is also configurable. In map and reduce tasks, performance may be influenced by adjusting parameters influencing the concurrency of operations and the frequency with which data will hit disk. Monitoring the filesystem counters for a job- particularly relative to byte counts from the map and into the reduce- is invaluable to the tuning of these parameters.

Map Parameters

A record emitted from a map will be serialized into a buffer and metadata will be stored into accounting buffers. As described in the following options, when either the serialization buffer or the metadata exceed a threshold, the contents of the buffers will be sorted and written to disk in the background while the map continues to output records. If either buffer fills completely while the spill is in progress, the map thread will block. When the map is finished, any remaining records are written to disk and all on-disk segments are merged into a single file. Minimizing the number of spills to disk can decrease map time, but a larger buffer also decreases the memory available to the mapper.

Other notes

• If either spill threshold is exceeded while a spill is in progress, collection will continue until the spill is finished. For example, if mapreduce.map.sort.spill.percent is set to 0.33, and the remainder of the buffer is filled while the spill runs, the next spill will include all the collected records, or 0.66 of the buffer, and will not generate additional spills. In other words, the thresholds are defining triggers, not blocking.
• A record larger than the serialization buffer will first trigger a spill, then be spilled to a separate file. It is undefined whether or not this record will first pass through the combiner.

Shuffle/Reduce Parameters

As described previously, each reduce fetches the output assigned to it by the Partitioner via HTTP into memory and periodically merges these outputs to disk. If intermediate compression of map outputs is turned on, each output is decompressed into memory. The following options affect the frequency of these merges to disk prior to the reduce and the memory allocated to map output during the reduce.

Other notes

• If a map output is larger than 25 percent of the memory allocated to copying map outputs, it will be written directly to disk without first staging through memory.
• When running with a combiner, the reasoning about high merge thresholds and large buffers may not hold. For merges started before all map outputs have been fetched, the combiner is run while spilling to disk. In some cases, one can obtain better reduce times by spending resources combining map outputs- making disk spills small and parallelizing spilling and fetching- rather than aggressively increasing buffer sizes.
• When merging in-memory map outputs to disk to begin the reduce, if an intermediate merge is necessary because there are segments to spill and at least mapreduce.task.io.sort.factor segments already on disk, the in-memory map outputs will be part of the intermediate merge.

Configured Parameters

The following properties are localized in the job configuration for each task's execution:

Note: During the execution of a streaming job, the names of the "mapreduce" parameters are transformed. The dots ( . ) become underscores ( _ ). For example, mapreduce.job.id becomes mapreduce_job_id and mapreduce.job.jar becomes mapreduce_job_jar. To get the values in a streaming job's mapper/reducer use the parameter names with the underscores.

Task Logs

The standard output (stdout) and error (stderr) streams and the syslog of the task are read by the NodeManager and logged to ${HADOOP_LOG_DIR}/userlogs.

Distributing Libraries

The DistributedCache can also be used to distribute both jars and native libraries for use in the map and/or reduce tasks. The child-jvm always has its current working directory added to the java.library.path and LD_LIBRARY_PATH. And hence the cached libraries can be loaded via System.loadLibrary or System.load. More details on how to load shared libraries through distributed cache are documented at Native Libraries.

Job Control

Users may need to chain MapReduce jobs to accomplish complex tasks which cannot be done via a single MapReduce job. This is fairly easy since the output of the job typically goes to distributed file-system, and the output, in turn, can be used as the input for the next job.

However, this also means that the onus on ensuring jobs are complete (success/failure) lies squarely on the clients. In such cases, the various job-control options are:

• Job.submit() : Submit the job to the cluster and return immediately.
• Job.waitForCompletion(boolean) : Submit the job to the cluster and wait for it to finish.

InputSplit

InputSplit represents the data to be processed by an individual Mapper.

Typically InputSplit presents a byte-oriented view of the input, and it is the responsibility of RecordReader to process and present a record-oriented view.

FileSplit is the default InputSplit. It sets mapreduce.map.input.file to the path of the input file for the logical split.

RecordReader

RecordReader reads <key, value> pairs from an InputSplit.

Typically the RecordReader converts the byte-oriented view of the input, provided by the InputSplit, and presents a record-oriented to the Mapper implementations for processing. RecordReader thus assumes the responsibility of processing record boundaries and presents the tasks with keys and values.

OutputCommitter

OutputCommitter describes the commit of task output for a MapReduce job.

The MapReduce framework relies on the OutputCommitter of the job to:

1. Setup the job during initialization. For example, create the temporary output directory for the job during the initialization of the job. Job setup is done by a separate task when the job is in PREP state and after initializing tasks. Once the setup task completes, the job will be moved to RUNNING state.
2. Cleanup the job after the job completion. For example, remove the temporary output directory after the job completion. Job cleanup is done by a separate task at the end of the job. Job is declared SUCCEDED/FAILED/KILLED after the cleanup task completes.
3. Setup the task temporary output. Task setup is done as part of the same task, during task initialization.
4. Check whether a task needs a commit. This is to avoid the commit procedure if a task does not need commit.
5. Commit of the task output. Once task is done, the task will commit it's output if required.
6. Discard the task commit. If the task has been failed/killed, the output will be cleaned-up. If task could not cleanup (in exception block), a separate task will be launched with same attempt-id to do the cleanup.

FileOutputCommitter is the default OutputCommitter. Job setup/cleanup tasks occupy map or reduce containers, whichever is available on the NodeManager. And JobCleanup task, TaskCleanup tasks and JobSetup task have the highest priority, and in that order.

Task Side-Effect Files

In some applications, component tasks need to create and/or write to side-files, which differ from the actual job-output files.

In such cases there could be issues with two instances of the same Mapper or Reducer running simultaneously (for example, speculative tasks) trying to open and/or write to the same file (path) on the FileSystem. Hence the application-writer will have to pick unique names per task-attempt (using the attemptid, say attempt_200709221812_0001_m_000000_0), not just per task.

To avoid these issues the MapReduce framework, when the OutputCommitter is FileOutputCommitter, maintains a special ${mapreduce.output.fileoutputformat.outputdir}/_temporary/_${taskid} sub-directory accessible via ${mapreduce.task.output.dir} for each task-attempt on the FileSystem where the output of the task-attempt is stored. On successful completion of the task-attempt, the files in the ${mapreduce.output.fileoutputformat.outputdir}/_temporary/_${taskid} (only) are promoted to ${mapreduce.output.fileoutputformat.outputdir}. Of course, the framework discards the sub-directory of unsuccessful task-attempts. This process is completely transparent to the application.

The application-writer can take advantage of this feature by creating any side-files required in ${mapreduce.task.output.dir} during execution of a task via FileOutputFormat.getWorkOutputPath(Conext), and the framework will promote them similarly for succesful task-attempts, thus eliminating the need to pick unique paths per task-attempt.

Note: The value of ${mapreduce.task.output.dir} during execution of a particular task-attempt is actually ${mapreduce.output.fileoutputformat.outputdir}/_temporary/_{$taskid}, and this value is set by the MapReduce framework. So, just create any side-files in the path returned by FileOutputFormat.getWorkOutputPath(Conext) from MapReduce task to take advantage of this feature.

The entire discussion holds true for maps of jobs with reducer=NONE (i.e. 0 reduces) since output of the map, in that case, goes directly to HDFS.

RecordWriter

RecordWriter writes the output <key, value> pairs to an output file.

RecordWriter implementations write the job outputs to the FileSystem.

Submitting Jobs to Queues

Users submit jobs to Queues. Queues, as collection of jobs, allow the system to provide specific functionality. For example, queues use ACLs to control which users who can submit jobs to them. Queues are expected to be primarily used by Hadoop Schedulers.

Hadoop comes configured with a single mandatory queue, called 'default'. Queue names are defined in the mapreduce.job.queuename> property of the Hadoop site configuration. Some job schedulers, such as the Capacity Scheduler, support multiple queues.

A job defines the queue it needs to be submitted to through the mapreduce.job.queuename property, or through the Configuration.set(MRJobConfig.QUEUE_NAME, String) API. Setting the queue name is optional. If a job is submitted without an associated queue name, it is submitted to the 'default' queue.

Counters

Counters represent global counters, defined either by the MapReduce framework or applications. Each Counter can be of any Enum type. Counters of a particular Enum are bunched into groups of type Counters.Group.

Applications can define arbitrary Counters (of type Enum) and update them via Counters.incrCounter(Enum, long) or Counters.incrCounter(String, String, long) in the map and/or reduce methods. These counters are then globally aggregated by the framework.

DistributedCache

DistributedCache distributes application-specific, large, read-only files efficiently.

DistributedCache is a facility provided by the MapReduce framework to cache files (text, archives, jars and so on) needed by applications.

Applications specify the files to be cached via urls (hdfs://) in the Job. The DistributedCache assumes that the files specified via hdfs:// urls are already present on the FileSystem.

The framework will copy the necessary files to the slave node before any tasks for the job are executed on that node. Its efficiency stems from the fact that the files are only copied once per job and the ability to cache archives which are un-archived on the slaves.

DistributedCache tracks the modification timestamps of the cached files. Clearly the cache files should not be modified by the application or externally while the job is executing.

DistributedCache can be used to distribute simple, read-only data/text files and more complex types such as archives and jars. Archives (zip, tar, tgz and tar.gz files) are un-archived at the slave nodes. Files have execution permissions set.

The files/archives can be distributed by setting the property mapreduce.job.cache.{files|archives}. If more than one file/archive has to be distributed, they can be added as comma separated paths. The properties can also be set by APIs Job.addCacheFile(URI)/ Job.addCacheArchive(URI) and Job.setCacheFiles(URI[])/ Job.setCacheArchives(URI[]) where URI is of the form hdfs://host:port/absolute-path#link-name. In Streaming, the files can be distributed through command line option -cacheFile/-cacheArchive.

The DistributedCache can also be used as a rudimentary software distribution mechanism for use in the map and/or reduce tasks. It can be used to distribute both jars and native libraries. The Job.addArchiveToClassPath(Path) or Job.addFileToClassPath(Path) api can be used to cache files/jars and also add them to the classpath of child-jvm. The same can be done by setting the configuration properties mapreduce.job.classpath.{files|archives}. Similarly the cached files that are symlinked into the working directory of the task can be used to distribute native libraries and load them.

Profiling

Profiling is a utility to get a representative (2 or 3) sample of built-in java profiler for a sample of maps and reduces.

User can specify whether the system should collect profiler information for some of the tasks in the job by setting the configuration property mapreduce.task.profile. The value can be set using the api Configuration.set(MRJobConfig.TASK_PROFILE, boolean). If the value is set true, the task profiling is enabled. The profiler information is stored in the user log directory. By default, profiling is not enabled for the job.

Once user configures that profiling is needed, she/he can use the configuration property mapreduce.task.profile.{maps|reduces} to set the ranges of MapReduce tasks to profile. The value can be set using the api Configuration.set(MRJobConfig.NUM_{MAP|REDUCE}_PROFILES, String). By default, the specified range is 0-2.

User can also specify the profiler configuration arguments by setting the configuration property mapreduce.task.profile.params. The value can be specified using the api Configuration.set(MRJobConfig.TASK_PROFILE_PARAMS, String). If the string contains a %s, it will be replaced with the name of the profiling output file when the task runs. These parameters are passed to the task child JVM on the command line. The default value for the profiling parameters is -agentlib:hprof=cpu=samples,heap=sites,force=n,thread=y,verbose=n,file=%s.

Debugging

The MapReduce framework provides a facility to run user-provided scripts for debugging. When a MapReduce task fails, a user can run a debug script, to process task logs for example. The script is given access to the task's stdout and stderr outputs, syslog and jobconf. The output from the debug script's stdout and stderr is displayed on the console diagnostics and also as part of the job UI.

In the following sections we discuss how to submit a debug script with a job. The script file needs to be distributed and submitted to the framework.

Data Compression

Hadoop MapReduce provides facilities for the application-writer to specify compression for both intermediate map-outputs and the job-outputs i.e. output of the reduces. It also comes bundled with CompressionCodec implementation for the zlib compression algorithm. The gzip, bzip2, snappy, and lz4 file format are also supported.

Hadoop also provides native implementations of the above compression codecs for reasons of both performance (zlib) and non-availability of Java libraries. More details on their usage and availability are available here.

Skipping Bad Records

Hadoop provides an option where a certain set of bad input records can be skipped when processing map inputs. Applications can control this feature through the SkipBadRecords class.

This feature can be used when map tasks crash deterministically on certain input. This usually happens due to bugs in the map function. Usually, the user would have to fix these bugs. This is, however, not possible sometimes. The bug may be in third party libraries, for example, for which the source code is not available. In such cases, the task never completes successfully even after multiple attempts, and the job fails. With this feature, only a small portion of data surrounding the bad records is lost, which may be acceptable for some applications (those performing statistical analysis on very large data, for example).

By default this feature is disabled. For enabling it, refer to SkipBadRecords.setMapperMaxSkipRecords(Configuration, long) and SkipBadRecords.setReducerMaxSkipGroups(Configuration, long).

With this feature enabled, the framework gets into 'skipping mode' after a certain number of map failures. For more details, see SkipBadRecords.setAttemptsToStartSkipping(Configuration, int). In 'skipping mode', map tasks maintain the range of records being processed. To do this, the framework relies on the processed record counter. See SkipBadRecords.COUNTER_MAP_PROCESSED_RECORDS and SkipBadRecords.COUNTER_REDUCE_PROCESSED_GROUPS. This counter enables the framework to know how many records have been processed successfully, and hence, what record range caused a task to crash. On further attempts, this range of records is skipped.

The number of records skipped depends on how frequently the processed record counter is incremented by the application. It is recommended that this counter be incremented after every record is processed. This may not be possible in some applications that typically batch their processing. In such cases, the framework may skip additional records surrounding the bad record. Users can control the number of skipped records through SkipBadRecords.setMapperMaxSkipRecords(Configuration, long) andSkipBadRecords.setReducerMaxSkipGroups(Configuration, long). The framework tries to narrow the range of skipped records using a binary search-like approach. The skipped range is divided into two halves and only one half gets executed. On subsequent failures, the framework figures out which half contains bad records. A task will be re-executed till the acceptable skipped value is met or all task attempts are exhausted. To increase the number of task attempts, use Job.setMaxMapAttempts(int) and Job.setMaxReduceAttempts(int)

Skipped records are written to HDFS in the sequence file format, for later analysis. The location can be changed through SkipBadRecords.setSkipOutputPath(JobConf, Path).

Source Code

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.net.URI;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.Counter;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.util.StringUtils;

public class WordCount2 {

  public static class TokenizerMapper
       extends Mapper<Object, Text, Text, IntWritable>{

    static enum CountersEnum { INPUT_WORDS }

    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    private boolean caseSensitive;
    private Set<String> patternsToSkip = new HashSet<String>();

    private Configuration conf;
    private BufferedReader fis;

    @Override
    public void setup(Context context) throws IOException,
        InterruptedException {
      conf = context.getConfiguration();
      caseSensitive = conf.getBoolean("wordcount.case.sensitive", true);
      if (conf.getBoolean("wordcount.skip.patterns", true)) {
        URI[] patternsURIs = Job.getInstance(conf).getCacheFiles();
        for (URI patternsURI : patternsURIs) {
          Path patternsPath = new Path(patternsURI.getPath());
          String patternsFileName = patternsPath.getName().toString();
          parseSkipFile(patternsFileName);
        }
      }
    }

    private void parseSkipFile(String fileName) {
      try {
        fis = new BufferedReader(new FileReader(fileName));
        String pattern = null;
        while ((pattern = fis.readLine()) != null) {
          patternsToSkip.add(pattern);
        }
      } catch (IOException ioe) {
        System.err.println("Caught exception while parsing the cached file '"
            + StringUtils.stringifyException(ioe));
      }
    }

    @Override
    public void map(Object key, Text value, Context context
                    ) throws IOException, InterruptedException {
      String line = (caseSensitive) ?
          value.toString() : value.toString().toLowerCase();
      for (String pattern : patternsToSkip) {
        line = line.replaceAll(pattern, "");
      }
      StringTokenizer itr = new StringTokenizer(line);
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        context.write(word, one);
        Counter counter = context.getCounter(CountersEnum.class.getName(),
            CountersEnum.INPUT_WORDS.toString());
        counter.increment(1);
      }
    }
  }

  public static class IntSumReducer
       extends Reducer<Text,IntWritable,Text,IntWritable> {
    private IntWritable result = new IntWritable();

    public void reduce(Text key, Iterable<IntWritable> values,
                       Context context
                       ) throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable val : values) {
        sum += val.get();
      }
      result.set(sum);
      context.write(key, result);
    }
  }

  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    GenericOptionsParser optionParser = new GenericOptionsParser(conf, args);
    String[] remainingArgs = optionParser.getRemainingArgs();
    if (!(remainingArgs.length != 2 || remainingArgs.length != 4)) {
      System.err.println("Usage: wordcount <in> <out> [-skip skipPatternFile]");
      System.exit(2);
    }
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount2.class);
    job.setMapperClass(TokenizerMapper.class);
    job.setCombinerClass(IntSumReducer.class);
    job.setReducerClass(IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    List<String> otherArgs = new ArrayList<String>();
    for (int i=0; i < remainingArgs.length; ++i) {
      if ("-skip".equals(remainingArgs[i])) {
        job.addCacheFile(new Path(remainingArgs[++i]).toUri());
        job.getConfiguration().setBoolean("wordcount.skip.patterns", true);
      } else {
        otherArgs.add(remainingArgs[i]);
      }
    }
    FileInputFormat.addInputPath(job, new Path(otherArgs.get(0)));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs.get(1)));

    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

Sample Runs

Sample text-files as input:

$ bin/hdfs dfs -ls /user/joe/wordcount/input/ 
/user/joe/wordcount/input/file01 
/user/joe/wordcount/input/file02 

$ bin/hdfs dfs -cat /user/joe/wordcount/input/file01 
Hello World, Bye World! 

$ bin/hdfs dfs -cat /user/joe/wordcount/input/file02 
Hello Hadoop, Goodbye to hadoop.

Run the application:

$ bin/hadoop jar wc.jar WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output

Output:

$ bin/hdfs dfs -cat /user/joe/wordcount/output/part-r-00000 
Bye 1 
Goodbye 1 
Hadoop, 1 
Hello 2 
World! 1 
World, 1 
hadoop. 1 
to 1

Notice that the inputs differ from the first version we looked at, and how they affect the outputs.

Now, lets plug-in a pattern-file which lists the word-patterns to be ignored, via the DistributedCache.

$ bin/hdfs dfs -cat /user/joe/wordcount/patterns.txt 
\. 
\, 
\! 
to

Run it again, this time with more options:

$ bin/hadoop jar wc.jar WordCount2 -Dwordcount.case.sensitive=true /user/joe/wordcount/input /user/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt

As expected, the output:

$ bin/hdfs dfs -cat /user/joe/wordcount/output/part-r-00000 
Bye 1 
Goodbye 1 
Hadoop 1 
Hello 2 
World 2 
hadoop 1

Run it once more, this time switch-off case-sensitivity:

$ bin/hadoop jar wc.jar WordCount2 -Dwordcount.case.sensitive=false /user/joe/wordcount/input /user/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt

Sure enough, the output:

$ bin/hdfs dfs -cat /user/joe/wordcount/output/part-r-00000 
bye 1 
goodbye 1 
hadoop 2 
hello 2 
horld 2

Highlights

The second version of WordCount improves upon the previous one by using some features offered by the MapReduce framework:

• Demonstrates how applications can access configuration parameters in the setup method of the Mapper (and Reducer) implementations.
• Demonstrates how the DistributedCache can be used to distribute read-only data needed by the jobs. Here it allows the user to specify word-patterns to skip while counting.
• Demonstrates the utility of the GenericOptionsParser to handle generic Hadoop command-line options.
• Demonstrates how applications can use Counters and how they can set application-specific status information passed to the map (and reduce) method.

Java and JNI are trademarks or registered trademarks of Oracle America, Inc. in the United States and other countries.

MapReduce 튜토리얼

• MapReduce 튜토리얼
  • 목적
  • 선결 요건
  • 개요
  • 입력 및 출력
  • 예 : WordCount v1.0
    • 소스 코드
    • 용법
    • 워크 쓰루
  • MapReduce - 사용자 인터페이스
    • 유효 탑재량
      • 매퍼
      • 감속기
      • 분할 자
      • 계수기
    • 작업 구성
    • 작업 실행 및 환경
      • 메모리 관리
      • 지도 매개 변수
      • 셔플 / 축소 매개 변수
      • 구성된 매개 변수
      • 작업 로그
      • 라이브러리 배포
    • 작업 제출 및 모니터링
      • 작업 제어
    • 작업 입력
      • InputSplit
      • RecordReader
    • 작업 출력
      • OutputCommitter
      • 작업 부작용 파일
      • RecordWriter
    • 기타 유용한 기능
      • 대기열에 작업 제출
      • 카운터
      • DistributedCache
      • 프로파일 링
      • 디버깅
      • 데이터 압축
      • 잘못된 레코드 건너 뜀
    • 예 : WordCount v2.0
      • 소스 코드
      • 샘플 실행
      • 하이라이트

import java.io.IOException; 
import java.util.StringTokenizer; 

import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.IntWritable; 
import org.apache.hadoop.io.Text; 
import org.apache.hadoop.mapreduce.Job; 
import org.apache.hadoop.mapreduce.Mapper; 
import org.apache.hadoop.mapreduce.Reducer; 
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 

공용 클래스 WordCount { 

  public static class TokenizerMapper는 
       매퍼를 확장합니다. <Object, Text, Text, IntWritable> { 

    개인 최종 정적 IntWritable one = 새 IntWritable (1);
    개인 텍스트 단어 = 새 텍스트 (); 

    public void map (Object key, Text value, Context context 
                    ) throws IOException, InterruptedException { 
      StringTokenizer itr = new StringTokenizer (value.toString ()); 
      while (itr.hasMoreTokens ()) { 
        word.set (itr.nextToken ()); 
        context.write (word, one); 
      } 
    } 
  } 

  public static class IntSumReducer는 
       Reducer <Text, IntWritable, Text, IntWritable>을 확장합니다. { 
    private IntWritable result = new IntWritable (); 

    public void reduce (텍스트 키, Iterable <IntWritable> 값, 
                       컨텍스트 컨텍스트
                       ) IOException을 던집니다, InterruptedException { 
      int sum = 0; 
      for (IntWritable val : 값) { 
        sum + = val.get (); 
      } 
      result.set (합계); 
      context.write (key, result); 
    } 
  } 

  public static void main (String [] args) 예외를 throw합니다. { 
    Configuration conf = new Configuration (); 
    작업 작업 = Job.getInstance (conf, "word count"); 
    job.setJarByClass (WordCount.class); 
    job.setMapperClass (TokenizerMapper.class); 
    job.setCombinerClass (IntSumReducer.class); 
    job.setReducerClass (IntSumReducer.class); 
    job.setOutputKeyClass (Text.class);
    job.setOutputValueClass (IntWritable.class); 
    FileInputFormat.addInputPath (job, newPath (args [0])); 
    FileOutputFormat.setOutputPath (job, 새 경로 (args [1])); 
    System.exit (job.waitForCompletion (true)? 0 : 1); 
  } 
}

export JAVA_HOME = / usr / java / default 
export PATH = $ JAVA_HOME / bin : $ PATH 
export HADOOP_CLASSPATH = $ JAVA_HOME / lib / tools.jar

    public void map (Object key, Text value, Context context 
                    ) throws IOException, InterruptedException { 
      StringTokenizer itr = new StringTokenizer (value.toString ()); 
      while (itr.hasMoreTokens ()) { 
        word.set (itr.nextToken ()); 
        context.write (word, one); 
      } 
    }

    job.setCombinerClass (IntSumReducer.class);

    public void reduce (텍스트 키, Iterable <IntWritable> 값, 
                       컨텍스트 컨텍스트 
                       ) throw IOException, InterruptedException { 
      int sum = 0; 
      for (IntWritable val : 값) { 
        sum + = val.get (); 
      } 
      result.set (합계); 
      context.write (key, result); 
    }

<property> 
  <name> mapreduce.map.java.opts </ name> 
  <value> 
    -Xmx512M -Djava.library.path = / home / mycompany / lib -verbose : gc -Xloggc : /tmp/@taskid@.gc 
    -dcom.sun.management.jmxremote.authenticate = false -Dcom.sun.management.jmxremote.ssl = false 
  </ 
property> 

<property> 
  <name> mapreduce.reduce.java.opts </ name> 
  < / 값> 
    -Xmx1024M -Djava.library.path = / home / mycompany / lib -verbose : gc -Xloggc : /tmp/@taskid@.gc 
    -Dcom.sun.management.jmxremote.authenticate = false -Dcom.sun.management .jmxremote.ssl = false 
  </ value> 
</ property>

import java.io.BufferedReader; 
import java.io.FileReader; 
import java.io.IOException; 
import java.net.URI; 
import java.util.ArrayList; 
import java.util.HashSet; 
import java.util.List; 
import java.util.Set; 
import java.util.StringTokenizer; 

import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.IntWritable; 
import org.apache.hadoop.io.Text; 
import org.apache.hadoop.mapreduce.Job; 
import org.apache.hadoop.mapreduce.Mapper; 
import org.apache.hadoop.mapreduce.Reducer; 
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 
import org.apache.hadoop.mapreduce.Counter; 
org.apache.hadoop.util.GenericOptionsParser 가져 오기; 
import org.apache.hadoop.util.StringUtils; 

공용 클래스 WordCount2 { 

  public static class TokenizerMapper는 
       Mapper <Object, Text, Text, IntWritable>를 확장합니다. { 

    static enum CountersEnum {INPUT_WORDS} 

    개인 최종 정적 IntWritable 하나 = 새 IntWritable (1); 
    개인 텍스트 단어 = 새 텍스트 (); 

    private 부울 caseSensitive; 
    개인 집합 <String> patternsToSkip = 새 HashSet <String> (); 

    개인 구성 conf; 
    private BufferedReader fis; 

    @보수
    공용 무효 설치 (컨텍스트 컨텍스트) IOException, 
        InterruptedException { 
      conf = context.getConfiguration () throw합니다 . 
      caseSensitive = conf.getBoolean ( "wordcount.case.sensitive", true); 
      if (conf.getBoolean ( "wordcount.skip.patterns", true)) { 
        URI [] patternsURIs = Job.getInstance (conf) .getCacheFiles (); 
        for (URI 패턴 URI : patternsURIs) { 
          Path patternsPath = 새 경로 (patternsURI.getPath ()); 
          String patternsFileName = patternsPath.getName (). toString (); 
          parseSkipFile (patternsFileName); 
        } 
      } 
    } 

    private void parseSkipFile (String fileName) { 
      try {
        fis = new BufferedReader (새 FileReader (fileName)); 
        문자열 패턴 = null; 
        while ((pattern = fis.readLine ())! = null) { 
          patternsToSkip.add (pattern); 
        } 
      } catch (IOException ioe) { 
        System.err.println ( "캐싱 된 파일을 구문 분석하는 동안 예외가 발생했습니다." 
            + StringUtils.stringifyException (ioe)); 
      } 
    } 

    @Override 
    public void map (Object key, Text value, Context context 
                    ) IOException, InterruptedException { 
      String line = (caseSensitive)? 
          value.toString () : value.toString (). toLowerCase ();
      for (String pattern : patternsToSkip) { 
        line = line.replaceAll (pattern, ""); 
      } 
      StringTokenizer itr = new StringTokenizer (line); 
      while (itr.hasMoreTokens ()) { 
        word.set (itr.nextToken ()); 
        context.write (word, one); 
        카운터 카운터 = context.getCounter (CountersEnum.class.getName (), 
            CountersEnum.INPUT_WORDS.toString ()); 
        counter.increment (1); 
      } 
    } 
  } 

  public static class IntSumReducer는 
       Reducer <Text, IntWritable, Text, IntWritable>을 확장합니다. { 
    private IntWritable result = new IntWritable ();

    public void reduce (텍스트 키, Iterable <IntWritable> 값, 
                       컨텍스트 컨텍스트 
                       ) throw IOException, InterruptedException { 
      int sum = 0; 
      for (IntWritable val : 값) { 
        sum + = val.get (); 
      } 
      result.set (합계); 
      context.write (key, result); 
    } 
  } 

  public static void main (String [] args) 예외를 throw합니다. { 
    Configuration conf = new Configuration (); 
    GenericOptionsParser optionParser = 새로운 GenericOptionsParser (conf, args); 
    String [] remainingArgs = optionParser.getRemainingArgs (); 
    if (! (remainingArgs.length! = 2 || remainingArgs.length! = 4)) {
      System.err.println ( "Usage : wordcount <in> <out> [-skip skipPatternFile]"); 
      System.exit (2); 
    } 
    Job job = Job.getInstance (conf, "word count"); 
    job.setJarByClass (WordCount2.class); 
    job.setMapperClass (TokenizerMapper.class); 
    job.setCombinerClass (IntSumReducer.class); 
    job.setReducerClass (IntSumReducer.class); 
    job.setOutputKeyClass (Text.class); 
    job.setOutputValueClass (IntWritable.class); 

    목록 <String> otherArgs = 새 ArrayList <String> (); 
        (새 경로 (remainingArgs [++ i]). toUri 
    (int i = 0; i <remainingArgs.length; ++ i) { 
      if ( "-skip".equals (remainingArgs [i])) { job.addCacheFile ());
        job.getConfiguration (). setBoolean ( "wordcount.skip.patterns", true); 
      } else { 
        otherArgs.add (remainingArgs [i]); 
      } 
    } 
    FileInputFormat.addInputPath (job, newPath (otherArgs.get (0))); 
    FileOutputFormat.setOutputPath (작업, 새 경로 (otherArgs.get (1))); 

    System.exit (job.waitForCompletion (true)? 0 : 1); 
  } 
}

지원 플랫폼

• GNU / Linux는 개발 및 생산 플랫폼으로 지원됩니다. Hadoop은 2000 노드를 가진 GNU / Linux 클러스터에서 시연되었습니다.
• Windows도 지원되는 플랫폼이지만 다음 단계는 Linux에만 해당됩니다. Windows에서 Hadoop을 설정하려면 wiki 페이지를 참조 하십시오 .

필수 소프트웨어

Linux에 필요한 소프트웨어는 다음과 같습니다.

1. Java ™가 설치되어 있어야합니다. 권장 Java 버전은 HadoopJavaVersions에 설명되어 있습니다 .
2. ssh가 설치되어 있어야하며 원격 Hadoop 데몬을 관리하는 Hadoop 스크립트를 사용하려면 sshd가 실행 중이어야합니다.

소프트웨어 설치

클러스터에 필수 소프트웨어가 없다면 설치해야합니다.

예를 들어 Ubuntu Linux의 경우 :

  $ sudo apt-get install ssh 
  $ sudo apt-get install rsync

구성

다음을 사용하십시오.

etc / hadoop / core-site.xml :

<configuration> 
    <property> 
        <name> fs.defaultFS </ name> 
        <value> hdfs : // localhost : 9000 </ value> 
    </ property> 
</ configuration>

etc / hadoop / hdfs-site.xml :

<configuration> 
    <property> 
        <name> dfs.replication </ name> 
        <value> 1 </ value> 
    </ property> 
</ configuration>

설치 passphraseless ssh

이제 passphrase없이 localhost로 ssh 할 수 있는지 확인하십시오 :

  $ ssh localhost

passphrase없이 localhost로 ssh 할 수 없다면 다음 명령을 실행하십시오 :

  $ ssh-keygen -t dsa -P "-f ~ / .ssh / id_dsa 
  $ cat ~ / .ssh / id_dsa.pub >> ~ / .ssh / authorized_keys

실행

다음 지시 사항은 MapReduce 작업을 로컬에서 실행하는 것입니다. YARN에서 작업을 실행하려면 단일 노드의 YARN을 참조하십시오 .

1. 파일 시스템을 포맷하십시오 :

     $ bin / hdfs namenode -format

2. NameNode 데몬 및 DataNode 데몬 시작 :

     $ sbin / start-dfs.sh

   hadoop 데몬 로그 출력은 $ HADOOP_LOG_DIR 디렉토리에 기록됩니다 (기본값은 $ HADOOP_HOME / logs입니다 ).

3. NameNode에 대한 웹 인터페이스를 탐색하십시오. 기본적으로 다음 위치에서 사용할 수 있습니다.
   • NameNode - http : // localhost : 50070 /
4. MapReduce 작업을 실행하는 데 필요한 HDFS 디렉토리를 만듭니다.

     $ bin / hdfs dfs -mkdir / 사용자 
     $ bin / hdfs dfs -mkdir / user / <username>

5. 입력 파일을 분산 파일 시스템에 복사하십시오.

     $ bin / hdfs dfs -put etc / hadoop 입력

6. 제공되는 몇 가지 예제를 실행하십시오.

     $ bin / hadoop jar share / hadoop / mapreduce / hadoop-mapreduce-examples-2.5.2.jar grep 입력 출력 'dfs [az.] +'

7. 출력 파일을 검사하십시오.

   출력 파일을 분산 파일 시스템에서 로컬 파일 시스템으로 복사하고 검사하십시오 :

     $ bin / hdfs dfs -get 출력 출력 
     $ cat 출력 / *

   또는

   분산 파일 시스템의 출력 파일보기 :

     $ bin / hdfs dfs -cat 출력 / *

8. 완료되면 데몬을 중지하십시오.

     $ sbin / stop-dfs.sh

단일 노드의 YARN

몇 개의 매개 변수를 설정하고 ResourceManager 데몬과 NodeManager 데몬을 추가로 실행하여 의사 배포 모드에서 YARN의 MapReduce 작업을 실행할 수 있습니다.

다음 명령어 는 위의 명령어 의 1 ~ 4. 단계 가 이미 실행 되었다고 가정합니다 .

1. 다음과 같이 매개 변수를 구성하십시오.

   etc / hadoop / mapred-site.xml :

   <configuration> 
       <property> 
           <name> mapreduce.framework.name </ name> 
           <value> 원사 </ value> 
       </ property> 
   </ configuration>

   etc / hadoop / yarn-site.xml :

   <configuration> 
       <property> 
           <name> yarn.nodemanager.aux-services </ name> 
           <value> mapreduce_shuffle </ value> 
       </ property> 
   </ configuration>

2. ResourceManager 데몬 및 NodeManager 데몬 시작 :

     $ sbin / start-yarn.sh

3. ResourceManager의 웹 인터페이스를 탐색하십시오. 기본적으로 다음 위치에서 사용할 수 있습니다.
   • ResourceManager - http : // localhost : 8088 /
4. MapReduce 작업을 실행하십시오.
5. 완료되면 데몬을 중지하십시오.

     $ sbin / stop-yarn.sh