mapreduce

mapreduce

mapreduce,What is MapReduce,Principles,Programming,Manual

Visit Site →

Official MapReduce website, What is MapReduce, principles, programming, manual

What is MapReduce?

MapReduce is a programming model and computational framework for processing and analyzing large datasets. Originally proposed by Google, it is widely used in the Apache Hadoop project.Hadoopproject.

Official MapReduce website:https://hadoop.apache.org/

Main Functions of MapReduce

The MapReduce framework provides automatic task parallelization, data partitioning, cross-node communication, and fault recovery. It can run on large clusters, utilizing the computing power and storage space of multiple computers for efficient distributed computing.

MapReduce is highly effective for processing large datasets, parallel computing, and distributed storage. It has become one of the important tools for handling big data, widely used in data analysis, search engines, log processing, and other fields.

One of the core components of Hadoop: the MapReduce solution for distributed computing, which is a programming model for parallel computation of large datasets, with Map (mapping) and Reduce (reduction).

MapReduce is both a programming model and a computing component, with the processing process divided into two stages: Map stage and Reduce stage. The Map stage is responsible for decomposing tasks into multiple small tasks, while the Reduce stage is responsible for summarizing the results of these tasks. The Map stage mainly takes a Key-Value pair as input, outputs a Key-Value pair after map computation, then merges the same Key into a Key-Value set, and finally transfers this Key-Value set to the Reduce stage for computation and outputs the final Key-Value result set.

MapReduce can achieve concurrent work on thousands of servers, providing very strong data processing capabilities. If a single server fails, the computing task will automatically be transferred to another node for execution, ensuring high fault tolerance; however, MapReduce is not suitable for real-time computing and streaming computing, as the data to be computed is static.

Official MapReduce manual

Overview

Hadoop MapReduce is a software framework that can easily create large clusters on commercial machines to process large amounts of data in a reliable and fault-tolerant manner.

A typical MapReduce job usually splits the input dataset into independent blocks. Map tasks process these blocks in a fully parallel manner. The framework sorts the output of the map and provides it as input to the reduce task. Typically, the input and output of the job are stored in a file system. The framework is responsible for scheduling tasks, monitoring tasks, and re-executing failed tasks.

Typically, computing nodes and storage nodes are the same, meaning that the MapReduce framework and HDFS run on the same set of nodes. This configuration ensures that the framework can effectively schedule tasks on nodes with existing data and achieve a very high total bandwidth between different clusters.

The MapReduce framework consists of a single master ResourceManager, a NodeManager on each cluster node, and an MRAppMaster for each application.

The application must specify the input/output locations and provide map and reduce functionality by implementing appropriate interfaces and abstract classes. These, along with other job parameters, make up the job configuration (configuration).

Then, the Hadoop job client submits the job (jar/executable files, etc.) and the configuration to the ResourceManager. The ResourceManager is responsible for distributing software/configuration, scheduling and supervising tasks, and providing status and diagnostic information to the job client.

Although the Hadoop framework is implemented in Java, MapReduce applications do not need to be written in Java.

Input and Output

The MapReduce framework operates completely in<key-value>form, meaning that the framework treats the input data given to the job as<key-value>pairs and produces a<key-value>pair set as the job's output.

The key and value classes must be serialized through the framework, so they need to implement the Writable interface. In addition, the key class must implement the WritableComparable interface to assist the framework in sorting.

The input and output types of a MapReduce job are as follows:

(input) <k1, v1>-> map -> <k2, v2>-> combine -> <k2, v2>-> reduce -> <k3, v3>(output)

Example: WordCount v1.0

Before delving into the details, let's look at a MapReduce application, which is very useful for helping us understand how it works.

WordCount is a simple application that calculates the number of occurrences of each word in an input set.

This example is applicable to local single-node, pseudo-distributed, and fully distributed Hadoop installations.

Source code:

  1. importjava.io.IOException;
  2. importjava.util.StringTokenizer;
  4. importorg.apache.hadoop.conf.Configuration;
  5. importorg.apache.hadoop.fs.Path;
  6. importorg.apache.hadoop.io.IntWritable;
  7. importorg.apache.hadoop.io.Text;
  8. importorg.apache.hadoop.mapreduce.Job;
  9. importorg.apache.hadoop.mapreduce.Mapper;
  10. importorg.apache.hadoop.mapreduce.Reducer;
  11. importorg.apache.hadoop.mapreduce.lib.input.FileInputFormat;