Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

CK404 
Created at Feb 28, 2012 09:00:46 
389   0   0   0  
Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes.

In other words, Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.

Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.

Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System (see below HDFS Architecture Diagram) are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster.

Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.

Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.

Although the Hadoop framework is implemented in Java, MapReduce applications need not be written in Java.

Hadoop Streaming is a utility which allows users to create and run jobs with any executables (e.g. shell utilities) as the mapper and/or the reducer.
Hadoop Pipes is a SWIG- compatible C++ API to implement MapReduce applications (non JNI based).

 

References

- http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Inputs+and+Outputs

Tags: Blob Chunk Cloud Cloud Computing Computers & Internet File-System HBase HDFS Hadoop Distributed File System Hadoop Mascot Hadoop Pipe Hadoop Streaming JNI Java JobTracker MapReduce NoSQL Parallel Processing SWIG TaskTracker Share on Facebook Share on X

◀ PREVIOUS
Apache Hadoop is designed to scale up from single servers to thousands of machines

▶ NEXT
HDFS(Hadoop Distributed File System) is designed to run on commodity hardware – Low cost hardware

  Comments 0
Login for comment
SIMILAR POSTS

Apache Hadoop is designed to scale up from single servers to thousands of machines

(created at Feb 28, 2012)

HDFS(Hadoop Distributed File System) is designed to run on commodity hardware – Low cost hardware

(created at Feb 28, 2012)

Apache HBase is a storage system, with roots in Hadoop, and uses HDFS for underlying storage.

(created at Mar 01, 2012)

Hive provides SQL-like query language on HDFS(Hadoop Distributed File System)

(created at Mar 14, 2012)

With AWS cloud,we met our reliability and performance objectives at a fraction of the cost – Mr. Chun Kang – Pricipal Engineer, Samsung

(updated at Apr 27, 2024)

If you want to copy clipboard image on livewriter as local image, use Clipboard Capture or Clipboard Live – Live Writer Plug-in

(created at Jul 25, 2012)

Web Cache function in Network Gateway could cause internet service trouble

(created at Oct 15, 2010)

Samsung Movies and TV Shows User Interface

(created at Aug 13, 2014)

FNC Batch Renames Digital Photos by Date

(created at Aug 21, 2009)

OTHER POSTS IN THE SAME CATEGORY

Microsoft and Qualcomm Collaborate to Bring On-Device AI to Windows Devices

(updated at May 22, 2024)

Flash Memory has lifetime, so you have to consider backup just in case

(updated at Apr 27, 2024)

Samsung Movies and TV Shows User Interface

(created at Aug 13, 2014)

The way can see the picture on the received email in outlook

(updated at Apr 27, 2024)

If you want to copy clipboard image on livewriter as local image, use Clipboard Capture or Clipboard Live – Live Writer Plug-in

(created at Jul 25, 2012)

With AWS cloud,we met our reliability and performance objectives at a fraction of the cost – Mr. Chun Kang – Pricipal Engineer, Samsung

(updated at Apr 27, 2024)

Hive provides SQL-like query language on HDFS(Hadoop Distributed File System)

(created at Mar 14, 2012)

Apache HBase is a storage system, with roots in Hadoop, and uses HDFS for underlying storage.

(created at Mar 01, 2012)

HDFS(Hadoop Distributed File System) is designed to run on commodity hardware – Low cost hardware

(created at Feb 28, 2012)

Apache Hadoop is designed to scale up from single servers to thousands of machines

(created at Feb 28, 2012)

Web Cache function in Network Gateway could cause internet service trouble

(created at Oct 15, 2010)

How to add groups to active resistered user in phpBB ?

(updated at Apr 27, 2024)

The simplest and most reliable application or method for editing digital photo EXIF data

(created at Aug 25, 2009)

FNC Batch Renames Digital Photos by Date

(created at Aug 21, 2009)

Enhancing Online Privacy - Special Tools for Hiding Your IP Address

(updated at Apr 28, 2024)

Anonymous Proxy Servers for hiding IP address

(updated at Apr 27, 2024)

Online Gaming IP Ban Proxy Servers Internet Security Digital Identity

(updated at Apr 27, 2024)

Simple Steps to Mask Your IP Address on Home Networks

(updated at Apr 27, 2024)

Hiding Your IP Address Made Easy: Using HTTP Proxy Servers

(updated at Apr 27, 2024)

Changing Your IP Address: Easy Methods for Privacy Protection

(updated at Apr 27, 2024)

UPDATES

REPORTBANK migration from Wordpress

(updated at Aug 13, 2024)

Why Maintenance is never ending in the Interenet-based Service?

(created at Jun 28, 2024)

Employer Requirements under California Labor Code Section 6401.9

(created at Jun 26, 2024)

Adding a Personal Touch - Embed Your Copyright into Your Nikon’s EXIF Data

(updated at May 31, 2024)

Microsoft and Qualcomm Collaborate to Bring On-Device AI to Windows Devices

(updated at May 22, 2024)

Crafting the Right Reasons for Leaving Your Job in Your Resume

(updated at May 16, 2024)

12 Proven Strategies to Work Smarter, Not Harder

(updated at May 16, 2024)

Evaluating Intangible Traits in Employee Performance Reviews

(updated at May 16, 2024)

The Example Turning a Negative into a Positive - How to Make Criticism Count?

(updated at May 16, 2024)

What is Sexual Harassment? It's Never Okay!

(updated at May 16, 2024)

The AI Revolution of 2024: Gemini vs. ChatGPT-4o

(updated at May 16, 2024)

To keep my mascara from rubbing off underneath my eyes throughout the day

(updated at May 15, 2024)

Understanding Cataracts

(updated at May 15, 2024)

Can cataract surgery be done with laser?

(updated at May 15, 2024)

About ReportBank

(updated at May 15, 2024)

Navigating the Realm of Copyright: Safeguarding Creativity in the Digital Age

(updated at May 15, 2024)

How to Tackle Post-Meal Sleepiness: Understanding Blood Sugar Spikes

(updated at May 13, 2024)

What is an Optician?

(updated at May 08, 2024)

Understanding the Link: Cirrhosis and Liver Cancer Risk

(updated at Apr 29, 2024)

Choosing Between Glucosamine and Chondroitin: Which Is Better for Joint Health?

(created at Apr 29, 2024)

Enhancing Online Privacy - Special Tools for Hiding Your IP Address

(updated at Apr 28, 2024)

How to Flatter with Words and Navigate Its Nuances

(updated at Apr 28, 2024)

Joan of Arc and Zandark

(updated at Apr 28, 2024)

France's Hundred Years' War

(updated at Apr 28, 2024)

Changing Your IP Address: Easy Methods for Privacy Protection

(updated at Apr 27, 2024)

Hiding Your IP Address Made Easy: Using HTTP Proxy Servers

(updated at Apr 27, 2024)

Simple Steps to Mask Your IP Address on Home Networks

(updated at Apr 27, 2024)

Online Gaming IP Ban Proxy Servers Internet Security Digital Identity

(updated at Apr 27, 2024)

The history of the excimer laser

(updated at Apr 27, 2024)

Anonymous Proxy Servers for hiding IP address

(updated at Apr 27, 2024)