Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

CK CK 3434 
Created at  
2,079 0 0
Hadoop MapReduce is a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes.

In other words, Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.

Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.

Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System (see below HDFS Architecture Diagram) are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster.

Hadoop MapReduce is a software framework for processing vast amounts of data in-parallel on large clusters

The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs' component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.

Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTracker which then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.

Although the Hadoop framework is implemented in Java, MapReduce applications need not be written in Java.

Hadoop Streaming is a utility which allows users to create and run jobs with any executables (e.g. shell utilities) as the mapper and/or the reducer.
Hadoop Pipes is a SWIG- compatible C++ API to implement MapReduce applications (non JNI based).

 

References

- http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Inputs+and+Outputs
Tags Blob Chunk Cloud Cloud Computing Computers & Internet File-System HBase HDFS Hadoop Distributed File System Hadoop Mascot Hadoop Pipe Hadoop Streaming JNI Java JobTracker MapReduce NoSQL Parallel Processing SWIG TaskTracker Facebook X
Comments 0
Similar posts
  1. Apache Hadoop is designed to scale up from single servers to thousands of machines
    1,880
  2. HDFS(Hadoop Distributed File System) is designed to run on commodity hardware – Low cost hardware
    1,926
  3. Apache HBase is a storage system, with roots in Hadoop, and uses HDFS for underlying storage.
    1,914
  4. Hive provides SQL-like query language on HDFS(Hadoop Distributed File System)
    1,922
  5. With AWS cloud,we met our reliability and performance objectives at a fraction of the cost – Mr. Chun Kang – Pricipal Engineer, Samsung
    2,347
  6. Web Cache function in Network Gateway could cause internet service trouble
    1,692
  7. Samsung Movies and TV Shows User Interface
    2,019
  8. FNC Batch Renames Digital Photos by Date
    1,922
  1. Discovering OpenBugBounty: A Free Tool for Website Security Insights
    2,323
  2. Microsoft and Qualcomm Collaborate to Bring On-Device AI to Windows Devices
    2,260
  3. Flash Memory has lifetime, so you have to consider backup just in case
    4,126
  4. The way can see the picture on the received email in outlook
    3,571
  5. If you want to copy clipboard image on livewriter as local image, use Clipboard Capture or Clipboard Live – Live Writer Plug-in
    1,837
  6. How to add groups to active resistered user in phpBB ?
    4,000
  7. The simplest and most reliable application or method for editing digital photo EXIF data
    1,833
  8. Enhancing Online Privacy - Special Tools for Hiding Your IP Address
    2,192
  9. Anonymous Proxy Servers for hiding IP address
    2,104
  10. Online Gaming IP Ban Proxy Servers Internet Security Digital Identity
    2,193
  11. Simple Steps to Mask Your IP Address on Home Networks
    4,515
  12. Hiding Your IP Address Made Easy: Using HTTP Proxy Servers
    2,599
  13. Changing Your IP Address: Easy Methods for Privacy Protection
    2,589
Recently updated
  1. Navigating App Store Guideline 4.3(a): Key Pitfalls to Avoid and How It Differs from 4.3(b)
    33
  2. Apple App Store Review Guideline Update: Section 4.3(b) Spam Provision
    40
  3. US Tax Strategies for Digital Service Operators & AdSense Earners
    203
  4. Guide to Updating Company Legal Name on D-U-N-S Profile (AAA to BBB)
    389
  5. Health Insurance Options and Monthly Costs for a Family of Four in the US (Without Employer Coverage)
    985
  6. Essential US Business Setup: Automated Payroll, Benefits, Tax Services, and Monthly Costs
    207
  7. YouTube Language Usage in India: Comprehensive Statistics and Data Breakdown
    1,061
  8. India's Linguistic Landscape: Top Languages, Regional Distribution, and English Usage
    1,131
  9. Decoding Global TikTok Revenue: Monetization Trends, Market Tiers, and Social Commerce Growth
    1,286
  10. Countries Where Fortune-Telling and Amulets Are Strictly Illegal
    204
  11. Global YouTube Usage Analysis: Top 100 Countries by User Population
    1,030
  12. Meta Unveils Muse Charm: The Keychain AI Assistant That Could Replace Your Smartwatch
    1,367
  13. Why iCloud Passwords is useful?
    1,042
  14. Good Company to Work For, but Pretty Pathetic once you Look Inside
    2,146
  15. Pre-Departure Google Workspace Personal Data Cleanup
    3,538
  16. The Harmful Effects of Squeezing Blemishes: Why You Should Avoid It
    1,978
  17. Finding Your Path to Clear Skin: The Best Way to Treat Acne
    2,202
  18. Choosing the Right Cosmetics and Cleansers for Acne-Prone Skin
    2,336
  19. Why Does Vietnamese Cable TV Have Almost No Ads? A Look at the Rules
    3,539
  20. In 2026, Advertising Is Converging on the TV Screen - 5 Opportunities OTT Operators Should Be Watching
    4,641
  21. Musk's Masterclass: Prioritizing, Delegating & Verifying
    861
  22. Life is Short: HubSpot Founder's Cliff-Edge Wisdom for Leaders
    3,781
  23. The Making of OpenAI: Birth, Battles, and Breakthroughs
    948
  24. Brazil's Digital & Demographic Deep Dive: Economy & Top Streaming Services
    1,550
  25. World Population Top 30 (2016-2025)
    1,167
  26. SouthEast Asia Advertising Market - Comprehensive Analysis by Country, Company, Platform & Revenue
    2,001
  27. Comprehensive Report on Portugal: Demographics, Digital Usage, and Top Streaming Companies
    1,003
  28. Mongolia Unpacked: Demographics, Digital Economy, and Leading Streaming Services
    1,717
  29. Italy: A Comprehensive Guide to Demographics, Digital Trends, and Its Top 10 Video Streaming Companies
    998
  30. Spain: Demographics, Digital Usage, Economic Insights & Top 10 Streaming Services
    1,441
  31. France's Digital Economy: Demographics, Trends, and Streaming Leaders
    1,008
  32. Germany's Digital Deep Dive: Demographics, Internet Usage, Market Trends, and Top 10 Video Streaming Powerhouses
    1,025
  33. United Kingdom: Demographics, Digital Life, and Streaming's Top 10 Players
    913
  34. Russia's Digital Blueprint: Economy, Demographics, and Streaming Landscape
    1,161
  35. Singapore's Top 10 Video Streaming Companies
    1,088
  36. Canada's Digital Pulse: Demographics, Connectivity & Top 10 Streaming Services Unveiled
    999
  37. Korea's Streaming Story: Top 10 Platforms in a Digital Nation
    2,091
  38. Malaysia's Top 10 Video Streaming Companies 2026
    1,536
  39. Indonesia's Top 10 Video Streaming Companies
    1,608
  40. Japan's Top 10 Video Streaming Companies
    1,091
  41. United States's Top 10 Video Streaming Companies
    1,025
  42. The Evolving Role of Engineering Organizations in the AI Era
    3,408
  43. exif_filename - a Powerful Command-Line Tool Designed for Automatically Renaming Image Files based on EXIF metadata
    781
  44. The Correlation between Income Level and Television Viewing Time
    2,261
  45. Problem-Solving Narratives in Startup Programs
    822
  46. Vietnamese Top 10 Video Streaming Companies 2026
    2,052
  47. Thailand's Top 10 Video Streaming Companies 2026
    1,959
  48. Phillippine's Top 10 Video Streaming Companies 2026
    1,572