CS570 - Big Data Processing - MapReduce Programming Notes

CS570 - Big Data Processing - MapReduce Programming

Objectives

To learn how to use Apache Hadoop framework To understand MapReduce pattern To analyze big data application and apply MapReduce pattern To learn HBase and NoSql DB To learn Apache Mahout machine learning library

Schedule

Week# Topic 01 Introduction to UNIX/Linux - extra 01 Introduction to Java - extra 01 Introduction - extra 02~04 MapReduce Short Course . - extra 02 Develope MapReduce Program in a Virtual Box VM - extra 02 Amazon AWS - extra 02 Cloudera Hadoop - extra Objectives: (1) Overview of this course Instructive Coverage & Activities: (1) Intended Audience (2) Pattern Format (3) The Examples in This Book (4) Conventions Used in This Book (5) Using Code Examples (6) Safari Books Online (7) How to Contact Us (8) Acknowldgements Assignments: (1) Drawing tables to show MapReduce algorithms Quiz / Exam: 05 Preface 05 Design Patterns and MapReduce Objectives: (1) Overview of design patterns Instructive Coverage & Activities: (1) Design Patterns and MapReduce (2) Design Patterns (3) MapReduce History (4) MapReduce and Hadoop Refresher (5) Hadoop Example Word Count (6) Pig and Hive Assignments: (1) Reading course material Quiz / Exam: Quiz 06 Summarization Patterns Objectives: (1) Summarization Patterns Instructive Coverage & Activities: (1) Summarization Patterns (2) Numerical Summarizations (3) Pattern Description 1 (4) Intent 1 (5) Motivation 1 (6) Applicability 1 (7) Structure 1(8) Consequences 1 (9) Known uses 1 (10) Resemblances 1 (11) Performance analysis 1 (12) Numerical Summarization Examples (13) Minimum maximum and count example (14) MinMaxCountTuple code (15) Mapper code 1 (16) Reducer code 1 (17) Combiner optimization 1 (18) Data flow diagram (19) Average example (20) Mapper code 2 (21) Reducer code 2 (22) Combiner optimization 2 (23) Data flow diagram 2 (24) Median and standard deviation (25) Mapper code 3 (26) Reducer code 3 (27) Combiner optimization 3 (28) Memory-conscious median and standard deviation (29)Mapper code 4 (30) Reducer code 4 (31) Combiner optimization 4 (32) Data flow diagram 4 (33) Inverted Index Summarizations (34) Pattern Description 5 (35) Intent 5 (36) Motivation 5 (37) Applicability 5 (38) Structure 5 (39) Consequences 5 (40) Performance analysis 5 (41) Inverted Index Example (42) Wikipedia reference inverted index (43) Mapper code 6 (44) Reducer code 6 (45) Combiner optimization 6 (46) Counting with Counters (47) Pattern Description 6 (48) Intent 6 (49) Motivation 6 (50) Applicability 6 (51) Structure 6 (52) Consequences 6 (53) Known uses 6 (54)Performance analysis 6 (55) Counting with Counters Example (56) Number of users per state (57) Mapper code 7 (58) Driver code 7 Assignments: (1) Inverted Index Quiz / Exam: Quiz 07 Filtering Patterns Midterm Exam Objectives: (1) Filtering Patterns Instructive Coverage & Activities: (1) Filtering (2) Pattern Description 1 (3) Intent 1 (4) Motivation 1 (5) Applicability 1 (6) Structure 1 (7) Consequences 1 (8) Known uses 1 (9)Resemblances 1 (10) Performance analysis 1 (11) Filtering Examples (12) Distributed grep (13) Mapper code 2 (14) Simple Random Sampling 2 (15)Mapper Code 3 (16) Bloom Filtering (17) Pattern Description 3 (18) Intent 3 (19) Motivation 3 (20) Applicability 3 (21) Structure 3 (22)Consequences 3 (23) Known uses 3 (24) Resemblances 3 (25) Performance analysis 3 (26) Bloom Filtering Examples (27) Hot list (28) Mapper code 4(29) HBase Query using a Bloom filter (30) Mapper Code 5 (31) Top Ten (32) Pattern Description (33) Intent 6 (34) Motivation 6 (35) Applicability 6(36) Structure 6 (37) Consequences 6 (38) Known uses 6 (39) Resemblances 6 (40) Performance analysis 6 (41) Top Ten Examples (42) Top ten users by reputation (43) Mapper code 7 (44) Reducer code 7 (45) Distinct (46) Pattern Description (47) Intent 7 (48) Motivation 7 (49) Applicability 7 (50)Structure 7 (51) Consequences 7 (52) Know uses 7 (53) Getting distinct values 7 (54) Protecting from an inner join explosion 7 (55) Resemblances 7(56) Performance analysis 7 (57) Distinct Examples (58) Distinct user IDs (59) Mapper code 8 (60) Reducer code 8 Assignments: (1) Distinct Design Pattern Quiz / Exam: Quiz 07 Bloom Filters Objectives: (1) Bloom Filters Instructive Coverage & Activities: (1) Use Cases (2) Representing a Data Set (3) Reduce Queries to External Database (4) Google BigTable (5) Downsides (6) Tweaking Your Bloom Filter Assignments: (1) Exercise Quiz / Exam: None 08~09 Data Organization Patterns Objectives: (1) Data Organization Patterns Instructive Coverage & Activities: (1) Structured to Hierarchical (2) Pattern Description 1 (3) Intent 1 (4) Motivation 1 (5) Applicability 1 (6) Structure 1 (7) Consequences 1 (8) Known uses 1 (9) Resemblances 1 (10) Performance analysis 1 (11) Structured to Hierarchical Examples (12) Post comment building on StackOverflow (13)Mapper code 2 (14) Reducer code 2 (15) Question_answer building on StackOverflow (16) Mapper code 3 (17) Reducer code 3 (18) Partitioning (19)Pattern Description 4 (20) Intent 4 (21) Motivation 4 (22) Applicability 4 (23) Structure 4 (24) Consequences 4 (25) Known uses 4 (26) Performance analysis 4 (27) Partitioning Examples (28) Partitioning users by last access date (29) Mapper code 5 (30) Partitioner code 5 (31) Reducer code 5 (32)Binning (33) Pattern Description (34) Intent 6 (35) Motivation 6 (36) Structure 6 (37) Consequences 6 (38) Resemblances 6 (39) Performance analysis 6 (40) Binning Examples (41) Binning by Hadoop-related tags (42) Mapper code 7 (43) Total Order Sorting (44) Pattern Description 8 (45) Intent 8(46) Motivation 8 (47) Applicability 8 (48) Structure 8 (49) Consequences 8 (50) Resemblances 8 (51) Performance analysis 8 (52) Total Order Sorting Examples (53) Sort users by last visit (54) Driver code (55) Analyze mapper code (56) Order mapper code (57) Shuffling (58) Pattern Description (59) Intent 9 (60) Motivation 9 (61) Structure 9 (62) Consequences 9 (63) Resemblances 9 (64) Performance analysis 9 (65) Shuffle Examples (66) Anonymizing StackOverflow comments (67) Mapper code 10 Assignments: (1) Exercise Quiz / Exam: Quiz 10~11 Join Patterns Objectives: (1) Join Patterns Instructive Coverage & Activities: (1) A Refresher on Joins (2) Reduce Side Join (3) Pattern Description 1 (4) Intent 1 (5) Motivation 1 (6) Applicability 1 (7) Structure 1 (8)Consequences 1 (9) Resemblances 1 (10) Performance analysis (11) Reduce Side Join Example (12) User and comment join (13) Driver code (14)User mapper code (15) Comment mapper code (16) Reducer code 1 (17) Combiner optimization 1 (18) Reduce Side Join with Bloom Filter (19)Reputable user and comment join (20) User mapper code (21) Comment mapper code (22) Replicated Join (23) Pattern Description (24) Intent (25)Motivation (26) Applicability (27) Structure (28) Consequences (29) Resemblances (30) Performance analysis (31) Replicated Join Examples (32)Replicated user comment example (33) Mapper code 2 (34) Composite Join (35) Pattern Description (36) Intent 3 (37) Motivation 3 (38) Applicability 3 (39) Structure 3 (40) Consequences 3 (41) Performance analysis 3 (42) Composite Join Examples (43) Composite user comment join (44) Driver code 4 (45) Mapper code 4 (46) Reducer and combiner (47) Cartesian Product (48) Pattern Description (49) Intent 5 (50) Motivation 5 (51)Applicability 5 (52) Structure 5 (53) Consequences 5 (54) Resemblances 5 (55) Performance Analysis 5 (56) Cartesian Product Examples (57)Comment Comparison (58) Input format code (59) Driver code 6 (60) Record reader code 6 (61) Mapper code 6 Assignments: (1) Exercise Quiz / Exam: Quiz 12 Metapatterns Objectives: (1) Metapatterns Instructive Coverage & Activities: (1) Job Chaining (2) With the Driver (3) Job Chaining Examples (4) Basic job chaining (5) Job one reducer (6) Job two mapper (7) Driver code (8)Parallel job chaining (9) Mapper code 1 (10) Reducer code 1 (11) Driver code 1 (12) With Shell Scripting (13) Bash example (14) Bash script (15)Sample run (16) With JobControl (17) Job control example (18) Main method (19) Helper methods (20) Chain Folding (21) The ChainMapper and ChainReducer Approach (22) Chain Folding Example (23) Bin users by reputation (24) Replicated join mapper code (25) Reducer code (26) Binning mapper code (27) Driver code (28) Job Merging (29) Job Merging Examples (30) Anonymous comments and distinct users (31) TaggedText WritableComparable (32) Merged mapper code (33) Merged reducer code (34) Driver code Assignments: (1) Create random data on fly Quiz / Exam: Quiz 13 Input and Output Patterns Objectives: (1) Input and Output Patterns Instructive Coverage & Activities: (1) Customizing Input and Output in Hadoop (2) InputFormat (3) RecordReader (4) OutputFormat (5) RecordWriter (6) Generating Data (7) Pattern Description (8) Intent 1 (9) Motivation 1 (10) Structure 1 (11) Consequences 1 (12) Resemblances 1 (13) Performance analysis 1 (14) Generating Data Examples (15) Generating random StackOverflow comments (16) Driver Code 2 (17) InputSplit code (18) InputFormat code (19) RecordReader code(20) External Source Output (21) Pattern Description (22) Intent 3 (23) Motivation 3 (24) Structure 3 (25) Performance analysis 3 (26) External Source Output Example (27) Writing to Redis instances (28) OutputFormat code (29) RecordReader code (30) Mapper Code (31) Driver Code (32) External Source Input (33) Pattern Description (34) Intent 4 (35) Motivation 4 (36) Structure 4 (37) Consequences 4 (38) Performance analysis 4 (39) External Source Input Example (40) Reading from Redis Instances (41) InputSplit code 5 (42) InputFormat code 5 (43) RecordReader code 5 (44) Driver code 5 (45) Partition Pruning (46) Pattern Description 6 (47) Intent 6 (48) Motivation 6 (49) Structure 6 (50) Consequences 6 (51) Resemblances 6 (52)Performance analysis 6 (53) Partition Pruning Examples (54) Partitioning by last access date to Redis instances (55) Custom WritableComparable code (56) OutputFormat code 7 (57) RecordWriter code 7 (58) Mapper code 7 (59) Driver code 7 (60) Querying for user reputation by last access date(61) InputSplit code 8 (62) InputFormat code 8 (63) RecordReader code 8 (64) Driver code 8 Assignments: (1) Exercise Quiz / Exam: Quiz 14 Mahout Final Thoughts and the Future of Design Patterns Objectives: (1) Final Thoughts and the Future of Design Patterns Instructive Coverage & Activities: (1) Trends in the Nature of Data (2) Images Audio and Video (3) Streaming Data (4) The Effects of YARN (5) Patterns as a Library or Component (6)How You Can Help Assignments: (1) None Quiz / Exam: Quiz 15 Review / Final Exam Objectives: (1) Final review (2) Final exam Assignments: None Quiz/Exam: (1) Final Exam

Week 2  VM system setup & Word Count

Q6 of http://npu85.npu.edu/~henry//npu/classes//mapreduce/word_count/slide/word_count_combiner.html

Your homework asnwer must include a MapReduce program.

Note: You can use any of the following VMs to do the homework
http://npu85.npu.edu/~henry/npu/classes/cloud_computing/cloudera_hadoop/slide/cloudera_vm.html

http://npu85.npu.edu/~henry/npu/classes/cloud_computing/eclipse/slide/saraswati.html

============== course material ================
http://npu85.npu.edu/~henry/npu/classes/index_course.html

============= Grading Policy ==============
http://npu85.npu.edu/~henry/npu/classes/mapreduce/intro/slide/CS570A_GRADING_POLICY.pdf

Week3  Map Reduce for Average

http://npu85.npu.edu/~henry/npu/classes/mapreduce/mean/slide/exercise_mean.html

Q4 ==> MapReduce for Average 

Week4 Pi

http://npu85.npu.edu/~henry/npu/classes/mapreduce/pi/slide/exercise_pi.html

Q1 ==> Pi 

Week5 XML Parsing

http://npu85.npu.edu/~henry/npu/classes/mapreduce/preface/slide/exercise_preface.html

Q2 => XML parsing + MapReduce 

Week6 Summarization Pattern

http://npu85.npu.edu/~henry/npu/classes/mapreduce/summarization_pattern/slide/exercise_summarization_pattern.html

Q10  ==> MinMax + Counter 

Week8 Min Word Length

http://npu85.npu.edu/~henry/npu/classes//mapreduce/word_count/slide/exercise_word_count_101.html

Q37 ==> Minimum Word Length