MapReduce and Design Patterns - Chain Folding Pattern Example

In this video we are going to discuss chain folding pattern example

So we shall go for one practical implementation of this concept. So in this example we will provide posts.xml and also users dot XML

These two XML files will be there to separate the users where the reputation is above

3000 and below 3000. So this is the assignment we are going to have and going to implement in our next demonstration

using Java coding and how to run it, how to get the output, will be showing that one also

in the demonstration. In this example, we are discussing chain folding pattern example and it is falling under

the meta pattern design pattern. So here is the problem is that we are supposed to separate users where reputation is greater

than 3000 and also below 3,000. So in this way we shall have to separate them

are going to have two XML files one is the posts dot XML another one is the users dot XML

This post dot XML is under the folder slash input slash post and users dot XML will be under

the folder slash input slash user. So these two XML files will be using and here you see we're

having this slash input slash user posts dot XML and let me go for slash input slash users

users.xm. So let me show you the respective file content. So at first we're going for

users.coml under the user's tag we're having multiple number of rows, two of them I have

shown and each and every row has got multiple attributes like your ID, reputation, creation date

display name and so on. So these are the multiple different rows will be there in the main

users.coml. So now we shall discuss what will be the current content of posts

dot XML under the posts tag we're having multiple row tags with multiple attributes

ID post type ID accepted answer ID accretion dates code and so many other attributes are

there so this posts dot XML will be also having multiple records now we shall

concentrate on our Java program this Java program will be having only a single

class only one Java program we shall be happy and here we'll be having the

respective two mappers and the respective reducers so here the name of the Java

class is our chain folding MR task that is the name of the Java class within

this Java class we have defined multiple public static final strings that is a

average calculation group multiple outputs below 3,000 multiple outputs above 3,000 so there are multiple final strings are defined and they have got

initialize with this so you see multiple outputs below 300 3,000 so below

3,000 whatever the value and above 3,000 whatever the value we put against

them will be creating our folders now we shall go for one inner class that is

the user count mapper and this user count mapper will be extending the map reduce

base class and implementing the mapper interface so mapper interface implementing means it must be implementing the map method within that we have

defined one static final string that is a records counter name initialized with

this records that is a record records counter name initialized with the records it

is a final that means constant also at defining private static final long

writable one is equal to new long writable one so here it is one will be

containing one there and also the output key of the type of text so these are the

variables are defined within this class and now within the within the map

method we're overriding the map method we're having XML parsed which will be

instantiated by the output argument of XML to map function this function will

actually convert the XML content to the hash map object and that has been that

will initialize XML parsed now from this XML parts we supposed to get the at get the user ID that is the owner user ID will be getting this one and that will initialize the user ID here

So get user ID from the posts. So if user ID is equal to null or is not equal to null then we'll be setting that one to

this output key and output. That is a output key and one

Now we are going to have our map reduce. So extending the map reduce base class and implement

the mapper interface so this is the respective class here so that is a user

reputation mapper so in this in this class we're having one output key of the

type of text and one hash map object that is a user reputation map and which

has been instantiated the key value pair will be of the type of string and string

we're having this method configured we're overriding that one it is having one user

reputation map. clear we're clearing that reputation map and then we are creating

one uri one uri array name of name of that is files that is the uri array which is

being instantiated with this distributed cache get cache files the respective one has been

replicated but it is working fine in our code and job will be the input argument

so if files is equal to is equal to null or the files dot length is equal to is equal to

0 that means that if the array is empty in that case we are throwing one exception

otherwise read all files in the distributed cache so for uri URI URI for each

contain in the files the files array and we'll be defining path P is equal to

new path URI file system fs is equal to file system dot get job so they are

getting instantiated now to read data from the file we have created one buffer

reader object RDR which will be which will be actually required to read data

from the file now within this while we'll be having this XML parse again initialized with this XML to map

which will convert the line from the line it will be parsing that one and that will be

initializing this XML parsed within this while we will be reading line by line

So users has been initialized with this get ID reputation has been initialized with this

Excel parse reputation. Now we shall go for one if if block that if user is not equal to

null and reputation is not equal to null. user reputation map dot put user comma reputation so we are putting that one in the

respective map user and reputation and it has been enclosed within the try catch block

properly so I think we're getting my point how we are doing now you know that we are

having this map interface so map method has to be overwritten here so here we're

going for that is a reputation is equal to user reputation map dot key that is a key to

string we're converting this key to the string and that will be assigned to the string

reputation and we will be having if the reputation is not equal to null then

output key will be having this there is a key then tab then reputation and

output dot call it output key and value so key value period will be kept in

this output. So in this way the map method body has been written and the value has been passed

as input argument. Next we're going to have our map reduce base so we are in inheriting that one

are also implementing the reducer. interface in long sum reducer so here we're supposed to we're bound to write or

define the reduce method body so here I and t sum is equal to zero while values

dot has next if the if next values are available sum plus equal to values dot next

dot get so getting each and every value and getting it added with the sum so

output value dot set sum so some will be kept in the output value and output dot collect key

output value so we'll be writing the key value pair onto the output

So in this way the reduced method has been defined. So now we shall go for another, there is a mapper

There is a user bin mapper. We'll be going for the user bin mapper. After writing this Q and value pair will be going for user bin mapper

So let me go for that. And this user bin mapper is extending this map reduce base and implementing the mapper interface the configure method so that is a multiple outputs multiple output object is MOS has

been initialized with null and so within this configured we're writing MOS is

equal to new multiple outputs configuration we are passing the configuration as

input conf and we're bound to override this map method because we're

implementing this mapper interface so within the map method we're writing integer dot parts I and T key to string dot split slash T one if it is less than

3000 that means we're converting this key to string and splitting it with the tab and

going for the second argument so second value so if it is if the count is below

3,000 then we'll be going for a ms.gat collector multiple outputs below 3,000

reporter dot collect key comma value so similarly the same thing will be happening for

these multiple outputs above 3,000 so that will be kept into this moss dot get

collector now we are going for this close and we are just closing that one within

this try catch block as usual so XML to map we have discussed already so let

me come to this main function now this main function requires three parameters the

first one is the respective class name and then the post folder and then users

folder and then the output folder so so arc zero arcs one arcs two and otherwise

system exit two we're just initializing the post input from the argument CHEERING

but initializing post input from the argument zero user input from the argument

one and then output directory from this argument two now here we're setting

multiple mappers and reducers so at first we're setting up the set jar by class

there is a chain folding m r tux dot class here we're going for chain mapper

dot add mapper so here we are introducing three mapper we're adding and one

reducer we're adding so respective the parameters have been passed so we're having the

config user count mapper dot class long writable dot class text.class text dot class and wrong

writable dot class so respective values and key value peers everything the the respective class

types are mentioned the class names have got many. for the full declaration and there is a text class and the long writable class

and then new job configuration false so in this way you can find that we have

just adding this with this chain mapper we're adding this this user account

mapper dot class in this four lines we're doing all these things in this four

lines we're doing we're just if these four lines are there and the upper one the

four lines are there so now here we're going for another mapper that is here we did the user count mapper here we are going for

this user reputation mapper and then some long sum reducer long sum reducer

also we are passing the respective parameters so we're adding them in a chain

reducer now add mapper and another one is the user being mapper everything

we define in the previous case we can see that we have defined everything now we shall

go for this set combiner so here the reducer class and has been treated as a combiner class so config.set input format that is a text input format

format.com and text input format dot set input paths will be config and post input. So all the

variables whatever you have defined earlier they are being used. Configure multiple outputs

So set output format that is a null output format.class file output format set output path is

config and output dar which you got the from the third argument common line

argument add the name output and also the add name output so multiple outputs

above 3,000 multiple outputs below 3,000 and with the respective classes as

required these are the I'm doing the rights call you just check now we shall go

for the output key class will be text class and output value class will be the long writeable class at the user files to the distributed cache now so file status array will be the user files

is a file status array so file system dot get config user list status user input and

then from this user's file that is a file status array for each and every

status we're going for distributed cache dot at cache file status dot get path

2 URI comma config so this we're adding all this these cash files were adding here onto the

distributed cache now we shall check then now we shall check one the job that is

we're defining one job so after running all this adding this one the config we

defined earlier now we shall go for this job and the job client dot run job config

so job object running job object has been created if the job has not been

completed then wait for the five seconds and again check again otherwise we shall

after coming out we shall go for if job is successful then return zero

otherwise one so in this way the whole Java program we have explained line by

line whatever the variables we defined earlier they have been used in the later on

now the now the tax you know that is we are going to create the JR file that

means file package right button click explode and go for the jar folder and the jar file name we have done that one earlier so we are not

going to do that one I have shown you that one in the other videos multiple times now let me go

for this command that is the command to execute so hadup jar respective the jar folder and

the jar file name then you shall go for this package and then the class name then the input

slash input slash post which will be content the posts dot XML slash input slash

user which will be content the users dot XML and slash output where the below

3000 and above 3,000 those two folders will be below average and above average

folders will be created so we are executing so we know that here we are separating

users depending upon the reputation whether the reputation is above 3,000 or below 3,000 so that was our task in this

chaining folding chain folding pattern example so let me let me see that

whether those two folders have got created under the output folder or not and

then we shall see the part file contents and accordingly so the command is

executing I hope it will not show any errors yes it has got perfectly executed

let me come to this here so we shall we shall go for this output the output

this output is there so now we are having this above 3,000 and below 3,000 so above

3,000 and below 3,000 two of part files have got created so now we shall go for the

respective print so we shall execute this is DFS dFS minus ghat and those two file names

we are having under the output folder so that is above star I'm just putting

I'm not writing the full file name so here you can find the respective output

of that file under that output folder and now we shall go for this below

below start enter So these outputs have been have been shown here now let me delete this output folder

I hope that you have got this process how the process has to be carried out how to write the Java code

how to execute the command so let me delete this output folder I think now you are getting

confidence to run this example thanks for watching this video