Sorting using Hadoop - TotalOrderPartitioner

Please Visit: http://lifelongprogrammer.blogspot.com



Sorting using Hadoop – TotalOrderPartitioner

http://pipiper.wordpress.com/2013/05/02/sorting-using-hadoop-totalorderpartitioner/

job.setPartitionerClass(TotalOrderPartitioner.class);

job.setSortComparatorClass(SortKeyComparator.class);

job.setPartitionerClass(TotalOrderPartitioner.class);



Path inputDir = new Path(partitionLocation);

Path partitionFile = new Path(inputDir, "partitioning");

TotalOrderPartitioner.setPartitionFile(job.getConfiguration(), partitionFile);



int numSamples = numReduceTasks;

int maxSplits = numReduceTasks - 1;



InputSampler.Sampler sampler = new InputSampler.RandomSampler(pcnt, numSamples, maxSplits);

InputSampler.writePartitionFile(job, sampler);



from Google Plus RSS Feed for 101157854606139706613 http://pipiper.wordpress.com/2013/05/02/sorting-using-hadoop-totalorderpartitioner

via LifeLong Community