lucene/src/main/java/com/twitter/elephantbird/mapreduce/input/LuceneIndexInputFormat.java
@Override
  public List<InputSplit> getSplits(JobContext job) throws IOException, InterruptedException {

    // load settings from job conf
    loadConfig(HadoopCompat.getConfiguration(job));

    // find all the index dirs and create a split for each
    PriorityQueue<LuceneIndexInputSplit> splits = findSplits(HadoopCompat.getConfiguration(job));

    // combine the splits based on maxCombineSplitSize
    List<InputSplit> combinedSplits = combineSplits(splits, maxCombinedIndexSizePerSplit,
      maxNumIndexesPerSplit);

    return combinedSplits;
  }
@Override
  public List<InputSplit> getSplits(JobContext job) throws IOException, InterruptedException {

    // load settings from job conf
    loadConfig(HadoopCompat.getConfiguration(job));

    // find all the index dirs and create a split for each
    PriorityQueue<LuceneIndexInputSplit> splits = findSplits(HadoopCompat.getConfiguration(job));

    // combine the splits based on maxCombineSplitSize
    List<InputSplit> combinedSplits = combineSplits(splits, maxCombinedIndexSizePerSplit,
      maxNumIndexesPerSplit);

    return combinedSplits;
  }
@ Override public List < InputSplit > getSplits ( JobContext job ) throws IOException , InterruptedException { // load settings from job conf loadConfig ( HadoopCompat . getConfiguration ( job ) ) ; // find all the index dirs and create a split for each PriorityQueue < LuceneIndexInputSplit > splits = findSplits ( HadoopCompat . getConfiguration ( job ) ) ; // combine the splits based on maxCombineSplitSize List < InputSplit > combinedSplits = combineSplits ( splits , maxCombinedIndexSizePerSplit , maxNumIndexesPerSplit ) ; return combinedSplits ; }
Creates splits with multiple indexes per split
(if they are smaller than maxCombinedIndexSizePerSplit).
It is possible for a split to be larger than maxCombinedIndexSizePerSplit,
if it consists of a single index that is
larger than maxCombinedIndexSizePerSplit.
<p>
All inputPaths will be searched for indexes recursively
<p>
The bin-packing problem of combining splits is solved naively:
<ol>
<li>Sort all indexes by size</li>
<li>Begin packing indexes into splits until adding the next split would cause the split to
exceed maxCombinedIndexSizePerSplit</li>
<li>Begin packing subsequent indexes into the next split, and so on</li>
</ol>
Creates splits with multiple indexes per split ( if they are smaller than maxCombinedIndexSizePerSplit ) . It is possible for a split to be larger than maxCombinedIndexSizePerSplit if it consists of a single index that is larger than maxCombinedIndexSizePerSplit . <p > All inputPaths will be searched for indexes recursively <p > The bin - packing problem of combining splits is solved naively : <ol > <li > Sort all indexes by size< / li > <li > Begin packing indexes into splits until adding the next split would cause the split to exceed maxCombinedIndexSizePerSplit< / li > <li > Begin packing subsequent indexes into the next split and so on< / li > < / ol >