Solve the problem of too many small files

---

1. Use hive Self contained concatenate command , Automatically merge small files

Usage method ：

2. Adjust the parameters to reduce map Number

In execution map Forward small file merge , stay mapper Combine multiple files into one split As input . adjustment split At least the size of

3. Reduce Reduce The number of

reduce The number of output files depends on the number of output files , So you can adjust reduce Control the number of hive The number of files in the table .

4. HAR file

Use hadoop Of archive File small files , Can package multiple small files into one har file

5. jvm reusing

Hadoop The default configuration is usually to use derivation JVM To execute map and Reduce Mission . At this time JVM The startup process can be quite expensive , Especially for execution job There are hundreds of thousands of them task Mission status .JVM Reuse allows JVM Instance in the same job Reuse in N Time .

The downside of this feature is that , Turn on JVM Reuse will always be used to task slot , For reuse , Not released until the mission is complete .