Small file special

Hive Small file merge parameters ;

Spark Small file merging ideas ：

Adoption community SPARK-24940 How to deal with , With the help of SQL hint Way to merge small files .

INSERT ... SELECT /*+ COALESCE(numPartitions) */ ... INSERT ... SELECT /*+ REPARTITION(numPartitions) */ ...

Add auto merge small file result file .

• The user side ： When spark.sql.shuffle.partitions The setting is relatively large and the result data set is relatively small , There's a lot of small files , newly added spark.sql.result.partitions Parameter to control the number of final output files .
• Platform side ： Trigger small file detection when the data falls into the disk , stay InsertIntoHiveTable If small file merge is enabled , And the average size of the file is lower than the threshold, the merge is performed , Do it after merging loadTable perhaps loadPartition operation .（ The platform side is enabled by default ）

Dynamic setting Shuffle Partition.

Spark Adaptive Execution Function support Shuffle Operate downstream Stage According to the upstream Stage Produced Shuffle Data volume automatically adjusts downstream Stage Of Task Count , namely Shuffle Read Multiple small files Partition hand