Spark parameter adjustment and tuning

1.1 --num-executors 100
Parameter interpretation ： Tasks can be applied for Excutor The largest number , It's not a one-time distribution 100 individual Excutor;Excutor The quantity will be dynamically adjusted during the operation of the task , Yes job be in pending Status then apply Excutor, One Excutor If the idle time is too long, remove it ;Excutor The number of determines the parallelism of tasks ;

apply Excutor： When a task is in pending state （ a backlog ） Over a certain period of time , I think there are insufficient resources , Application required Excutor;

When to apply for ： When pending The backlog exceeds spark.dynamicAllocation.schedulerBacklogTimeout（1 second ） On the application
How much to apply for ： Number of applications = Running and pending Number of tasks * spark.dynamicAllocation.executorAllocationRatio（1）/ Parallelism
remove Excutor：

spark.dynamicAllocation.enabled（false） Decide whether to use dynamic resource allocation ; External... Must be turned on shuffle;
spark.dynamicAllocation.executorIdleTimeout （60s） Free 60s It will be recycled ( And no cache );
Determine the parallelism of tasks ：executor The number of nodes is the number of work nodes , It directly determines the parallelism of tasks ; Exactly by executor*core Decisive ; This is only the maximum degree of parallelism physically provided , The actual parallelism of the task is determined by the parallelism set in the program , That is to say RDD The number of partitions ;

1.2 --executor-memory 5g
Parameter interpretation ： Every execu