spark reports an error OutOfMemory "recommended collection"

Hello everyone, meet again, I'm your friend Quanstack Jun.

When I use spark for analysis recently, the amount of tens of millions of data does not feel much, but it runs very slowly

OutOfMemory

1. Then partition repartition before map where there is use of map

2.join will have shuffle, shuffle will also produce data overflow

3.map can also be replaced with mapPartitions and appropriately adjust the number of partitions 200 400

There are many others. That's all I use and the task can run.

What else is wrong, such as reset by peer, what is wrong with 255

The error message is not recorded

Then use the configuration file to adjust

config.set("spark.network.timeout","100000")config.set("spark.executor.heartbeatInterval","100s")config.set("spark.executor.memory", "50g")config.set("spark.shuffle.blockTransferService", "nio")

Copy

Useless

Mainly a code issue...

Publisher: Full-stack programmer, please indicate the source: https://javaforall.cn/127866.htmlOriginal link: https://javaforall.cn