Yan required executor memory is above the max threshold (8192mb) of this cluster!

Problem description ：

Sparn on Yarn In this scenario Spark When the task , Single executor Of memeory Set more than 8G The following errors will be reported , This is a yarn Unreasonable settings lead to ,yarn Default single nodemanager by 8G resources , This is not reasonable , So I make adjustments on the cluster .

Main log performance ：

---

Solution ：

modify yarn-site.xml

<property>
    <description>The minimum allocation for every container request at the RM,
        in MBs. Memory requests lower than this won't take effect,
        and the specified value will get allocated at minimum.</description>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>2048</value>
</property>

<property>
    <description>The maximum allocation for every container request at the RM,
        in MBs. Memory requests higher than this won't take effect,
        and will get capped to this value.</description>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>16384</value>
</property>

<property>
    <description>yarn Assigned to nodemanager Maximum number of cores </description>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>16</value>
</property>

Adjusted performance