Yarn restart applications Record recovery

modify yarn-core.xml The configuration file

ResourceManager Restart recovery

1. take yarn-site.xml Medium yarn.resourcemanager.recovery.enabled Configuration item set to true（ The default is false）

   <property>
       <name>yarn.resourcemanager.recovery.enabled</name>
       <value>true</value>
   </property>
   

2. To configure yarn.resourcemanager.store.class Parameters , This parameter is used to specify RM On which storage medium to save your state before restarting , There are 3 Storage options

   • org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
     The default value is , File system based storage （ Local storage or HDFS）. You can specify yarn.resourcemanager.fs.state-store.uri As a storage path , If you specify this yarn.resourcemanager.fs.state-store.uri You have to set .

   • org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
     be based on ZooKeeper The storage , When you enable RM High availability , You can only choose this way . Because the two one. RM Can be active （ Think you are the real RM）, And then brain crack . be based on ZK Storage of can be isolated （fence） Status data prevent brain cracking . You can specify hadoop.zk.address（ZK Node address list ） and yarn.resourcemanager.zk-state-store.parent-path（ The root node path of the state data ） Parameters .

   • org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore
     be based on LevelDB The storage . It is more lightweight than the first two , Occupied storage space and I/O Much smaller , And support better atomic operations . When there are extreme requirements for performance . You can specify yarn.resourcemanager.leveldb-state-store.path As a storage path .

     <property>
         <name>yarn.resourcemanager.store.class</name>
         <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore</value>
     </property>
     

3. To configure yarn.resourcemanager.fs.state-store.uri, If yarn.resourcemanager.store.class yes org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore Need configuration , Configuration here HDFS Storage

   <property>
       <name>yarn.resourcemanager.fs.state-store.uri</name>
       <value>hdfs://hadoop-master:9010/rmstore</value>
   </property>
   

4. Final configuration yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms, It means from RM After restart, from each NM Sync Container The waiting time of information , After that, new Container. The default value is 10000（10 second ）, Generally, it does not need to be changed .

   <property>
       <name>yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms</name>
       <value>10000</value>
   </property>
   

To configure NodeManager Restart automatic recovery

1. take yarn-site.xml Medium yarn.nodemanager.recovery.enabled Configuration item set to true（ The default is false）

   <property>
       <name>yarn.nodemanager.recovery.enabled</name>
       <value>true</value>
   </property>
   

2. To configure yarn.nodemanager.recovery.dir Parameters , Appoint NM Before you restart , take Container The status of is written to this local path . The default value is ${hadoop.tmp.dir}/yarn-nm-recovery

   <property>
       <name>yarn.nodemanager.recovery.dir</name>
       <value>/opt/topology/db_data/hadoop-data/yarn-nm-recovery</value>
   </property>
   

3. To configure yarn.nodemanager.address Parameters , The parameter is NM Of RPC Address , The default is ${yarn.nodemanager.hostname}:0, That is, temporary ports are used randomly . Be sure to specify a fixed port （ Such as 8041）, otherwise NM The port will be changed after restart , Can't recover Container The state of

   <property>
       <name>yarn.nodemanager.address</name>
       <value>hadoop-master:45454</value>
   </property>