[e-mr] error recovery record of namenode

Come to the point

Using Alibaba cloud's E-MR Cloud platform , Discover the use of Hive The service cannot query data . Through relevant error reporting , Open the console interface of Alibaba cloud platform , Found in this E-MR On the interface of HDFS Of NameNode It's gone .

Problem handling

The problem now is NameNode I'm dead , Now let's restart this NameNode No, it's OK ？
Specific implementation steps ：

1. On the Alibaba cloud console , adopt web ui To restart this E-MR Of NameNode Service for . Found that the solution failed . Click the restart button on the web page , Unable to restart this program normally .
2. Started shaking people on nails . Alibaba cloud experts and our relevant personnel were invited to analyze the problem . By holding a meeting all morning , It is judged that the resources of cloud platform machines are too low . There are big problems in this deployment process . First Master There is only one node , Configuration is 4c16G. But it has NameNode role ,SecondaryNameNode,ResourceManager role , also HMaster role . There are also master nodes of other services . First of all, from this point of view , Many services are concentrated on this machine node . This 4C 16G The pressure of the machine is great, which is understandable , So under the solution of this part , It's about the configuration upgrade or capacity expansion of the node .
3. If I'm the kind of person who waits for others to deal with the second point , Maybe there will be no water , In this area , I got it actually master The account and password on the node . I was thinking , Why should I go to the web page and click start NameNode The node of , Why can't I operate directly on this machine ？ So I started it manually NameNode, Now the service is running stably .（ Of course, the subsequent operations of server capacity expansion and configuration upgrade are also required ）

summary

although , Ali cloud, E-MR The web side of can easily operate instances of these services . If there is a problem , If you can't solve the problem on the web . Then we might as well log in to these instance nodes , With the knowledge we have , Handle these problems manually .