How to monitor micro web services

Reading guide Finally I realized , The risk of any server I want to write is very low , If they go down occasionally 2 Hours is no big deal , I just need to set up some very simple monitoring to help them keep running . Hello ! I recently started running some servers （ Nginx playground、mess with dns、dns lookup）, So I've been thinking about monitoring .
At first, I didn't completely know how to monitor these websites , So I want to quickly write down how I did it .
I'm not going to talk about how to monitor large 、 Serious mission critical websites , Just talk about small and unimportant websites .

DSC0000.jpg(115.64 KB, amount of downloads : 0)

Download the attachment

yesterday  10:47 Upload

The goal is ： It takes little time to operate I hope the website can work normally most of the time , But I also hope not to spend time on continuous operation .
I was initially very vigilant about running servers , Because in my last job , I am a 24/7 Take turns on duty , Responsible for some key services , In my impression ,“ Responsible for the server ” signify “ In the morning 2 Click to be called up to repair the server ” and “ There are many complex dashboards ”.
So for a while I only did static websites , So I don't have to think about the server .
But finally I realized , The risk of any server I want to write is very low , If they go down occasionally 2 Hours is no big deal , I just need to set up some very simple monitoring to help them keep running .
It's bad not to monitor At first , I haven't set up any monitoring for my server at all . The result of this is very predictable ： Sometimes the website breaks down , But I didn't find , Until someone told me ！
step 1：uptime The viewer The first step is to establish a uptime The viewer . There are many such things outside , I'm using updown.io and uptime robot. I prefer updown User interface and pricing structure （ It is on request, not on a monthly basis ）, but uptime Robots have a more generous free package .
They will ：

• Check whether the website is normal
  
• If there is a fault , It will email me
  

I find email notification is a good notification level for me , If the website goes down , I will soon find , But it won't wake me up or do anything else to disturb .
step 2： End to end health check Next , Let's talk about “ Check whether the website is normal ” What does it mean .
At first , I just turn one of my health check endpoints into a function , It will come back anyway 200 OK.
This is quite useful – It tells me that the server is started ！
But as expected , I have a problem , Because it didn't check API Is it true that Work – Sometimes the health check-up succeeds , Although other parts of the service have actually entered a bad state .
So I updated it , Let it really send API request , And make sure it succeeds .
All my services have done very little （nginx playground There's only one endpoint ）, So setting up a health check-up is very easy , It actually runs through most of the actions that services should do .
Here is nginx playground What the end-to-end health check handler looks like . It's very basic ： It just sends out a POST request （ Give yourself ）, And check whether the request succeeds or fails .

  
 
 

   
  
  
func healthHandler(w http.ResponseWriter, r *http.Request) {// make a request to localhost:8080 with `healthcheckJSON` as the body
   
  
  
// if it works, return 200
   
  
  
// if it doesn't, return 500
   
  
  
client := http.Client{}
   
  
  
resp, err := client.Post("(healthcheckJSON))
   
  
  
if err != nil {
   
  
  
log.Println(err)
   
  
  
w.WriteHeader(http.StatusInternalServerError)
   
  
  
return
   
  
  
}
   
  
  
if resp.StatusCode != http.StatusOK {
   
  
  
log.Println(resp.StatusCode)
   
  
  
w.WriteHeader(http.StatusInternalServerError)
   
  
  
return
   
  
  
}
   
  
  
w.WriteHeader(http.StatusOK)
   
  
  
}
  
 
 

Now? , Most of my health checks run every hour , Some every 30 Run every minute .
I run every hour , because updown.io The price of is calculated according to the number of health checks , I'm monitoring 18 Different URL, And I want to keep my health examination budget at 5 dollar / The lowest level of the year .
Take an hour to find that one of these websites is down , It's OK for me – If there are questions , I can't guarantee to repair it soon .
If you can run them more often , I may every 5-10 Run every minute .
step 3： The third step ： If the health check fails , Automatic restart Some of my websites are fly.io On ,fly It has a fairly standard function , I can configure one for a service HTTP health examination , If the health check fails , Just restart the service .
“ Frequent restart ” It's a very useful strategy to make up for what I haven't fixed bug, For a while ,nginx playground There is a process leak ,nginx The process was not terminated , So the memory of the server has been running out .
Pass the health check , As a result, , This happens every other day or so ：

• The server is out of memory
  
• The health check began to fail
  
• It was restarted
  
• It's all right again
  
• After a few hours, repeat the whole legend again
  

Final , I started actually fixing process leaks , But I'm glad to have a solution that I can delay repairing bug Keep running .
These health checks used to decide whether to restart the service run more frequently ： Every time 5 About minutes .
This is not the best way to monitor large Services This may be obvious , I have said it at the beginning , however “ Write a HTTP health examination ” Not the best way to monitor large and complex services . But I won't discuss it in depth , Because this is not the subject of this article .
It has been running well so far ！ I was at first 3 I wrote this article in April three months ago , But I waited until now to release it to make sure the whole setup works .
This makes a big difference – Before I encounter some very stupid downtime problems , Now in the past few months , The running time of the website has reached 99.95%！

From the original ：