The intern left a big hole when he ran away and made two online problems, which made me miserable

Xiao Bian was hurt by interns today ...

Yesterday's online question

There was an online problem yesterday , Today, there is another online problem .

An online problem occurred yesterday .

Monitoring begins to report a large number of exceptions pvlost, Interface access reports 500, Report on pvlost 了 .

Xiaobian quickly logs in to the problem instance , Go up manually curl I made this request , Log in to this instance manually curl No abnormal information .

Return is normal 200. continuity curl After more than ten times, it's no problem .

It's strange .

Yesterday, I thought about adding some logs, printing some logs and looking at them . This problem should be solved with the customer first , I'm going to add my diary today .

As a result, I haven't had time to add a log today ...

Today, there is another online problem

Another online problem has popped up , Then continue to look at this new online problem .

It is found that the function of querying the faulty computer room is not very easy to use .

Now there are two computer rooms on the line .A The computer room and B Computer room .

• When A Computer room tsdb If there is a problem with the time series database, go to query B Computer room tsdb Time series database .
• When B Computer room tsdb If there is a problem with the time series database, go to query A Computer room tsdb Time series database .

Customer op You will find out which computer room has problems , Will go to the database mysql Insert a piece of data of the faulty machine room .

In the picture above idcA There is a fault in the corresponding time period .

Currently, there are M Service and D Service query tsdb Time series database .

stay D The service implements the function of querying the faulty machine room , Just check the database to see if there is a problem in the computer room , If you find the query start and end time periods and mysql In the database idcA The fault time has intersection , Then it means A If the computer room is faulty, read the configuration file idcB Computer room url Inquire about idcB Computer room tsdb Timing data .

because D The service implements , Just use M Service to query D service , because D The affordability of the service is not good ,M The query pressure of the service is very high , therefore M Service query D A layer of cache is provided for the service .

The cache service code is really awesome , Come up with two online questions .

    @Cacheable(value =  "cacheManager", key =  "#root.method")
    public String getMalfunctionStatus(long queryStartTime, long queryEndTime) {

        QueryDateDto queryDateDto = new QueryDateDto();
        String startTime = String.valueOf(queryStartTime);
        String endTime = String.valueOf(queryEndTime);

        queryDateDto.setStartTime(startTime);
        queryDateDto.setEndTime(endTime);
        String malfunctionStatus =  "";
        try {
            malfunctionStatus  =  Inquire about D Service data http call .
        } catch (Exception e) {
            LOG.error( "request dashboard failed. e: {}", e.getMessage());
            
        }
         return malfunctionStatus;
    }

Cache analysis

to getMalfunctionStatus This method adds caching ,key Is the name of this method . Suppose the parameters passed queryStartTime：2022-05-17 10:00:00 queryEndTime 2022-05-17 12:00:00

Suppose this time period queries the downstream D Service http call , Return is fault state

Cache time expiration is 5 minute .

this 5 This in minutes getMalfunctionStatus This method returns fault state .

If this 5 Within minutes, the user transmits parameters queryStartTime：2022-05-17 12:00:00 queryEndTime 2022-05-17 14:00:00 The query period is different , But with the cache , This one returns fault state .

This cache adds loneliness .

It is better not to add .

At least you can query the downstream without adding D The service is the correct return result , Add this 5 The result is wrong within minutes ...

Interns leave a big hole for Xiaobian ！

Intern Niu X!

The second pit

If the service finds that the current computer room is faulty .

According to the downstream of the query D The status of the service found a fault , You need to switch and query the address of the computer room .

So the addresses of the two computer rooms are written in the configuration file , Then read the configuration file .

Recently, the user has added a faulty machine room , A large number of interfaces are returned 500 error .

There is no port in the configuration file when switching the computer room . This causes downstream service messages to be queried 500.

summary

I was too kind to interns before , In the future, the intern's code still needs a good look , Interns hold it for a day 300 My internship salary , Wrote a bunch of code for the pit , Come up with two online questions , Mixed a large factory internship experience , Pat your ass and leave , Xiaobian is here to wipe the intern's ass .

Uncomfortable ...