Generate OOM records in a production environment. Conclusion: Don't be lazy to query useless fields unless you are completely sure.

1、背景

When I got off work today, I saw a message from the technical director in the company's WeChat group,The content is that the company website is down due to code execution,Then he asked the boss of the website department to immediately find the corresponding one in the background logbug来源,I have been involved in this project before seeing it,This is the log message I wrote,Then the cause of the problem is that the collection of objects queried from the database is too large,Directly burst the memory,It is said to have been reached20G,Then the website went down

2、伪代码

This time, the following pseudo code is used to illustrate the code logic

public class Test {
    

    // Mock information database
    private static List<A> dataList = new ArrayList<>();

    static {
    
        imitateMysql();
    }

    /** * Simulate pagination of output data */
    public static void main(String[] args) {
    
        // 总数据量
        int total = dataList.size();
        System.out.println("》》》》》》The total amount of data to be processed：" + total + ";当前时间：" + DateUtil.now(DateUtil.DEFAULT_DATE_FORMAT));
        // 页面容量
        int pageSize = 100;
        // 总页码
        int totalPage = total % pageSize == 0 ? total / pageSize : total / pageSize + 1;
        // Paginate data
        for (int currentPage = 1; currentPage <= totalPage; currentPage++) {
    
            List<A> pageList = page(currentPage, pageSize);
            for (A entity : pageList) {
    
                indexData(entity.getId());
            }
            // 回收资源
            pageList = null;
        }
        System.out.println("》》》》》》All data has been processed");
    }

    /** * Generate mock database data */
    private static void imitateMysql() {
    
        for (int i = 0; i < 210000; i++) {
    
            A a = new A();
            a.setId(UUIDUtil.create().toString().replace("-", ""));
            a.setContent("Imagine for yourself that there is a lot of text here");
            dataList.add(a);
        }
    }

    /** * 根据数据idindex data toElasticsearch,No more simulation here * @param id 数据id */
    private static void indexData(String id) {
    
        // 根据数据id查询数据

        // index data toElasticsearch
        System.out.println("index data toElasticsearch完成,id = " + id);
    }

    /** * Simulate paging to get data * @param currentPage 当前页码 * @param pageSize 页面容量 * @return */
    private static List<A> page(int currentPage, int pageSize) {
    
        List<A> pageList = new ArrayList<>(pageSize);
        int start = (currentPage - 1) * pageSize;
        int end = start + pageSize - 1;
        for (int i = 0; i < dataList.size(); i++) {
    
            if (i >= start && i <= end) {
    
                pageList.add(dataList.get(i));
            }
        }
        return pageList;
    }
}

3、分析

我们来分析一下上面的代码,Paging to obtain data and recycle resources are all to reduce memory space usage,但是我们忽略了一个事实,That is, in a real environmentcontent内容巨大,因此可能造成pageList很大,In severe cases even lead toOOM,Although we empty the paginated data collection,But the garbage collector does not complete garbage collection so quickly

For operations that paginate data from the database,It is completely possible to get data only by pagingid集合,without getting other fields,The benefit of this is that paginated queries are faster,占用内存空间更小,And I didn't think about it that much at the time,Then lazily used the company's default pagination method to get all the field data,导致了OOM

4、结论

when fetching data from the database,Get only the data you need as much as possible,Don't be lazy and get useless data,Smaller will slow down the efficiency of the program,Larger causes the programOOM,影响客户正常使用,造成生产事故