04. Project blog log

Project blog log

• The system has no logs , It means that people have no eyes

• First of all , Access log access log（server The most important log at the end ）

• second , Custom log （ Include custom events 、 Error records, etc ）

1. nodejs File operations

File read

Basic file reading methods ：

const fs = require("fs");
const path = require("path");

//  Gets the name of the current folder  data.txt  Location path 
const filePath = path.resolve(__dirname, "data.txt");

//  Read file contents 
fs.readFile(filePath, (err, data) => {
    
  if (err) {
    
    console.error(err);
    return;
  }
  // data  It's a binary type , Need to be converted to a string 
  console.log(data.toString());
});

Read the contents of the file in one breath and then output . This brings risks , If the file to be read is particularly large , Memory can't bear .

File is written to

Basic file writing methods ：

//  write file 
const content = " This is what's new \n";
const opt = {
    
  flag: "a",
};
// writeFile  Four parameters ：
// filePath:  Destination file path  content:  What to write 
// opt:  How to write  'a'  Represents append write , For covering  'w'
//  The last parameter is the callback function 
fs.writeFile(filePath, content, opt, (err) => {
    
  if (err) {
    
    console.error(err);
  }
});

Every time I write, I have to get the file path and open it for writing , This will consume a lot of memory . in addition , If you want to write a particularly large content , The whole content should be stored in memory , Memory can't bear .

Judge whether the file exists

//  Check whether synchronization exists 
console.log(fs.existsSync(filePath)); // true

//  Check asynchronously for existence 
fs.exists(filePath, (exist) => {
    
  console.log(exist);
});

2. stream

2.1 IO Performance bottlenecks in operations

• IO Include ” The Internet IO“ and ” file IO“
• Compared with CPU Computing and memory reading and writing ,IO The outstanding feature of is slow
• How to improve with limited hardware resources IO Operational efficiency

2.2 stream Popular introduction

stream It's flow , Before, reading the contents of the file in one breath is like carrying a bucket , Directly carried the whole bucket away . But most people don't have that strength . A better way is to , Connect a water pipe , Transfer water stably to other places through pipes , Reduced cost （ Children can complete ）.stream It's similar to this , It greatly reduces the requirements of hardware resources .

2.1 stream demonstration

Standard input and output

process.stdin.pipe(process.stdout);

The input content of the terminal will be continuously output to the terminal through the pipeline .

Request content output to response content

const http = require('http')
const server = http.createServer((req, res) => {
    
    if (req.method === 'POST') {
    
        req.pipe(res)  //  Most importantly 
    }
})
server.listen(8000)

Now? ,req and res adopt pipe Connect , once req Received something , It will be output to res in .

stream Copy files

adopt fs.createReadStream Create a file to read stream object ,fs.createWriteStream Create a write file stream object . adopt pipe Connections allow data to flow .

//  Two file paths 
const fileName1 = path.resolve(__dirname, 'data.txt')
const fileName2 = path.resolve(__dirname, 'data-bak.txt')

//  Read the file  stream  object 
const readStream = fs.createReadStream(fileName1)
//  Write to file  stream  object 
const writeStream = fs.createWriteStream(fileName2)

//  adopt  pipe  Execute Copy 
readStream.pipe(writeStream)

//  It can monitor the content of data transmission 
readStream.on('data', chunk => {
    
    console.log(chunk.toString())
})

//  After listening to the completion of reading , Execution callback 
readStream.on('end', () => {
    
    console.log('copy done')
})

stream Read file contents

To launch a get Request to read the file , Just read the file stream and output it to response It's OK .

const http = require('http')
const fs = require('fs')
const path = require('path')
const fileName1 = path.resolve(__dirname, 'data.txt')
const server = http.createServer((req, res) => {
    
    if (req.method === 'GET') {
    
        const readStream = fs.createReadStream(fileName1)
        readStream.pipe(res)
    }
})
server.listen(8000)

3. Write the log

3.1 Create relevant methods for writing logs

Create... In the project logs Folder , Log files are stored inside . At the same time src directories creating utils Used to put commonly used tools and methods , Inside create log.js, It writes the relevant methods of writing logs .

const fs = require("fs");
const path = require("path");

//  Write the log 
const writeLog = (writeStream, log) => {
    
  //  Write information and wrap 
  writeStream.write(log + "\n");
};

//  Generate  writeStream
const createWriteStream = (fileName) => {
    
  const fullFileName = path.resolve(__dirname, "../", "../", "logs", fileName);
  const writeStream = fs.createWriteStream(fullFileName, {
    
    //  Append write 
    flags: "a",
  });
  return writeStream;
};

//  Write access logs ,error log  and  event log  Empathy 
const accessWriteStream = createWriteStream("access.log");
const access = (log) => {
    
  writeLog(accessWriteStream, log);
};

module.exports = {
    
  access,
};

3.2 Write to the log every time you receive a request

app.js Inside serverHandle Each call is when the request is received , At this point, write the access log . Other logs , for example error and event Write when appropriate .

const serverHandle = async (req, res) => {
    
  //  Record access information 
  access(
    `${
      req.method} -- ${
      req.url} -- ${
       req.headers["user-agent"] } -- ${
      Date.now()}`
  );
	...... Other main codes 
}

Realization effect , When accessing, I successfully wrote log file ：

4. Log splitting

• The content of the log will slowly accumulate , It's not easy to put it in a file
• Log files by time , Easy to locate
• Realization way ：Linux Of crontab command , That is, scheduled tasks

crontab

• Set the format of scheduled tasks ： ( minute ) ( Hours ) ( date ) ( month ) ( week ) command, If you don't need to fill in, use an asterisk * Instead of
• When it's time , take access.log Copy and rename to the current time log, Such as
• Empty access.log file , Continue to accumulate logs

4.1 Implementation method

The following process applies to Linux and macOS.

Write sh Script

Write a script copy.sh Used to generate logs with date marks , Command line execution sh coppy.sh See the effect ：

#!/bin/sh
cd logs Absolute path to folder 
#  Copy file 
cp access.log $(date +%Y-%m-%d).access.log
#  Empty  access.log
echo "" > access.log

Ideally , Generate a new log file with a date , also access.log The contents of the are emptied .

Use crontab

Execute on the command line crontab -e Enter edit mode , Write the following , It means the... Of every day 0 Hours to perform sh command .

* 0 * * * sh copy.sh The absolute path of 

see crontab Mission

crontab -l You can view the scheduled tasks of the current system .

5. Log analysis

• As for access.log journal , analysis Chrome The proportion of
• Logs are stored in rows , A line is a log
• Use nodejs Of readline Read line by line （ be based on stream, Efficient ）

such as , The following log contents , The first four are Safari The browser runs , The last two are Chrome browser .

GET -- /api/blog/detail?id=1 -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 -- 1655271842104
GET -- /favicon.ico -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 -- 1655271843024
GET -- /api/blog/detail?id=1 -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 -- 1655271853955
GET -- /api/blog/detail?id=2 -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 -- 1655271854350
GET -- /api/blog/detail?id=1 -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36 -- 1655271857708
GET -- /api/blog/detail?id=1 -- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36 -- 1655271859548

It can be seen that ,Safari When the browser sends the request , There's no Chrome Mark . We can judge according to this Chrome The proportion of （ Of course, it is not rigorous , Demo uses ）

Calculation Chrome The script of the proportion is as follows ：

// utils/readline.js
const fs = require("fs");
const path = require("path");
const readline = require("readline");

//  File path 
const fullFileName = path.resolve(
  __dirname,
  "../",
  "../",
  "logs",
  "access.log"
);
//  establish  readStream
const readStream = fs.createReadStream(fullFileName);

//  establish  readline  object 
const rl = readline.createInterface({
    
  input: readStream,
});

let chromeNum = 0;
let sum = 0;

//  Read line by line 
rl.on("line", (lineData) => {
    
  if (!lineData) {
    
    return;
  }

  //  Record the total number of rows 
  sum++;

  const [method, url, userAgent] = lineData.split(" -- ");
  if (userAgent?.includes("Chrome")) {
    
    //  Add up  Chrome  Number 
    chromeNum++;
  }
});

//  Listen read complete 
rl.on("close", () => {
    
  console.log(`Chrome  Proportion ：${
      chromeNum / sum}`);
});

Running results ：

Chrome  Proportion ：0.3333333333333333

6. summary

• Log pair server The importance of the end , Equivalent to human eyes
• IO Performance bottleneck , Use stream Improve performance
• Use crontab Split log files , Use readline Analyze log content