当前位置：网站首页>The middle stage of vivo Monkey King activity

The middle stage of vivo Monkey King activity

2020-11-09 11:30:00 【Vivo Internet technology】

Classic architecture design can span time and language , To pass on .

—— .

One 、 background

Wu Kong activities in Taiwan technical article series and you meet again , It's getting cold , Be sure to keep warm .

In the previous series of technical articles, we mainly focus on sharing all aspects of front-end technology , Such as the state management of micro components , Cross platform exploration of micro components , And damaging layout , Performance optimization and so on . Students who haven't paid attention to , If you're interested, check out previous articles .

Today's technology theme is a little different , Let's talk about the technology construction of the application service layer in Wukong activities .

In the design of the technical architecture of Wukong activity stage , We fully embrace JavaScript ecology , Hope to push forward JavaScript The full stack development process of , So on the server side of the application layer , We chose Node As BFF(Backend For Fronted) Layer solution .

I hope to make full use of it JavaScript The effectiveness of , Can be more efficient 、 Finish the iteration of the product with higher quality .

Discover through practice JavaScript The benefits of the full stack development process ：

  1. Front and rear end use JavaScript To build , Make the front and back end more integrated and efficient . In the reuse of front-end and back-end code, some modules and components can not only be used in the front-end , It can also be used in the back end .
  2. Reduced a lot of communication costs . Design and iteration around product requirements , Front end engineers switch seamlessly between front-end and back-end development , Ensure the fast landing of business .
  3. From the perspective of developers' overall development . Give the front-end engineers a chance to get from the product 、 front end 、 Think about problems and technological innovation from a back-end perspective .

Of course Node It's just part of service application development . When we need to store business data , We also need a data persistence solution . In Wukong activity, Taiwan chooses mature and reliable MySQL As our data storage database . Then we need to think Node and MySQL How to match to better release each other's ability , Now let's go on the road of exploration .

Two 、Node Data persistence layer status and thinking

1、 Pure MySQL drive

Node-MySQL yes Node Connect MySQL The driver , Use pure JavaScript Development , Avoid the underlying module compilation . Let's see how to use it , First we need to install this module .

Examples are as follows ：

npm install mysql #  Before 0.9 You need to install this version of  npm install mysqljs/mysql

The normal use process is as follows ：

var mysql      = require('mysql');
var connection = mysql.createConnection({
  host     : '..', // db host
  user     : '..', // db user
  password : '',   // db password
  database : '..'  // which database
});

connection.connect();

connection.query(
  'SELECT id, name, rank FROM lanaguges', 
  function (error, results, fields) {
    if (error) throw error;
    /**
     *  Output ：
     * [ RowDataPacket {
     *    id: 1,
     *    name: "Java",
     *    rank: 1
     *  },
     *  RowDataPacket {
     *    id: 2,
     *    name: "C",
     *    rank: 2
     *  }
     *]
     *
     *
     */
    console.log('The language rank is: ', results);
});

connection.end();

By the above example , We are right. MySQL There is a simple understanding of how modules are used , The basic way to use it is to create connections , perform SQL sentence , Get the results , Close connections, etc .

In actual projects, we rarely use this module directly , Generally, it will be packaged on the basis of this module , Such as ：

  • By default, database connection pool is used to improve performance .
  • improvement callback The style of callback function , Migrate to promise,async/await More modern JavaScript The asynchronous processing scheme of .
  • Use more flexible transaction processing .
  • For complexity SQL Compiling , String splicing is a painful way , Need more semantic SQL Writing ability .

2、 Mainstream ORM

At present, in the data persistence layer technology solution ORM Still the mainstream technology solution ,ORM yes " object - Relation mapping "（Object/Relational Mapping） Abbreviation , Simply speaking ORM Through the syntax of instance objects , The technology of completing the operation of relational database , Pictured -1.

Whether it's Java Of JPA Technical specifications and Hibernate And so on , perhaps Ruby On Rails Of ActiveRecord, Or Django Of ORM. Almost every language has its own ecology ORM Technical implementation scheme of .

chart -1 O/R Mapping

ORM Mapping a database to an object ：

  • Table of database （table） => class （class）
  • Record （row, Row data ）=> object （object）
  • Field （field）=> Object properties （attribute）

Node stay ORM On the technical scheme of , Communities have different perspectives to explore , It fully embodies the diversity of the community , For example, it is very popular at present  Sequelize.Sequelize It's based on Promise Of Node.js ORM, At present, we support PostgreSQL、MySQL、SQLite as well as SQL-Server. It has strong transaction support 、 Connections 、 read-ahead 、 Delay loading 、 Read copy and other functions . If above MySQL Use case , If you use Sequelize ORM To achieve , The code is as follows ：

//  Definition ORM The data of and model mapping 
const Language = sequelize.define('language', {
  //  Definition id, int type  &&  Primary key 
  id: {
    type: DataTypes.INTEGER,
    primaryKey: true
  },
  //  Definition name, string Type mapping database varchar
  name: {
    type: DataTypes.STRING,
  },
  //  Definition rank, string Type mapping database varchar
  range: {
    type: DataTypes.INTEGER
  },
}, {
  //  Do not generate timestamps 
  timestamps: false
});

//  Query all 
const languages = await Language.findAll()

3、 The Future Star Talents Competition TypeORM

Ever since TypeScript after , Let's look at the tool chain and ecology of the front end from another perspective ,TypeScript The typological system of the system gives us more imagination , Code static check error correction 、 restructure 、 Automatic prompt, etc . With these new perspectives, the community has become more popular  TypeORM. It is also worth learning from .

chart -2 TypeORM

TypeORM Fully integrate TypeScript, Provide a better development experience . The goal is to always support the latest JavaScript function , And other features to help you develop any type of application that uses a database , From small applications with a small number of tables to large enterprise applications with multiple databases .

With all the other JavaScript ORM Different ,TypeORM Support Active Record （RubyOnRails Of ORM At the heart of ） and Data Mapper （Django Of ORM The core design pattern of ） Pattern , This means that we can write high quality... In the most efficient way 、 Loose coupling 、 Telescopic 、 Maintainable applications .

4、 Rational thinking

In software development as we all know , There is no real silver bullet program ,ORM It brings us faster iteration speed , There are still some shortcomings . Embodied in ：

  • For simple scenes CRUD Very fast , For multiple tables and complex associated queries, it will be a bit inadequate .
  • ORM Libraries are not lightweight tools , It takes a lot of energy to learn and set up .
  • For complex queries ,ORM Or it can't be expressed , Or it's not as good as native SQL.
  • ORM Abstracting away the database layer , Developers can't understand the underlying database operations , You can't customize some special SQL.
  • Easy to produce N+1 Query questions .

We began to think about how ORM On the basis of , Keep the tough SQL The ability to express ？ Final , Our eyes are fixed on Java Community is a very popular semi-automatic ORM On the frame of MyBatis.

3、 ... and 、 Exploration of data persistence layer in Wukong activity

Thinking through , Let's go back to the beginning and reexamine this problem , We think SQL It is the best domain language for program and database interaction , Simple and easy to learn, strong versatility and no need to avoid SQL In itself . meanwhile MyBatis We are inspired by the architecture design of , Technically, it can be preserved SQL Flexible and powerful , At the same time, take into account from SQL Flexible mapping to objects .

1、 What is? MyBatis ？

MyBatis Is an excellent persistence layer framework , It supports customization SQL、 Stored procedures and high-level mappings .MyBatis It dispenses with almost everything JDBC Code and the work of setting parameters and getting result sets .MyBatis It can be done by simple XML Or annotations to configure and map primitive types 、 Interface and Java POJO（Plain Old Java Objects, Ordinary old-fashioned Java object ） For records in the database .MyBatis The best design is in object mapping and native SQL There's a good balance between the powers .

SQL To configure 

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper
  PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
  "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="org.mybatis.example.BlogMapper">
  <select id="selectBlog" resultType="Blog">
    select  name  from blog where id = #{id}
  </select>

SQL Inquire about 

BlogMapper mapper = session.getMapper(BlogMapper.class);
Blog blog = mapper.selectBlog(101);

So we started building Node Of MyBatis, Technically Node-MyBatis With characteristic ：

  • Easy to learn . The code implementation is small and simple . There is no third party relying on , Easy to use .
  • flexible .Node-Mybatis No impact on the existing design of the application or database . With the help of ES6 Of string template To write SQL, Flexible and direct .
  • relieve SQL Coupling with program code .

By providing DAO layer , Separate business logic from data access logic , Make the design of the system clearer , Easier to maintain , Easier unit testing .

  • Support dynamic SQL . avoid SQL String splicing .
  • prevent SQL Inject . Automatic dynamic parameters SQL Anti Injection .
  • Declarative transaction mechanism . With the help of decorator It's easier to declare transactions .
  • combination Typescript The type of . According to the table structure of data, automatically generate data type definition file , Code upgrade and complement , Enhance the development experience .

2、Node-MyBatis Solution

In our business development , Our construction SQL It must be judged and dynamically spliced according to the business , If each one SQL It's all manual stitching back to MySQL Simple model , If you're not careful, you create a lot of SQL Injection and other problems , What shall we do then ？ This is the time to call out Node-MyBatis The dynamics of the SQL Of uilder The model .

（1）SQL-Builder

\# expression

： For the dynamic SQL Placeholder in , The most common scene we come across is Placeholder for string ,# This is followed by the name of the variable that will be dynamically replaced in the future . Such as ：

SELECT
    id as id,
    book_name as bookName
    publish_time as publishTime
    price as price
  FROM t_books t
  WHERE
    t.id = #data.id AND t.book_name = #data.bookName
    
 --  The  SQL  adopt  Node-MyBatis  At the bottom  SQL Compile  After analysis , Generated  SQL as follows ,
 -- data  Parameter is : {id: '11236562', bookName: 'JavaScript Red Data Book ' }
 
 SELECT
    id as id,
    book_name as bookName
    publish_time as publishTime
    price as price
  FROM t_books t
  WHERE
    t.id = '11236562' AND t.book_name = 'JavaScript Red Data Book '

$ expression

$: Placeholder for dynamic data , The occupier will be in our sql template After compiling, insert the value of the variable dynamically SQL , as follows ：

SELECT 
  id, name, email 
FROM t_user t
WHERE t.state=$data.state AND t.type in ($data.types)

--  The  SQL  adopt  Node-MyBatis  At the bottom  SQL Compile  After analysis , Generated  SQL as follows 
-- data  Parameter is : {state: 1, types: [1,2,3]}

SELECT 
  id, name, email 
FROM t_user t
WHERE t.state=0 AND t.type in (1,2,3)

<%%>  Code block

Templates are also languages , That's Turing complete , loop 、 Branching is essential . We need to provide dynamic programming capabilities to deal with more complex SQL scene , How to mark code blocks ？ Wukong uses a similar EJS Grammatical features of templates <%%> Code tag , And to reduce SQL The difficulty of template learning . Here's a demonstration in SQL How to use the template .

--  loop 
SELECT
    t1.plugin_id as pluginId,
    t1.en_name  as pluginEnName,
    t1.version as version,
    t1.state as state
 FROM test_npm_list  t1
    WHERE t1.state = '0'
  <% for (let [name, version] of data.list ) { %>
     AND t1.en_name = #name AND t1.version=#version
  <% } %>

--  Branch judgment 
SELECT 
   id,
   name, 
   age 
FROM users
WHERE name like #data.name
<% if(data.age > 10) {%>
AND age = $data.age
<% } %>

How to achieve the above functions ？

We use ES6 Of String Template Can implement a very simple template system . Let's use the template string to output the template result .

let template = `
<ul>
  <% for(let i=0; i < data.users.length; i++) { %>
    <li><%= data.users[i] %></li>
  <% } %>
</ul>
`;

The above code is in the template string , Put a regular template . This template uses   <%...%\> place JavaScript Code , Use <%= ... %> Output JavaScript expression . How to compile this template string ？ The idea is to turn it into JavaScript Expression string , The goal is to convert it to the following string .

print('<ul>');
for(let i = 0; i < data.users.length; i++) {
  print('<li>');
  print(data.users[i]);
  print('</li>');
};
print('</ul>');

First of all ： The regular expression is used for matching transformation .

let evalExpr = /<%=(.+?)%>/g;
let expr = /<%([\s\S]+?)%>/g;

template = template
  .replace(evalExpr, '`); \n  print( $1 ); \n  echo(`')
  .replace(expr, '`); \n $1 \n  print(`');

template = 'print(`' + template + '`);';
console.log(template);

//  Output 

echo(`
<ul>
  `);
  for(let i=0; i < data.supplies.length; i++) {
  echo(`
    <li>`);
  echo(  data.supplies[i]  );
  echo(`</li>
  `);
  }
  echo(`
</ul>
`);

second ： take template Regular encapsulation returns in a function . In this way, the ability of template compilation is realized , The complete code is as follows ：

function compile(template){
  const evalExpr = /<%=(.+?)%>/g;
  const expr = /<%([\s\S]+?)%>/g;

  template = template
    .replace(evalExpr, '`); \n  print( $1 ); \n  echo(`')
    .replace(expr, '`); \n $1 \n  print(`');

  template = 'print(`' + template + '`);';

  let script =
  `(function parse(data){
    let output = "";

    function print(html){
      output += html;
    }

    ${ template }

    return output;
  })`;

  return script;
}

Third ： adopt compile function , We got a SQL Builder Of Higher order function , Pass parameters , To get the final SQL Template string .

let parse = eval(compile(template));
parse({ users: [ "Green", "John", "Lee" ] });
//   <ul>
//     <li>Green</li>
//     <li>John</li>
//     <li>Lee</li>
//   </ul>

According to the idea of this template , We design our own sqlCompile To generate SQL Code for .

sqlCompile(template) {
    template =
        'print(`' +
        template
            //  analysis # Dynamic expressions 
            .replace(/#([\w\.]{0,})(\W)/g, '`); \n  print_str( $1 ); \n  print(`$2')
            //  analysis $ Dynamic expressions 
            .replace(/\$([\w\.]{0,})(\W)/g, '`); \n  print( $1 ); \n  print(`$2')
            //  analysis <%%> Dynamic statements 
            .replace(/<%([\s\S]+?)%>/g, '`); \n $1 \n  print(`') +
        '`);'
    return `(function parse(data,connection){
      let output = "";
      function print(str){
        output += str;
      }
      function print_str(str){
       output += "\'" + str + "\'";
      }
      ${template}
      return output.replace(/[\\r\\n]/g,"");
    })`
  }

（2）SQL Anti Injection

SQL Supporting splicing may exist SQL The possibility of Injection ,Java in MyBatis  $ The use of dynamic expressions also has injection risks , because $ Variables can be replaced without enclosing character quotes , The community does not recommend using $ Symbols to put together SQL. about Node-MyBatis Come on , Because it keeps $ The ability of , So we need to deal with it SQL The risk of Injection . Reference resources MyBatis Of Node-MyBatis Tool usage is also relatively simple , Examples are as follows ：

// data = {name: 1}
`db.name = #data.name` // =>  Character substitution , Will be taken as   db.name = "1"
`db.name = $data.name` // =>  Complete replacement , Will be taken as   db.name =  1

Inject the scene 

// SQL  Templates 
`SELECT * from t_user WHERE username = $data.name and paasword = $data.passwd`
// data  The data is  {username: "'admin' or 1 = 1 --'", passwd: ""}
//  This way  SQL Comment structure   To form the SQL The injection of 
`SELECT * FROM members WHERE username = 'admin' or 1 = 1 -- AND password = ''`

// SQL  Templates 
`SELECT * from $data.table`
// data  The data is  {table: "user;drop table user"}
//  This way  SQL Comment structure   To form the SQL The injection of 
`SELECT * from user;drop table user`

For common splicing SQL Scene , Let's not talk about them one by one . The following will start from the common inevitable splicing common , Explain to you Node-Mybatis To avoid it . The program uses MySQL Built in escape Method or SQL Keyword intercepting method is used to avoid parameter value passing .

escape escape , Use $ To transfer values , The bottom of the template will go first escape Method to escape , We do this with data that contains different data types escape Capability testing , Such as ：

const arr = escape（[1,"a",true,false,null,undefined,new Date()]）;

//  Output 
( 1,'a', true, false, NULL, NULL, '2019-12-13 16:19:17.947')

Keyword blocking , stay SQL Need to use database keywords , Table name 、 Column names and function keywords where、 sum、count 、max 、 order by 、 group by etc. . If you assemble it directly SQL The sentence will be more obvious SQL Injection hidden danger . So we have to restrict $ The range of values used by the symbol of . Special business scenarios , Such as dynamic sorting 、 A dynamic query 、 Dynamic grouping 、 Dynamic condition judgment, etc , Developers need to pre enumerate to determine the possible deterministic values, and then pass in SQL.Node-MyBatis By default, high-risk $ Input key words .

if(tag === '$'){
  if(/where|select|sleep|benchmark/gi.test(str)){
    throw new Error('$ value not allowed include where、select、sleep、benchmark keyword !')
  }
  //...
}

Configure interception , We want to control SQL The injection risk of , stay SQL Query does not support the execution of multiple statements by default .MySQL The underlying driver has the same options , Off by default . stay MySQL The driver's documentation provides a detailed explanation as follows ：

Connection options - Connection properties

multipleStatements: 

  • Allow multiple mysql statements per query. Be careful with this, it could increase the scope of SQL injection attacks. (Default: false)
  • Each query allows multiple mysql sentence .  Please pay attention to this , It may increase SQL Inject the scope of the attack . （ The default value is ：false）

Node-MyBatis By default, the configuration of multi line execution statements and $ Common use scenarios .

if(tag === '$'){
  if(this.pool.config.connectionConfig.multipleStatements){
    throw new Error('$ and multipleStatements mode not allowed to be used at the same time !')
  }
  //...
}

SQL Injection detection

sqlmap  Is an open source penetration testing tool , It can be used for automatic detection , utilize SQL Inject holes , Get database server permissions . It has a powerful detection engine , Function options of penetration testing for different types of databases , Including getting the data stored in the database , Accessing operating system files can even execute operating system commands through an external data connection .sqlmap Support MySQL, Oracle, PostgreSQL, Microsoft SQL Server, Microsoft Access, IBM DB2, SQLite, Firebird, Sybase and SAP MaxDB And other database security vulnerabilities detection .

sqlmap Five different injection modes are supported ：

  • Blind annotation based on Boolean That is, the injection of true or false conditions can be judged according to the returned page ;
  • Time based blind annotation That is, no information can be judged based on the content returned from the page , Use conditional statement to check whether time delay statement is executed （ That is, whether the page return time increases ） To judge ;
  • Based on error reporting injection That is, the page will return an error message , Or return the result of the injected statement directly to the page ;
  • Joint query injection have access to union In the case of Injection ;
  •   Heap query injection It can execute multiple statements at the same time .

chart -3 - SQLmap Use

install & Use 

// Installation method 
git clone --depth 1 https://github.com/sqlmapproject/sqlmap.git sqlmap-dev

// Usage method 
sqlmap -u 'some url' --flush-session --batch --cookie="some cookie"

Common command parameters

  • -u Set up the website you want to verify url
  • --flush-session Clear the history of the past
  • --batch Batch validation Injection
  • --cookie If you need to log in Set up cookie value

clear sqlmap   After using the method , In the actual project packaging process, we can base on sqlmap Build our custom test script , After submitting the code , adopt GitLab The integration tool of automatic trigger for engineering verification .

（3） Declarative transactions

stay Node And database interaction , For updated SQL scene , We need to manage things , Managing transactions manually is time-consuming and laborious ,Node-MyBatis Provides a better transaction management mechanism , Provides declarative transaction management capabilities , Freeing us from complex transactions , Get the connection 、 Close the connection 、 Transaction submission 、 Roll back 、 Exception handling and other operations will be handled automatically .

Declarative transaction management uses AOP Realized , The essence is to intercept the target method before and after execution . Join or create a transaction before the target method is executed , After the execution method is executed , Choose to commit or roll back the transaction according to the actual situation . There is no need to write transaction related code in business logic code , Just configure or use annotations in the configuration file （@Transaction）, It's not intrusive .

In terms of code implementation , We use ES7 The specification of decorators in the specification , To implement the target class , Method , Modification of attributes . Decorators are very simple to use , It's essentially a function wrapper . Let's encapsulate a simple log Decorator function .

Decoration 

function log(target, name, descriptor) {
  console.log(target)
  console.log(name)
  console.log(descriptor)
}

@log
class User {
  walk() {
    console.log('I am walking')
  }
}

const u = new User()
u.walk()

Decoration method 

function log(target, name, descriptor) {
  console.log(target)
  console.log(name)
  console.log(descriptor)
}

class Test {
  @log //  Decorators for decorating class methods 
  run() {
    console.log('hello world')
  }
}

const t = new Test()
t.run()

The decorator function has three parameters , Its meaning is not used when decorating different properties . In the decoration class , The first parameter represents the function itself of the class . Before log Output is as follows ：

[Function: User]
undefined
undefined
I am walking

When decorating class methods , The first parameter represents the prototype of the class ( prototype ), The second parameter represents the method name , The third parameter represents the property of the decorated parameter . Before log Output is as follows :

Test { run: [Function] }
run
{
  value: [Function],
  writable: true,
  enumerable: true,
  configurable: true
}
hello world

Third describe Parameters have the following properties :

  1. configurable - Can control delete 、 Can modify descriptor In itself .
  2. writable - Control whether the value can be modified .
  3. enumerable - Control can enumerate attributes .
  4. value - Control the corresponding value , The method is just a value It's a property of the function .
  5. get and set - Read and write logic that controls access .

Implementation mechanism

About ES7 Some powerful features and usage of decorator can refer to TC39 Proposal for , I don't want to talk about it here . Come and see us @Transaction The implementation of the ：

//  Encapsulating higher-order functions 
function Transaction() {
  //  Returns the proxy method 
  return (target, propetyKey, descriptor) => {
    //  Get the current proxy method 
    let original = descriptor.value;
    //  Intercept extension method 
    descriptor.value = async function (...args) {
     try {
       //  Get the bottom layer mysql Transaction scope for 
        await this.ctx.app.mysql.beginTransactionScope(async (conn) => {
          //  Bind database connection 
          this.ctx.conn = conn
          //  Execution method 
          await original.apply(this, [...args]);
      }, this.ctx);
     } catch (error) {
       //  Error handling ...
      this.ctx.body = {error}
     }
    };
  };
}

stay Transaction In the decorator of , We use the bottom layer egg-mysql Object extended beginTransactionScope Auto-Control , Transactions with scope .

API：beginTransactionScope(scope, ctx)

  • scope:   One generatorFunction, It will execute all of the SQL.
  • ctx:   The context object of the current request , It will ensure that even in the case of nested transactions , There is only one active transaction in a request at the same time . 
const result = yield app.mysql.beginTransactionScope(function* (conn) {
  //  There is no need to manually handle transaction opening and rollback 
  yield conn.insert(table, row1);
  yield conn.update(table, row2);
  return { success: true };
}, ctx); // ctx  Execution context

combination Midway Use 

import { Context, controller, get, inject, provide } from "midway";

@provide()
@controller("/api/user")
export class UserController {

  @get("/destroy")
  @Transaction()
  async destroy(): Promise<void> {
    const { id } = this.ctx.query;
    const user = await this.ctx.service.user.deleteUserById({ id });
    //  If it fails , The above database operations are automatically rolled back 
    const user2 = await this.ctx.service.user.deleteUserById2({ id });
    this.ctx.body = { success: true, data: [user,user2] };
  }
}

（4） More feature iterations

Data caching , In order to improve the efficiency of data query , We are developing iteratively Node-MyBatis Cache mechanism , Reduce the pressure on database data query , Improve the efficiency of overall data query .

MyBatis Provides a level 1 Cache , And L2 cache , We also refer to the architecture , The architecture is shown in the following figure -4.

The first level cache is SqlSession Level cache , In the same SqlSession Two times in the same SQL sentence , After the first execution, the query data in the database will be written to the cache （ Memory ）, The second time you get data from the cache, you will no longer query from the database , So as to improve query efficiency . When one SqlSession When it's over SqlSession The first level cache in does not exist . Different SqlSession The cache data regions between are not affected by each other .

The L2 cache is Mapper Level cache , Multiple SqlSession To operate the same Mapper Of SQL Statement to get data will exist in the secondary cache area , Multiple SqlSession Second level cache can be shared , Second level cache is cross SqlSession Of .

chart -4 Cache architecture diagram

Custom methods and tags , stay SQL In template , We go through #、$、<%%> To achieve SQL Dynamic construction of , However, in the actual project, we found a lot of repetition SQL Splicing scenes , For these scenarios, we are developing in SQL Templates support custom methods and tags , Directly built into the template to improve development efficiency , And provide plug-in mechanism , It is convenient for developers to develop their own custom methods and tags . Let's take a look at using custom methods and tags to SQL Some small examples of building .

Insert data into

The current way of data insertion , Keep it up native SQL The way , however , When there are so many fields in the database , It's tiring to list the fields inserted one by one . Especially when our norms emphasize SQL When inserting, you must specify the name of the inserted column , Avoid inconsistent data insertion .

INSERT INTO 
  test_user 
(name, job, email, age, edu)
 VALUES 
  (#data.name, #data.job, #data.email, #data.age, #data.edu)

Node-MyBatis Built-in methods - quickInsert()

-- user = {
--  name: 'test', 
--  job: 'Programmer', 
--  email: '[email protected]', 
--  age: 25, 
--  edu: 'Undergraduate'
-- }

-- sql builder
INSERT INTO test_user <% quickInsert(data.user) %>

--  adopt  SQL compiler  Auto output 

INSERT INTO 
  test_user (name, job, email, age, edu) 
  VALUES('test', 'Programmer', '[email protected]', 25, 'Undergraduate')



-- userList =   [
--  {name: 'test', job: 'Programmer', email: '[email protected]',  age: 25, edu: 'Undergraduate'}, 
--  {name: 'test2', job: 'Programmer', email: '[email protected]',  age: 30, edu: 'Undergraduate'}
-- ]

--  Batch insert 
INSERT INTO test_user <% quickInsert(data.userList)%>

--  adopt  SQL compiler  Auto output 

INSERT INTO 
  test_user (name, job, email, age, edu) 
  VALUES 
    ('test', 'Programmer', '[email protected]', 25, 'Undergraduate'),
    ('test2', 'Programmer', '[email protected]', 30, 'Undergraduate')

Node-MyBatis Built in tags - <Insert />

-- user = {
--  name: 'test', 
--  job: 'Programmer', 
--  email: '[email protected]', 
--  age: 25, 
--  edu: 'Undergraduate'
-- }

-- sql builder
<Insert table="test_user" values={data.user}></Insert>

--  adopt  SQL compiler  Auto output 
INSERT INTO 
  test_user (name, job, email, age, edu) 
  VALUES('test', 'Programmer', '[email protected]', 25, 'Undergraduate')


-- userList =   [
--  {name: 'test', job: 'Programmer', email: '[email protected]',  age: 25, edu: 'Undergraduate'}, 
--  {name: 'test2', job: 'Programmer', email: '[email protected]',  age: 30, edu: 'Undergraduate'}
--]

-- sql builder
<Insert table="test_user" values={data.userList}></Insert>

-- adopt  SQL compiler  Auto output 
INSERT INTO 
  test_users (name, job, email, age, edu) 
  VALUES 
    ('test', 'Programmer', '[email protected]', 25, 'Undergraduate'),
    ('test2', 'Programmer', '[email protected]', 30, 'Undergraduate')

at present Node-MyBatis be based on Midway The plug-in specification is built into the project , At present, independent modules are being separated to form independent solutions , And then for Midway To adapt and dock , Enhance the independence of the program .

3、Node-MyBatis actual combat

（1）API

/**
 *  Query database records that meet all criteria 
 * @param sql: string sql character string 
 * @param params  Pass to sql String dynamic variable object 
 */
query(sql, params = {})

/**
 *  Query the database for a record that meets the criteria 
 * @param sql: string sql character string 
 * @param params  Pass to sql String dynamic variable object 
 */
queryOne(sql, params = {})

/**
 *  Insert or update database records 
 * @param sql: string sql character string 
 * @param params  Pass to sql String dynamic variable object 
 */
exec(sql, params = {})

（2） Project structure

Because we choose to use Midway As our BFF Of Node frame , So our directory structure follows the standard Midway Structure .

.
├── controller                #  entrance  controller  layer 
│   ├── base.ts               # controller  Common base class 
│   ├── table.ts
│   └── user.ts
├── extend                    #  Yes  midway  An extension of 
│   ├── codes
│   │   └── index.ts
│   ├── context.ts
│   ├── enums                 #  Enumerated values 
│   │   ├── index.ts
│   │   └── user.ts
│   ├── env                   #  Extended environment 
│   │   ├── index.ts
│   │   ├── local.ts
│   │   ├── prev.ts
│   │   ├── prod.ts
│   │   └── test.ts
│   ├── helper.ts             #  Tool method 
│   └── nodebatis             # nodebatis  Core code 
│       ├── decorator         #  Declarative transaction encapsulation 
│       ├── plugin            #  Custom tool method 
│       ├── config.ts         #  Core configuration item 
│       └── index.ts
├── middleware                #  Middleware layer 
│   └── error_handler.ts      #  Extended error handling 
├── public
└── service                  #  Business  service  layer 
    ├── Mapping              # node-mybatis Of  mapping layer 
    │   ├── TableMapping.ts
    │   └── UserMapping.ts
    ├── table.ts             # table service  and  db Related calls  TableMapping
    └── user.ts              # user service  and  db Related calls  UserMapping

（3） Business scenario

According to the user id Query user information , When Node The service receives a query request for user information , according to URL The regular routing of the request is assigned to UserController Of getUserById Methods to deal with ,getUserById The way to do this is through UserService Call to complete the relevant data acquisition ,UserService adopt Node-MyBatis Complete the query of database user information .

Controller layer 

//controller/UserController.ts
import {  controller, get, provide } from 'midway';
import BaseController from './base'

@provide()
@controller('/api/user')
export class UserController extends BaseController {
  /**
   *  According to the user id Query all user information 
   */
  @get('/getUserById')
  async getUserById() {
    const { userId } = this.ctx.query;
    let userInfo:IUser = await this.ctx.service.user.getUserById({userId})
    this.success(userInfo)
  }
}

Service layer 

// service/UserService.ts
import { provide } from 'midway';
import { Service } from 'egg';
import UserMapping from './mapping/UserMapping';

@provide()
export default class UserService extends Service {
  getUserById(params: {userId: number}): Promise<{id: number, name: string, age: number}> {
     return this.ctx.helper.queryOne(UserMapping.findById, params);
  }
}

DAO layer 

// service/mapping/UserMapping.ts
export default {
  findById: `
    SELECT 
      id,
      name,
      age
    FROM users t1
    WHERE
      t1.id=#data.userId
  `
}

4、 Engineering system

（1） Type system

stay Node Service development , We need more engineering capabilities , Such as code prompt and automatic completion 、 Code checking 、 Reconstruction, etc , So we choose TypeScript As our development language , meanwhile Midway It also provides a lot for Typescript The support of .

We hope our Node-MyBatis You can also plug in the type of wings , According to the query data, it can automatically check, correct and complete . And then we are born tts (table to typescript system) Solution ,  It can be automatically generated according to the metadata of the database TypeScript The type definition file of .

chart -5 Database table structure

adopt tts -t test_user Automatically generate the type definition file of the form , as follows ：

export interface ITestUser {
  /**
   *  user id
   */
  id: number

  /**
   *  user name 
   */
  name: string

  /**
   *  User state 
   */
  state: string

  /**
   *  User mailbox 
   */
  email: string

  /**
   *  User age 
   */
  age: string
}

In this way, you can use this type of file in development , Bring some convenience to our daily development . Coupled with the TypeScript Advanced type containers such as Pick、Partial、Record、Omit The complex types can be adapted according to the query fields .

chart -6 tts The use of type files

（2）LSP

VSCode Basically become the first choice of the front-end development editor , In addition, through VSCode In the architecture LSP（ Language service agreement ） You can do a lot of IDE The function of , Provide more intelligent help for our development .

For example, we are using Node-MyBatis You need to write a lot of SQL character string , about VSCode Come on , This is a common JavaScript String , Nothing special .

But we look forward to going further , For example, it can automatically identify SQL Key words of , Syntax highlighting , And realize SQL Automatic beautification of . By developing VSCode plug-in unit , Yes SQL Intelligent analysis of grammatical features of , The following effects can be achieved , Realization SQL Code highlighting and formatting , It will support SQL Automatic completion of .

chart -7 SQL String template highlighting and formatting

in addition , Because it supports custom templates and methods , To write SQL The efficiency has been improved , about SQL Generation becomes less intuitive , You need to know at runtime SQL The string content of . How to solve this pain point , In fact, it can also be through LSP Solve this problem , Balance efficiency and maintainability , By developing   VSCode Plug in for , Intelligent analysis SQL Template structure is generated by suspending prompt in real time SQL .

Four 、 summary

Here comes the article , It's coming to an end , Thank you for your attention and company . In this paper, we review the Wukong activities Node Some thoughts and explorations on the design of service data layer persistence solution , We want to keep SQL Simple, universal and powerful , It can guarantee the ultimate development experience , Hope to pass Node-MyBatis The design of our design has fulfilled our thinking .

author ： vivo Wukong middle stage R & D team

Previous reading ：

版权声明
本文为[Vivo Internet technology]所创，转载请带上原文链接，感谢

边栏推荐

猜你喜欢

随机推荐