当前位置：网站首页>3000 words to teach you how to use mot

3000 words to teach you how to use mot

2022-06-11 15:48:00 【Gauss squirrel Club】

Last issue We introduced MOT Features and deployment . Said why to use , The next step is how to use MOT. Use MOT It's simple , The following sections will describe .

openGauss Allow applications to use MOT And standard disk based tables .MOT For the most active 、 High contention and throughput sensitive application tables , It can also be used for all application tables .

The following command describes how to create MOT, And how to convert existing disk based tables to MOT, To accelerate the database related performance of the application .MOT It is especially beneficial for tables that have proved to be bottlenecks .

Workflow Overview

The following is related to the use of MOT A brief overview of related tasks ：

This section also describes how to perform various and MOT Related additional tasks , as well as MOT SQL Coverage and limitations .

1. Grant user rights

To grant database users access to MOT Take the access permission of the storage engine as an example . Each database user executes only once , This is usually done during the initial configuration phase .

  explain ： MOT Through the external data wrapper （Foreign Data Wrapper,FDW） Mechanism and openGauss Database integration , Therefore, you need to authorize user permissions .

To enable specific users to create and access MOT（DDL、DML、SELECT）, The following statement is executed only once ：

GRANT USAGE ON FOREIGN SERVER mot_server TO <user>;

All keywords are case insensitive .

2. establish / Delete MOT

establish MOT It's simple . Only MOT The create and delete table statements in are the same as openGauss The statements of disk based tables in are different .SELECT、DML and DDL The syntax of all other commands for MOT Table and openGauss Disk based tables are the same .

  • establish MOT：

    create FOREIGN table test(x int) [server mot_server];

  • In the above sentence ：

    • Always use a FOREIGN Keyword reference MOT.
    • Creating MOT Table time ,[server mot_server] Part is optional , because MOT Is an integrated engine , Instead of a separate server .
    • Above to create a named test Memory table of （ There is one in the table called x Integer column of ） For example . In the next section （ Create index ） A more realistic example will be provided .
    • If postgresql.conf Incremental checkpoints are enabled in , You can't create MOT. So please create MOT Former general enable_incremental_checkpoint Set to off.

  • Delete the name test Of MOT：

    drop FOREIGN table test;

3. by MOT Create index

Supporting the standard PostgreSQL Create and delete index statements .

for example ：

create index text_index1 on test(x) ;
 Create one for TPC-C Of ORDER surface , And create indexes ：
create FOREIGN table bmsql_oorder ( 
  o_w_id       integer      not null, 
  o_d_id       integer      not null, 
  o_id         integer      not null, 
  o_c_id       integer not null, 
  o_carrier_id integer,          
  o_ol_cnt     integer, 
  o_all_local  integer, 
  o_entry_d    timestamp, 
  primary key (o_w_id, o_d_id, o_id) 
); 
create index  bmsql_oorder_index1 on bmsql_oorder(o_w_id, o_d_id, o_c_id, o_id) ;

  explain ：  stay MOT You don't need to specify a name before FOREIGN keyword , Because it is only used for commands to create and delete tables .

of MOT Index limits , Please see the “MOT SQL Coverage and limitations ” The index part of .

4. Convert disk table to MOT

The disk table is directly converted to MOT Not yet achieved , This means that there is no way to convert disk based tables to MOT Of ALTER TABLE sentence .

The following describes how to manually convert disk based tables to MOT, How to use gs_dump Tools export data , And how to use it gs_restore Tools import data .

4.1 Precondition check

Check to be converted to MOT Whether the schema of the disk table contains all the required columns .

Check whether the schema contains any unsupported column data types , Specific see “ Unsupported data type ” chapter .

If a specific column is not supported , It is recommended to first create a spare disk table with updated mode . This mode is the same as the original table , Only all unsupported types have been converted to supported types .

Use the following script to export the spare disk table , Then import to MOT in .

4.2 transformation

To convert a disk based table to MOT, Please perform the following steps ：

  1. Pause application activity .
  2. Use gs_dump The tool dumps the table data to a physical file on disk . Please make sure to use data only.
  3. Rename the original disk based table .
  4. Create a with the same name and pattern MOT. Please make sure to use to create FOREIGN Keyword specifies that the table is MOT.
  5. Use gs_restore Load the data of the disk file / Restore to database table .
  6. Browse or manually verify that all raw data is correctly imported into the new MOT in . Here is an example .
  7. Resume application activity .

  instructions ：  Because the table name remains unchanged , Application queries and related database stored procedures will have seamless access to new data MOT, Without changing the code . Please note that ,MOT Cross engine multi table query is not supported at present （ If you use Join、Union And subquery ） And cross engine multi table transactions . therefore , If you query in multiple tables 、 Accessing the original table in a stored procedure or transaction , All relevant disk tables must be converted to MOT, Or change the relevant code in the application or database .

4.3 Conversion example

Suppose you want to change the database benchmarksql A disk based table in customer Migrate to MOT in .

take customer Table migrated to MOT, The operation steps are as follows ：

  1. Check the source table column type . verification MOT Supports all types of , Please refer to “ Unsupported data type ” chapter .

    benchmarksql-# \d+ customer 
                       Table "public.customer" 
 Column |  Type   | Modifiers | Storage | Stats target | Description 
--------+---------+-----------+---------+--------------+------------- 
 x      | integer |           | plain   |              | 
 y      | integer |           | plain   |              | 
Has OIDs: no 
Options: orientation=row, compression=no

  2. Please check the source table data .

    benchmarksql=# select * from customer; 
 x | y 
---+--- 
 1 | 2 
 3 | 4 
(2 rows)

  3. Only use gs_dump Dump table data .

    
$ gs_dump -Fc benchmarksql -a --table customer -f customer.dump -p 16000
gs_dump[port='15500'][benchmarksql][2020-06-04 16:45:38]: dump database benchmarksql successfully 
gs_dump[port='15500'][benchmarksql][2020-06-04 16:45:38]: total time: 332  ms

  4. Rename table source .

    benchmarksql=# alter table customer rename to customer_bk; 
ALTER TABLE

  5. Create exactly the same... As the source table MOT.

    benchmarksql=# create foreign table customer (x int, y int); 
CREATE FOREIGN TABLE 
benchmarksql=# select * from customer; 
 x | y 
---+--- 
(0 rows)

  6. Import the source dump data into a new MOT in . 
    $ gs_restore -C -d benchmarksql customer.dump -p 16000
restore operation successful 
total time: 24  ms 
Check that the data was imported successfully. 
benchmarksql=# select * from customer; 
 x | y 
---+--- 
 1 | 2 
 3 | 4 
(2 rows) 
      
benchmarksql=# \d 
                                List of relations 
 Schema |    Name     |     Type      | Owner  |             Storage 
--------+-------------+---------------+--------+---------------------------------- 
 public | customer    | foreign table | aharon | 
 public | customer_bk | table         | aharon | {orientation=row,compression=no} 
(2 rows)

5. Query native compilation

MOT Another characteristic of , Before the precompiled full query needs to be executed , Can be in native format （ Use PREPARE sentence ） Prepare and parse these queries .

This native format facilitates more efficient execution later （ Use EXECUTE command ）. This type of execution is much faster , Because the native format bypasses multiple database processing layers during execution , For better performance .

This division of labor avoids repeated analytical operations . Queries and transaction statements can be executed interactively . This feature is sometimes called instant （Just-In-Time,JIT） The query compiler .

5.1 The query compiler ：PREPARE sentence

If you want to use MOT Native Query compilation , Call... Before executing the query PREPARE Client statement .MOT Precompiled queries and （ or ） Preload previously precompiled code from the cache .

Here is SQL in PREPARE Examples of Syntax ：

PREPARE name [ ( data_type [, ...] ) ] AS statement

PREPARE Create a preprocessing statement in the database server , This statement is a server-side object that can be used to optimize performance .

5.2  Run the command

issue EXECUTE On command , Will parse 、 analysis 、 Rewrite and execute preprocessing statements . This division of labor avoids repeated analytical operations , At the same time, make the execution plan dependent on specific settings .

The following is the Java Call in application PREPARE and EXECUTE Examples of statements .

conn = DriverManager.getConnection(connectionUrl, connectionUser, connectionPassword); 

// Example 1: PREPARE without bind settings 
String query = "SELECT * FROM getusers";  
PreparedStatement prepStmt1 = conn.prepareStatement(query); 
ResultSet rs1 = pstatement.executeQuery()) 
while (rs1.next()) {…} 

// Example 2: PREPARE with bind settings 
String sqlStmt = "SELECT * FROM employees where first_name=? and last_name like ?"; 
PreparedStatement prepStmt2 = conn.prepareStatement(sqlStmt); 
prepStmt2.setString(1, "Mark"); // first name “Mark” 
prepStmt2.setString(2, "%n%"); // last name contains a letter “n” 
ResultSet rs2 = prepStmt2.executeQuery()) 
while (rs2.next()) {…}

MOT The supported and unsupported features of compilation are shown below .

5.3 Lightweight execution of supported queries

The following query types are suitable for lightweight execution ：

  • Simple point query

    • SELECT (including SELECT for UPDATE)
    • UPDATE
    • DELETE

  • INSERT Inquire about

  • The range of the complete prefix that references the primary key UPDATE Inquire about

  • The range of the complete prefix that references the primary key SELECT Inquire about

  • JOIN Inquire about , One part or two parts overlap into point queries

  • Reference the complete prefix of the primary key in each connection table JOIN Inquire about

5.4 Lightweight execution of unsupported queries

Any special query properties are not suitable for lightweight execution . In particular, if any of the following conditions apply , Then the query is not suitable for lightweight execution . For more information , see also “ Native compilation and lightweight execution of unsupported queries ”.

One point needs to be emphasized , If the query statement is not suitable for native compilation and lightweight execution , Do not report errors to clients , The query is still executed in a normal and standardized way .

of MOT Details of native compilation functions , see also “ Query native compilation ” or “ Query native compilation （JIT）” About .

6. Retry abort transaction

In optimistic concurrency control （OCC） in , stay COMMIT Transaction period before phase （ Use any isolation level ） Records are not locked . This is a powerful advantage that can significantly improve performance . Its disadvantages are , If another session tries to update the same record , Then the update may fail . So you have to abort the whole transaction . These so-called update conflicts are caused by MOT Detected by the version checking mechanism at the time of submission .

  explain ：  Engines that use pessimistic concurrency control , As standard Postgres and openGauss Disk based tables , When using SERIALIZABLE or REPEATABLE-READ At the isolation level , A similar abnormal abort will occur .

This update conflict is common in OLTP Very rare in the scene , In the use of MOT It is especially rare when . however , Since this is still possible , Developers should consider using transaction retry code to solve this problem .

Let's take multiple sessions trying to update the same table at the same time as an example , Explains how to retry the table command . For more details , see also “OCC And 2PL Examples of the differences between ” part . Let's say TPC-C Take payment transactions as an example .

int commitAborts = 0; 

while (commitAborts < RETRY_LIMIT) { 

    try {                         
        stmt =db.stmtPaymentUpdateDistrict; 
        stmt.setDouble(1, 100); 
        stmt.setInt(2, 1); 
        stmt.setInt(3, 1); 
        stmt.executeUpdate(); 

        db.commit();                      

        break; 
    }               
    catch (SQLException se) { 
        if(se != null && se.getMessage().contains("could not serialize access due to concurrent update")) { 
            log.error("commmit abort = " + se.getMessage()); 
            commitAborts++; 
            continue; 
        }else { 
            db.rollback(); 
        } 

        break; 
    } 
}

7. MOT External support tools

For support MOT, Modified the following external openGauss Tools . Make sure you are using the latest version of the tool . The following will introduce and MOT Related usage . A complete description of these tools and how to use them , see also openGauss Tool reference .

gs_ctl（ Full and incremental ）

This tool is used to create a standby server from the primary server , And when the server's timeline deviates , Synchronize the server with its replica .

At the end of the operation , The tool will get the latest MOT checkpoint , Consider at the same time checkpoint_dir Configuration values .

Checkpoint from the source server checkpoint_dir Read to the target server checkpoint_dir.

at present MOT Incremental checkpoints are not supported . therefore ,gs_ctl Incremental build for MOT It doesn't work incrementally , But work in a full-scale way .Postgres Disk tables can still be built incrementally .

gs_basebackup

gs_basebackup Used to prepare the basic backup of the running server , Does not affect other database clients .

MOT Checkpoints are also obtained at the end of the operation . however , The location of the checkpoint is from the source server checkpoint_dir Acquired , And transfer to the source data directory , In order to back up correctly .

gs_dump

gs_dump Used to export database schema and data to a file . Support MOT.

gs_restore

gs_restore Used to import database schema and data from files . Support MOT.

8. MOT SQL Coverage and limitations

MOT The design can almost cover SQL And future feature sets . for example , Most support standard Postgres SQL, It also supports common database features , Like stored procedures 、 Custom functions, etc .

Here are a variety of SQL Coverage and limitations .

8.1 Unsupported feature

MOT The following features are not supported ：

  • Cross engine operation ： Cross engine is not supported （ disk +MOT） Query for 、 View or transaction . Plans to 2021 Realize this feature in .
  • MVCC、 Isolation ： No snapshot is supported / Serializable isolation . Plans to 2021 Realize this feature in .
  • Native compilation （JIT）：SQL Limited coverage . Besides , The of stored procedures is not supported JIT compile .
  • Local memory is limited to 1GB. A transaction can only be changed less than 1GB The data of .
  • Capacity （ data + Indexes ） Limited by available memory . The future will provide Anti-caching And data tiering .
  • Full text indexing is not supported .
  • The logical copy feature is not supported .

Besides , Below is a detailed list of MOT、MOT Indexes 、 Query and DML Various general restrictions on syntax , Features and limitations of Native Query compilation .

8.2 MOT Limit

MOT Functional limitation ：

  • Partition by range
  • AES encryption
  • Flow operation
  • Custom type
  • Sub business
  • DML trigger
  • DDL trigger
  • “C” or “POSIX” Sorting rules other than

8.3 Don't support DDL operation

  • Modify table structure
  • establish including surface
  • establish as select surface
  • Partition by range
  • Create a no logging clause （no-logging clause） Table of
  • Create a deferrable constraint primary key （DEFERRABLE）
  • Rebuild index
  • Table space
  • Use the subcommand to create the schema

8.4 Unsupported data type

  • UUID
  • User-Defined Type (UDF)
  • Array data type
  • NVARCHAR2(n)
  • NVARCHAR(n)
  • Clob
  • Name
  • Blob
  • Raw
  • Path
  • Circle
  • Reltime
  • Bit varying(10)
  • Tsvector
  • Tsquery
  • JSON
  • HSTORE
  • Box
  • Text
  • Line
  • Point
  • LSEG
  • POLYGON
  • INET
  • CIDR
  • MACADDR
  • Smalldatetime
  • BYTEA
  • Bit
  • Varbit
  • OID
  • Money
  • Unlimited varchar/character varying
  • HSTORE
  • XML

8.5 Unsupported index DDL And index

  • Create indexes on decimal and numeric types

  • Create an index on a nullable column

  • The total number of indexes created in a single table >9

  • In key size >256 Create index on table

    The key size includes the column size in bytes + Column additional size , This is the cost of maintaining the index . The following table lists the additional column sizes for different column types .

    Besides , If the index is not unique , Additional needs 8 byte .

    Here is the pseudo code to calculate the key size ：

    keySize =0; 
      
for each (column in index){ 
      keySize += (columnSize + columnAddSize); 
} 
if (index is non_unique) { 
      keySize += 8; 
}

    Column type

    Column size

    Column additional size

    varchar

    N

    4

    tinyint

    1

    1

    smallint

    2

    1

    int

    4

    1

    longint

    8

    1

    float

    4

    2

    double

    8

    3

Types not specified in the above table , The additional size of the column is zero （ For example, time stamp ）.

8.6 Don't support DML

  • Merge into
  • Select into
  • Lock table
  • Copy from table

8.7 Native compilation and lightweight execution of unsupported queries

  • The query involves more than two tables
  • The query has any of the following conditions ：
    • Aggregation of non-native types
    • Window function
    • Subquery sublink
    • Distinct-ON Modifier （distinct Clause from DISTINCT ON）
    • recursive （ Has been designated WITH RECURSIVE）
    • modify CTE（WITH There is INSERT/UPDATE/DELETE）

The following sentences do not support lightweight execution ：

  • Returning list
  • Group By clause
  • Grouping sets
  • Having clause
  • Windows clause
  • Distinct clause
  • Sort clause that does not conform to native index order
  • Set operations
  • Constraint dependencies

Okay , Today's article is here . Have you learned ？

原网站

版权声明
本文为[Gauss squirrel Club]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/162/202206111535103214.html

边栏推荐

猜你喜欢

随机推荐