QT writing the Internet of things management platform 38- multiple database support

One 、 Preface

At the beginning of the design of this system, it is required to support a variety of different databases , such as sqlite、mysql、postgres、sqlserver etc. , Even domestic databases, such as NPC Jincang kingbase etc. ,（ Due to the vigorous promotion of localization , Domestic databases must also be supported ）,Qt Database components encapsulated in sql modular , All databases are abstracted , This is a good prerequisite for the application to do a variety of database support , So more in the details , For example, database date range query , Different databases are handled differently , This requires different processing in the components encapsulated by yourself , There is also Database Paging , The paging query statements of different databases are different , such as sqlite and mysql It uses limit keyword ,postgres and kongbase The database uses limit offset keyword ,sqlserver It's using top keyword ,oracle The most complex needs to be combined with hidden columns , This requires special treatment in the wheels they package , Encapsulated into a function and called directly .

Some experience summary of database ：

• stay sqlite There is no concept of data type in the database , You set up int You can still store strings . Other databases have data types .
• Try to avoid database keywords and function keywords when designing database fields , such as year、month、plan etc. , Execute if necessary sql The corresponding field should be enclosed in quotation marks .
• except sqlite Outside the database , Other databases have length requirements , For example, it's set 2, You have to insert the length 3 May be an error , Execution unsuccessful .
• varchar Type stores the string ,gbk Code the next Chinese character 2 byte ,utf8 Code the next Chinese character 3 byte , So we must consider well , Like content ‘ Enable ’ It is best to set the length 6 byte .
• In order to minimize the volume of database files , It is recommended to set as long as possible when the field length is known .
• Database tables with a large number of record rows , Tables that often need to be queried , Index must be set , Otherwise it's very slow .
• For example, a table with a small amount of data has fewer than a few thousand items , And there is little need to change , Then it is unnecessary to set the index , This will speed up the insertion .
• The integer fields INT、INTEGER Cannot set length , There is no need to set the length , Otherwise, in the sqlserver An error will be reported when the database is executed .
• It is recommended to set a primary key field for each table , In particular, tables that need to be indexed must have primary key fields .
• linux Installation on mysql Client commands ：apt-get install libmysqlclient-dev .

About Qt Some cold knowledge of database development .

• Qt That is, it supports direct communication with the database in the form of a library , Also support ODBC The form of data source communicates with various databases , So it covers all the situations .
• Qt The database program is packaged and released , All premises ： Pay attention to distinguish between 32/64 position , Your program is 32 You have to take it with you 32 Bit Library ,64 You must bring 64 Bit Library , this Qt This is also the requirement for the library .mysql Publishing is the easiest , Take one mysql The dynamic library file of （windows On is libmysql.dll）, It's simple .sqlserver No need to bring it , Because he is the son of Microsoft , The general operating system comes with .postgres Need to bring libpq.dll、libintl-8.dll、libiconv-2.dll、libeay32.dll、ssleay32.dll Just a few files .oracle Need to bring oci.dll、oraociei11.dll（ This file is very big 130MB+）, If not, it is recommended to install one directly oracle client Client software , Then corresponding bin Just set the directory to the environment variable .
• Test it after packaging and publishing , Find out 32 Bit programs can also be connected normally 64 Bit mysql,64 Bit programs can also be connected normally 32 Bit mysql, Therefore, the judgment is as long as it is consistent with the number of bits in the library of the program （ The same rule applies when compiling ,32 Bit Qt The program compilation database plug-in also needs to use 32 Bit database link library .）, It does not need to be consistent with the number of digits in a specific database , tested mysql、sqlserver、postgresql Databases are similar rules .
• A lot of tests are compared , adopt odbc A large number of data records are inserted in batch by means of data source and direct database , Direct connection is faster , about 5% about , Therefore, it is recommended to adopt this method as far as possible , It is only in the environment without this method that odbc The way of data sources ,Qt Default by oneself odbc Database plug-ins .
• Different databases are executing sql Script time , The table name or field name will be automatically converted to uppercase or lowercase ,mysql The table name will be converted to lowercase 、postgresql The table name and field name will be converted to lowercase 、oracle The table name and field name will be capitalized . This leads to the use of QSqlTableModel call setTable When setting the database table name , It must be consistent with the table name in the database , Case sensitive , So that's right postgresql and oracle You must pay attention to the database , I have been stuck here for a long time , Almost put this huge shit basin in Qt Of BUG On .

void DbHelper::bindTable(const QString &dbType, QSqlTableModel *model, const QString &table)
{
    
    //postgresql All lowercase ,oracle All capitals , The two databases strictly distinguish the case of table names and field names 
    QString flag = dbType.toUpper();
    if (flag == "POSTGRESQL") {
    
        model->setTable(table.toLower());
    } else if (flag == "ORACLE") {
    
        model->setTable(table.toUpper());
    } else {
    
        model->setTable(table);
    }
}

• Qt It supports opening a database without specifying a database name , Because sometimes it is necessary to connect to the database server , perform sql Statement create database . How to connect to the database does not exist yet , Tests found sqlite、mysql、sqlserver、postgresql All support this feature . The premise of deleting and creating a database is that the database is not occupied by other programs , For example, if other programs have opened the database, the execution will fail . Here I have tortured many times , Why did the execution fail ？ Later, I found that the third-party database tool has opened the database , Just turn off the tools ok 了 .

QSqlDatabase database = QSqlDatabase::addDatabase("QMYSQL");
//database.setDatabaseName("dbtool");
database.setHostName("127.0.0.1");
database.setPort(3306);
database.setUserName("root");
database.setPassword("root");

if (database.open()) {
    
    QSqlQuery query(database);
    qDebug() << " Delete database " << query.exec("drop database dbtool");
    qDebug() << " Create database " << query.exec("create database dbtool");
    if (query.exec("select * from userinfo")) {
    
        while (query.next()) {
    
            qDebug() << " Query the database " << query.value(0);
        }
    }
} else {
    
     qDebug() << " Open database " << database.lastError().text();
}

• use QSqlQueryModel+QTableView Display the data ,int Data of type , If exceeded 100 ten thousand , Will become a scientific count display , This is very annoying , Definitely not the result you want . Search the Internet , Finally found a friend with the same problem , You need to add an empty delegate to this column . It is not possible to find an empty delegate later , exceed 1000 Ten thousand pieces are awesome again , Need the ultimate Dafa overloaded data model display .

ui->tableView->setItemDelegateForColumn(0, new QItemDelegate);

// Here is the ultimate Dharma 
QVariant SqlQueryModel::data(const QModelIndex &index, int role) const
{
    
    QVariant value = QSqlQueryModel::data(index, role);
    // exceed 100 The value of ten thousand will be displayed by scientific counting. It needs to be converted here into a string display 
    if (role == Qt::DisplayRole) {
    
        int result = value.toInt();
        if (result >= 1000000) {
    
            value = QString::number(result);
        }
    }
    return value
}

• mysql Databases have multiple database engines , among MyIsam Database transactions are not supported , The default is usually this engine , So when you use Qt Medium transaction After the method commit At the time of submission , You will find it unsuccessful , In fact, it is successful , It is correct to check the corresponding results in the database . There are two ways , The first is to change the database engine to InnoDB, The second is to make a wrong judgment after submitting if (database.commit() || !database.lastError().isValid()) , Error unavailability also indicates success .
• If the odbc Data source communication , You only need to set the database name setDatabaseName、 Set user name setUserName、 Set user password setPassword These three parameters are sufficient , Because the corresponding host address, port and associated database name have been set when configuring the data source , So in use odbc When communicating with the data source, you only need to verify the user information again . What we should pay special attention to here is setDatabaseName To set the database name, fill in the name of the data source configuration .
• After a lot of comparative tests , Including the insert 、 Delete 、 Batch 、 Inquire about 、 Paging and other operations , Tens of millions of data , stay Qt The response speed of the database part , The friendliness ranking is sqlite > postgresql > oracle > mysql > odbc . Tens of millions or more postgresql > oracle > mysql > sqlite > odbc . Above 100 million is oracle > postgresql > other . The above tests are based on the level of beginners , As for sub database and sub table 、 The joint query 、 cache 、 Memory database and other advanced knowledge points are useless .
• mysql There are mainly two versions ,mysql5.7 and mysql8, Officially it's 8 Than 5 Much faster , Personal test down ,5.7 Than 8 Much faster , Whether it's a query , Or batch insert data , I don't know why , This is also the result of online search （https://www.coder4.com/archives/7596）, Everybody says 8 A lot slower .
• mysql There is a branch called mariadb, Than mysql More pure , It is said that all aspects are hanged mysql（https://blog.csdn.net/x275920/article/details/123847792）, Personal comparison tests show that the performance of batch insert and query is better , And fully compatible mysql, Even the library file can be renamed directly , For example, will libmariadb.dll Change to libmysql.dll You can use it directly , And it's eight times smaller , This is good , When it was released, there were several megabytes less .
• If it is Qt+mysql Program , When publishing, the library version should be consistent with the corresponding database version of the plug-in , Otherwise, there may be no database transaction feature ,database.driver()->hasFeature(QSqlDriver::Transactions) For false .
• QSqlTableModel It's very well packaged , Not all data is loaded at once , Instead, the required data is loaded as the scroll bar is pulled , Test 100 million watches , Very fast , The same speed as thousands of watches .
• When connecting to the network database , If your local network has a proxy , For example, the agent is used github Wait for the website , You will find Qt Database program of cannot be connected , You need to set not to use local proxy settings QNetworkProxyFactory::setUseSystemConfiguration(false) . If this place is not careful, it will find problems and find you doubting life .

Two 、 Functional characteristics

2.1 Software modules

1. Equipment monitoring module , Including data monitoring （ Show... In tabular form ）、 Device panel （ Display in panel form ）、 Map monitoring （ Display in map form ）、 Curve monitoring （ Show in curve form ）.
2. Data query module , Including alarm records 、 Operation record 、 Operation record .
3. System setup module , Including basic settings 、 Port Management 、 Controller management 、 Detector Management 、 Alarm linkage 、 Type setting, etc .
4. Other setting modules , Including user management 、 Map Management 、 Position adjustment 、 Configuration design 、 Equipment commissioning, etc .

2.2 Basic function

1. Device data acquisition , Support serial port 、 The Internet , Serial port can set serial port number 、 Baud rate , The network can be set IP Address 、 Communication port .
2. Each port supports acquisition cycle time , Default 1 One device per second .
3. Support to set communication timeout times , Default 3 Time .
4. Support maximum reconnection time , For rereading offline devices .
5. Controller information , Can add controller name , Select the controller address 、 Controller model , Set the number of detectors under the controller .
6. Detector information , Can add tag number 、 Detector model 、 Gas type 、 Gas symbol 、 High reported value 、 Under reported value 、 Buffer value 、 Zero value 、 Is it enabled? 、 Alarm sound 、 Background map 、 Storage cycle 、 The number of decimal places for numerical conversion 、 Alarm delay time 、 Type of alarm （HH,LL,HL） etc. .
7. Type management configurable controller model 、 Detector model 、 Gas type 、 Gas symbols, etc .
8. Map supports importing and deleting , The positions of all detectors on the map can be dragged and saved freely .
9. Port information 、 Controller information 、 Detector information 、 Type information 、 User information, etc , Both support import 、 export 、 Export to excel、 Print .
10. Operation record 、 Alarm records 、 Operation record , Both support multi condition combined query , For example, time period 、 controller 、 Detector, etc , All records support export to excel/pdf And print .
11. Operation record 、 Alarm records 、 All operation records can delete data within a specified time range .
12. The system can select the maximum number of records saved in the corresponding table , Automatically clean up early data , Leave enough space for important data .
13. Alarm SMS forwarding , Support multiple receiving mobile numbers , The sending interval can be set , Like instant delivery or 6 Send all alarm messages once an hour , The message is too long , Automatically split multiple messages .
14. Alarm mail forwarding , Support multiple receiving mailboxes , The sending interval can be set , Like instant delivery or 6 Send all alarm messages once an hour , Support attachment sending .
15. Set the Chinese title of the software 、 English title 、logo route 、 All rights reserved .
16. The switch is set for startup and operation 、 Alarm sound 、 automatic logon 、 Remember the password, etc .
17. Alarm sound can be set to play times , Interface style provides 18 Set skin file selection .
18. User management , Including user permission configuration , Different users can have different module permissions .
19. User login and user logout , Can remember password and auto login , More than three error prompts and close the program .
20. Four monitoring modes , Equipment panel monitoring 、 Map monitoring 、 Form data monitoring 、 Curve data monitoring , Free to switch , The collected data is displayed in real time in all four modes , Alarm flashing, etc .
21. Alarm relay linkage , A tag number can link multiple modules and relay numbers across serial ports , Support many to many .

2.3 Special function

1. Communication protocol supports modbus_com、modbus_tcp_rtu, Later expansion mqtt Such agreement .
2. The data source is in addition to the real hardware device collection , Optional database acquisition , In this way, users can arrange other programmers, such as java The programmer puts the data collected by the front end into the database , The system can collect data directly from the database . Database collection mode can be used as a general system , More suitable for multi person and multi system cooperation .
3. Smart skip timeout device , Speed up the acquisition of online devices , Especially useful when there are a large number of devices .
4. For smart skip timeout devices , Automatically collect once at the set reconnection time , In order to detect whether the equipment is online again .
5. Each detector can be controlled whether it is enabled , If it is not enabled, it will not collect , It will not be displayed on the interface , It is equivalent to temporary shutdown in the operation phase .
6. The detector can set the buffer value and alarm delay time , An alarm generated by fluctuations near this value , Not included in the alarm , Only when it is continuously at the alarm value and exceeds the alarm delay time can it be considered as a real alarm , This can avoid many false positives caused by fluctuations .
7. The detector can set the storage cycle , Store an operation record according to the set time , The storage cycle can be shorter for those with high importance according to the degree of importance , The unimportant setting is bigger , This can save a lot of storage space , It also ensures the timely storage of important data .
8. The detector can set the zero clearing value , When some high-precision and sensitive equipment may leave the factory, the default value may not be 0, The reset value needs to be set to represent the initial value .
9. The detector can set the decimal point , It is used to control the display of decimal places for the calculated real data , Equivalent to divided by 10、 Divide 100、 Divide 1000, In this way, most of the detector data directly control the real converted value through the decimal point setting , If special conversion is required, it can be agreed in the communication protocol .
10. The type of detector alarm supports multiple types , Some devices are higher than a certain value and report high , Under reporting below a certain value , And some equipment is in the range of minimum and maximum value, which is high alarm , Under reporting below the minimum value , Above maximum normal . In this way, it can be handled according to the situation , Covers various alarm types .
11. Original data import 、 export 、 Printer system , Cross platform does not rely on any components , Export data instantaneously .
12. Export to excel The records support all excel、wps Wait for the form file version , Do not rely on excel Such as software .
13. High color 、 Low color 、 Normal color 、 Default values: color, etc , Can be set freely .
14. Support cloud data synchronization , Synchronize the data collected locally to the cloud in real time .
15. Support network forwarding and network receiving , When network reception is turned on , Software from udp Receive data for parsing . Network forwarding supports multiple targets IP, In this way, the local acquisition software is realized , Free to transfer data to client , Check the collected data at any time .
16. Automatically remember the last user interface and other configuration information , Automatic application after restart .
17. The alarm automatically switches to the corresponding map , The detector button flashes , The table data is displayed in the corresponding color .
18. Double click the probe Icon , Pop up the corresponding detector details , The back control operation can be customized as required .
19. The database supports a variety of , Include sqlite、mysql、sqlserver、postgresql、oracle、 People's Congress, Jincang, etc .
20. The data collected by the local device is uploaded to the cloud in real time , So that cell phones APP perhaps web And so on .
21. Built in device simulation tools , Support data simulation of multiple devices of different models , At the same time, it also has database data simulation , In order to test the data when there is no equipment .
22. standard modbus agreement , Various controller types 、 Detector type 、 species 、 All custom symbols, etc , Very flexible and powerful , The communication protocol sample data is very complete , General various modbus Protocol system , It is applicable to various application scenarios .
23. At the same time, it integrates serial communication 、 Network communication 、 Database communication 、 Data import / export printing 、 Communication protocol analysis 、 Interface UI、 Many components and knowledge points such as global skin change , Very suitable for beginners and advanced .
24. Support xp、win7、win10、、win11、linux、mac、 All kinds of domestic systems （UOS、 Winning Qilin 、 Galaxy unicorn, etc ）、 The embedded linux Such as system .
25. Note complete , The project structure is clear , Super detailed complete use of the development manual , Accurate to the function description of each code file , Keep iterating over versions .

3、 ... and 、 Experience address

1. Domestic site ：https://gitee.com/feiyangqingyun
2. International sites ：https://github.com/feiyangqingyun
3. Personal home page ：https://blog.csdn.net/feiyangqingyun
4. Zhihu Homepage ：https://www.zhihu.com/people/feiyangqingyun
5. The product page ：https://blog.csdn.net/feiyangqingyun/article/details/97565652
6. Online document ：https://feiyangqingyun.gitee.io/qwidgetdemo/iotsystem/
7. Experience address ：https://pan.baidu.com/s/1ZxG-oyUKe286LPMPxOrO2A Extraction code ：o05q file name ：bin_iotsystem.zip.
8. Article navigation ：https://qtchina.blog.csdn.net/article/details/121330922

Four 、 design sketch

5、 ... and 、 Related codes

void frmConfigDb::on_btnConnect_clicked()
{
    
    {
    
        // Initialize database connection information structure data 
        DbInfo dbInfo;
        initDbInfo(dbInfo, connName);

        QString dbType = AppConfig::LocalDbType.toUpper();
        if (dbType == "SQLITE") {
    
            dbInfo.dbName = DbHelper::getDbDefaultFile(connFlag);
            if (QFile(dbInfo.dbName).size() <= 4) {
    
                QUIHelper::showMessageBoxError(" Database file does not exist !", 5);
                return;
            }
        }

        QSqlDatabase database;
        if (DbHelper::initDatabase(true, dbType, database, dbInfo)) {
    
            if (database.open()) {
    
                database.close();
                QUIHelper::showMessageBoxInfo(" Open database successfully !", 3);
            } else {
    
                QString error = database.lastError().text();
                QUIHelper::showMessageBoxError(" Failed to open database !\n" + error, 3);
            }
        } else {
    
            QString error = database.lastError().text();
            QUIHelper::showMessageBoxError(" Failed to connect to database !\n" + error, 3);
        }
    }

    QSqlDatabase::removeDatabase(connName);
}

void frmConfigDb::on_btnInit_clicked()
{
    
    if (QUIHelper::showMessageBoxQuestion(" Are you sure you want to initialize the database ?  All data will be cleared and cannot be restored !") != QMessageBox::Yes) {
    
        return;
    }

    QString sqlName = QString("%1/db/%2.sql").arg(QUIHelper::appPath()).arg(connFlag);
    QFile file(sqlName);
    if (!file.open(QIODevice::ReadOnly | QIODevice::Text)) {
    
        QUIHelper::showMessageBoxError(" Database script file open failed !", 3);
        return;
    }

    QElapsedTimer time;
    time.start();
    {
    
        // Initialize database connection information structure data 
        DbInfo dbInfo;
        initDbInfo(dbInfo, connName);

        QString dbType = AppConfig::LocalDbType.toUpper();
        if (dbType == "SQLITE") {
    
            dbInfo.dbName = DbHelper::getDbDefaultFile(connFlag);
            // If the file exists, delete the original database file first , Looks like win Not on 
            QFile file(dbInfo.dbName);
            if (file.exists()) {
    
                bool ok = file.remove();
                if (!ok) {
    
                    qDebug() << TIMEMS << "remove error" << dbInfo.dbName;
                }

                // Empty all tables 
                QStringList tables = QSqlDatabase::database().tables();
                foreach (QString table, tables) {
    
                    DbHelper::clearTable(table, dbType);
                    qDebug() << TIMEMS << "clearTable" << table;
                }

                // Close the default database connection 
                QSqlDatabase::database().close();
            }
        }

        // Initialize the database connection and open the database 
        QSqlDatabase database;
        if (!DbHelper::initDatabase(true, dbType, database, dbInfo)) {
    
            QString error = database.lastError().text();
            QUIHelper::showMessageBoxError(" Failed to connect to database !\n" + error, 3);
            return;
        }
        if (!database.open()) {
    
            QString error = database.lastError().text();
            QUIHelper::showMessageBoxError(" Failed to open database !\n" + error, 3);
            return;
        }

        QSqlQuery query(QSqlDatabase::database(connName));

        // First step : Delete the original database 
        QString sql = QString("DROP DATABASE %1;").arg(dbInfo.dbName);
        qDebug() << TIMEMS << "sql:" << sql << "result:" << query.exec(sql);

        // The second step : New database 
        sql = QString("CREATE DATABASE %1;").arg(dbInfo.dbName);
        qDebug() << TIMEMS << "sql:" << sql << "result:" << query.exec(sql);

        // The third step : Switch to the newly created database and execute the CREATE TABLE statement 
        database.close();
        if (!DbHelper::initDatabase(false, dbType, database, dbInfo)) {
    
            QString error = database.lastError().text();
            QUIHelper::showMessageBoxError(" Failed to connect to database !\n" + error, 3);
            return;
        }
        if (!database.open()) {
    
            QString error = database.lastError().text();
            QUIHelper::showMessageBoxError(" Failed to open database !\n" + error, 3);
            return;
        }

        // An error will be executed sql Statement is output to a file for easy viewing 
        QString fileName2 = QString("%1/db/error.sql").arg(QUIHelper::appPath());
        QFile file2(fileName2);

        QSqlQuery query2(QSqlDatabase::database(connName));

        sql = "BEGIN;";
        qDebug() << TIMEMS << "sql:" << sql << "result:" << query2.exec(sql);

        while (!file.atEnd()) {
    
            sql = QString::fromUtf8(file.readLine());
            sql.replace("\n", "");

            // Some statements that are not supported by the database jump over 
            if (DbHelper::existNoSupportSql(sql)) {
    
                continue;
            }

            // Re correct sql sentence 
            DbHelper::checkSql(dbType, sql);

            if (!query2.exec(sql)) {
    
                // Print and output error messages 
                QString error = query2.lastError().text();
                qDebug() << TIMEMS << "sql:" << sql << error;

                // If you don't open it, open it first 
                if (!file2.isOpen()) {
    
                    file2.open(QFile::WriteOnly | QFile::Append);
                }

                QString msg = QString(" Time [%1]  sentence : %2  error : %3\n").arg(DATETIME).arg(sql).arg(error);
                file2.write(msg.toUtf8());
            }
        }

        sql = "COMMIT;";
        qDebug() << TIMEMS << "sql:" << sql << "result:" << query2.exec(sql);
        database.close();

        //sqlite If the database is compressed, reduce the volume 
        if (dbType == "SQLITE") {
    
            DbHelper::execSql("VACUUM;");
        }
    }

    QSqlDatabase::removeDatabase(connName);
    double ms = time.elapsed();
    QString info = QString(" Database script executed successfully ,  Total time  %1  second !\n Remember to restart the program !").arg(QString::number(ms / 1000, 'f', 1));
    QUIHelper::showMessageBoxInfo(info, 3);
}

void frmConfigDb::on_cboxDbType_activated(int)
{
    
    // Automatically switch the default port number and other information 
    QString hostPort, userName, userPwd;
    QString dbType = ui->cboxDbType->currentText().toUpper();
    DbHelper::getDbDefaultInfo(dbType, hostPort, userName, userPwd);
    if (!hostPort.isEmpty() && AppConfig::LocalAutoInfo) {
    
        QStringList hostInfo = ui->txtHostInfo->text().split(":");
        ui->txtHostInfo->setText(hostInfo.at(0) + ":" + hostPort);
        ui->txtUserName->setText(userName);
        ui->txtUserPwd->setText(userPwd);
    }
}