Continuous test (CT) practical experience sharing

Preface

The traditional automobile industry is developing in the direction of Internet automobile

In the automotive industry “ New four modernizations ” Driven by the trend , Traditional distributed EE The architecture is developing towards domain centralized architecture , With the introduction of the concept of domain controller , The software will also be classified and integrated according to the corresponding functional domain , The amount of domain controller code is also increasing .SOA The development mode realizes the decoupling between the software function modules , Help the software team shorten the development cycle , Fast iteration software version .OTA The upgrade technology rapidly deploys the software to each domain controller , Provide guarantee for rapid iteration of software products . With the change of development mode , Iterative testing becomes short and frequent , Continuous testing will provide rapid quality assurance for software products .

What is continuous testing ？

Continuous testing comes from DevOps（Development and Operations） It is advocated in the book CI（ Continuous integration ）- CO( Continuous deployment ) - CT（ Keep testing ）- CD（ Continuous delivery ）, Through process and continuous integration / Continuous testing tool platform , automation “ Software delivery ”, Make build 、 test 、 Faster release of software 、 Frequent and reliable .

Why does the automotive industry need continuous testing ？

In the traditional automobile software development model , Testing methods include manual testing and automation after writing test scripts manually / Semi automated testing . It is characterized by “ people ” Centered , Repetitive manual testing by test engineers 、 Constantly update test scripts or configurations 、 Record test problems and test status tracking . Test cycle and quality , Depends on the number of test engineers 、 Experience, ability and number of tools .

chart 1： Traditional testing process

The method of continuous testing takes people 、 Testing process 、 Combination of testing tools , Through the combination of a series of test tool chains and automated test engineering , The test process begins when the new version of the software is submitted , Quickly find software defects , Solve the contradiction between testing and development in time , Help the software development team achieve the goal of software product speed and quality .

This article will focus on continuous testing , Share based on Jenkins and Vector Practical experience of software and hardware implementation .

Introduction to test scheme

System framework

chart 2： System diagram

The main function

Test process management

• Test project management module 、 Test requirements management module 、 Test case management module 、 Test task management module 、 Test defect management module 、 Test report management module 、 Remote device management module

Continuously verify the scheduling management

• Test object 、 Test equipment scheduling management 、 Code version library call management 、 Test process construction and implementation

IDL Script conversion tool integration

• Generate test script through test data file

The test script / Test data version control management

• be based on GitLab/SVN Or self built database management test script / Test data , Build an effective knowledge base

Data flow engine driven test data analysis

• Analyze based on test data , Get test results

The test report engine automatically tests reports

• Generate customized test reports based on test data analysis results

Case introduction

be based on Jenkins and Vector Tool implementation CAN/CANFD/LIN/FlexRay/ETH Network and diagnostic testing .

chart 3：CT technological process

The core of continuous testing is to automatically 、 Intelligently organize , And provide status query 、 Notification and other services , The implementation steps are as follows ：

• test plan ： Establish the corresponding test plan according to the software version submitted by the developer
• Test environment construction ： The test tool chain is connected with the test object , Meet the requirements of the test environment
• Update test cases ： Establish a test case library corresponding to the software version , And version management , Facilitate regression testing
• The test triggers ： Monitor developer submission status , The end of submission is the beginning of the test , use Jenkins Manage the process of test execution
• Test sample / Equipment management ： Automatic slave test sample / Select the environment in which the test can be executed in the device library , Improve the utilization of testing resources
• Automated test case scripts ： According to standardized test cases and databases , Automatically generate testable projects
• Data acquisition and analysis ： Use automated test engineering instead of test engineers to perform tests manually , Both shorten the test cycle , It also avoids the problems of false detection and missing detection caused by manual testing ; Analyze the test data , Automatically generate test reports and test problems , Avoid the repetitive work of test engineers to manually enter test problems
• Test report version management ： The test case 、 Test Engineering 、 Test report 、 The test data corresponds to the software version , It is convenient to trace and locate test problems
  The following will be prepared from the test 、 The test triggers 、 The steps of setting up a test environment are introduced in detail from three aspects: test execution and test problem recording .

Test preparation

The test preparation process configures various tools that need to be used in the test execution , It includes the following process ：

• Test plan establishment ： Use beihui self research PAVELINK perhaps JIRA Wait for the test management software to establish a test plan , Contains the test software version information , Test case library , Test responsible person, etc
• Update test cases ： Establish a test case library according to the newly released software version , Use GitLab/SVN Version management
• Test sample / Equipment management ： Manage test samples / Equipment status and test progress
• Test environment construction ： Test equipment and test samples for test environment debugging , To support automated testing . Use of test tools Vector Bus interface devices and CANoe Software
• Jenkins Test process management ： To configure Jenkins Job, Manage the process of test execution
  

chart 4： Traditional bus network and Ethernet test system

The test triggers

The automated test process passed Jenkins Job Configuration management .Jenkins Job Can be manually 、 timing 、 monitor GitLab/SVN Version or by Jenkins API Trigger call, etc . This program monitors and tests the software version of the sample , After the software is upgraded to the specified version , Use Jenkins API The scheduling mode triggers the start of the test process .

chart 5：Jenkins API

Test execution and test problem record

Jenkins Job Define a test process , Include the generation of automated test scripts 、 The test execution 、 Test question record .

• Automated test script generation ： Generate test scripts by parsing database files

  

chart 6： Test scripts are automatically generated

• The test execution ： adopt CANoe Software COM The method of interface call realizes the choice of test case Engineering 、 The test execution 、 Read back test data and test results

chart 7：CANoe COM Interface specification

• Test question record ： The test results read back are automatically uploaded to the test problem management system , Use beihui self-developed PAVELINK Test results of platform update test cases , Upload test data and create new test questions

chart 8：PAVELINK Problem management platform

• Jenkins With email sending function , At the end of the test execution , The test results and other information can be sent to the relevant responsible person by email .

Continuous testing is not equivalent to automated testing , Continuously test a series of processes , Include automated testing , Automated testing is a key step in continuous testing . By combining with software development iteration in the process , Quickly find software defects .

At the end

Using continuous testing can solve the problem of long manual testing cycle , Improved testing efficiency , Improve the productivity of testers . At the same time, control the test output and software development version , Meet the requirements of forward development and test management .

Continuous testing as a tool to improve testing efficiency , It does bring “ One button ” Work experience of hosting and executing tests . But as a tool , Continuous testing cannot meet all test scenarios . First , Continuous testing requires executable test cases to be automated , If you need to manually input parameters or change the test environment during the test , It is difficult to achieve the effect of improving the efficiency of testing . secondly , The test script can be generated automatically or the test script does not need to be modified , Facing the software products of each iteration , Writing or modifying test projects manually will also bring heavy workload to testers .

There are many and complex tool chains used in continuous testing , The overall stability of the test system needs special attention . Therefore, whether to conduct continuous testing needs to analyze the test scenario first , Nor can we abandon the accuracy of testing in order to pursue sustainability .

Beihui information follows the market trend , I have also accumulated a lot of experience in continuous testing , In addition to based on Jenkins Continuous testing program , Beihui information also self researched based on CANoe Software test management tools , Looking forward to communicating with colleagues in the industry .

reference
【1】 Talk about cars OTA test ： An overview
【2】 SOA framework / Test phase interface description language conversion scheme
【3】 White paper on the development of automotive basic software in China 2.0
【4】 Jenkins Authoritative guide

author ： Beihui information - Crazy Pipi

If you like this article, remember to comment and praise the collection More technical articles live courses , Please continue to pay attention to beihui information ️ For business consultation, please send private information to beihui or leave a message on the official website ️