On-off and on-off of quality system construction

Preface

Jia Pengbo is a test and Development Engineer of sound network , Responsible for sound network SDK And establish and improve relevant testing tools . Worked in a financial company , Look at the stock , The Commission is calculated , There is content distribution 、 Online retailers 、 Small game related testing experience .

This article is based on Jia Pengbo's 「 Sound network developer entrepreneurship Lecture • Phase III - how to ensure product quality in the early stage of entrepreneurship 」 Second sorting of shared content in the activity .

01 Background of quality system construction

First , From the figure 1 You know , Create from requirements , To assess 、 establish 、 Testing and distribution of requirements , And then to the final test , The workflow of requirements is very long , We need to build and track workflows , In order to better ensure business operation . secondly , Data records are used for double disk 、 Improvement and efficiency . Because the construction of quality system covers all aspects , It also includes the landing of some documents , With the increase of release and testing tasks , Various versions and test data will be accumulated accordingly , In this process, data analysis should be carried out , Get the data and analyze the duplicate 、 improvement , To promote the construction of quality system .

Last , The purpose of building a quality system is to get high-quality products . From R & D self-test 、 Advance of the testing stage 、 Requirements standard alignment 、 Develop test specifications 、Jira Information visualization and tracking , And other aspects of follow-up . All the services we do are for the launch of the version , Provide users with high quality 、 Highly reliable products and services .

■ chart 1

chart 2 It comprehensively shows before and now the establishment of the quality system , Quality level 、 Online problem 、 Test the closed loop of the problem 、 Process improvement 、 Use case addition and how to coordinate manual case And automation case . adopt “ before ” Graph , It is difficult to draw relevant conclusions , Only establish a system for continuous analysis , Will get the corresponding conclusions or indicators to improve . From the figure 2 You know “ before ” how , Now, what effects can be achieved by establishing a system , And the measures to be taken to realize the system . These measures are in all aspects , Not only need to land , Some may have to constantly align and repeat .

■ chart 2

02 methodology

1、 Small step run

Trot is used to quickly go online , Sometimes the need can be very urgent , Simply understand the purpose , After docking with the development or product manager, it is time to start work , Test cases may be too late to maintain and update . And because it is a stage of rapid growth , You may also need to go through parallelism .

The so-called parallelism is shown in the figure 3 Shown , Many people have N A need , this N Needs are tracked by different people , But these requirements are new , Just add it and go online quickly . The characteristics of this period are , Demand is new , Only test the part you are responsible for , Other needs have nothing to do with yourself , There is no need to return , You can directly add online . It should be noted that , Because only focus on fast online , So as long as a certain quality is guaranteed , The control of details is not in place .

■ chart 3

2、 Seek in stability “ - ” The high quality of

Pay attention to quality when the company is in a period of peace , Many considerations are needed . A requirement may be divided into N Sub requirements , These sub requirements are tested by different people . in addition , In the process of testing, we need to interact constantly , Delivery is achieved through different test points and covering different test contents , The whole test process may be relatively long , Because it needs to be verified in many aspects , Including manual testing 、 automated testing 、 Performance testing 、 Laboratory tests, etc , Use the comprehensive test results to determine whether the version has backtracking or quality problems . If the whole is in good condition , You can publish .

This kind of stability “ - ” The method is more suitable for companies with strict quality requirements , Such as banks , Because it has strict control over money , So its duration is very long , Pay close attention to the quality problems in the testing process , It requires different people to cross and complete relevant test tasks by exchanging test content and test cases , Is the characteristic of this period .

3、 Documents that standardize the process

At this time, the company is still in a period of peace , In the process of going through small steps, fast runs and long iterations of testing requirements , Has accumulated a variety of experience , These experiences need to be translated into process improvement and function improvement , How to find more problems is a test of demand sorting and data recording ability , See the picture for details 4 Shown ：

■ chart 4

Design test cases from requirements , Test cases may include Xmind and Excel, Next, develop review dependent case, At this point, you may want to stand in the perspective of code design, product use and QA Improved angle case, Modify to meet the online requirements . Of course , You may encounter various problems during the online process , These problems need to cover the function 、 performance , And it requires that in the process of continuous iteration （ Whether it's the test environment , Or pre release environment 、 Online environment, etc ）, All functions are normal , There is no fallback problem . The main feature of this stage is that the requirements need to be tracked from the beginning to the end 、 To sum up . meanwhile , You need to sort out the documents and pay attention to the specifications , Avoid duplicate documents, etc .

■ chart 5

chart 5 The current version of bug Number and proportion , Different teams bug The number can reflect the time to solve the problem to a certain extent , It helps to summarize the test progress , And control highlight Mission , So as to promote the timely launch of functions . chart 6 The problem solving time and the length difference of problem solving are recorded . In fact, the document data recorded by the document school is to better promote the test progress , Ensure timely or real-time online .

■ chart 6

4、 The tool flow of flowers competing

At present, various testing tools have been launched , Corresponding testing tools can be used according to different businesses to improve efficiency , Timely delivery of test tasks . chart 7 Eight categories of test tools are shown , This is just a simple enumeration , There are far more testing tools on the market .

■ chart 7

5、 The development flow of excellence

At this time, the company may be in a mature stage , The current tool flow can no longer support the test task , It needs to be developed for business with the help of tool flow , such as STF（Smartphone Test Farm） And Tencent Bugly, They are all secondary development to fit their own business . Of course , Now there are many tool flows , Include sonic、httprunner etc. , They cover all aspects , Including interface automation performance 、 Performance test, etc , These can be adapted to the business for secondary development . The main characteristics of this period are , High requirements for code , Need to fully understand the business , And meet business needs through code capabilities .

chart 8 It shows the secondary development and related practices of our company for the current tools , I will briefly elaborate later .

■ chart 8

03 Landing and practice

1、SDK Real time and efficient closed-loop processing of crash data

Our company mainly tests SDK, Compared with App There may be some differences , Its landing practice is mainly SDK Real time and efficient closed-loop processing of crash data . There are some such tools on the market , such as Bugly, But we found that Bugly Processing SDK There are still some deficiencies in the process of reporting the crash , such as SDK There was a crash , however Bugly No monitoring . So we have made some improvements in this regard , The overall process is shown in the figure 9 Shown , our SDK It serves customers , Embedded in the customer's App in , Its whole closed loop is , The program executes normally to monitor whether crash, And then by monitoring crash State finding daemon aware capture , Next generate dmp File and submit to the crash collector , To analyze the relevant system , At this time, some relevant information will be stored , Finally, integrate the information and send it to users .

■ chart 9

chart 10 It shows that SDK Real time and efficient closed-loop processing of crash data , First, extract the stack key information , Then judge according to the hash value , If the same version Hash The value is , Then it is related JIRA. The processing logic of this part is to provide the crash log to relevant developers for timely processing after obtaining it .

■ chart 10

chart 11 It shows how to collect logs and compile them through the parsing system crash after , stay JIRA Show and assign . First, get the crash information , Then report it to JIRA With some key information , At the same time, inform relevant developers , Make it repair . Simply speaking ,SDK The real-time and efficient closed-loop processing of crash data is in SDK The embedded App after , Determine whether it is our SDK the crash, Because some may be customers App Of itself crash, We will judge this part , if SDK Of crash It will be reported directly, and then parsed by the compilation system and reported to JIRA, In this process, it will be submitted to relevant developers for processing , Complete the whole link .

■ chart 11

2、 Circuit plugging

This part mainly introduces the experiment of plugging headphones , Because of our SDK You may need to constantly plug and unplug headphones during the test . chart 12 It is a simplification of the circuit diagram , Simply put, it's on the motherboard , Weld the wire in the earphone with the wire in the circuit diagram .

The original circuit diagram is very complicated , Students who have studied integrated circuits should know , There are various nodes , Test these points , Combined with the principle of circuit diagram , The function of plugging in and out headphones can be achieved through the program control circuit . We used to spend a lot of manpower in the process of testing and plugging headphones , Conservatively estimate a person 3 All the days online You can finish all the work on it case. But after finishing this circuit diagram , It only takes one hour to finish all the related case. We will show the relevant data on the circuit diagram （ Including the analysis of Web data , And the tailoring of performance indicators ） Integration , It is equivalent to the commercialization of earphone plugging .

■ chart 12

3、HENGE Test platform

chart 13 Is based on STF Cloud platform for secondary development , Its main link is to dynamically obtain the test package to get its information , Then dynamically send it to Yunzhen machine for testing , Then submit the relevant test results to the relevant testers , It is convenient for them to obtain the test report in order to draw the test conclusion . The figure mainly shows our test App After the process is issued, the package is loaded and operated , View the report after running , Of course, this includes crash Log and related functions .

■ chart 13

4、Moonlight

Our products Moonlight Is open source （https://github.com/AgoraIO-Community/MoonLight） 了 , It's a SDK, For automatic performance data collection , Interested students can check the relevant code . Some performance tests on it show , It can obtain very accurate information with low resource consumption CPU、memory And so on .

04 System diagram

From why to build a quality system , Different strategies may be adopted in the next five periods , As well as various gadgets that may be born in the process of strategy, they all serve the high-quality products we provide , This is the last and only indicator . All testing tasks , Including the construction of quality system is to maintain the current business development . So no matter what strategy you use , Whether it's document School 、 Tool flow , Or secondary development , All node or task events , They all serve their own business systems .

chart 14 It shows a quality system , Including the requirements phase 、 The development phase 、 Code integration 、 Testing phase 、 Before going online 、 deliver 、 Quality tracking , This is from 0 To 1 The milestone of , Because you can see the standard code access principles in the development process of requirements management 、 Various test index items of the test 、 Various alignments before going online 、 The summary of the report after delivery is repeated , Monitor the whole data until it is finally launched , Completed a basic quality system diagram .

■ chart 14

06 Q & A

1、 How to check coverage ？

The coverage check uses tools bullseye, This tool is in CI During editing, a Cov file , Operational correlation case Of Cov The code coverage of the development of the corresponding module in the file will change , Finally, after all modules run, they can be mapped and compared according to the standard .

2、 In startups CI/CD Which stage is it mainly in ？

Entrepreneurship may need to be put into the testing stage , We are mainly responsible for CI Outgoing package , because SDK After compiling the package , Will directly generate relevant zip Package and trigger compilation related App, These are our testing tools . Because the sound network mainly does SDK test , So we will put our SDK Embedded in self-developed tests App For testing tasks . Of course, our company divides it into two sets , The development department has relevant UT, After submitting the code, there will be relevant code checks , If UT But it cannot be submitted , This is the development part ; The test part is mainly used for Tools out of the bag , Because our company SDK The particularity of the test , If it is on the market, most workers do not need to be embedded sdk They are all developed by their own functions app, It is generally divided into two parts . After the developed submission code , Some people may have test case, There may be backup test environment before the test environment , In this environment, there will be relevant test tasks . Put relevant tests test case Run through , Achieve a certain pass rate , Then the submission will be successful , Proceed to the next test . Of course , It depends on whether you have the energy to maintain two sets of environment , Because this environment is before code testing , And the relevant requirements will be higher . If the developed code UT After that , Then release it to the test environment to run test case,case There may be two kinds , One is the new function , This is not acceptable , Because there are no new functions . The other is the existing old function , You may need to hit tag To determine whether to cover the test cases , If it is an old function , You can determine whether to call back by setting the general rate , So it needs to be adjusted according to different business types .

3、 How to measure the end-to-end performance test ？ Do you have any tools ？ What if the buried point data is not accurate ？

There are many end-to-end performance testing tools ,test home Someone seems to have shared Android tools , You can go and have a look . That tool is open source , After installation, run related App when , It will display relevant data . The other is open source tools that make tidevice, It's through Python Retrieved Apple Of API To get data . If it is iOS or Mac It can be used moonlight,Windows There are also some performance tools that can directly monitor . If you want to try secondary development ,iOS Just transfer it to private api, Android is adb call . Apple officially has instrument,iOS and Mac It's all through instrument Call to get new data .

For the problem of inaccurate burial point , In fact, burying points is more a development task , Because the development is the log writing of related tasks in the code , I think it should be developed to bury these points , Then it will have a series of call chains . Our task is to understand the requirements of development , Then get these data to verify whether its burial point is correct .