On-off and on-off of quality system construction

Preface

Jia Pengbo is a test and Development Engineer of sound network , Responsible for sound network  SDK  And establish and improve relevant testing tools . Worked in a financial company , Look at the stock , The Commission is calculated , There is content distribution 、 Online retailers 、 Small game related testing experience .

This article is based on Jia Pengbo's 「

Sound network developer entrepreneurship Lecture •  Phase III - how to ensure product quality in the early stage of entrepreneurship

」 Second sorting of shared content in the activity . Official account 「

Sound network developer

」, Reply key words 「

JT0611

」 You can download activity related  PPT  Information .

01  Background of quality system construction

First , From the figure  1  You know , Create from requirements , To assess 、 establish 、 Testing and distribution of requirements , And then to the final test , The workflow of requirements is very long , We need to build and track workflows , In order to better ensure business operation . secondly , Data records are used for double disk 、 Improvement and efficiency . Because the construction of quality system covers all aspects , It also includes the landing of some documents , With the increase of release and testing tasks , Various versions and test data will be accumulated accordingly , In this process, data analysis should be carried out , Get the data and analyze the duplicate 、 improvement , To promote the construction of quality system .

Last , The purpose of building a quality system is to get high-quality products . From R & D self-test 、 Advance of the testing stage 、 Requirements standard alignment 、 Develop test specifications 、Jira  Information visualization and tracking , And other aspects of follow-up . All the services we do are for the launch of the version , Provide users with high quality 、 Highly reliable products and services .

■ chart  1

chart  2  It comprehensively shows before and now the establishment of the quality system , Quality level 、 Online problem 、 Test the closed loop of the problem 、 Process improvement 、 Use case addition and how to coordinate manual  case  And automation  case . adopt “ before ” Graph , It is difficult to draw relevant conclusions , Only establish a system for continuous analysis , Will get the corresponding conclusions or indicators to improve . From the figure  2  You know “ before ” how , Now, what effects can be achieved by establishing a system , And the measures to be taken to realize the system . These measures are in all aspects , Not only need to land , Some may have to constantly align and repeat .

■ chart  2

02  methodology

1、 Small step run

Trot is used to quickly go online , Sometimes the need can be very urgent , Simply understand the purpose , After docking with the development or product manager, it is time to start work , Test cases may be too late to maintain and update . And because it is a stage of rapid growth , You may also need to go through parallelism .

The so-called parallelism is shown in the figure  3  Shown , Many people have  N  A need , this  N  Needs are tracked by different people , But these requirements are new , Just add it and go online quickly . The characteristics of this period are , Demand is new , Only test the part you are responsible for , Other needs have nothing to do with yourself , There is no need to return , You can directly add online . It should be noted that , Because only focus on fast online , So as long as a certain quality is guaranteed , The control of details is not in place .

■ chart  3

2、 Seek in stability “ - ” The high quality of

Pay attention to quality when the company is in a period of peace , Many considerations are needed . A requirement may be divided into  N  Sub requirements , These sub requirements are tested by different people . in addition , In the process of testing, we need to interact constantly , Delivery is achieved through different test points and covering different test contents , The whole test process may be relatively long , Because it needs to be verified in many aspects , Including manual testing 、 automated testing 、 Performance testing 、 Laboratory tests, etc , Use the comprehensive test results to determine whether the version has backtracking or quality problems . If the whole is in good condition , You can publish .

This kind of stability “ - ” The method is more suitable for companies with strict quality requirements , Such as banks , Because it has strict control over money , So its duration is very long , Pay close attention to the quality problems in the testing process , It requires different people to cross and complete relevant test tasks by exchanging test content and test cases , Is the characteristic of this period .

3、 Documents that standardize the process

At this time, the company is still in a period of peace , In the process of going through small steps, fast runs and long iterations of testing requirements , Has accumulated a variety of experience , These experiences need to be translated into process improvement and function improvement , How to find more problems is a test of demand sorting and data recording ability , See the picture for details  4  Shown ：

■ chart 4

Design test cases from requirements , Test cases may include  Xmind  and  Excel, Next, develop  review  dependent  case, At this point, you may want to stand in the perspective of code design, product use and  QA  Improved angle  case, Modify to meet the online requirements . Of course , You may encounter various problems during the online process , These problems need to cover the function 、 performance , And it requires that in the process of continuous iteration （ Whether it's the test environment , Or pre release environment 、 Online environment, etc ）, All functions are normal , There is no fallback problem . The main feature of this stage is that the requirements need to be tracked from the beginning to the end 、 To sum up . meanwhile , You need to sort out the documents and pay attention to the specifications , Avoid duplicate documents, etc .

■ chart  5

chart  5  The current version of  bug  Number and proportion , Different teams  bug  The number can reflect the time to solve the problem to a certain extent , It helps to summarize the test progress , And control  highlight  Mission , So as to promote the timely launch of functions . chart  6  The problem solving time and the length difference of problem solving are recorded . In fact, the document data recorded by the document school is to better promote the test progress , Ensure timely or real-time online .

■ chart  6

4、 The tool flow of flowers competing

At present, various testing tools have been launched , Corresponding testing tools can be used according to different businesses to improve efficiency , Timely delivery of test tasks . chart  7  Eight categories of test tools are shown , This is just a simple enumeration , There are far more testing tools on the market .

■ chart  7

5、 The development flow of excellence

At this time, the company may be in a mature stage , The current tool flow can no longer support the test task , It needs to be developed for business with the help of tool flow , such as  STF（Smartphone Test Farm） And Tencent  Bugly, They are all secondary development to fit their own business . Of course , Now there are many tool flows , Include  sonic、httprunner  etc. , They cover all aspects , Including interface automation performance 、 Performance test, etc , These can be adapted to the business for secondary development . The main characteristics of this period are , High requirements for code , Need to fully understand the business , And meet business needs through code capabilities .

chart  8  It shows the secondary development and related practices of our company for the current tools , I will briefly elaborate later .

■ chart  8

03  Landing and practice

1、SDK  Real time and efficient closed-loop processing of crash data

Our company mainly tests  SDK, Compared with  App  There may be some differences , Its landing practice is mainly  SDK  Real time and efficient closed-loop processing of crash data . There are some such tools on the market , such as  Bugly, But we found that  Bugly  Processing  SDK  There are still some deficiencies in the process of reporting the crash , such as  SDK  There was a crash , however  Bugly  No monitoring . So we have made some improvements in this regard , The overall process is shown in the figure  9  Shown , our  SDK  It serves customers , Embedded in the customer's  App  in , Its whole closed loop is , The program executes normally to monitor whether  crash, And then by monitoring  crash  State finding daemon aware capture , Next generate  dmp  File and submit to the crash collector , To analyze the relevant system , At this time, some relevant information will be stored , Finally, integrate the information and send it to users .

■ chart  9

chart  10  It shows that  SDK  Real time and efficient closed-loop processing of crash data , First, extract the stack key information , Then judge according to the hash value , If the same version  Hash  The value is , Then it is related  JIRA. The processing logic of this part is to provide the crash log to relevant developers for timely processing after obtaining it .

■ chart  10

chart  11  It shows how to collect logs and compile them through the parsing system  crash  after , stay  JIRA  Show and assign . First, get the crash information , Then report it to  JIRA  With some key information , At the same time, inform relevant developers , Make it repair . Simply speaking ,SDK  The real-time and efficient closed-loop processing of crash data is in  SDK  The embedded  App  after , Determine whether it is our  SDK  the  crash, Because some may be customers  App  Of itself crash, We will judge this part , if  SDK  Of  crash  It will be reported directly, and then parsed by the compilation system and reported to  JIRA, In this process, it will be submitted to relevant developers for processing , Complete the whole link .

■ chart  11

2、 Circuit plugging

This part mainly introduces the experiment of plugging headphones , Because of our  SDK  You may need to constantly plug and unplug headphones during the test . chart  12  It is a simplification of the circuit diagram , Simply put, it's on the motherboard , Weld the wire in the earphone with the wire in the circuit diagram .

The original circuit diagram is very complicated , Students who have studied integrated circuits should know , There are various nodes , Test these points , Combined with the principle of circuit diagram , The function of plugging in and out headphones can be achieved through the program control circuit . We used to spend a lot of manpower in the process of testing and plugging headphones , Conservatively estimate a person  3  All the days  online  You can finish all the work on it  case. But after finishing this circuit diagram , It only takes one hour to finish all the related  case. We will show the relevant data on the circuit diagram （ Including the analysis of Web data , And the tailoring of performance indicators ） Integration , It is equivalent to the commercialization of earphone plugging .

■ chart  12

3、HENGE Test platform

chart  13  Is based on  STF  Cloud platform for secondary development , Its main link is to dynamically obtain the test package to get its information , Then dynamically send it to Yunzhen machine for testing , Then submit the relevant test results to the relevant testers , It is convenient for them to obtain the test report in order to draw the test conclusion . The figure mainly shows our test  App  After the process is issued, the package is loaded and operated , View the report after running , Of course, this includes  crash  Log and related functions .

■ chart  13

4、Moonlight

Our products  Moonlight  Is open source （

https://github.com/AgoraIO-Community/MoonLight

） 了 , It's a  SDK, For automatic performance data collection , Interested students can check the relevant code . Some performance tests on it show , It can obtain very accurate information with low resource consumption  CPU、memory  And so on .

04  System diagram

From why to build a quality system , Different strategies may be adopted in the next five periods , As well as various gadgets that may be born in the process of strategy, they all serve the high-quality products we provide , This is the last and only indicator . All testing tasks , Including the construction of quality system is to maintain the current business development . So no matter what strategy you use , Whether it's document School 、 Tool flow , Or secondary development , All node or task events , They all serve their own business systems .

chart  14  It shows a quality system , Including the requirements phase 、 The development phase 、 Code integration 、 Testing phase 、 Before going online 、 deliver 、 Quality tracking , This is from  0  To  1  The milestone of , Because you can see the standard code access principles in the development process of requirements management 、 Various test index items of the test 、 Various alignments before going online 、 The summary of the report after delivery is repeated , Monitor the whole data until it is finally launched , Completed a basic quality system diagram .

■ chart  14

06  Q & A

1、 How to check coverage ？

The coverage check uses tools  bullseye, This tool is in  CI  During editing, a  Cov  file , Operational correlation  case  Of  Cov  The code coverage of the development of the corresponding module in the file will change , Finally, after all modules run, they can be mapped and compared according to the standard .

2、 In startups  CI/CD  Which stage is it mainly in ？

Entrepreneurship may need to be put into the testing stage , We are mainly responsible for  CI  Outgoing package , because  SDK  After compiling the package , Will directly generate relevant  zip  Package and trigger compilation related  App, These are our testing tools . Because the sound network mainly does  SDK  test , So we will put our  SDK  Embedded in self-developed tests  App  For testing tasks . Of course, our company divides it into two sets , The development department has relevant  UT, After submitting the code, there will be relevant code checks , If  UT  But it cannot be submitted , This is the development part ; The test part is mainly used for   Tools out of the bag , Because our company  SDK  The particularity of the test , If it is on the market, most workers do not need to be embedded sdk They are all developed by their own functions  app, It is generally divided into two parts . After the developed submission code , Some people may have  test case, There may be backup test environment before the test environment , In this environment, there will be relevant test tasks . Put relevant tests  test case  Run through , Achieve a certain pass rate , Then the submission will be successful , Proceed to the next test . Of course , It depends on whether you have the energy to maintain two sets of environment , Because this environment is before code testing , And the relevant requirements will be higher . If the developed code  UT  After that , Then release it to the test environment to run  test case,case  There may be two kinds , One is the new function , This is not acceptable , Because there are no new functions . The other is the existing old function , You may need to hit  tag  To determine whether to cover the test cases , If it is an old function , You can determine whether to call back by setting the general rate , So it needs to be adjusted according to different business types .

3、 How to measure the end-to-end performance test ？ Do you have any tools ？ What if the buried point data is not accurate ？

There are many end-to-end performance testing tools ,test home  Someone seems to have shared Android tools , You can go and have a look . That tool is open source , After installation, run related  App  when , It will display relevant data . The other is open source tools that make  tidevice, It's through  Python  Retrieved  Apple  Of  API  To get data . If it is  iOS  or  Mac  It can be used  moonlight,Windows  There are also some performance tools that can directly monitor . If you want to try secondary development ,iOS  Just transfer it to private  api, Android is  adb  call . Apple officially has  instrument,iOS  and  Mac  It's all through  instrument  Call to get new data .

For the problem of inaccurate burial point , In fact, burying points is more a development task , Because the development is the log writing of related tasks in the code , I think it should be developed to bury these points , Then it will have a series of call chains . Our task is to understand the requirements of development , Then get these data to verify whether its burial point is correct .

Activity Notice

7  month  16  On the afternoon of Sunday , Sound network developer entrepreneurship Lecture  •  The first  4  Expect to 「

How does the entrepreneurial team ensure the safety and compliance of the product business ？

」 entitled , Invitation letter 、 You family 、 The technical experts from three excellent enterprises in baishanyun bring you wonderful sharing .

Action is better than action , Scan the QR code or click 「

Read the original

」 Sign up ！