Explore Netease's large-scale automated testing solutions see here see here

This article is from the official account of the project. ：“AirtestProject”
Copyright notice ： Allowed to reprint , But reprint must keep the original link ; Do not use for commercial or illegal purposes

One 、 Introduction to private cloud solution

1. development history

Airtest project （UI Automated testing framework ） On 2014 Launched in , After successfully serving dozens of projects within Netease , Invited at 2018 Year of GDC Jointly released with Google at the Conference . Open source projects so far , Tens of thousands of developers and thousands of companies are using .

But in the internal practice process , We found that only the underlying testing framework is far from enough to meet the needs of enterprises for large-scale automated testing , such as , How to build a stable and efficient equipment cluster 、 How to manage and dispatch equipment scientifically 、 How to manage test resources and continuous integration   etc. , They are all problems that need to be solved by a complete set of automated testing solutions . For the above needs , We developed the mobile phone cabinet by ourselves 、 Equipment management system DeviceKeeper、Airlab Cloud test platform and other software and hardware , Set up a set of   Complete automated testing solutions  , Not only serve Netease internal , It also serves many external enterprises , They have achieved very good results .

2. Project introduction

Netease's automated test cluster solution , It includes the construction of equipment clusters 、 Equipment management 、 scripting 、 Equipment scheduling 、 Script multi machine execution 、 Report generation 、 Complete solutions such as continuous integration . It is software and hardware “ Integrated ” Products , It can help us quickly build a private mobile cloud testing platform within the enterprise .

chart Netease automated testing solutions - Private cloud

Two 、 Detailed introduction of scheme products

We will start from   Cluster construction 、 Cluster management 、 Script writing and platform construction   this 4 Let's introduce our entire private cloud solution in detail from three aspects .

1. Cluster construction

Traditional equipment management , It is managed manually by the device administrator , Testers also need   Frequently borrow and return equipment offline  , To complete their own complex testing work . In addition, with the gradual increase of internal test equipment , Equipment management will also consume a lot of time and manpower , And the utilization rate of equipment will be very low . Think about it ,1 Device administrators , Manual matching 100 Routine on / off maintenance of this equipment , How long will it take ？

So how to manage test equipment efficiently ？ How to provide stable equipment resources for testing ？ Our plan is to build a real machine cluster . Real machine clusters can uniformly manage equipment resources （  Device resource pooling  ）, In this way, the maintenance colleagues can   Online batch management 、 Maintain these equipment  ; It also allows test colleagues to quickly and flexibly obtain designated devices or a large number of devices for regression testing / Compatibility test .

In order to build such a real machine cluster , Netease has independently developed an industrial mobile phone cluster cabinet ：

chart 8*4 Professional cabinet of specifications

chart Cabinet of Netease automatic test room

The equipment carrying capacity and floor height of the cabinet support flexible customization , existing 8 * 4、10 * 8、10 * 9 And other cabinets of different specifications . The equipment drawers on each floor can be pulled out and support independent work . Good ventilation design of cabinet 、 Built in industrial network solution , It can guarantee the continuity test 、 Or equipment stability and network fluency during batch equipment tasks . In addition to cabinet cabinet （ Suitable for placing in the machine room ）, We have also developed portable portable cabinets ：

chart Portable portable cabinet

Compared with cabinet , The equipment carrying capacity of portable portable cabinet is relatively small , most 10 platform , But small and flexible , Where mentioned can be realized , Where to measure , It is not limited to placing in the equipment room . It is very suitable for some   Test effect demonstration scenario 、 Outdoor environment test scenario   etc. , Even testing directly on the workstation is a very good choice .

in addition , In order to solve the problem of battery shell expansion or short service life caused by long-term online charging of cabinet equipment , We are also responsible for USB HUB Customized upgrade , The optimization has become a version that can intelligently control electricity , Through the program, it is convenient to control the on-off and data on-off of the equipment  , Carry out shallow charging and discharging of the equipment regularly , It greatly extends the service life of the device battery , It reduces the maintenance and update cost of equipment hardware .

chart Cabinet - Intelligent power control hub

2. Cluster management

After the successful establishment of the device cluster , The next thing we need to consider is cluster management ：

One side , We need to It is very convenient to monitor and maintain our device cluster , To ensure the long-term stability of the equipment cluster ;

On the other hand , We also need Flexibly manage and allocate our device clusters , Distinguish between public equipment and project specific equipment , While ensuring high utilization of equipment , We also need to ensure the privacy of private devices . In order to solve the above pain points , We have developed hardware with cabinets “ Matching ” Equipment management system based on DeviceKeeper：

chart NetEase DeviceFarm Cluster solution

DeviceKeeper Not only can the device administrator remotely view the online status of the device cluster , It also supports one click batch maintenance 、 Monitor our device cluster ：

chart DeviceKeeper- Check the online status of the cluster

chart DeviceKeeper- Abnormal monitoring

Of course , It also supports the management and distribution of the entire cabinet or the equipment on the cabinet , Ensure that only specific project groups or specific users can access and use private devices （ It is more in line with the internal device use needs and privacy ）：

chart DeviceKeeper- Cabinet management

It is also worth mentioning that , The familiar cloud real machine platform , Also integrated in our DeviceKeeper Equipment management system , Convenient for testers Carry out some manual tests remotely on the web ：

chart DeviceKeeper- Cloud real machine page

3. scripting

The purpose of automated testing is , In order to find product problems faster and better , Effectively improve product quality . When we solve the problem of equipment resources for automated testing , You need to obtain these equipment resources , To automate scripting 、 The script runs and generates test reports to find product problems .

Netease's private cloud solution not only provides us with an underlying automated testing framework , It also provides tools such as editors for rapid production scripts and test reports ：

chart automated testing - Framework and tools

Selection of automated testing framework , We need to consider the versatility of the platform and the difficulty of getting started , Platform commonality It can ensure that the framework is compatible with mainstream Android、iOS、 Applet 、H5 And various game engines and other platforms ; The threshold is low , It can facilitate enterprises to quickly access automated testing and quickly implement automated testing solutions .

Netease open source Airtest and Poco frame , Based on these pain points , Cooperate with the editor dedicated to the framework AirtestIDE,0 Code based testers can also produce some very quickly “ Simple ” Automated test scripts for （ Community Edition AirtestIDE website ：airtest.netease.com/）：

chart Pain points and solutions of automated testing

chart Airtest- Framework principle and examples

chart Poco- Framework principle and examples

in addition , The enterprise version of the private cloud solution IDE In the Community Edition IDE On the basis of , Developed more Advanced features , such as Run in bulk （ Small batch compatibility test / regression testing ）、 Generate aggregate report with one click 、 Connect devices in HD mode （ Good picture compatibility 、 More fluid ） etc. , It is more in line with the requirements of enterprise level automated scripting .

chart Enterprise Edition IDE- Run tasks in batches

chart Enterprise Edition IDE- Overview of aggregated reports

besides , The private cloud solution also provides a  Klay Unified installer  , This is an enhanced tool library . It encapsulates some of our common functions , such as Automatically click pop-up 、 Slide according to the direction 、 Use slide to unlock the screen And so on , And some Airtest API Enhanced functionality . This library reduces the difficulty of writing some scripts , When testers encounter similar scenarios , Directly call the provided API that will do , No additional packaging is required .

4. Platform building

Although the enterprise version IDE Support the completion of small batch compatibility tests / regression testing （ The number of equipment is about a dozen ）, But larger regression tests and device compatibility tests , You just need one Efficient automatic test case running platform Here we go .

Netease private cloud solution provides Airlab Cloud test platform , Not only Support the scheduling and running test of basic batch test scripts , also Support unified management of test resources 、 Automated test report analysis and continuous integration etc. ：

chart Airlab- Platform function diagram

Use Airlab Cloud test platform , Testers only need to prepare test resources and submit task configuration , You can start the test task , And run the test on a large number of equipment .

chart Airlab- Regression test task configuration

The whole test process does not need human intervention , After the task is completed, the complete running report of this test task will be provided automatically , The content of the report includes the overview of the report , Performance report , Error reporting , Screen recording of running test cases 、 Script log, etc , It can help testers find product problems .

chart Airlab- Overview of regression test task report

Regression test and compatibility test for stable operation function , Just write the test script once , Save on this platform , You can repeat the automatic operation for many times , It can maximize the efficiency of testing and save the labor cost of testing .

It is also worth mentioning ,Airlab The automated test platform can Support the automatic packaging of batch equipment 、 Next Patch And login requirements , Help testers reduce packaging 、 It takes time to start the application and login process .

The project team only needs to provide the test package 、 Login case 、 Log in scripts, test accounts and other test resources to Airlab Automated test platform , And in Airlab Configure a compatibility test task for test login on the automated test platform , Select the equipment to be packed , The platform automatically completes the installation of test applications 、 Next patch And login operation , And inform the project team to test after the task is completed ：

chart Airlab- Packing task

3、 ... and 、 Program advantages

1. Flexibility of equipment cluster construction

We   Support a variety of forms 、 Build real machine clusters of various sizes  , It can meet the different cluster construction needs of enterprises ： For small-scale clusters , And the cluster can be flexibly placed in the office / Station and other places , You can choose our portable portable cabinet , Can accommodate 10 device , Support Android and iOS, but 1 Portable portable cabinets can only be deployed 1 A platform device ;

For medium and large-scale clusters , You can choose our industrial cabinet , Common specifications are 4 * 8（32 device ）、 8 * 10（80 device ）、 9 * 10（90 device ） etc. , We also Support customization of cabinet specifications according to the actual equipment scale of the enterprise ; in addition , Industrial cabinets also support customized floor heights , It can meet the needs of enterprises for special equipment on the shelves , For example, car engine 、 Translation pen 、 Special flat plate, etc ;

Of course , Our industrial cabinets also support Android and iOS equipment , also   Support mixed deployment by row  , for instance ,4*8 Specifications of industrial cabinet , Altogether 4 That's ok , Each row 8 device , We can choose the second 1 Rows are deployed as iOS node , shelves iOS equipment ; The remaining 3 Row deployment Android node , shelves Android equipment .

2. Professionalism of cabinet

Our industrial cabinet has   Experienced several generations of hardware trial and error  , Finally confirm the existing hardware scheme , It can ensure the stability of cluster operation to a large extent ; And we are still constantly improving and optimizing our cabinets , such as 2021 Over the years , We have put ordinary usb hub It has been upgraded to be able to control electricity intelligently usb hub, Cooperate with intelligent power control system , Greatly extend the battery life of the device ;

Another example is the cabinet intelligent door lock scheme we are testing and using , It can increase the security of the cabinet , And it is convenient for test colleagues to scan code, open door and borrow equipment offline .

3. Scalability of the underlying test framework

Our image recognition framework Airtest And control recognition framework Poco All pure python Framework encapsulation of language ; You can freely encapsulate quadratic functions when using 、 Script encapsulates the call , Flexible organization of test sets .

In addition, our framework also supports the combination of our own testing services （ Such as game command service ）, Call instructions freely in the script 、 Combined with the test framework api Complete more in-depth test functions .

4. The integrity of the tool chain

Netease private cloud solution , Build from clusters 、 Cluster management 、 scripting 、 Automated tasks run to report generation , That is, from the hardware layer to the business layer , Have provided us with detailed solutions .

Enterprises can according to their actual needs , Flexibly purchase the software and hardware products provided by our private cloud solutions , Including cabinet customization 、 Software scheme selection, etc .

5. Privacy of privatized deployment

Now there are many enterprises in the market that can provide public equipment testing services , Press / Charge for the length of use , It can also meet certain testing machine needs of enterprises .

But the shortcomings of public equipment services , First, there is no guarantee that all equipment models can meet the testing needs of enterprises ; Second, the privacy of the whole testing process cannot be guaranteed , After all, we need resources such as test packages and scripts , Upload to a public platform , The issue of confidentiality cannot be underestimated .

therefore , Now many companies are more and more inclined to privatize the deployment of equipment clusters , The private cloud solution provided by Netease , It also deploys cabinets and software platforms inside the enterprise , It effectively ensures the privacy of equipment use and testing process , meanwhile , Privatization deployment can also ensure that enterprises can flexibly arrange their large-scale return / Compatibility test task .

Four 、 Scheme procurement and implementation process

Usually , External enterprises purchase our private cloud solutions , You need to go through the following process ：

chart Netease private cloud solution procurement process

among , Enterprise demand confirmation , To understand the different private cloud deployment needs of each enterprise / Automation requirements , After clearly understanding the needs of the enterprise , Our operations / Only business students can give a plan suitable for the enterprise and the quotation of the plan .

When the enterprise confirms that the scheme can meet its own needs , We will confirm the quotation and sign the contract again . After the contract is signed and the enterprise pays the advance payment , We will arrange the hardware （ Cabinet and cabinet accessories ） Mail door-to-door matters , Then we will send engineers to deploy hardware and software , Ensure that the system is successfully deployed inside the enterprise and can be used normally .

At this time, the enterprise can check and accept our products , The final payment will be made after the acceptance . thus , The scheme is purchased and implemented inside the enterprise . In the future, we will provide special technical support groups for enterprises , It is convenient for enterprises to have questions or other problems in the process of using the scheme , Can get timely response and solution .