In the field of software engineering, we have been doing scientific research for ten years!

The development of technology is inseparable from the innovation of scientific research and the landing of projects , The author of this article Liu Yang's programming life starts from interest , It has become a result of programming and scientific research day and night , Then he combined engineering with scientific research , Conduct in-depth research and Practice on software engineering digitalization . In this paper , He shared today when open source devours software , Experience of exploring the digitalization of software engineering .

writing | Liu Yang

Produce |《 New programmers 》 Editorial Office

Over the past 20 years , The world has experienced the transformation from informatization to digitalization . We see that enterprises meet the challenges of Technology , It is also clear that the whole industry now has higher requirements and standards for software management and information security . We do scientific research in school , Publish research results with professional papers and reports , At the same time, it also focuses on helping enterprises solve key problems , Provide some more effective technical support and enabling methods , Continue to let technology improve business value .

As a scientific researcher in the field of software engineering , I will sort out my thoughts and experiences over the past ten years for your reference . About the code 、 Program 、 Software and applications , I hope that through the narration of this article , Discuss and practice with everyone .

This article is excerpted from 《 New programmers 》004,『 Paper book + Electronic journal 』 It's officially on sale

My way of programming

I remember when I was in the sixth grade of primary school , A very accidental opportunity , I signed up for an input method training class , Thus, I have a strong interest in computers . Later, I leaned by chance and could type , Unexpectedly, he became a special student and entered a key junior high school . From then on, I decided to learn computer , It has also become my ideal and belief . But the college entrance examination results are not ideal , The score failed to reach the admission line of computer major of Zhejiang University , I chose the major of electrical information automation . Although electrical information sounds a little related to computers , But the course is completely different . To coincide with CSDN Of 《 The programmer 》 The magazine （《 New programmers 》 The predecessor ） stay 2000 Issued at the beginning of the year , Look at the story of programmers and the introduction of cutting-edge technologies , It even stimulated my desire for computer learning .

Life is sometimes full of twists , Three months after college , I was very lucky to get the opportunity to study in Singapore , Successfully entered the National University of Singapore, majoring in computer science, undergraduate , Finally, I realized my childhood dream . Four years of undergraduate , Learn the program from the beginning 、 Write program , Go to team up and write the program , Enjoy it every day . What impressed me most during my college years was , because 《 The programmer 》 It's a technical publication , It's not easy to buy , Every time I go back to China, I find newspapers and magazines in the streets , Every time I finish reading, I get a full harvest . By graduation , I have 50000 lines of code , I feel like I have finally become a programmer , There is a sense of pride .

When you graduate from college , The computer industry is in Dot-com bubble （ Internet bubble , Also known as the science and technology network foam ） After the low tide , So I decided to stay in school and continue my doctoral studies （ Zhibo ）, Mainly engaged in the research of program verification . At that time, it was not very clear which research directions were interesting , I just feel like “ Program ” Relevant is very good . At first, I worked as an assistant for my senior brother , Writing algorithm , I didn't expect the experimental effect to be good , Relying on the development experience accumulated in University internship, the algorithm is gradually made into several small tools , Finally, it gradually evolved into a software verification platform . During the doctoral period, I started to write programs 、 Design procedure , Finally, it gradually turns into an analysis program . Four years , A total of nearly a million lines of code have been written , At that time, I felt that I finally began to understand the program , It has also constantly expanded my understanding of Technology .

Relying on the software verification platform made during my blog reading , I was very lucky to get the teaching post of Nanyang Technological University in Singapore , Set up a network security laboratory （ See the picture 1）, It is here that I really start the scientific research of software engineering .

chart 1 Group photo of the scientific research team in the network security laboratory of Nanyang Technological University

In the last decade , Our research scope basically covers all steps in the software development process （ Demand analysis 、 Architecture design 、 Development 、 test 、 Operations etc. ）, Various quality attributes of software （ correctness 、 Security 、 reliability 、 Performance etc. ）, Various types of software and systems （ Mobile phone application 、 Artificial intelligence model 、 The Internet of things 、 Autopilot system 、 Smart contracts, etc ）.

The current situation and challenges of software development in the new era

With the continuous development of the software industry , The software system itself is becoming more and more complex , The corresponding software development process has gradually become a system 、 Comprehensive engineering science . Any one with excellent functions 、 Behind the complex software system , Can not be separated from the huge R & D and management team . However , When the complexity of software system increases explosively , In the past, repeated stacking of labor 、 The engineering means of rushing forward to expand production capacity gradually fails in software development , Various software development process models 、 Management methods are constantly emerging .

From the waterfall 、 The incremental 、 Spiral and other early models , To 21 Agile development in the 21st century 、DevOps etc. , Generation after generation of software development models emerge in endlessly . The supporting system of software development is becoming more and more perfect and systematic , Such as interactive teamwork 、 Model Driven Development 、 Microservices 、 Low code 、 No concepts such as code development 、 technology 、 Tools have entered our world . However , These technical means are constantly changing the process of software development methods , It is also gradually increasing the complexity of the process , This puts forward higher requirements for the process management of software engineering .

Besides , Large complex systems with commercial companies from top to bottom “ Cathedral ” Type of software development is completely different , Another kind Focus on users 、 Advocate early release, such as “ Market ” The free software development model has withstood the test of time . among , The most typical case is Linux The success of the system . just as 《 Cathedral and market 》 As mentioned ,Linus Torvalds Pursue “ Publish as much as possible as soon as possible , Delegate everything you can , Open to all changes and integration ” Software development philosophy , Developers who will spread around the world , Only through the loose cooperation of the Internet , It makes a world-class operating system . Open source software created in this free environment is free 、 The nature of openness , It is gradually incorporated into the development process of mainstream software , Among them, many excellent open source projects are widely integrated 、 Used in the projects of mainstream software manufacturers , Thus avoiding “ Repeat the wheel ”, It greatly improves the efficiency of software development .

Review our research experience in software engineering , It also grows step by step around software development . Starting from the quality of software , At first, formal methods were used to model the software , And verify the correctness of the software . We have developed a series of complete software formal modeling languages and verification algorithms to analyze concurrent systems 、 Real time systems and probabilistic systems , At the same time, these methods are applied to distributed systems 、 Security protocols 、 The Internet of things etc. . Finally, these capabilities are integrated into a system called PAT（Process Analysis Toolkit） Formal verification platform （ See the picture 2）.

chart 2 PAT Formal verification platform architecture

But when the complexity of software increases or the scale becomes larger , Formal methods face scalability challenges . meanwhile , Developers also have very high requirements for mathematical modeling ability . therefore , We will gradually transform the method of scientific research into the technology of program analysis ： Static analysis and dynamic testing . Although these two methods cannot provide the same perfect result guarantee as the formal method , But in terms of practicality , Has brought about a significant improvement .

We use program analysis technology to do various applications , Including defect and vulnerability detection 、 Software performance evaluation 、 Software architecture is reverse 、 Software evolution and development efficiency analysis . Especially in terms of software vulnerabilities , Developed a series of dynamic testing techniques , Found a large number of unknown vulnerabilities in commercial software and websites , This improves the security of the software . Up to now , We have begun to use the method of combining dynamic and static , Analyze and locate possible defects and vulnerabilities with static programs , Then guide the test algorithm to quickly trigger the vulnerability （ See the picture 3）.

As the amount of code increases and open source code is widely used , We can access a lot of code data , Naturally, I began to think about how to use code big data and artificial intelligence algorithms to enable software development . We apply the algorithm of artificial intelligence to requirement understanding 、 Clone detection 、 Code search 、 Code completion 、 Vulnerability detection and software testing . These attempts let us really think about the importance of program semantics , And how to integrate program semantic representation with deep neural network . therefore , We learn the variable types of weak typed languages from strong typed languages through transfer learning , It involves the interpretability of program semantic understanding and deep learning . Although these efforts have made some valuable progress , But the application of artificial intelligence in software engineering is still in its early stage , There are still many challenges that need to be solved urgently .

Based on open source , The exploration of software engineering digitalization

Because software development is becoming more and more complex , In order to effectively alleviate the tight coupling of software development process and improve development efficiency , The industry faces the output of products in the process of software development 、 Verification and end-to-end traceability put forward higher requirements , For example, the expression of requirements documents in the forward development process 、 Design and verification of software architecture model 、 Reverse traceability of requirements documents 、 Verification of the specific implementation of the architecture design model at the code level 、 Reverse traceability from code implementation to design model and requirements . With the large-scale use of open source software , Because of its freedom 、 to open up 、 Out of control , It further intensifies the urgent need for end-to-end verification of software engineering , This finally prompted us to gradually clarify and embark on the road of exploring the digitalization of open source software engineering .

According to the Gartner data ,99% 's organizations use open source software in their information systems .Sonatype The company is also interested in 3000 We have conducted a survey on the use of open source software in three enterprises , The results show that every enterprise downloads 5000 Multiple open source software . The trend that the modern software industry is highly dependent on the open source system cannot be changed , But while open source software provides convenience for enterprises , There are also some risk problems that can not be ignored , Including intellectual property 、 Open source protocol component conflict 、 Security vulnerabilities 、 Data leakage, etc . For example, in the past six months Log4Shell And Spring4Shell“ Nuclear grade ” Vulnerability Events 、 developer Marak Squires“ Delete the library and run away ” event 、Elasticsearch License change event 、node-ipc anti-war “ Poisoning ” event , And high frequency npm Malicious component attack , These are all tearing apart the fragile open source supply chain .

under these circumstances , Want to do a good job in software quality and safety management at the same time , It has become a very challenging thing . This also strengthens our next research direction ： Explore better ways to develop programs , Develop more standardized tools to analyze , Understand and manage software and its development process , And improve the management mechanism of the whole open source . Especially when open source code is widely used , Open source governance and software supply chain governance Become a key scientific research topic .

Based on this , We are 2017 In, it incubated a software security company for open source component testing Scantist. Take software security as the starting point , Based on software component analysis , From the development of open source software itself 、 maintain 、 Release 、 In the process of being relied on , Seek effective maintenance and management strategies for open source software , And through in-depth understanding of various programming languages 、 The working mode of open source core modules such as package manager , Explore the formation of open source supply chain 、 Transmission and evolution , Form a set of solutions for open source supply chain governance .

In addition to the current security vulnerabilities and license related appeals that the industry is more concerned about , We have also carried out a lot of research work on open source software quality assessment . Based on team in software engineering 、 Precipitation of many years' scientific research work in the field of program analysis , Gradually condense a set of multi-dimensional software portrait framework for open source software , Around open source software code 、 The development process 、 The development team digitises , From quality 、 Security 、 Open source ingredients 、 Maintainability 、 maturity 、 Activity, business risk and other dimensions , Measure the health of open source software itself , And form a team 、 Understand the health status of the core development team of open source projects from different perspectives such as development risks . So as to realize the precise use of open source software in the process of software development （ Such as technical selection recommendation 、 Risk analysis, etc ）, Identify the key basic components in the open source environment , Monitor the health of its open source projects and development teams , And guide relevant business enterprises 、 The community actively participates to improve the quality of open source basic components , So as to properly manage the whole open source ecosystem .

Based on the image of open source software , At the same time, we put forward the idea of software engineering digital platform , Digitally decompose and record the whole process of software development , By taking the software architecture and open source components as the software skeleton, the whole process of software development is connected , Make the software development process transparent 、 Visualization and traceability , Understand the development process quantitatively 、 Development quality 、 Personnel effectiveness and development bottlenecks . Of course, the development and investment of this platform is very huge , I hope to work with more partners to promote .

Almost every commercial project uses open source software to save development time 、 Reduce company costs 、 Avoid making wheels again , However, software companies may not pay enough attention to the quality and source of these codes . So I want to put forward some ideas and suggestions about the open source software industry ：

• Be a firm long-term believer . The correct and safe use of open source software requires the power of the community , We should jointly safeguard the open source market and the ecosystem , Bring more vitality to the open source software industry , Practice in the balance of rules and freedom .

• Open source software has become an important infrastructure in cyberspace . Now many enterprises have embraced open source software , All parties should also actively carry out the source code detection project of open source software , Promote the internal construction of open source software governance system , Form a long-term mechanism for open source software management . The government 、 Enterprises 、 developer 、 Software practitioners, users and other parties should establish perfect legal norms for open source software , Import approval 、 Technical evaluation 、 Compliance use 、 Vulnerability detection 、 Update maintenance 、 Emergency response 、 Stop, quit and other systems .

• The stars and the sea . Regardless of technical or scientific background , We should actively participate in and carry out forward-looking technology exploration and product research and development , Strive to promote the development of open source , Strengthen the standard guidance and education on open source , From a technical 、 Make corresponding contributions at the application and legal levels . The open source community （ Including code hosting platform 、 Software Association 、 Industry Alliance ）、 Open source organizations such as the open source foundation need to make full use of their own and collaborative advantages , Call on more contributors , While developing and expanding the influence of open source projects , Shoulder the responsibility and mission of open source security and continue to move forward .

Conclusion

Software engineering is a complex and important engineering activity , Its application promotes our work and life to be more efficient 、 facilitate . I learned programming from a student , To researchers who study software , Then the founder of a start-up company led the team to provide automated tools for software development , The transformation of identity also drives me to carry out self iteration .

These twenty years , I deeply feel the importance of technology and software , And strive to extend them to product services and application scenarios to play a greater value , Hope to accumulate through theory and practice , Cultivate compound talents with a sense of social responsibility , Accumulate strength for the development of the industry . I also hope to accompany programmers to grow all the way , Help them find “ Own business ”. In the next five to ten years , I will continue to promote the research and transformation of software engineering digitalization , And more reliable 、 Make the best contribution to a safer digital world .

The authors introduce ： Liu Yang , Nanyang University of technology, Singapore (NTU) Professor of computer college ,NTU Director of network security laboratory 、HP-NTU Laboratory director , Deputy director of Singapore National Centre of excellence , Network security company Scantist Co founder and CEO, Specializing in software engineering 、 Network security and artificial intelligence .

— Recommended reading  —

《 New programmers 001-004》 Fully listed

Welcome to scan the QR code below or click to subscribe now

You can enjoy e-books and exquisite paper books