In the field of software engineering, we have been doing scientific research for ten years!

writing | Liu Yang
Produce |《 New programmers 》 Editorial Office

The development of technology is inseparable from the innovation of scientific research and the landing of projects , The author of this article Liu Yang's programming life starts from interest , It has become a result of programming and scientific research day and night , Then he combined engineering with scientific research , Conduct in-depth research and Practice on software engineering digitalization . In this paper , He shared today when open source devours software , Experience of exploring the digitalization of software engineering .

Over the past 20 years , The world has experienced the transformation from informatization to digitalization . We see that enterprises meet the challenges of Technology , It is also clear that the whole industry now has higher requirements and standards for software management and information security . We do scientific research in school , Publish research results with professional papers and reports , At the same time, it also focuses on helping enterprises solve key problems , Provide some more effective technical support and enabling methods , Continue to let technology improve business value .

As a scientific researcher in the field of software engineering , I will sort out my thoughts and experiences over the past ten years for your reference . About the code 、 Program 、 Software and applications , I hope that through the narration of this article , Discuss and practice with everyone .

This article is excerpted from 《 New programmers 》004,『 Paper book + Electronic journal 』 It's officially on sale

My way of programming

I remember when I was in the sixth grade of primary school , A very accidental opportunity , I signed up for an input method training class , Thus, I have a strong interest in computers . Later, I leaned by chance and could type , Unexpectedly, he became a special student and entered a key junior high school . From then on, I decided to learn computer , It has also become my ideal and belief . But the college entrance examination results are not ideal , The score failed to reach the admission line of computer major of Zhejiang University , I chose the major of electrical information automation . Although electrical information sounds a little related to computers , But the course is completely different . To coincide with CSDN Of 《 The programmer 》 The magazine （《 New programmers 》 The predecessor ） stay 2000 Issued at the beginning of the year , Look at the story of programmers and the introduction of cutting-edge technologies , It even stimulated my desire for computer learning .

Life is sometimes full of twists , Three months after college , I was very lucky to get the opportunity to study in Singapore , Successfully entered the National University of Singapore, majoring in computer science, undergraduate , Finally, I realized my childhood dream . Four years of undergraduate , Learn the program from the beginning 、 Write program , Go to team up and write the program , Enjoy it every day . What impressed me most during my college years was , because 《 The programmer 》 It's a technical publication , It's not easy to buy , Every time I go back to China, I find newspapers and magazines in the streets , Every time I finish reading, I get a full harvest . By graduation , I have 50000 lines of code , I feel like I have finally become a programmer , There is a sense of pride .

When you graduate from college , The computer industry is in Dot-com bubble （ Internet bubble , Also known as the science and technology network foam ） After the low tide , So I decided to stay in school and continue my doctoral studies （ Zhibo ）, Mainly engaged in the research of program verification . At that time, it was not very clear which research directions were interesting , I just feel like “ Program ” Relevant is very good . At first, I worked as an assistant for my senior brother , Writing algorithm , I didn't expect the experimental effect to be good , Relying on the development experience accumulated in University internship, the algorithm is gradually made into several small tools , Finally, it gradually evolved into a software verification platform . During the doctoral period, I started to write programs 、 Design procedure , Finally, it gradually turns into an analysis program . Four years , A total of nearly a million lines of code have been written , At that time, I felt that I finally began to understand the program , It has also constantly expanded my understanding of Technology .

Relying on the software verification platform made during my blog reading , I was very lucky to get the teaching post of Nanyang Technological University in Singapore , Set up a network security laboratory （ See the picture 1）, It is here that I really start the scientific research of software engineering .

chart 1 Group photo of the scientific research team in the network security laboratory of Nanyang Technological University

In the last decade , Our research scope basically covers all steps in the software development process （ Demand analysis 、 Architecture design 、 Development 、 test 、 Operations etc. ）, Various quality attributes of software （ correctness 、 Security 、 reliability 、 Performance etc. ）, Various types of software and systems （ Mobile phone application 、 Artificial intelligence model 、 The Internet of things 、 Autopilot system 、 Smart contracts, etc ）.

The current situation and challenges of software development in the new era

With the continuous development of the software industry , The software system itself is becoming more and more complex , The corresponding software development process has gradually become a system 、 Comprehensive engineering science . Any one with excellent functions 、 Behind the complex software system , Can not be separated from the huge R & D and management team . However , When the complexity of software system increases explosively , In the past, repeated stacking of labor 、 The engineering means of rushing forward to expand production capacity gradually fails in software development , Various software development process models 、 Management methods are constantly emerging .

From the waterfall 、 The incremental 、 Spiral and other early models , To 21 Agile development in the 21st century 、DevOps etc. , Generation after generation of software development models emerge in endlessly . The supporting system of software development is becoming more and more perfect and systematic , Such as interactive teamwork 、 Model Driven Development 、 Microservices 、 Low code 、 No concepts such as code development 、 technology 、 Tools have entered our world . However , These technical means are constantly changing the process of software development methods , It is also gradually increasing the complexity of the process , This puts forward higher requirements for the process management of software engineering .

Besides , Large complex systems with commercial companies from top to bottom “ Cathedral ” Type of software development is completely different , Another focus on users 、 Advocate early release, such as “ Market ” The free software development model has withstood the test of time . among , The most typical case is Linux The success of the system . just as 《 Cathedral and market 》 As mentioned ,Linus Torvalds Pursue “ Publish as much as possible as soon as possible , Delegate everything you can , Open to all changes and integration ” Software development philosophy , Developers who will spread around the world , Only through the loose cooperation of the Internet , It makes a world-class operating system . Open source software created in this free environment is free 、 The nature of openness , It is gradually incorporated into the development process of mainstream software , Among them, many excellent open source projects are widely integrated 、 Used in the projects of mainstream software manufacturers , Thus avoiding “ Repeat the wheel ”, It greatly improves the efficiency of software development .

Review our research experience in software engineering , It also grows step by step around software development . Starting from the quality of software , At first, formal methods were used to model the software , And verify the correctness of the software . We have developed a series of complete software formal modeling languages and verification algorithms to analyze concurrent systems 、 Real time systems and probabilistic systems , At the same time, these methods are applied to distributed systems 、 Security protocols 、 The Internet of things etc. . Finally, these capabilities are integrated into a system called PAT（Process Analysis Toolkit） Formal verification platform （ See the picture 2）.

chart 2 PAT Formal verification platform architecture

But when the complexity of software increases or the scale becomes larger , Formal methods face scalability challenges . meanwhile , Developers also have very high requirements for mathematical modeling ability . therefore , We will gradually transform the method of scientific research into the technology of program analysis ： Static analysis and dynamic testing . Although these two methods cannot provide the same perfect result guarantee as the formal method , But in terms of practicality , Has brought about a significant improvement .

We use program analysis technology to do various applications , Including defect and vulnerability detection 、 Software performance evaluation 、 Software architecture is reverse 、 Software evolution and development efficiency analysis . Especially in terms of software vulnerabilities , Developed a series of dynamic testing techniques , Found a large number of unknown vulnerabilities in commercial software and websites , This improves the security of the software . Up to now , We have begun to use the method of combining dynamic and static , Analyze and locate possible defects and vulnerabilities with static programs , Then guide the test algorithm to quickly trigger the vulnerability （ See the picture 3）.

As the amount of code increases and open source code is widely used , We can access a lot of code data , Naturally, I began to think about how to use code big data and artificial intelligence algorithms to enable software development . We apply the algorithm of artificial intelligence to requirement understanding 、 Clone detection 、 Code search 、 Code completion 、 Vulnerability detection and software testing . These attempts let us really think about the importance of program semantics , And how to integrate program semantic representation with deep neural network . therefore , We learn the variable types of weak typed languages from strong typed languages through transfer learning , It involves the interpretability of program semantic understanding and deep learning . Although these efforts have made some valuable progress , But the application of artificial intelligence in software engineering is still in its early stage , There are still many challenges that need to be solved urgently .

Based on open source , The exploration of software engineering digitalization

Because software development is becoming more and more complex , In order to effectively alleviate the tight coupling of software development process and improve development efficiency , The industry faces the output of products in the process of software development 、 Verification and end-to-end traceability put forward higher requirements , For example, the expression of requirements documents in the forward development process 、 Design and verification of software architecture model 、 Reverse traceability of requirements documents 、 Verification of the specific implementation of the architecture design model at the code level 、 Reverse traceability from code implementation to design model and requirements . With the large-scale use of open source software , Because of its freedom 、 to open up 、 Out of control , It further intensifies the urgent need for end-to-end verification of software engineering , This finally prompted us to gradually clarify and embark on the road of exploring the digitalization of open source software engineering .

According to the Gartner data ,99% 's organizations use open source software in their information systems .Sonatype The company is also interested in 3000 We have conducted a survey on the use of open source software in three enterprises , The results show that every enterprise downloads 5000 Multiple open source software . The trend that the modern software industry is highly dependent on the open source system cannot be changed , But while open source software provides convenience for enterprises , There are also some risk problems that can not be ignored , Including intellectual property 、 Open source protocol component conflict 、 Security vulnerabilities 、 Data leakage, etc . For example, in the past six months Log4Shell And Spring4Shell“ Nuclear grade ” Vulnerability Events 、 developer Marak Squires“ Delete the library and run away ” event 、Elasticsearch License change event 、node-ipc anti-war “ Poisoning ” event , And high frequency npm Malicious component attack , These are all tearing apart the fragile open source supply chain .

under these circumstances , Want to do a good job in software quality and safety management at the same time , It has become a very challenging thing . This also strengthens our next research direction ： Explore better ways to develop programs , Develop more standardized tools to analyze , Understand and manage software and its development process , And improve the management mechanism of the whole open source . Especially when open source code is widely used , Open source governance and software supply chain governance have become a key scientific research topic .

Based on this , We are 2017 In, it incubated a software security company for open source component testing Scantist. Take software security as the starting point , Based on software component analysis , From the development of open source software itself 、 maintain 、 Release 、 In the process of being relied on , Seek effective maintenance and management strategies for open source software , And through in-depth understanding of various programming languages 、 The working mode of open source core modules such as package manager , Explore the formation of open source supply chain 、 Transmission and evolution , Form a set of solutions for open source supply chain governance .

In addition to the current security vulnerabilities and license related appeals that the industry is more concerned about , We have also carried out a lot of research work on open source software quality assessment . Based on team in software engineering 、 Precipitation of many years' scientific research work in the field of program analysis , Gradually condense a set of multi-dimensional software portrait framework for open source software , Around open source software code 、 The development process 、 The development team digitises , From quality 、 Security 、 Open source ingredients 、 Maintainability 、 maturity 、 Activity, business risk and other dimensions , Measure the health of open source software itself , And form a team 、 Understand the health status of the core development team of open source projects from different perspectives such as development risks . So as to realize the precise use of open source software in the process of software development （ Such as technical selection recommendation 、 Risk analysis, etc ）, Identify the key basic components in the open source environment , Monitor the health of its open source projects and development teams , And guide relevant business enterprises 、 The community actively participates to improve the quality of open source basic components , So as to properly manage the whole open source ecosystem .

Based on the image of open source software , At the same time, we put forward the idea of software engineering digital platform , Digitally decompose and record the whole process of software development , By taking the software architecture and open source components as the software skeleton, the whole process of software development is connected , Make the software development process transparent 、 Visualization and traceability , Understand the development process quantitatively 、 Development quality 、 Personnel effectiveness and development bottlenecks . Of course, the development and investment of this platform is very huge , I hope to work with more partners to promote .

Almost every commercial project uses open source software to save development time 、 Reduce company costs 、 Avoid making wheels again , However, software companies may not pay enough attention to the quality and source of these codes . So I want to put forward some ideas and suggestions about the open source software industry ：

Be a firm long-term believer . The correct and safe use of open source software requires the power of the community , We should jointly safeguard the open source market and the ecosystem , Bring more vitality to the open source software industry , Practice in the balance of rules and freedom .

• Open source software has become an important infrastructure in cyberspace . Now many enterprises have embraced open source software , All parties should also actively carry out the source code detection project of open source software , Promote the internal construction of open source software governance system , Form a long-term mechanism for open source software management . The government 、 Enterprises 、 developer 、 Software practitioners, users and other parties should establish perfect legal norms for open source software , Import approval 、 Technical evaluation 、 Compliance use 、 Vulnerability detection 、 Update maintenance 、 Emergency response 、 Stop, quit and other systems .

• The stars and the sea . Regardless of technical or scientific background , We should actively participate in and carry out forward-looking technology exploration and product research and development , Strive to promote the development of open source , Strengthen the standard guidance and education on open source , From a technical 、 Make corresponding contributions at the application and legal levels . The open source community （ Including code hosting platform 、 Software Association 、 Industry Alliance ）、 Open source organizations such as the open source foundation need to make full use of their own and collaborative advantages , Call on more contributors , While developing and expanding the influence of open source projects , Shoulder the responsibility and mission of open source security and continue to move forward .

Conclusion

Software engineering is a complex and important engineering activity , Its application promotes our work and life to be more efficient 、 facilitate . I learned programming from a student , To researchers who study software , Then the founder of a start-up company led the team to provide automated tools for software development , The transformation of identity also drives me to carry out self iteration .

These twenty years , I deeply feel the importance of technology and software , And strive to extend them to product services and application scenarios to play a greater value , Hope to accumulate through theory and practice , Cultivate compound talents with a sense of social responsibility , Accumulate strength for the development of the industry . I also hope to accompany programmers to grow all the way , Help them find “ Own business ”. In the next five to ten years , I will continue to promote the research and transformation of software engineering digitalization , And more reliable 、 Make the best contribution to a safer digital world .

The authors introduce ： Liu Yang , Nanyang University of technology, Singapore (NTU) Professor of computer college ,NTU Director of network security laboratory 、HP-NTU Laboratory director , Deputy director of Singapore National Centre of excellence , Network security company Scantist Co founder and CEO, Specializing in software engineering 、 Network security and artificial intelligence .

---

–END–

《 New programmers 》001-004 Fully listed

Welcome to scan the QR code below or click to subscribe now to enjoy e-books and exquisite paper books .