What is the experience of maintaining Wanxing open source vector database

“ Dear Xiao Fan , Congratulations on becoming Milvus Project Maintainer！ As Milvus The maintainer of the community , This role has many rights ： Small to submit and review project code , Big enough to lead the daily operation of the community , Even make decisions about the technical route of the whole technical project . But compared with rights, there is a greater responsibility ...” ——《Milvus Maintainer Invitation》

This is my bravest decision in ten years .

Joining a startup 、 One year after participating in the open source community , I became Milvus  Community  [https://milvus.io]  Of Maintainer, Received the invitation letter above . Except for the joy of dreams coming true , I can't help sighing about the ups and downs along the way . Also on the “ It's on paper , We must know that we must do it ” This sentence has a deeper understanding , Maintenance experience of open source projects in this year , It taught me that I want to maintain an open source community not only by passion , It needs more “ physical strength ”、“ mental ” and “ Mind power ”.

Before I choose to devote myself Milvus The open source community , Before joining this startup , I have made three very important decisions ： for the first time , Is to join after graduation Oracle, Decide to engage in database related work ; The second time , Is to follow Cassandra founder Avinash Join a software definition storage startup Hedvig, Learn more about how to design and implement distributed systems ; third time , Is to decide to return home and join Alibaba , Build an autonomous and controllable cloud native multi-mode database Lindorm. Fortunately, , Although these three choices are difficult modes , But they all gained a lot , Let my outlook on life and technology have been further improved .

Before deciding to start a new journey , I once asked myself what I wanted , After all the previous experiences , The new answer is on the paper ： I saw the AI New opportunities for the explosive growth of unstructured data in the era , Want to be on this new track , From zero to one, make a product for AI Database products 、 Make a masterpiece of your engineer career . So I resolutely gave up the million annual salary and the familiar old track , Start to pursue their open source ideals and technological aspirations .

If you want to know how to maintain an open source community , Or you are like me , I hope to join a meaningful open source community , Improve your technical influence , Do something difficult and right to help more people . Maybe next , What I want to share can help you , Let friends on the open source road take fewer detours .

Open source maintainers 101 - Physical strength

As a maintainer of an active open source community , The hardest thing is to program day after day , Almost every day I browse emails and clean up Github notice [https://github.com/milvus-io/]  At the beginning . Sometimes because of a business trip , Without clearing up the news in time ,Github The notification interface will look like this , This is for the “ Red dots eliminate obsessive-compulsive disorder ” It's really a disaster for me .

Stuffed with notices Github inbox

Github It brings not only a surprising number of on-site notifications and email reminders , It also contains a lot of inexplicable descriptions issue、discussion, Repeated questions including common errors , Non conforming items CI Quality requirements and failure to follow process specifications PR Contribution submission , And a lot of daily trivial project warehouse related management affairs .

After struggling in a lot of attempts , I have found some ways to help developers of open source projects .

One 、 Pay attention to the maintenance of documents

Good documentation can reduce a lot of meaningless repetitive work , So please update the documents of the projects you maintain .

Be sure to check whether your open source project includes a meaningful Of “README.md”, And confirm that in this document , Have a clear introduction to your project vision , And the information users need to start using and contributing to the project .

maintain A good “ Project roadmap ”, So that users can know when the functions they need will be released . meanwhile , It can also facilitate contributors to quickly find their own interesting and suitable content to contribute .

Open Basic rules and regulations and operation norms of the community , such as “DEVELOPMENT.md” It can help new students understand the development process ,“CONTRIBUTING.md” So that contributors can clearly understand the contribution process and review rules ,“MAINTAINER / CONTRIBUTOR” List file , It can help people find community members who can ask for help .

update User documentation and technical architecture documentation , Can significantly reduce the cost of interpretation . When community users encounter common problems , I usually encourage users to consult project documents by themselves , Find the answer you want through your own strength .

Two 、 Attach importance to Automation

The more parts of automation, the better , Code is better than a thousand words .

Repetitive and innovative work , Can automate , such as Prow It can help you save a lot GitOps Related work ,Mergify You can check the rules , Auto merge code .

The code coverage of our previous projects is very low , In the introduction of Code Coverage And complete the minimum code coverage limit , The coverage of the project soon increased to 80% above .

When we introduce Golint after , It helped us complete the project Go Unified code style . Of course , It also took us a few weeks , Conduct centralized problem repair . And then , In a large number of references Uber Of Go Language development specifications  ① after , We also have strict requirements for the format of code submission information .

Besides , We also use Github Action To build the CentOS、Ubuntu、MacOS、MinGW The development environment of these different systems , Give Way Milvus Be able to compile and test automatically on multiple platforms .

3、 ... and 、 Learn to refuse

Open source maintainers should learn to say no .

Maybe human nature is to add , However, add an immature function , Or accept one that does not conform to community norms PR, It usually doesn't help the community get better .

Before accepting a function from a community contributor , Ask yourself first ：“ Whether this contribution conforms to the code specification of the community ”？“ Whether to comply with the original design intention of the project ”？“ Meet your own technical aesthetics ”？“ Is it really elegant and efficient ”？

When there are many “ confused ” Of issue Or the quality is not up to standard PR Contribution time , Think about whether you provide users and developers with good issue or PR Template , Let participants have examples to follow .

If your project has not been used “GitHub Template ” function , I strongly suggest you try . I mentioned above , Give priority to establishing clear rules .

When users and developers have problems , Please also point out the problem to the contributors immediately , Which rule did he break , And put forward improvement plans to him or send him relevant documents for reference .

Open source maintainers 102 - Brain work

Just investing a lot of physical strength can not become a qualified open source project maintainer , More importantly Continue to think and precipitate the methodology of how to maximize the value of the project .

methodology

In my submission , A good open source project must first have its unique value , It is better to be able to README The first sentence in is reflected .

With K8s Community examples （README ② The first sentence in ）：

"Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications."

The above 20 words , The problem to be solved by the project is explained to the user concisely . A maintainer of the open source community , You must think over and over again about the true value of the project to users , And continue to precipitate , Refine the vision that meets the expectations of community users .

When the project has a clear enough vision , The next most urgent thing is to develop a solution to the problem of user scenarios MVP product . And define what is “ Minimum feature set ” It also tests the wisdom of open source defenders . We once stepped on a big hole in this link , Start up Milvus 2.0 Project time , We have set too ambitious goals , So that in the whole 2021 Mid year , The development team is deeply involved in repairing Bug And reconstruction .

Most of the time , Open source projects want to be successful , When starting, you should often choose niche markets , Solve a small user core pain point at a low cost , And continue to expand iterations on this basis , Instead of excessive pursuit of large and complete beauty of form . Recommended reading 《RethinkDB： Why did we fail 》 This article , The article starts from a CEO It explains how important it is for an open source infrastructure product to deliver the minimum feature set on time .

During project iterations , The development process of the community has been gradually improved through continuous thinking and discussion . I want to sincerely thank you here @yanliang567  ③ classmate , As the quality director of the community, he undertook a lot of detailed work disassembly and task planning ： From the first preview to 2.0 During the release of the official version , We have experienced as much as 8 individual RC edition , Every experience a version , Our software quality indicators become clearer .

In this process, we gradually increase the data volume of stress testing 、 For a long time 、 Chaos test is added 、 Rolling upgrade test and data compatibility test . The achievement of goals in each stage , Have provided us with a lot of positive feedback , It greatly improves the development team's attitude towards the product “ Stable and reliable 、 Safe and fast ” Confidence in the goal .

With Milvus For example , Continuously improve the project release Standards

In the process , The development process of the community has also experienced gradual thinking and improvement , Get an efficient train release model ： Release a reliable major version every year , Release a large version every three months , Publish one every two weeks bugfix edition . The complete workflow is as follows ：

1. When the maintenance period of each version is approaching the end , By the community Maintainer Organizing meetings , Maintainers discuss the functions to be released in the next version , And confirm the specific time point of release .
2. When the new milestone of the project is formulated , Claimed by team members and community contributors , Make a reliable development plan , perfect Milvus Enhancement Plan And initiate a discussion in the community technical meeting . This stage is also called Debt Week, In addition to completing the discussion of new function design , We will also focus on repairing some technical bonds and Bug. The community strictly abides by “ Document first ” Principles , The developer of the claim function should not only implement the code , It also needs to be maintained and designed “ file ”, So that we can carry out a series of tests later 、 Users can have a clear function use document .
3. In the intense development process , The development leader of each function needs to align the project progress every week , And adjust the function according to priority and implementation status “ Get on the train ” still “ Get off the bus ”.
4. The last month of the version development cycle , We usually choose to freeze code , by QA The team pulls out a branch for testing . In this month , We will also focus on solving the problems that have not been solved before 、 Higher priority Bug, When Bug Triange The team judged that the software had solved all blocking releases Issue、 After judging that the quality can meet the release standard , The software can be released to meet users .

I personally think something like this “ Train model ”, It is very suitable for the open source community, which is a relatively loose team . Can greatly reduce because “ Human single point ” Cause the project release to be blocked . in fact , Maintain a relatively frequent release rhythm , For the project in GitHub The activity on the platform will also help , Whenever I release a version , Can see GitHub Warehouse and Docker The number of image downloads has increased significantly .

Open source community operation

At the end of this section , I want to talk about the operation of open source community .

Many students with engineering backgrounds are more concerned about the value of open source project code itself , And easy to ignore “ file 、 Community operations 、 well Demo” These three pieces bring a huge value bonus to the project .

In the process of learning open source operation , Book of night sky official account ④ and 《 The sales Acceleration Formula 》⑤ This book brings me a lot of new thoughts , If you are interested in open source operation , Welcome to discuss and exchange .

here , I share three tips about open source operation ：

1. Not only for data operations , We should also face the correct data operation

About whether it is necessary to build an index system for the operation of open source communities , There have been heated discussions in the community and within the company , I am a loyal supporter of digital operation . Personal view , Not just open source operations , For any “inbound” The way to reach , The established user funnel is the only way to get customers .

At present Milvus In the community user system , It also includes such a model ：

Milvus Community participant funnel

For different types of community participants , The help we should provide is different .

such as ： For junior users, we should provide more friendly access documents and Bootcamp, Help users run the product in half an hour , Judge whether the project can solve their problems ; And for deep users , We should pay more attention to whether wechat groups are active 、Github Issue Can I get a reply on the same day , Does the community have enough high-quality technical documents to help them understand the technical details , Are there enough auxiliary tools in the project to help them complete the monitoring 、 Log collection 、 Common requirements such as data migration ; For contributors ,Milvus The community launched Mentorship project , adopt “ Old brings new ” To establish the connection between new contributors and old contributors , Help them quickly integrate into the project , Accomplish the goal you want to achieve faster ; Last , For deep contributors , The community should establish a more perfect honor system and incentive mechanism , Including but not limited to Membership Mechanism , Clearly acknowledge contributors in the release notes , Of course, you can also thank contributors through small gifts full of affection .

My personal suggestion is for different stages of the community , Try to choose different 、 Reasonable quantitative indicators , Using an index system that adapts to the stage of community development will be of great help to community decision-making .

Even though “Star” The quantity of is a very important quantitative index , It is often used to measure the popularity of an open source project , However, paying too much attention to this indicator will lead to the funnel staying in the initial stage of customer acquisition for a long time , Loss of activation 、 More important contents such as retention and final transformation . therefore Star These relatively simple numerical indicators should never be the focus of open source project maintainers , Even the only data indicator concerned .

2. At the right stage , Do the right thing

Peter, master of Management . Drucker once pointed out ：“ Efficiency is doing things the right way , Efficiency is doing the right thing , For enterprises , The indispensable first is efficiency ”. in fact , For the community , As long as you don't deviate from the project mission , Not against the spirit of open source , User oriented , Most things are in the right direction . The real effectiveness challenge lies in facing the different stages of the community , How to make a correct implementation plan .

for instance , We used to build it ourselves Milvus Online forum , For community operation, it is a relatively failed attempt . The original intention of building the forum is that we hope to be able to send users to Slack And discussed in wechat 、 Organize and disclose knowledge that is difficult to precipitate . Therefore, it draws lessons from TiDB Community AskTUG After the successful experience of , We're online, too Milvus Community own Forum . However, this forum did not get the expected good feedback , Even Milvus Of Maintainer Teams rarely use forums to communicate .

Come back later , Community operation should not be simple “ Copy homework ”, Instead, measures should be taken according to local conditions . User groups of the two communities 、 Number of users , There is a gap between defenders , Simply setting up a forum is more formal than practical . Besides, GitHub Discussion and Slack It can well meet the core needs of users' questions and discussions , In fact, there is no need to introduce new forms .

3. Only a simple process can last

Developing community processes is also the core work of community managers , But the first thing to make clear is that the community is not made up of cold code and files , The final service object of the process is “ people ”, That is, community members and community users .

Before you figure out the meaning of a process , Don't rush to introduce any new processes , Instead, you need to ask yourself a few questions ：

First of all , Whether the objectives and participants of the process are clear ;

second , Is there a clear goal for each step of the process ;

Third , Whether the expectations and feedback of community participants on this matter are clear ？ For example, what developers expect in the submission process ？reviewer What's your expectation ？CI/CD What are the defenders' expectations ？

It is necessary to ensure that the process can be effectively implemented by the participants in each link , Instead of being an obstacle to the furnishings or contributions of participants .

The benefit of a simple process is that the cost of implementation and compliance will be significantly reduced , The revenue will increase significantly . I have read nearly 60 Page Alibaba Development Manual , Try to promote , In fact, it has little effect , Even myself as reviewer Nor can you write code that perfectly conforms to the development manual . by comparison , A more reliable way is to find or customize a code style that can be checked 、 And automatically judge whether the code can be merged according to the rules IDE plug-in unit , Completely automate the whole thing .

About the process , A more obvious benefit of simplicity is iteratibility . Good processes are used , It needs continuous publicity and encourages everyone to follow , It also requires special maintainers to continuously pay attention to whether the process is running normally . Simple processes may exist in the early stage of application “ Loophole ”（ Some scenes are missing ）, But it is convenient for maintainers to constantly supplement details , And too complex processes can easily lead us into formalism and lose the original intention of doing these things .

Open source maintainers 103 - Mental force

About the psychological construction of open source defenders , I don't think any article is better than 《How to Use Open Source and Shut the Fxxk Up At the Same Time》⑥ The introduction in the article is clearer . Open source projects have high popularity due to their openness , It not only brings technical influence and other positive effects to the maintainers of open source projects , It also brings a lot of unexpected troubles .

As a contributor to open source projects , It's hard to completely ignore the voice of the community , Especially in the process of project maintenance , Will frequently encounter a large number of stretch out party for stability 、 Complaints about performance 、 Requirements for various customized functions . During technical route decision , Maintainers need to introduce to users with different cognitive levels again and again when、why、what and how This series of questions . These problems also bothered me for a long time , It slowly erodes the patience of defenders including me , Fortunately, there are always friendly users when we need them most , Give us encouragement , Cheer us up , Let's feel that all our efforts are worth it , It seems that I have found my original intention to make open source software ：

Thank you for cheering us up

As an open source contributor , Always maintain a tolerant and friendly attitude is a lifelong compulsory course . Learn to face all kinds of criticism and complaints calmly , Look at all positive and negative evaluations dialectically , Whether the point of view is correct or not, we can discuss it as it is , It is the biggest growth I have gained in this year . Mr. Lu Xun once said “ It's enough to have a confidant in life , Si Shi should regard it with the same mind ”, Be able to find the world outlook of 32 in the community 、 sense of worth 、 A small partner with a highly convergent view of Technology , It has been my greatest happiness as an open source contributor , After all, our original intention is to do a meaningful thing with a group of sentient and righteous people ！

Open source contributors work hard to build the next level , In my opinion, I can be a person with temperature . The open source world needs more idealism and altruism , Not just cold code fragments , Or full of simple and rigid rule-based arguments ."Community over code" yes Apache The code of conduct of the community , The success of the community is based on code , It's more from how community defenders treat people and things , Whether we can make reasonable decisions .

In my career as a programmer , Several predecessors have had a far-reaching impact on my view of technology and the world , such as Cassandra The original author Avinash、Apache HBase The first in the East eighth District Committer Tianwu , Of course, it also includes Milvus The initiator of the project, xingjue . It is under the careful guidance and cultivation of these predecessors , Only then can I become familiar with open source technology 、 Learn about open source culture 、 Grow in an open source environment , Finally, I became a maintainer of open source projects .

A problem is just a small effort for defenders , Maybe it's as simple as typing an order , For some community users, it may be a headache , Even affect their “ Jobs ”. So I keep practicing myself , Hope to be friendly to users and developers in the community , Maybe just a little bit of goodwill can change one's view of open source projects .

Last , The construction of mental power often comes from inner contradictions .

as everyone knows , Open source is not a business model , Pure open source projects cannot bring any economic benefits to open source contributors . In terms of Technology , Open source is the best way to create software , Because it can accelerate product feedback and innovation , The resurgence of open source software in recent years has benefited from “ Paid support ”： Such as Open-core model and SaaS When the business model matures . Behind a large number of successful communities and software, there is often a business company that continues to invest .

For contributors like me who work in an open source company ,"You always wear two hats", Simply speaking , It is necessary to pay attention to the healthy development of the community , Safeguard the interests of community users and developers , We should also ensure that the business services provided by the company have sufficient value and obtain commercial success . This requires a good balance between open source and business . But it's not easy , Open source companies often fail to commercialize , And as a result, the halberd fell to the sand , For example, these three typical problems ：

There are three scenarios in which the commercialization of open source software fails

Want to break , Enterprise decision makers and open source maintainers need to have a general pattern and peace of mind .

in my opinion , Open source software must provide users with core capabilities of sufficient value in order to achieve good commercialization , Instead of trying to make money by providing some small user value through micro innovation . If a good open source project wants to be commercialized, it must be able to form a flywheel effect 、 Form technological innovation 、 Form community scale , And the virtuous circle of business value . therefore , Another practice of mental strength lies in enough restraint , It is a restraint to the pace of commercialization , It is also a restraint to adhere to the core values of the project and not excessively expand the boundaries of open source projects .“Dream big,Step small”, I firmly believe that after the open source software itself provides enough valuable functions to users , Naturally, it will bring other enterprise level demands that can generate business value , such as ： reliability 、 Performance cost 、 Tool plugins 、 Audit services 、 Security 、 trusteeship 、 Data exchange and so on .

summary

Lack of enough mental strength , It is difficult to keep your heart and happiness in the complicated daily work ; Lack of brain power , It will be like a headless fly bumping around without results ; Lack of physical strength , It will lead to vision greater than implementation , Insufficient accumulation of quantitative changes cannot produce qualitative changes .

It is my great honor to join Milvus In the construction of such an excellent open source community , Let me in “ physical strength ”、“ mental ” and “ Mind power ” There has been an all-round improvement .

References offer

1. https://github.com/uber-go/guide

2. https://github.com/kubernetes/kubernetes

3. https://github.com/yanliang567

4. https://tisonkun.org/tags/%E5%BC%80%E6%BA%90/

5. https://book.douban.com/subject/35466335/

6. https://hueniversedotcom.wordpress.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/

The authors introduce

Luan Xiaofan

Zilliz  Partner and technical director

Luan Xiaofan yes Zilliz Partner and technical director , At the same time LF AI & Data Member of the technical advisory committee of the foundation . Prior to joining Zilliz front , He is the R & D Manager of Alibaba cloud , be responsible for NoSQL database Lindorm R & D of . before , He has worked in Oracle Corporation and software definition storage company Hedvig As a software engineer . Luan Xiaofan holds a master's degree in computer engineering from Cornell University .

---

Thank you for seeing this , If you are tired of working in a big factory “ screw ”, Just like us Sincerely recognize the value of open source , Believe in long-term ism . Welcome to join Milvus The community and the business companies behind it Zilliz  [https://zilliz.com], Believe that our position list , There must be a chance for you to show your strength .

Expect to be good at AI、 database 、 You in the direction of high-performance computing 、 Like to participate in the operation of open source community, you are full of vitality 、 You who are meticulous and patient and can continuously improve documents , Go forward with us on the open source road , Make the best vector database in the world ！

This article is from WeChat official account. - ZILLIZ（Zilliztech）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .