当前位置：网站首页>Pulse style ｜ exclusive interview with Committee -- Tencent engineer Zhang Dawei calls you to eat "crab"

Pulse style ｜ exclusive interview with Committee -- Tencent engineer Zhang Dawei calls you to eat "crab"

2022-07-28 09:43:00 【StreamNative】

The interview time of this article is 2022 year 6 month , from StreamNative Interview and document .

Long time no see , I miss you so much , Meet you again in the exclusive interview with community figures Apache Pulsar The community has announced digital Committer, Let's watch the pulse style ！

The editor ：
Interviewee Zhang Dawei , tencent TEG Senior engineer of data platform department ,aka Baby King , Daily hobbies: walking children and playing with children . Community partners who have communicated on wechat will find that Dawei's Avatar changes , It's still the head with a baby .Pulsar There are many in the community Daddy role , Zhang Dawei Community experience after taking children What is the ？ The following is in the third person , Introduce new people Apache Pulsar Committer Zhang Dawei .

Zhang Dawei's wechat avatar

KoP “ Crab ” Team members

Dawei graduated from Northwestern Polytechnic University ,2016 I joined Tencent after graduation in , Inauguration TEG Data Platform Department , The main work direction is big data integration . in the meantime , He incubated an open source project with the team ——Apache InLong. Most of Dawei's work involves MQ And real time processing , Also doing Pulsar Flink Integrate .

Dawei paid attention to it very early Pulsar. Later on InLong The project is responsible for internal MQ when , Because the project needs a high consistency 、 Highly reliable MQ, In the end to 2020 Year begins Pulsar Land and contribute code .

When we ask “ How to introduce yourself to the community in one sentence ” when , David's answer is “ We are the first to use KoP The team ”, Enough to see that they are right “ The first person to eat crab ” Pride of identity . stay KoP（Kafka-on-Pulsar） Early stage of open source , In order to make the project available in the production environment , He and his team members have made a lot of contributions , There are some groundbreaking features , Such as the implementation [PIP -70] continuity offset. Ideas are constantly generated in the subsequent actual business operation scenarios 、 Find the problem , He was in Pulsar The stability and other performance improvements have also been highly contributed .

Dawei said , The most fulfilling moment in the community is to KoP When Tencent ran , They are The first in the industry will KoP The team in the production environment ！ From Tencent and StreamNative Work together to build KoP Open source ecology , To move most of the business to Pulsar, He and the team aim at the initial KoP Many optimization schemes are discussed in version .

Fearless Pulsar High threshold

Even as a senior Big Data Engineer , Dawei also admitted that he first came into contact Pulsar when , The feeling is Difficulty .Pulsar And ordinary MQ The difference is , Its code is asynchronous . Although there is a threshold , But after a long time of understanding, you will feel that its code and design are very intuitive .

Here we steal great lessons for everyone Pulsar How to get started ： It is difficult for beginners to read the code directly .Pulsar Every feature has PIP, Read first PIP The background of the design 、 There are relevant discussions about the plan , Deepen your understanding of code and design ; Then follow the process of data writing and reading 、 Follow the production and consumption process . Look at the process of data flow from a high level , Understanding will be more holistic .

After joining the community , The biggest feeling is to feel The advantages of the open source community . In Dawei's view ,Pulsar Community operations 、 Technology Extension 、 Technical documentation and community activities （ Such as TGIP And online and offline meetup） Are very suitable for developers , Whether in the company or in the open source community, it is of reference significance to do technology publicity and promotion . The support of the community has a very important impact on the internal support business .

Join in Pulsar The community has also expanded its own Technical ability and career development . David passed and Pulsar Community communication , Got to know a lot MQ Domain expert , such as come from StreamNative、 Technology lovers like didi and Huawei . The collaboration mode of the open source community has been verified by the huge community scale , This code is submitted to the community 、 Collaborative governance , Imperceptibly influenced the code processing at work .

Committer Insight
—— How to persuade the team to land Pulsar

to ground Pulsar front , Let's take an objective look first Pulsar The advantages and disadvantages of .Pulsar There is no doubt about its performance advantages , The design of cloud native architecture is novel , Easy operation and maintenance 、 Easy to expand and shrink .Pulsar stay Two level storage 、 Transactional production 、 In the financial scenario , Have leading competitiveness . however Pulsar There is still room for improvement in storage stability , The threshold for dealing with problems in the storage layer is relatively high , When it comes to BookKeeper, More can only find solutions by looking at the source code . To attract more users will Pulsar Run ,BookKeeper Relevant information is essential .

As the first to KoP The team that landed in the production environment , Zhang Dawei is teaching and persuading the team to land Pulsar I have a great say in my experience . Costs and benefits affect decisions , The reason why most teams are difficult to promote a project is the decision-making level's consideration of costs and benefits . In the process of system replacement , Unstable performance 、 Produce a loss 、 It is an unavoidable cost to spend time and manpower , Most decision makers are hard to accept . So we need to prove that system replacement can produce long-term 、 Considerable benefits . Although Tencent has now landed in large quantities Pulsar, But Dawei's team is also after the internal system performance cannot support the business , In the internal MQ And Pulsar After a long time of dual route operation , Before deciding to switch the system .

secondly , At present, many users use Pulsar Do big data integration , such as Pulsar - Flink. When the user selects a system , I hope to build a complete product link around my own scenario . Compared with analyzing the advantages of a system , More should be The solution brought by mining system .

Actually Pulsar The community has done a lot of support for business migration , such as KoP And other plug-in protocols , Users can only replace the server without replacing the client . Its technology has developed well 、 The community is active , Problems can also be solved in time .

Communicate effectively with the community

Do you have a resonance in this view —— Dawei concluded that in the open source community Terror The three things are ：

• carry PR No one reviewed
• No one discussed sending emails
• There is no context for the problem of small partners

Global developer remote 、 Cross time zone collaboration , Effective communication It is particularly important in the open source community .Pulsar It is a competitive open source project , The community is growing rapidly , Whether it's GitHub、 Mailing list, , Or wechat group , Daily discussions are endless . How to get the answer quickly , Dawei as Pulsar Senior members of the community , Suggestions are given ： Community members get together because of their love for open source and projects , Everyone takes time to give back to the community after work . Before asking questions , If possible Simple analysis 、 Location problem 、 Propose a solution conjecture 、 Fully describe and reproduce the steps , Then you will quickly understand the problem and discuss .Pulsar Technical documentation 、 There are various forms of introductory tutorials , comprehensive , General browsing can help beginners get started .

summary

Become Apache Pulsar Committer after , Zhang Dawei is still “ Be overweeningly ambitious ”：“ I will be in Pulsar Pay attention to technological progress in improving the stability and other characteristics of , And add corresponding support according to the business scenario , It is hoped that the transaction feature can be fully available at the production level . I will also contribute performance and stability in the future , For example, how to quickly locate and repair faults .” Dawei is still making continuous contributions , Team colleagues are also making an impact BookKeeper Committer.

We named the interview of community people 《 Pulse style 》 series , In addition to focusing on technology and community , I hope to see every GitHub ID The three-dimensional image of the contributors behind . The community is not just code , Friendship has accumulated behind every discussion . Pulse style will be displayed in more diversified forms , Introduce you to the friends behind the code .