Save the image with gaussdb (for redis), and the recommended business can easily reduce the cost by 60%

Abstract ： User profile storage is the core of recommendation business , But open source Redis incompetence . Huawei cloud Redis It is the best storage model , Easy cost reduction 60%, At the same time, enterprise level high stability .

This article is shared from Huawei cloud community 《 Hua Wei Yun GaussDB(for Redis) Uncover secrets 23 period ： use GaussDB(for Redis) Save a portrait , Recommend easy cost reduction for business 60%》, author ： gaussian Redis The official blog .

One 、 What is a recommendation system

I don't know if you've ever had such an experience , When your forefoot just bought a mobile phone on an e-commerce website , Open the e-commerce website in two days , There must be headphones on the home page 、 Following from 、 Bluetooth speakers and other mobile phone accessories . If you don't buy a mobile phone , It's a dress , Next time I open the e-commerce website , It must be pants and shoes matching with clothes .

Smart you can't help asking , Why e-commerce websites are so powerful , Can predict your preferences in advance , And recommend you the products you may like ？ In fact, behind this , Can't do without the powerful recommendation system .

What is a recommendation system ？ First, let's take a look at the definition on Wikipedia ： Recommendation system is an information filtering system , It can predict the user's response to the item according to the user's historical behavior “ score ” or “ Preference ”. Simply speaking , If you are an electronics enthusiast , Then the system will definitely recommend you all kinds of fresh ones 3C product , If you are one coder, Then your page must be full of all kinds of programming books . Recommendation system is very popular in recent years , Applied to all walks of life , Recommended targets include ： The movie 、 music 、 Journalism 、 Books 、 Academic papers 、 Search for 、 Focus classification 、 E-commerce shopping and game business .

Two 、 Recommend the architecture of the system

After knowing what a recommendation system is , Next, let's continue to introduce the architecture of the recommendation system , Take the game industry for example , The architecture design of a typical game business recommendation system is as follows ：

The recommendation system mainly consists of 3 Part of it is made up of , Namely ： Behavior log collection 、 Characteristic production and Characteristic consumption .

01. Behavior log

Big data business collects user behavior logs , Analyze and obtain user portraits , And save these user portraits in the distributed file system HDFS in

02. Characteristic production

Engineering business is responsible for providing a set of interface calls for big data business , Mainly “ Irrigation reservoir ”, That is to say, set a fixed time or according to a certain logic HDFS User portraits in are processed into features , Responsible for filling engineering business “KV Storage ”.

03. Characteristic consumption

The engineering business team is also responsible for implementing the recommended model of the algorithm team , They develop online reasoning components , from KV Extract feature data from storage 、 Analysis and calculation , Finally come to the recommended conclusion , Show it to the user .

3、 ... and 、 Recommend the storage pain points of the system

In the last section, we introduced the architecture of recommendation system in the game industry , This is also a typical architecture of recommendation system . As can be seen from the architecture diagram ,KV Stored in a complete set of links , Carrying an important connecting role . However , In the recommendation system KV There are two big pain points in storage , The first is the high cost , The second is the slow expansion .

01. The high cost

The first is the problem of high cost . Usually we build KV The first choice for storage is to build an open source Redis Cluster as KV The storage system .

One side , Open source Redis All of the data is stored in memory , As we all know, the storage cost of memory is very expensive , Only suitable for storing a small amount of data , If you have a lot of data , Storage costs will become a burden for enterprises ;

On the other hand , Open source Redis It's going on AOF There exists in the process of file rewriting fork Mechanism , Lead to open source Redis stay AOF On file rewrite , Its memory utilization is only 50%, This further adds open source Redis Memory usage cost .

02. Slow capacity expansion

Besides the high cost , Open source Redis There is also the problem of slow capacity expansion , In self built open source Redis In the cluster , As the business grows ,KV The storage capacity has to be expanded . But because of the original Redis Its own architectural characteristics , It is inevitable to happen in the process of capacity expansion key Cross slot transfer , As shown in the figure below , Span slot Migration takes a long time and the business is affected for a long time .

Four 、 Why recommend GaussDB(for Redis)

Know the pain point of the recommendation system , How to solve it ？ The fundamental reason is to reduce costs and increase efficiency , and GaussDB(for Redis) It can be said that it was born to solve these problems .

01. Reduced Edition

GaussDB(for Redis) from Two aspects Reduce KV The cost of data storage ：

On the one hand ,GaussDB(for Redis) All data of the tribe is stored , Compared to open source Redis Data is stored in memory , Its cost is reduced 75%~90%, Form a great price advantage . for instance , One 512GB The open source Redis colony , The cost is almost 5w/ month , And examples of the same specification , If it is replaced with GaussDB(for Redis), Its cost savings 40% above , Almost saved a labor cost in it . The following table shows the cost comparison between different levels of storage .

On the other hand , In the recommendation system ,KV The data mainly stores the information of user portraits , This information is basically through Protobuf Serialized information , and GaussDB(for Redis) Its own data compression function , The serialized information can be compressed with high compression ratio , The actual occupied space is only open source Redis Of 50% about , This further reduces KV The cost of data storage .

02. Efficiency

In addition to cost reduction , On the other hand, it increases efficiency . as everyone knows , Open source Redis In the framework of , As shown in the figure below , Its storage and calculation are not separated , This leads to node expansion , There will be piecemeal migration , As a result, the business will be affected .

GaussDB(for Redis) To solve this problem , It adopts the structure of separation of storage and calculation , As shown in the figure below ：

Under the structure of separation of storage and calculation , The underlying data can be accessed by any upper computing node , No data copy or relocation occurs during capacity expansion , Extremely fast ; At the same time, it also achieves minute level node expansion , Business second level perception ; Storage expansion business 0 perception . Whether expanding nodes or storage capacity , The impact on the business is almost 0.

5、 ... and 、 summary

This article briefly introduces what the recommendation system is , And take the game business as an example , It clarifies the architecture of the recommendation system and the existing storage pain points , meanwhile GaussDB(for Redis) How to solve these storage pain points . In the age of big data , There will be more and more application scenarios of the recommendation system , Data storage as a recommendation system ,GaussDB(for Redis) It perfectly makes up for open source Redis The short board , It can provide strong technical support for the recommendation system .

6、 ... and 、 appendix

• The author of this article ： Huawei cloud database GaussDB(for Redis) The team
• More product information , Welcome to the official blog ：bbs.huaweicloud.com/blogs/248875

Click to follow , The first time to learn about Huawei's new cloud technology ~