Hologres shared cluster helps Taobao subscribe to the extreme refined operation

author ： Eucommia ulmoides Oliv  （ flower ： Divine heaven ）   Taobao subscription development , Now mainly responsible for Taobao subscription business , Main research content features understanding .

Taobao subscription is in urgent need of refined content operation

Taobao subscription is based on C-B Users of the relationship / Business dual private domain products , User side and recommendation - Guess you like complementation , Build subscription - I like the mind . The merchant side is deeply linked with the merchant , structured , Automation introduces quality supply , Help businesses better operate fan members .

In the initial stage, the content release from the background of the merchant was built , Distribute recommendations to the algorithm , And then to the complete link of foreground consumption and data recycling ; In the later stage, in order to refine the operation , Improve the content recommendation experience , Start exploring content features , Build a content feature circle selection system .

1、 Content needs to use features when recommending distribution

• High quality content circle selection  ： Distribute content at the front desk of Taobao subscription , Support multiple dimension feature filtering methods .

• Low quality content filtering ： Yellow River, politics and meaningless content , Filter through feature filtering .

2、 Feature selection is required for content operation

• The core content is put on display ： The operation will select a batch of core in-depth operation contents for front office delivery , Through the circle selection system , Filter by different dimensions , The obtained content is used for the content aggregation page of the foreground

• Promote the content atmosphere ： A number of activities were selected by the operation circle , When the front desk is revealed, it will be greatly promoted to strengthen the atmosphere

• The flow of cooperative merchants is inclined ： Through the circle selection system , Circle the contents of a group of core cooperative businesses , Flow tilting is performed when the content on the front desk is displayed by hand .

Subscription features circle selection system engine selection

Content circle selection is a filtering operation for existing content , The selected content has many dimensions , Large amount of data , There are also certain requirements for data preview , Therefore, it is necessary to design a scheme as a whole , To make the circle selection more accurate . in addition , Technology also needs to consider future scalability , It makes it more convenient to add data indicator filtering in the future .

Architecture design of current subscription circle selection system

The following is the design scheme of the content feature generation and subscription circle selection system .

• Abstract the circle selection process into content id+ relation id+ Multi dimension indicator filtering , Get the filtered target content id The process of ;

• The process of creating an activity instance that contains a batch of content ;

• Configure the circlable information into filter items schema;

• Take the actual filter condition value as the filter item value.

therefore , You can turn the problem into a filter based problem schema And filter items value, Filter by different indicators , Data query operation .

The core demands of engine selection ： flexibility & High performance

The existing circle selection system has supported configuration , You can customize data sources and indicators to circle . During the circle selection ,

The process of translating multiple filter items into executable query statements is the core of the circle engine

. The filter engine needs to map different filter items to fields in different tables , Generate executable statements , Then execute in the filtering engine to get the filtering results . Based on business scenarios , We have summarized the core demands for the screening engine as follows ：

• Easy access , Reduce the complexity of filtering executable statement translation

• Performance and stability guarantee , The logic of circle selection changes with the operation strategy , Need to support complex query and rapid response .

• Support to add variable characteristic fields , Have some flexibility .

Through a lot of research inside and outside Alibaba Group , And finally made a detailed comparison between several products , As follows ：

adopt Hologres Subscription system built by cluster

Through research and testing , Finally, I chose Hologres The cluster serves as the computing engine of the subscription system . The following will introduce the subscription system based on Hologres Best practices for clustering .

Hologres colony ： Less data movement + Faster queries

1、 Low use cost

• Fast access ： Hologres The shared cluster only needs to establish an instance for quick use , It can facilitate the quick start of business , It basically meets most usage scenarios . When business development needs it , You can apply for an independent cluster and migrate , This is relatively friendly among many engines of the group . Subscription services are also built based on public clusters at the initial stage , In the later stage, independent clusters are gradually used .

• Seamless development ：Hologres Supported by the SQL Query syntax and common SQL The query is basically consistent , Basically seamless use . The visual interface supports the function of one key synchronization table structure , It is especially suitable for students whose table structure changes frequently .

• Reduce data movement ：Hologres The natural support is to read and store in the form of appearance MaxCompute Multiple project The data of , This allows you to aggregate data from different project Of offline data , Reduce the complexity of query , You can query directly without data import and export , It also lowers storage costs .

2、 High query efficiency

• Compared with MaxCompute Query for , High performance improvement . After many tests ： The data volume is billion , Appearance query complex statements （ Contains multiple tables JOIN） It takes about 8-9 second ; It takes time to filter and query a single table 2 About seconds . Suitable for offline / Quasi real time query scenario . The internal table query is about 60ms, It can be used for online inquiry .

• By supporting  UDF/ Expression push down , To achieve user-defined UDF Calculation ; Pushing down the expression can reduce the overhead of useless data transmission , Further improve performance .

adopt Hologres Best practices for setting up subscription systems

adopt Hologres The flow of the subscription circle selection system built by the shared cluster is shown in the following figure ：

The operation only needs to check the filter item and fill in the filter value on the background circle page , The circle selection system will automatically generate Hologres SQL sentence （ The following example ） And in Hologres Get data in , Finally, the data is returned to the front end , And put it on the front desk . The operation will continuously optimize the circle selection scheme according to the launch effect , Increase the effect of circle selection , To achieve the purpose of more refined operation .

The whole process , There is no need to import or export data between systems , Only by clicking on the page , Can be transformed into SQL Calculate . At the same time, the selected content can be adjusted according to the business logic , complex SQL It can also calculate the desired circle selection data results quickly and efficiently , Save time in getting data . Make the whole link very simple and efficient .

SELECT feed_id
FROM qn_xxx_provider AS a
WHERE a.xxx_pv > 30000
AND a.xxx_pctr > '0.1'
AND a.last_publish_time >= '2022-06-17 08:00:00'
AND a.biz_xxx_code = '111'
AND a.ds = MAX_PT('xxxxxx_table')
AND CAST(a.owner_xxx_id AS VARCHAR) IN (SELECT b.domain_xxx_id FROM xxxxxxx_table AS b WHERE b.rule_type = 12 AND b.channel_xxx_id = 137 AND b.dataset_xx_id = xxxxx AND b.ds = MAX_PT('xxxxx_odps_channel') )
and a.feed_id in (SELECT feed_id from xxxxx_submission_feed_hh where activity_id = 222 and approval_status=1 and ds = MAX_PT('xxxxx_submission_hh') and hh = '13')
;

Business value

adopt Hologres Taobao subscription system built by shared cluster , Supported 1000+ Field operation circle selection activity task , Supports double 11、618、 New forces week and other large promotion activities , Support the configuration of multiple secondary pages for subscribing to play scenes, etc , It simplifies the construction of the subscription system , The offline data can be accelerated directly without data import and export , Reduce the operation cost , It can make the business focus on business growth more efficiently .

some time , We will continue to use Hologres To enrich the functions of the subscription system , So as to maintain the rapid growth of business , We hope that the circle selection system can ：

• Support for more real-time features ： Hologres The performance of the internal meter is superior , Import live features into Hologres In the inner table , Support real-time feature query

• Reduce tuning GUC Use of parameters , for example ：set hg_foreign_table_max_partition_limit =128;（ Single adjustment query Number of access surface partitions ）,  It is expected that better product-oriented capabilities can be used to solve , Reduce GUC Use of parameters .