Jinshan cloud team shared | 5000 words to understand how Presto matches with alluxio

Introduction

• Jin Shan Yun - Enterprise Cloud team （ Zhao Kan 、 Li Jinhui ） In the interactive query scenario Presto And Alluxio Combined with a series of tests , And summed up some Presto collocation Alluxio Suggestions for use .
• This test does not use object storage , The network latency between the computing engine and storage is also relatively low . If storage IO Time consuming and network consuming ,Alluxio The acceleration benefit should be more obvious .

Test purpose

• Verify the impact Alluxio Factors that accelerate returns
• Record Alluxio Accelerated performance under appropriate conditions
• summary Alluxio Suggestions for use of

Test environment

Cluster deployment architecture diagram

• The computing engine uses Presto, Data source is Hive,Alluxio As caching .
• A total of 2 Clusters ：Presto+Alluxio colony ,Hadoop colony .

Presto+Alluxio Cluster configuration

• machine ：4 platform linux virtual machine
• cpu/ Memory ：32 nucleus 64G
• disk ：60GB SSD + 200GB SSD
• The Internet ： 10 trillion bandwidth
• Key configuration ：

Hadoop Cluster configuration

• machine ：5 platform linux virtual machine
• cpu/ Memory ：16 nucleus 32G
• disk ：60GB HDD + 1200GB HDD
• The Internet ： 10 trillion bandwidth

List of test data sets

• The data used in the test is tpc-ds Standard data set .
• Due to test needs , We imported several datasets of different sizes . The table structure in each dataset is exactly the same , The difference is the amount of data in the table , Partitions between some tables are also different .

Alluxio Strong acceleration performance

First let's take a look , When relatively suitable conditions are provided ,Alluxio Specific acceleration performance of .

• The dataset for the following query is tpc-ds Data sets tpcds_bin_partitioned_orc_100, The size of the entire dataset is 23.4G.
• The following is executed SQL Is in tpc-ds Provided sql Select the 7 Conditions are met , In the experiment 7 strip SQL Check repeatedly 10 Time , Record the time taken for each query .
• The interval between each query 8~10 second , to Alluxio Asynchronous cache for a certain time , Try to improve the hit rate of the next query .

Experimental results show

• Key points

￮ The unit of time consumption is ms.
￮ “avg” The value of the column is 2 Query to 10 The average number of queries .
￮“first” The value of the column is the 1 Time consuming for query .“ Accelerate revenue ” The columns are calculated as ：(first - avg) / first.

Analysis of experimental results

• The first query , Because the data has not been cached , Time consuming is the highest , In the future, the data will be stored asynchronously Alluxio, The query time decreases gradually , Wait for the full amount of data to enter Alluxio cache , At this time, the acceleration income reaches the highest , The query takes the least time and stops decreasing .
• We can see that compared with the most time-consuming first,avg The value of is greatly reduced , It takes only the original time 20%~30%. from “ Accelerate revenue ” In this column, we can make a more quantitative assessment of the accelerated benefits .
• Now let's verify the impact Alluxio Specific factors that accelerate returns . In the following experiments , You can see Alluxio In a more suitable environment , It can even reduce the query time to the original 5%.

Important factors affecting accelerated returns

• According to the results of the test , Describe in combination with official documents , Sum up a few factors that will affect Alluxio Accelerate revenue impact . Let's take a look at these factors
• Key points ： here Alluxio Accelerate revenue , It refers to how much faster the query time is after acceleration than before , It doesn't just mean IO How much faster did that part take . A more concrete description , It refers to how much time a single query consumes after acceleration is reduced by% compared with that before acceleration .

Factor one ： Of the query Hive The size of the underlying data file of the table is positively related to the acceleration benefit

1. Conclusion ： The queried data file is less than about 2MB when , The smaller the file ,Alluxio The lower the acceleration gain .

Because the file is too small , So each file is a split.Presto Resolve each split establish driver, Dispatch driver perform , This period of time cannot be accelerated , and operator The scanned data is too small , So disk IO The time consumption of will account for a lower proportion of the total time consumption ,Alluxio The benefit of accelerating queries will be reduced .

In the environment used for this test , Each data file is less than about 50KB when , In terms of the overall query time , The acceleration benefit will be relatively low .

The data file is larger than about 2MB when , The impact of file size on accelerated revenue is relatively less obvious .

This 2MB and 50KB It is only a rough threshold observed in this environment , It may behave differently in different environments .

2. verification ： With a simple SQL To test

【a. Description of experiment case 】

The test method is the same SQL Go to query 5 individual tpcds A table with the same name in a dataset , Guarantee SQL Same complexity 、 The table structure is the same , The partition table has the same number of partitions , Only the data file size is different .

Here is a simple SQL Statement to query repeatedly , Record the time taken for each query .

select * from web_sales where ws_wholesale_cost=1.where Condition field ws_wholesale_cost Not a partition field , Therefore, a full table scan will be triggered .

stay presto See only divided 2 individual stage

This experiment inquires about web_sales Table in 5 Related metadata in data sets

￮ For partitioned tables , Because the data file is small , Every file is a split.
￮ For a clearer contrast , The query of non partitioned tables is also introduced , The number of data files in non partitioned tables is small , Each data file size is in 100MB above ,Presto According to HDFS File block size split, Every split The data in is about 28~38MB about .

【b. Experimental results show 】

【c. Analysis of experimental results 】

￮ When each split Is less than 2MB when , Accelerated returns will be significantly lower . The smaller the data , The lower the acceleration gain ,split There are dozens of data KB when , The acceleration is very weak .
￮ We see when every split The data reached 2MB when , Accelerate revenue and 30MB It's not that different , Indicates that the data is greater than 2MB after , The data size has little effect on the acceleration .
￮ Now let's change one SQL、 Verify again with another table , We can see that the conclusion is the same .

3. verification ： use tpcds Medium Query55 To test

【a. Description of experiment case 】

query55 Contains aggregations 、 Sort 、 Size table join, Average complexity ,presto It will be divided into 7 individual stage

query55 The query fact table is store_sales.

store_sales Table related metadata

【b. Experimental results show 】

【c. Analysis of experimental results 】

￮ The conclusion is consistent with the last experiment , When split There are only a hundred years of data K When , The acceleration benefit is relatively weak , only 20%.
￮ split The data of is greater than 2M above , Data size has little impact on accelerating revenue .

Factor two ： Executive SQL The complexity of the statement is negatively correlated with the acceleration benefit

1. Conclusion ：SQL The more complex the statement ,Alluxio The lower the acceleration benefit

SQL The more complex the statement , The more time it takes to calculate , In the overall time consumption ,IO The time-consuming proportion will decrease , and Alluxio Can only accelerate IO Time consuming , So in the overall time-consuming acceleration , The benefits will be reduced .

In evaluation SQL Complexity when , We can refer to Presto In execution SQL Divided by time Stage The number of , To quantify SQL Complexity ,Stage The more, the more SQL It's more complicated . Of course, the complexity can also be roughly judged by the naked eye ：

a. Simple ： Group aggregation 、 Sort
b. complex ： Size table join、select Statement nesting
c. Extremely complex ： The big table join The big table 、 More tables participate join、 Multiple select Statement nesting

2. verification ： in the light of store_sales Test the fact table

【a. Description of experiment case 】

The test method ： Adopt selection tpc-ds Query the same fact table in query, To query the same data set . Ensure that the table structure is the same 、 Same number of files 、 Same file size 、 Same number of partitions , only SQL Different .

query Besides fact tables, dimension tables are also queried in , Different query The dimension tables involved may be different , But the data volume of the dimension table is very small ( Smaller than the fact table 1000 Times to 1w times ), And they are not partitioned tables , So here we can ignore the impact of different dimension tables .

When testing the same sql Repeat , Record the time taken for each execution . The interval between each query 8~10 second .

The following test data sets are selected tpcds_bin_partitioned_orc_100, Fact table selection store_sales.

store_sales Table related metadata ：

test SQL Use the following 6 strip

【b. Experimental results show 】

【c. Analysis of experimental results 】

￮ We see the simplest SQL, Acceleration gains are very high .
￮ In the face of general complex SQL,Alluxio Still very good acceleration income .
￮ On the whole ,Stage Quantity less than 15 when ,Alluxio Good performance .

3. verification ： When the queried data file is small and SQL The sentences are complicated

【a. Description of experiment case 】

Let's add another experiment here , When the queried data file is small , and SQL If the sentence is complicated , What about the acceleration benefits .

This experiment SQL Statements use query88, Check separately 2 Sets of data sets .

Table related metadata

【b. Experimental results show 】

【c. Analysis of experimental results 】

We see , It's not appropriate to be here Alluxio Under the condition of , The acceleration benefit will be relatively low .

Factor three ： Inquire about SQL The amount of data returned by the statement is negatively correlated with the acceleration gain

1. Conclusion ： The larger the amount of data returned by the query ,Alluxio The lower the acceleration benefit

Presto Finally, a Driver Come on reduce all Driver Calculated results of , More data ,reduce The longer it takes , This mainly includes the calculation time and network IO Time consuming . If you are scanning a table, you will filter very little data , Then the amount of data involved in the calculation will increase , The calculation time is increased , Then it will be reduced IO The proportion of time spent in the whole query time

2. verification ：

【a. Description of experiment case 】

We use 2 A simple SQL Query the same table , use where Conditions and limit Control the amount of data returned

Use tpcds_bin_partitioned_orc_100 Medium web_sales surface .

Table related metadata

￮ SQL1：select * from web_sales where ws_wholesale_cost=1- The number of data pieces returned is 7.35K, Data size is 2.12MB.
￮ SQL2：select * from web_sales limit 2000000- The number of data pieces returned is 2M, Data size is 558.87MB.

【b. Experimental results show 】

【c. Analysis of experimental results 】

￮ When a small amount of data is returned , The overall query time is very short , The acceleration benefit is very good
￮ When a large amount of data is returned , The overall time-consuming of the query has increased significantly , Accelerating returns is not good

Factor four ： Hardware environmental factors that affect accelerated revenue

Alluxio The acceleration benefit of is also affected by the following hardware factors ：

• Alluxio Cache read / write speed
• UFS Disk read and write speed
• network bandwidth

This part of the conclusion will not be verified .

Statement ： All tests were conducted under ideal conditions

• All the above tests , The size of the data file scanned at the same time exceeds Alluxio Caching , In the query , Data can be fully cached , Achieve the effect of all local hits . So we see that each query takes a long time 2 To 4 After the decrement of the query , It will fluctuate in a very small range , Generally speaking, the time consumption is relatively stable .
• In a production environment, many queries may be executed in parallel , Need to scan a lot of data , The total size of the data file will exceed Alluxio Cache size . Then it will trigger the elimination of data , It is not possible for every query to cache all the required data , So the query time will never be stable , Instead, it fluctuates in a large range .
• This test does not cover the scenario where data files are stored in object storage . If you use object storage S3, It is expected that there will be a more significant acceleration effect .

About Alluxio Used in production business 、 Optimization Suggestions

• Alluxio Good acceleration gains cannot be achieved in all scenarios , Want to play Alluxio Greater value , We've summed up some of the uses 、 Optimization Suggestions .

Recommendation 1 ： Use Shadow Cache testing Alluxio Fit with the production business

Introduce... Into production Alluxio Whether it can bring accelerated benefits , How much cache capacity should be set to maximize the benefits , Assessment and testing are required .

Alluxio Provides a Shadow Cache function , The bloom filter can be used to record which data the computing engine has accessed and the total amount of data . Every time the computing engine accesses data , Both count whether they hit once shadow cache Data in . combination shadow cache and Alluxio The hit rate of can detect whether the business is suitable for using cache .

￮ Business scenarios are not suitable for caching ： Statistics shadow cache shooting , If lower , It means that duplicate data is rarely accessed in the business scenario , It is not suitable to use caching .
￮ Alluxio Cache needs more space ： If shadow cache The hit rate is high , but Alluxio Cache hit rate is low , It indicates that the business scenario is suitable for caching , But the current Alluxio The cache space is too small . Because when the data file is accessed repeatedly , Deposited before Alluxio The cache of , It has been eliminated because the cache is full , So repeated visits will not get any acceleration . Then increase at this time Alluxio Cache space will get better acceleration benefits .
￮ Alluxio The benefit of caching has reached a better value ： If shadow cache High hit rate ,Alluxio Cache hit rate is also high , It means that at present Alluxio The acceleration benefit brought by caching to the business has reached a good level , Increase Alluxio Cache space doesn't make much sense .

Use Shadow Cache, To use Alluxio There is a more intuitive direction for optimization .

Recommendation two ： Preload thermal data

Alluxio The acceleration of needs to put the data load To cache , This requires waiting for the user to access the data first , That is to say, the first query does not accelerate the revenue . So for hot data , We can pre - visit before the user load To cache . For hot data that will not change for a long time , You can also put the data pin In the cache , Will not be eliminated .

Recommendation three ： Dynamically determine whether queries need to be accelerated

Some queries are Alluxio Caching can bring good acceleration benefits , Some don't . We can dynamically determine whether we need to make a query Alluxio cache . In the query , analysis SQL Statement to get the queried table , Go to get the metadata of the table first , Determine the data file size , At the same time, it also combines SQL The approximate complexity of , Then decide whether to query Alluxio.

Suggestion 4 ： Set single tier cache , Use SSD As a storage medium

Alluxio The official recommendation is not to set up multi-layer cache , The sinking and floating of data in multi-layer cache may have some impact on performance . For the comprehensive consideration of cost and performance , For single-layer cache storage media, it is recommended to use SSD.

Recommendation 5 ： Merge small files

Too many small files can affect Alluxio Accelerated benefits , Hot data can be , Perform a small file merge before querying .Alluxio It provides this function , It can be done to Hive Merge all files under the same partition , The merged file will not overwrite the original small file , It won't change Hive Metadata . When Alluxio To cache these small files , The merged files are cached directly , Greatly improve the reading speed .