Clickhouse learning (VII) table query optimization

---

Single table

prewhere

Its function and where identical , Used to filter data , First, the specified column data will be read , To judge data filtering , Wait for the data to be filtered before reading select Declared column fields complete the remaining attributes , Reduce io operation

explain syntax select WatchID, 
 JavaEnable, 
 Title, 
 GoodEvent, 
 EventTime, 
 EventDate, 
 CounterID, 
 ClientIP, 
 ClientIP6, 
 RegionID, 
 UserID, 
 CounterClass, 
 OS, 
 UserAgent, 
 URL, 
 Referer, 
 URLDomain, 
 RefererDomain, 
 Refresh, 
 IsRobot, 
 RefererCategories, 
 URLCategories, 
 URLRegions, 
 RefererRegions, 
 ResolutionWidth, 
 ResolutionHeight, 
 ResolutionDepth, 
 FlashMajor, 
 FlashMinor, 
 FlashMinor2
from datasets.hits_v1 where UserID='3198390223272470366';

prewhere If you don't close , On by default The shutdown command : set optimize_move_to_prewhere=0;

But in some scenarios, you have to specify it manually prewhere, So directly prewhere To replace the where The writing is simple and clear

Data sampling

SELECT Title,count(*) AS PageViews 
FROM hits_v1
SAMPLE 0.1 
WHERE CounterID =57
GROUP BY Title
ORDER BY PageViews DESC LIMIT 100;

from 1000 Before extracting data 10% The sample of , Only approximate data is not actual data

Column clipping and partition clipping

In general, column clipping is to select fields instead of *, Partition clipping is to read partition information

It should be avoided when the amount of data is too large select * operation , The fewer fields , The consumption of io The less resources , The higher the performance .

select WatchID, 
 JavaEnable, 
 Title, 
 GoodEvent, 
 EventTime, 
 EventDate, 
 CounterID, 
 ClientIP, 
 ClientIP6, 
 RegionID, 
 UserID
from datasets.hits_v1;

Partition clipping is to read only the needed partitions , Specify... In the filter criteria .

select WatchID, 
 JavaEnable, 
 Title, 
 GoodEvent, 
 ClientIP6, 
 RegionID, 
 UserID
from datasets.hits_v1
where EventDate='2014-03-23';

orderby combination where、limit Use

More than 10 million data sets order by It needs to be matched when querying where Conditions and limit Statement together

SELECT UserID,Age
FROM hits_v1 
PREWHERE CounterID=57
ORDER BY Age DESC LIMIT 1000

Avoid building virtual columns

Just try not to use as Create a new column
for example select a/b as t from test

uniqCombined replace distinct

Approximate de duplication uniqCombined
Count(distinct ) Will use uniqExact Accurate weight removal

select count(distinct rand()) from hits_v1;

SELECT uniqCombined(rand()) from datasets.hits_v1;

Multiple tables

preparation

Create a small table , Avoid memory explosion

CREATE TABLE visits_v2 
ENGINE = CollapsingMergeTree(Sign)
PARTITION BY toYYYYMM(StartDate)
ORDER BY (CounterID, StartDate, intHash32(UserID), VisitID)
SAMPLE BY intHash32(UserID)
SETTINGS index_granularity = 8192
as select * from visits_v1 limit 10000;

Create a result table for storing data , Avoid rendering explosion

CREATE TABLE hits_v2 
ENGINE = MergeTree()
PARTITION BY toYYYYMM(EventDate)
ORDER BY (CounterID, EventDate, intHash32(UserID))
SAMPLE BY intHash32(UserID)
SETTINGS index_granularity = 8192
as select * from hits_v1 where 1=0;

As far as possible with in no need join

insert into hits_v2
select a.* from hits_v1 a where a. CounterID in (select CounterID from 
visits_v1);

Must use join

!!! The principle of small table on the right must be met ,clickhouse The data of the right table will be loaded into memory and compared with the left table , No matter which connection method is used, it will only load the right table into memory

When the big table is on the right, the program will directly report an error , Not enough memory space