当前位置:网站首页>Clickhouse: Test on query speed of A-share minute data [Part 2]
Clickhouse: Test on query speed of A-share minute data [Part 2]
2022-07-01 09:25:00 【songroom】
explain :
1、my_db.stock_tb: It's a 22 Billion rows , about 15 Column and is a single table , The compressed space is 50G, The compression ratio is about 30 times ;
This table has been saved 2010 All the years since A Shares of 1 Minute line data . Why design a single table , I want to do a pressure test .
2、 I am on my home computer WSL2 Stand alone mode test under , Memory 32G.
Here is a simple test of the above table .
1、LIMIT N test
def get_data_from_ch(): # startdate : 2010-01-01
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
code = "600036.XSHG"
query_sql = f'SELECT * FROM {database_name}.{table_name} LIMIT 10000000'
print(query_sql)
data = client.execute(query_sql)
return data
t0 = time.time()
data = get_data_from_ch()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
Different tests were carried out LIMIT Number of tests :
2、 Conditional query test
(1)、 Single condition
def get_data_from_ch(): # startdate : 2010-01-01
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
code = "600036.XSHG"
query_sql = f"SELECT * FROM {database_name}.{table_name} WHERE code = '{code}' LIMIT 10"
print(query_sql)
data = client.execute(query_sql)
return data
t0 = time.time()
data = get_data_from_ch()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
(2)、 Multi condition query of object and date
def get_data_from_ch_by_code_and_datetime():
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
startdate = "2018-01-01"
enddate = "2022-02-02"
code = "600036.XSHG"
query_sql = f"SELECT * FROM {database_name}.{table_name} WHERE code ='{code}' AND toDate(datetime) >= toDate('{startdate}') AND toDate(datetime) <= toDate('{enddate}') "
print(query_sql)
data = client.execute(query_sql)
print(f"data : {len(data)}")
return data
t0 = time.time()
data = get_data_from_ch_by_code_and_datetime()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")

Now look at , If we don't optimize the design , The reading speed is still not good , It can't meet the business , Especially the requirements of quantitative platform . In general ,100 Reading 10000 lines of data , exceed 1 Seconds is too slow .
Where is the optimization of design ? Sub database and sub table ? Asynchronous read ?
边栏推荐
- Using closures to implement private variables
- nacos简易实现负载均衡
- 队列的实现和应用
- [ESP nanny level tutorial] crazy completion chapter - Case: ws2812 light control system based on Alibaba cloud, applet and Arduino
- Structure de l'arbre - - - arbre binaire 2 traversée non récursive
- 【pytorch】2.4 卷积函数 nn.conv2d
- Which method is good for the management of fixed assets of small and medium-sized enterprises?
- Click the screen with your finger to simulate F11 and enter the full screen
- Ranking list of domestic databases in February, 2022: oceanbase regained the "three consecutive increases", and gaussdb is expected to achieve the largest increase this month
- Microcomputer principle - bus and its formation
猜你喜欢

【pytorch】2.4 卷积函数 nn.conv2d

Jetson nano installs tensorflow GPU and problem solving

Vsync+ triple cache mechanism +choreographer

How to manage fixed assets efficiently in one stop?

Bird recognition app

MySQL optimization

Reproduced Xray - cve-2017-7921 (unauthorized access by Hikvision)

Why is the Ltd independent station a Web3.0 website!

Learning practice: comprehensive application of cycle and branch structure (II)

Preparing for the Blue Bridge Cup -- bit operation
随机推荐
Analysis and solution of JS this loss
【ESP 保姆级教程】疯狂毕设篇 —— 案例:基于阿里云、小程序、Arduino的温湿度监控系统
nacos简易实现负载均衡
【pytorch】transforms. Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
Serialization, listening, custom annotation
美团2022年机试
Using closures to implement private variables
[ESP nanny level tutorial] crazy completion chapter - Case: temperature and humidity monitoring system based on Alibaba cloud, applet and Arduino
Key points of NFT supervision and overseas policies
JS use toString to distinguish between object and array
MapReduce编程基础
记一次redis超时
Youqitong PE toolbox [vip] v3.7.2022.0106 official January 22 Edition
JS原型链
MySQL optimization
[ESP nanny level tutorial preview] crazy node JS server - Case: esp8266 + DHT11 +nodejs local service + MySQL database
MT7628K eCos开发入门
【电赛训练】红外光通信装置 2013年电赛真题
tensorrt yolov5_ trt. Py comments
利用闭包实现私有变量