当前位置:网站首页>Clickhouse: Test on query speed of A-share minute data [Part 2]
Clickhouse: Test on query speed of A-share minute data [Part 2]
2022-07-01 09:25:00 【songroom】
explain :
1、my_db.stock_tb: It's a 22 Billion rows , about 15 Column and is a single table , The compressed space is 50G, The compression ratio is about 30 times ;
This table has been saved 2010 All the years since A Shares of 1 Minute line data . Why design a single table , I want to do a pressure test .
2、 I am on my home computer WSL2 Stand alone mode test under , Memory 32G.
Here is a simple test of the above table .
1、LIMIT N test
def get_data_from_ch(): # startdate : 2010-01-01
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
code = "600036.XSHG"
query_sql = f'SELECT * FROM {database_name}.{table_name} LIMIT 10000000'
print(query_sql)
data = client.execute(query_sql)
return data
t0 = time.time()
data = get_data_from_ch()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
Different tests were carried out LIMIT Number of tests :
2、 Conditional query test
(1)、 Single condition
def get_data_from_ch(): # startdate : 2010-01-01
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
code = "600036.XSHG"
query_sql = f"SELECT * FROM {database_name}.{table_name} WHERE code = '{code}' LIMIT 10"
print(query_sql)
data = client.execute(query_sql)
return data
t0 = time.time()
data = get_data_from_ch()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
(2)、 Multi condition query of object and date
def get_data_from_ch_by_code_and_datetime():
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
startdate = "2018-01-01"
enddate = "2022-02-02"
code = "600036.XSHG"
query_sql = f"SELECT * FROM {database_name}.{table_name} WHERE code ='{code}' AND toDate(datetime) >= toDate('{startdate}') AND toDate(datetime) <= toDate('{enddate}') "
print(query_sql)
data = client.execute(query_sql)
print(f"data : {len(data)}")
return data
t0 = time.time()
data = get_data_from_ch_by_code_and_datetime()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
Now look at , If we don't optimize the design , The reading speed is still not good , It can't meet the business , Especially the requirements of quantitative platform . In general ,100 Reading 10000 lines of data , exceed 1 Seconds is too slow .
Where is the optimization of design ? Sub database and sub table ? Asynchronous read ?
边栏推荐
- 3D printing Arduino four axis aircraft
- MapReduce编程基础
- Microcomputer principle - bus and its formation
- Record a redis timeout
- 树结构---二叉树1
- NoSQL数据库的安装和使用
- Log4j 日志框架
- Day06 branch structure and cycle (III)
- Solution of EPS image blur by latex insertion
- Pain points and solutions of equipment management in large factories
猜你喜欢
Installation and use of NoSQL database
MySQL optimization
dsPIC30F6014a LCD 方块显示
【pytorch】nn. Crossentropyloss() and nn NLLLoss()
Imitation of Baidu search results top navigation bar effect
集成积木报表报错 org.apache.catalina.core.StandardContext.filterStart 启动过滤器异常
FAQ | FAQ for building applications for large screen devices
2.3 【pytorch】数据预处理 torchvision.datasets.ImageFolder
js作用域链与闭包
樹結構---二叉樹2非遞歸遍曆
随机推荐
Ranking list of domestic databases in February, 2022: oceanbase regained the "three consecutive increases", and gaussdb is expected to achieve the largest increase this month
美团2022年机试
Youqitong PE toolbox [vip] v3.7.2022.0106 official January 22 Edition
ES6 decoupling top-level objects from windows
2.2 【pytorch】torchvision. transforms
Differences between JS valueof and toString
Exception handling of classes in C #
phpexcel 里 获取某一列的列表 获取某一列的字母
Football and basketball game score live broadcast platform source code /app development and construction project
JS prototype trap
How to effectively align team cognition
Short circuit operator lazy evaluation
js valueOf 与 toString 区别
Principles of Microcomputer - internal and external structure of microprocessor
Jetson nano installs tensorflow GPU and problem solving
Naoqi robot summary 28
js变量提升(hoisting)
富文本实现插值
JS use toString to distinguish between object and array
Error org apache. catalina. core. StandardContext. FilterStart start filter exception