当前位置:网站首页>Clickhouse: Test on query speed of A-share minute data [Part 2]
Clickhouse: Test on query speed of A-share minute data [Part 2]
2022-07-01 09:25:00 【songroom】
explain :
1、my_db.stock_tb: It's a 22 Billion rows , about 15 Column and is a single table , The compressed space is 50G, The compression ratio is about 30 times ;
This table has been saved 2010 All the years since A Shares of 1 Minute line data . Why design a single table , I want to do a pressure test .
2、 I am on my home computer WSL2 Stand alone mode test under , Memory 32G.
Here is a simple test of the above table .
1、LIMIT N test
def get_data_from_ch(): # startdate : 2010-01-01
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
code = "600036.XSHG"
query_sql = f'SELECT * FROM {database_name}.{table_name} LIMIT 10000000'
print(query_sql)
data = client.execute(query_sql)
return data
t0 = time.time()
data = get_data_from_ch()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
Different tests were carried out LIMIT Number of tests :
2、 Conditional query test
(1)、 Single condition
def get_data_from_ch(): # startdate : 2010-01-01
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
code = "600036.XSHG"
query_sql = f"SELECT * FROM {database_name}.{table_name} WHERE code = '{code}' LIMIT 10"
print(query_sql)
data = client.execute(query_sql)
return data
t0 = time.time()
data = get_data_from_ch()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")
(2)、 Multi condition query of object and date
def get_data_from_ch_by_code_and_datetime():
client = Client('localhost')
database_name = "my_db"
table_name = 'stock_tb'
startdate = "2018-01-01"
enddate = "2022-02-02"
code = "600036.XSHG"
query_sql = f"SELECT * FROM {database_name}.{table_name} WHERE code ='{code}' AND toDate(datetime) >= toDate('{startdate}') AND toDate(datetime) <= toDate('{enddate}') "
print(query_sql)
data = client.execute(query_sql)
print(f"data : {len(data)}")
return data
t0 = time.time()
data = get_data_from_ch_by_code_and_datetime()
t1 = time.time()
print(f"get_data cost time : {t1-t0} s! ")

Now look at , If we don't optimize the design , The reading speed is still not good , It can't meet the business , Especially the requirements of quantitative platform . In general ,100 Reading 10000 lines of data , exceed 1 Seconds is too slow .
Where is the optimization of design ? Sub database and sub table ? Asynchronous read ?
边栏推荐
- FAQ | FAQ for building applications for large screen devices
- Rich text interpolation
- Serialization, listening, custom annotation
- Why is the Ltd independent station a Web3.0 website!
- js 使用toString 区分Object、Array
- Differences between JS valueof and toString
- PR training notes
- JS variable lifting
- Learning practice: comprehensive application of cycle and branch structure (II)
- Error org apache. catalina. core. StandardContext. FilterStart start filter exception
猜你喜欢

2.3 【pytorch】数据预处理 torchvision.datasets.ImageFolder

樹結構---二叉樹2非遞歸遍曆

Network counting 01 physical layer

js作用域链与闭包

2.4 activation function

【pytorch】2.4 卷积函数 nn.conv2d

2022.02.15_ Daily question leetcode six hundred and ninety

Phishing identification app

Football and basketball game score live broadcast platform source code /app development and construction project

nacos简易实现负载均衡
随机推荐
Meituan machine test in 2022
Which method is good for the management of fixed assets of small and medium-sized enterprises?
Tree structure --- binary tree 1
MySQL optimization
【ESP 保姆级教程 预告】疯狂Node.js服务器篇 ——案例:ESP8266 + DHT11 +NodeJs本地服务+ MySQL数据库
Solution of EPS image blur by latex insertion
类加载
[ESP nanny level tutorial preview] crazy node JS server - Case: esp8266 + DS18B20 temperature sensor +nodejs local service + MySQL database
韦东山板子编译内核问题解决
2022.02.15_ Daily question leetcode six hundred and ninety
Ranking list of domestic databases in February, 2022: oceanbase regained the "three consecutive increases", and gaussdb is expected to achieve the largest increase this month
闭包实现迭代器效果
【ESP 保姆级教程】疯狂毕设篇 —— 案例:基于阿里云和Arduino的化学环境系统检测,支持钉钉机器人告警
Mysql8.0 learning record 17 -create table
[pytorch] softmax function
Is it safe to dig up money and make new shares
序列化、监听、自定义注解
【pytorch】nn. Crossentropyloss() and nn NLLLoss()
【ESP 保姆级教程 预告】疯狂Node.js服务器篇 ——案例:ESP8266 + MQ系列 + NodeJs本地服务 + MySql存储
Implementation and application of queue