当前位置:网站首页>Nebula importer data import practice
Nebula importer data import practice
2022-07-07 20:20:00 【Figure database nebulagraph】
This article was first published in Nebula Graph Community official account
Preface
Nebula At present, as a relatively mature product , There is already a rich ecosystem . In terms of the dimension of data import, there are many choices . It's big and complete Nebula Exchange, Small and streamlined Nebula Importer, And for Spark / Flink The engine provides Nebula Spark Connector and Nebula Flink Connector.
Among many import methods , Which is more convenient ?
Introduction to the use scenario :
- Nebula Exchange
- Need to bring from Kafka、Pulsar Streaming data of the platform , Import Nebula Graph database
- From relational database ( Such as MySQL) Or distributed file systems ( Such as HDFS) Read batch data in
- Large quantities of data need to be generated Nebula Graph Recognable SST file
- Nebula Importer
- Importer Applicable to local CSV Import the contents of the file into Nebula Graph in
- Nebula Spark Connector:
- In different Nebula Graph Migrate data between clusters
- In the same Nebula Graph Migrate data between different graph spaces in the cluster
- Nebula Graph Migrate data with other data sources
- combination Nebula Algorithm Do graph calculation
- Nebula Flink Connector
- In different Nebula Graph Migrate data between clusters
- In the same Nebula Graph Migrate data between different graph spaces in the cluster
- Nebula Graph Migrate data with other data sources
Above excerpts from Nebula Official documents :https://docs.nebula-graph.com.cn/2.6.2/1.introduction/1.what-is-nebula-graph/
On the whole ,Exchange Instead of , It can be combined with most storage engines , Import to Nebula in , But it needs to be deployed Spark Environmental Science .
Importer Easy to use , Less dependency required , But you need to generate data files in advance , Good configuration schema Once and for all , But it does not support breakpoint continuation , Suitable for medium amount of data .
Spark / Flink Connector It needs to be combined with stream data .
Choose different tools for different scenarios , If used as a newcomer Nebula When importing data , It is recommended to use Nebula Importer Tools , It's easy and quick .
Nebula Importer Use
Before we touch Nebula Graph initial stage , At that time, the ecology was not perfect , In addition, only some businesses are migrated to Nebula Graph On , We are right. Nebula Graph Data import, whether full or incremental, adopts Hive Push table to Kafka, consumption Kafka Batch write Nebula Graph The way . Later, with more and more data and business switching to Nebula Graph, The efficiency of imported data is becoming more and more serious , Increase of import duration , So that the full amount of data is still imported at the peak of business , This is unacceptable .
For the above problems , Trying to Nebula Spark Connector and Nebula Importer after , Considering the convenience of maintenance and migration , use Hive table -> CSV -> Nebula Server -> Nebula Importer
The way to import the full amount , The overall time consumption has also been greatly improved .
Nebula Importor Related configuration of
System environment
[[email protected] importer]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
Stepping: 7
CPU MHz: 2499.998
BogoMIPS: 4999.99
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 36608K
NUMA node0 CPU(s): 0-15
Disk:SSD
Memory: 128G
Cluster environment
- Nebula Version:v2.6.1
- Deployment way :RPM
- The cluster size : Three copies , Six nodes
Data scale
+---------+--------------------------+-----------+
| "Space" | "vertices" | 559191827 |
+---------+--------------------------+-----------+
| "Space" | "edges" | 722490436 |
+---------+--------------------------+-----------+
Importer To configure
# Graph edition , Connect 2.x Is set to v2.
version: v2
description: Relation Space import data
# Whether to delete the temporarily generated logs and error data files .
removeTempFiles: false
clientSettings:
# nGQL Number of retries for statement execution failure .
retry: 3
# Nebula Graph Number of concurrent clients .
concurrency: 5
# Every Nebula Graph The cache queue size of the client .
channelBufferSize: 1024
# Specify the data to import Nebula Graph Graph space .
space: Relation
# Connection information .
connection:
user: root
password: ******
address: 10.0.XXX.XXX:9669,10.0.XXX.XXX:9669
postStart:
# configure connections Nebula Graph After the server , Some operations performed before inserting data .
commands: |
# The interval between the execution of the above command and the execution of the insert data command .
afterPeriod: 1s
preStop:
# Configure disconnect Nebula Graph Some operations performed before connecting to the server .
commands: |
# Error and other log information output file path .
logPath: /mnt/csv_file/prod_relation/err/test.log
....
Due to space Only show some globally relevant configurations , There are many configurations related to points and edges , Don't expand , For details, please refer to GitHub.
Set up Crontab,Hive After the table is generated, it is transferred to Nebula Server, Run when the traffic is low at night Nebula Importer Mission :
50 03 15 * * /mnt/csv_file/importer/nebula-importer -config /mnt/csv_file/importer/rel.yaml >> /root/rel.log
The total time is 2h, Complete the import of full data around 6 o'clock .
part log as follows , The maximum import speed is maintained at 200,000/s about :
2022/05/15 03:50:11 [INFO] statsmgr.go:62: Tick: Time(10.00s), Finished(1952500), Failed(0), Read Failed(0), Latency AVG(4232us), Batches Req AVG(4582us), Rows AVG(195248.59/s)
2022/05/15 03:50:16 [INFO] statsmgr.go:62: Tick: Time(15.00s), Finished(2925600), Failed(0), Read Failed(0), Latency AVG(4421us), Batches Req AVG(4761us), Rows AVG(195039.12/s)
2022/05/15 03:50:21 [INFO] statsmgr.go:62: Tick: Time(20.00s), Finished(3927400), Failed(0), Read Failed(0), Latency AVG(4486us), Batches Req AVG(4818us), Rows AVG(196367.10/s)
2022/05/15 03:50:26 [INFO] statsmgr.go:62: Tick: Time(25.00s), Finished(5140500), Failed(0), Read Failed(0), Latency AVG(4327us), Batches Req AVG(4653us), Rows AVG(205619.44/s)
2022/05/15 03:50:31 [INFO] statsmgr.go:62: Tick: Time(30.00s), Finished(6080800), Failed(0), Read Failed(0), Latency AVG(4431us), Batches Req AVG(4755us), Rows AVG(202693.39/s)
2022/05/15 03:50:36 [INFO] statsmgr.go:62: Tick: Time(35.00s), Finished(7087200), Failed(0), Read Failed(0), Latency AVG(4461us), Batches Req AVG(4784us), Rows AVG(202489.00/s)
Then at seven , According to time stamp , Consume again Kafka Import incremental data from the morning to seven o'clock of the day , prevent T+1 The full amount of data covers the incremental data of the day .
50 07 15 * * python3 /mnt/code/consumer_by_time/relation_consumer_by_timestamp.py
Incremental consumption may take time 10-15min.
The real time
according to MD5 The incremental data obtained after comparison , Import Kafka in , Real time consumption Kafka The data of , Ensure that the data delay does not exceed 1 minute .
In addition, unexpected data problems may occur and not be found in real-time for a long time , So every 30 Full data will be imported once a day , It's described above Importer Import . And then to Space Point and edge add TTL=35 Ensure that the data that is not updated in time will be Filter And subsequent recycling .
Some notes
Forum post https://discuss.nebula-graph.com.cn/t/topic/361 Here is a reference to CSV Common problems in importing , You can refer to it . In addition, based on experience, here are some suggestions :
- About concurrency , It is mentioned in the question that , This concurrency Designated as your cpu cores Can , Indicates how many client Connect Nebula Server. In practice , Want to go trade off The impact of import speed and server pressure . Test on our side , If concurrency is too high , Will cause disk IO Too high , Trigger some set alarms , It is not recommended to increase concurrency , You can make a trade-off according to the actual business test .
- Importer It can't be continued at breakpoints , If something goes wrong , Need to be handled manually . In practice , We will analyze the program Importer Of log, Handle according to the situation , If any part of the data has unexpected errors , Alarm notification , Artificial intervention , Prevent accidents .
- Hive After the table is generated, it is transferred to Nebula Server, This part of the task The actual time consumption is and Hadoop Resources are closely related , There may be insufficient resources leading to Hive and CSV Table generation time is slow , and Importer Normal running , This part needs to be predicted in advance . Our side is based on hive Task end time and Importer Compare the task start time , To determine whether or not Importer The process of is running normally .
Communication graph database technology ? Join in Nebula Communication group please first Fill in your Nebula Business card ,Nebula The little assistant will pull you into the group ~~
边栏推荐
猜你喜欢
[MySQL - Basic] transactions
How to test CIS chip?
ASP. Net learning & ASP's one word
Splicing and splitting of integer ints
一. 基础概念
Opencv learning notes high dynamic range (HDR) imaging
Force buckle 2319 Judge whether the matrix is an X matrix
使用高斯Redis实现二级索引
Dachang classic pointer written test questions
School 1 of vulnhub
随机推荐
实战:sqlserver 2008 扩展事件-XML转换为标准的table格式[通俗易懂]
使用高斯Redis实现二级索引
凌云出海记 | 易点天下&华为云:推动中国电商企业品牌全球化
kubernetes之创建mysql8
Mongodb learn from simple to deep
浅尝不辄止系列之试试腾讯云的TUIRoom(晚上有约,未完待续...)
如何在软件研发阶段落地安全实践
微服务远程Debug,Nocalhost + Rainbond微服务开发第二弹
力扣674. 最长连续递增序列
rk3128投影仪lcd显示四周显示不完整解决
理财产品要怎么选?新手还什么都不懂
When easygbs cascades, how to solve the streaming failure and screen jam caused by the restart of the superior platform?
pom. XML configuration file label: differences between dependencies and dependencymanagement
Force buckle 2315 Statistical asterisk
一. 基础概念
网络原理(1)——基础原理概述
Airiot helps the urban pipe gallery project, and smart IOT guards the lifeline of the city
数据孤岛是企业数字化转型遇到的第一道险关
Kubernetes -- detailed usage of kubectl command line tool
Open source heavy ware! Chapter 9 the open source project of ylarn causal learning of Yunji datacanvas company will be released soon!