Rapidly develop GraphScope graph analysis applications

我们开发了一个 C++ 模板库,Users only need to replace a few functions with custom logic,You can customize the graph analysis algorithm and run it GraphScope 上.

Graphs are an important data source for big data,Various analytical techniques surrounding graphs also have wide-ranging applications.The core of graph computing lies in algorithms,如 PageRank、K-Core Algorithms such as these have shown powerful capabilities in node importance analysis scenarios.GraphScope The graph analysis engine of GRAPE 系统的开源版本,A variety of graph analysis algorithms are built in,can be found in the path、Community discovery and other fields provide one-stop graph analysis solutions;然而,Users often need to customize some algorithms according to their own needs,因此,In order to facilitate users to develop custom graph analysis algorithms,我们提供了一个 C++ 模版库,The next part of this article will introduce how to use the template library.

Step1. 安装 GraphScope

首先,We need to install it in the local environment with the following command GraphScope:

pip3 install graphscope 

Step2. Develop algorithms using template libraries

使用浏览器打开template library GitHub 地址,Click to the right of the library Use this template button to create a code repository of your own,And clone the repository locally with the following command：

# Make sure to replace the <username> and <repo-name> to the right values.
git clone https://github.com/<username>/<repo-name>.git 
cd <repo-name>

接下来,我们通过修改 src 目录下的文件,Implement custom algorithm logic,其中:

• my_app.h: This file is used to implement the main logic of the algorithm,即 PEval 和 IncEval 函数;
• my_app_context.h: 该文件用于存储、Define the results of the algorithm when it runs and the data structures used.

Defines the data structures that the algorithm needs to run

To simplify the algorithm logic,Highlight the development process of the algorithm,在本例中,Our algorithm will take care of thatThe sum of the degrees of each node's neighbors in the graph.在这一步中,我们需要在 my_app_context.h 文件中：1) Defines the data structure of the algorithm runtime;2) 在 Init function to initialize these data structures.

The algorithm runtime data structure is as follows:

变量名	类型	作用
result	vertex_array_t<size_t>	Store the calculation result of each node
degree	vertex_array_t<size_t>	辅助数据结构,Stores the degree of each node

最终,my_app_context.h 实现如下:

Implement algorithm logic

After the data structure is defined,我们可在 my_app.h The corresponding algorithm logic is implemented in,即 PEval 和 IncEval 函数.关于 PEval 和 IncEval 函数的定义,可参考 PIE Introduction to the programming model.在这个例子中,

• PEval 函数: Get the degree of each internal node,and send messages along the outgoing edge;
• IncEval 函数: 1) 负责接收、Process messages from the previous round;2) Count the sum of the degrees of each node's neighbor nodes.

最终,完整的 my_app.h 实现如下:

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-8WsI3SVy-1658738305019)(https://files.mdnice.com/user/24781/653b7ec0-fca7-46d0-b9f9-3614d52b8f71.png)]

Step3. 构建并测试

Build and test your algorithm locally

使用如下命令,Compile the build algorithm：

make build & cd build
cmake .. && make    # compilation algorithm
make package        # Package the algorithm as available GraphScope resources running on it (.gar文件)

在 GraphScope Run the algorithm on the cluster

算法开发完成后,We can then assign the corresponding algorithm resources (gar文件) 运行在 GraphScope 集群上,具体步骤如下:

import graphscope
graphscope.set_option(show_log=True)

from graphscope.framework.app import load_app
from graphscope.dataset import load_p2p_network

sess = graphscope.session(cluster_type="hosts")
simple_graph = load_p2p_network(sess)._project_to_simple()

my_app = load_app('<path_to_your_gar_resource>')
result = my_app(simple_graph)
print(result.to_numpy("r"))

结语

本文详细介绍了如何基于 C++ The template library can quickly develop a template library GraphScope Graph analytics algorithms running on the cluster,In the future, we will also share more algorithm cases developed based on this template library in real scenarios,敬请期待！点击链接The source code of the template library can be obtained.同时也欢迎关注 GraphScope 公众号及GitHub,及时了解图计算相关的知识及 GraphScope 的研发进展.