Benchmark datasets, data loaders, and evaluators for graph machine learning

I plan on submitting several label propagation baselines. However, when using label propagation (which has 0 parameters), it seems quite weird (and unfair to it) to not use the validation labels for the purposes of propagation.

There's only one hyperparameter to tune (alpha) and no parameters learned. In addition, regardless of what hyperparameters you choose (and how you choose them), it's hard for me to imagine an instance where using train + validation labels will worsen your performance at inference.

For GNNs a validation set is often needed, as you need it for stuff like early stopping as well as tuning your (many) hyperparameters.

Essentially, if say, ogbn-arxiv was a realistic setting, I can't imagine a case where you wouldn't want to use validation labels for your label propagation.

However, I can understand if OGB would like to prohibit that, given that OGB is meant as a benchmark, which by their very nature, are somewhat synthetic.

Hi, could you explain the test data for ogbl-biokg in more detail?

Specifically, we corrupt each test triplet edges by replacing its head or tail with randomly-sampled 1,000 negative entities (500 for head and 500 for tail), while ensuring the resulting triplets do not appear in KG.

I'm not sure I fully understand. Does this mean that you randomly sample, from all nodes, each head and tail (500 for each) for each test edge? If we are to predict the existence of edges, what will this information be used for?

https://github.com/snap-stanford/ogb/blob/master/examples/linkproppred/collab/gnn.py#L106

it seems that gnn model takes the whole adjacency matrix (data.adj_t).

but as far as I know, in the standard setting, gnn takes an incomplete set of edges (split_edge['train']['edge']) and predicts the rest (split_edge['valid'] and split_edge['test']).

Should I fix it? or could you please give me some reference for this setting?

I really appreciate the great commitment of you all.

HI, I try to get the mol from pcqm4m-v2-train.sdf, and compare the structure with mol from rdkit.Chem.MolFromSmiles, for example: obabel -ixyz 1.xyz -osmi -O 1.smi, we get the smiles CC(=O)N(C)/C=C/c1ccc(cc1OC)OC, but origin is COc1cc(OC)ccc1/C=C/N(C(=O)C)C then I do gnn inference with these two graphs, the final result is a little different。 it seems that 2D graph is not the same? btw sdf does not provide xyz of H like pcqm4m-v2_xyz.zip ?

The ogb package can not be imported on console now and will generate errors for the below imports:

import torch import ogb from ogb.graphproppred import GraphPropPredDataset from ogb.graphproppred import PygGraphPropPredDataset

In addition log generation is not implemented in init files. I also noticed that the lines in init that blocks proper import have not yet been used anywhere else in the program so I have a PR to address these issues.

https://github.com/Zahra-Bakhtiari/GraphPropPred_bug_fix/pull/1/files

I also did some google search and noticed that this is a big struggle for the new ogb users, several open issues exist on it :)

Hi there!

I was trying to work with the obgl-biokg and execute the script at https://github.com/snap-stanford/ogb/blob/master/examples/linkproppred/biokg/run.py .

I was getting the following error:

Exception has occurred: AttributeError
Can't pickle local object 'main.<locals>.<lambda>'
  File "/.../src/ogb_original/dataloader.py", line 121, in one_shot_iterator
    for data in dataloader:
  File "/.../src/ogb_original/dataloader.py", line 112, in __next__
    data = next(self.iterator_tail)
  File "/.../src/ogb_original/model.py", line 238, in train_step
    positive_sample, negative_sample, subsampling_weight, mode = next(train_iterator)
  File "/...3/src/ogblbiokg/linkproppred_biokg.py", line 306, in main
    log = kge_model.train_step(kge_model, optimizer, train_iterator, args)
  File "/.../src/ogblbiokg/linkproppred_biokg.py", line 371, in <module>
    main(parse_args(params))

Could you help me how to handle this issue?

Hello, OGB team, hope you are doing great! I just downloaded the example code of ogbn-proteins and ran the cluster_gin.py. I found out you didn't use node features and the node species information has been changed from previous one-hot encoding (version 3) to taxonomy ID. However, in cluster_gin.py file, cluster_data.data.x = cluster_data.data.x.to(torch.float) this statement is incorrect cause there is no attribute called x now. You can check when you set the argumentuse_node_features to be True. Another question is that it is possible for us to use one-hot encoding features provided previously? Because we have no idea about the meaning represented by taxonomy ID of each protein or does the similarity between two proteins could be expressed by the difference of their taxonomy IDs? Thank you for replying in advance and have a good one!

https://github.com/snap-stanford/ogb/blob/f5534d99703ab549ae4f7279f2002c6cc79041dc/examples/linkproppred/vessel/gnn.py#L139

I found that the predictor is not set to predictor.eval() in the test function in gnn.py, which may result in the poor performance for GNN on this dataset. If predictor.eval() is added, even with hidden size 3, the test ROC-AUC of GCN may reach 70+%, although sometimes it is stuck at 50%.

I think there is a problem with imports in ogb. I tried to run the example:

import networkx as nx
print("a")
# from sklearn.metrics import roc_auc_score, average_precision_score
from ogb.graphproppred import PygGraphPropPredDataset
print("c")
from torch_geometric.loader import DataLoader

# Download and process data at './dataset/ogbg_molhiv/'
dataset = PygGraphPropPredDataset(name="ogbg-molhiv", root='dataset/')

split_idx = dataset.get_idx_split()
train_loader = DataLoader(dataset[split_idx["train"]], batch_size=32, shuffle=False)
valid_loader = DataLoader(dataset[split_idx["valid"]], batch_size=32, shuffle=False)
test_loader = DataLoader(dataset[split_idx["test"]], batch_size=32, shuffle=False)

a gets printed, but then Python gets stuck and does not print c. However I tried to add print("b") to various points in ogb, following the from ogb.graphproppred import PygGraphPropPredDataset line.

To import PygGraphPropPredDataset, we need to run the code in __init__.py:

from .evaluate import Evaluator
from .dataset import GraphPropPredDataset

try:
    from .dataset_pyg import PygGraphPropPredDataset
except ImportError:
    pass

try:
    from .dataset_dgl import DglGraphPropPredDataset
    from .dataset_dgl import collate_dgl
except (ImportError, OSError):
    pass

Even the first line does not get printed. So I went to the evaluate.py and there we have:

from sklearn.metrics import roc_auc_score, average_precision_score
print("b")

import pandas as pd
import os
import numpy as np

try:
    import torch
except ImportError:
    torch = None

And b does not get printed. So I added from sklearn.metrics import roc_auc_score, average_precision_score directly after print("a") in the original file. After this modification it works.

I had an opportunity to try this multiple times, on fresh installs of the entire OS (Windows 2 times, Linux 1 time), with three versions of Python (3.8, 3.9, 3.10) and 3 versions of PyTorch (3.9, 3.10, 3.11). The problem was identical every time, and every time adding the manual import worked.

Fixing this would be pretty nice, since this is a quickstart on the main page.

Hi I am able to run from torch_geometric.data import InMemoryDataset successfully. But whenever I try to import anything from ogb, the program stuck for infinite time. Import code is: from ogb.graphproppred import Evaluator

I followed the following command to install torch in conda environment:

CUDA=cu113
TORCH=1.11.0
pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-sparse -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-cluster -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-spline-conv -f https://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pip install torch-geometric

Note that, I am using Ubuntu. My Nvidia-smi output:

$ nvidia-smi
Wed Apr 20 05:09:04 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.119.03   Driver Version: 450.119.03   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
| 27%   36C    P8     9W / 151W |   1227MiB /  8116MiB |      1%      Default |
|                               |                      |                  N/A

I am working on a big machine learning project in which various features of tensorflow are used. So, while using an ogb dataset for a new graph-based module, I cannot switch to torch. I thought your library-agnostic loader provides a way to incorporate tensorflow, as here one reads

We also prepare library-agnostic dataset loaders that can be used with any other deep learning libraries such as Tensorflow and MxNet.

However, I still get the following error

Traceback (most recent call last):
  File "C:/Users/Matinking/PycharmProjects/RL/GNN_spektral_OGB.py", line 11, in <module>
    from ogb.nodeproppred import NodePropPredDataset
  File "C:\Users\Matinking\AppData\Local\Programs\Python\Python37\lib\site-packages\ogb\nodeproppred\__init__.py", line 2, in <module>
    from .dataset import NodePropPredDataset
  File "~\AppData\Local\Programs\Python\Python37\lib\site-packages\ogb\nodeproppred\dataset.py", line 5, in <module>
    from ogb.io.read_graph_raw import read_csv_graph_raw, read_csv_heterograph_raw,\
  File "~\AppData\Local\Programs\Python\Python37\lib\site-packages\ogb\io\__init__.py", line 1, in <module>
    from .save_dataset import DatasetSaver
  File "~\AppData\Local\Programs\Python\Python37\lib\site-packages\ogb\io\save_dataset.py", line 1, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

when I run, for example,

mport numpy as np
import matplotlib.pyplot as plt

from tensorflow.keras.losses import CategoricalCrossentropy
from tensorflow.keras.metrics import categorical_accuracy
from tensorflow.keras.optimizers import Adam

from spektral.data import DisjointLoader
from spektral.models import GeneralGNN

from ogb.nodeproppred import NodePropPredDataset

dataset = NodePropPredDataset("ogbn-proteins")
split_idx = dataset.get_idx_split()
train_idx, valid_idx, test_idx = split_idx["train"], split_idx["valid"], split_idx["test"]

np.random.seed(0)

batch_size = 16
learning_rate = 0.0001
epochs = 100

loader_tr = DisjointLoader(train_idx, batch_size=batch_size, epochs=epochs)
loader_te = DisjointLoader(test_idx, batch_size=batch_size)

model = GeneralGNN(dataset.labels, activation="softmax")

optimizer = Adam(learning_rate)
loss_fn = CategoricalCrossentropy()
model.compile(loss=loss_fn,
              optimizer=optimizer,
              metrics=categorical_accuracy)

history = model.fit(loader_tr.load(), steps_per_epoch=loader_te.steps_per_epoch, epochs=epochs)

plt.plot(history.history['loss'])
plt.plot(history.history['categorical_accuracy'])
plt.xlabel('epoch')
plt.legend(["Loss", "Categorical Accuracy"])
plt.show()

Thus, can you please guide me on how to use ogb datasets by tensorflow, instead of torch?

Thanks, Matin

Aware, of course, that you don't yet officially support Python3.9, but thought this might be useful for someone who needs to use it:

Had this error when trying to install the module on Python3.9:

/python3.9/site-packages/torch/__init__.py", line 153, in _load_global_deps
     ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
   File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__
     self._handle = _dlopen(self._name, mode)
 OSError: libcublas.so.11: cannot open shared object file: No such file or directory

Worked around the error by updating the torch version in our model's requirements.txt to include a different cuda version (when running on Python3.9 or greater) thus:

# Workaround for OGB issue on Python3.9
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.13.0+cu116; python_version >= '3.9'

Hello devs. I am trying to develop support for OGB Datasets in MLDatasets.jl. One of the bottlenecks we are facing is loading the .pt files. This implementation here using Pickle.jl hack results in substantial memory usage compared to python. With new support for TorchArrow can you support parquet files for loading the splits?

Hi OGB-Team, right now, heterogenous graphs are loaded as a Data objects by filling x_dict, edge_index_dict etc.. HeteroData in PyG on the other hand, stores features and edges by its node-type or edge-type identifier, e.g. node-related buffers like x are exposed as data[NODE_TYPE].buffer and edge-related buffers like edge_index are exposed as data[EDGE_TYPE].buffer.Those representations are quite similar besides basically the order of keys accessing the data, e.g. data.x_dict[NODE_TYPE] vs data[NODE_TYPE].x.

Is there any particular reason for this? If not, HeteroData would seem more flexible to me in the PyG framework, as it would allow for direct use with samplers or for transformations like to_undirected or to_homogeneous.