Batching Graphs in PygPCQM4MDataset

[edwardelson]

Hi thanks for preparing the processing code!

Was just thinking if batching the smiles graph into separate torch files would be a feasible solution to reduce memory requirement? I notice in the process() function of class PygPCQM4MDataset(InMemoryDataset), the list of graphs obtained from the smiles strings are all combined into a single dataset, and subsequently torch.save'd into one file (only to be split again later on to different dataloaders? during training and testing)

Since all of the graphs are independent of each other, would it be possible to perhaps save these into a couple of torch files, each made of batches of several graphs data to reduce RAM requirement?

Thanks!

[weihua916]

Hi! Yes, we thought about that option, but we think 8GB should be manageable for most of the RAM on the server.

[weihua916]

If RAM is an issue, you can use the smiles dataset and transform smiles strings into graph objects on the fly (using transform function in pytorch dataloader). It should give you descent speed if you set num_workers to be large enough.

from ogb.lsc import PCQM4MDataset
dataset = PCQM4MDataset(root = ROOT, only_smiles = True)

[edwardelson]

brilliant thanks!