A command-line tool to generate SBGN images for PC pathways

[jvwong]

This provides command line tools to generate a collection of image files of SBGN pathway representations, say from a list of pathways declared in a GMT file available from Pathway Commons. This would useful in providing PC pathway previews in the Search app, a PC pathway static site and/or improving the chance that pathway images are indexed by Google and picked up in user searches.

Details

• Command line programs
  • source: Download and extract a file
  • snapshot: Generate PNGs for pathways listed in a PC GMT-formatted file
• Dependencies
  • Syblars
    • Setup provided herein by Dockerfile.syblars and docker-compose.yml
• Configuration
  • SBGN_IMG_SERVICE_BASE_URL: Location of Syblars instance
  • SBGN_IMG_PATH: Where to save image files (default: public/img/pathways)

Performance

I did a test run using pathways in PathwayCommons12.All.hgnc.gmt.gz on my system (iMac 3.6 GHz 8-Core Intel Core i9) with a local Docker instance of Syblars outputting PNG files.

• Results
  • N=3971
  • Storage requirements: Directory 744 MB
  • Elapsed processing time: 73 Minutes

Notes

• Used the PC pathway URI as the filename, but had to mangle the URI to something legal for a file name (i.e. replace symbols with _). There may be a better way to handle this so there's no mapping needed, possibility of clashes.
• snapshot is configured to dump images to public/img/pathways so expressJS can serve it at img/pathways/:id, where id is the mangled pathway URI

Issues

Syblars

Getting Syblars to work locally required some fiddling, hence the choice to include the Docker.syblars. I also suspect a leak, as it would gradually hold onto RAM while processing images, crash with an out of memory error (> 4096 MB configured), then reboot clean:

Future considerations

• Incremental updates
• Rolling our own sbgn-2-image generator (see Feature: Top pathway hit #1425 (comment))
• A pathway data file, with image file name and associated metadata (genes, source, etc) which is unused currently.

Related

• Feature: Top pathway hit #1425
• Making biological network knowledge discoverable and accessible on search engines nrnb/GoogleSummerOfCode#218
• Feature: Top pathway hit #1425 (comment)

[maxkfranz]

Is Chromium/Syblars itself configured to use a maximum amount of memory? Even without a leak, browsers can be aggressive with allocating memory, because UI end-users don't like it when their tabs slow down or crash.

[jvwong]

Is Chromium/Syblars itself configured to use a maximum amount of memory? Even without a leak, browsers can be aggressive with allocating memory, because UI end-users don't like it when their tabs slow down or crash.

Not sure. I didn't customize either.
I ran this again with a Syblars container using the (mostly) original Dockerfile and the result was the same: It holds onto memory until it hits the default NodeJS limit (~2 GB) then the container exits.

[IgorRodchenkov]

We could use the md5 hash of a pathway URI (could still add the sanitized original URI as preix) for the file names @jvwong

[jvwong]

We could use the md5 hash of a pathway URI (could still add the sanitized original URI as preix) for the file names @jvwong

Right yes - during development I wanted to look up specific pathways by their filename/PC URI.