Improving figshare download modularity

[dhimmel]

Currently, there are three main issues we need to think about / address:

• cognoml/data.py and cognoml/figshare.py have duplicate functionality.

• We may want to make a subclass for FigshareDataSource in data.py. See cognoml refactor part 1 #3 (comment)

• We need to find a way to expose figshare download functionality to other cognoma repositories. For example, this could help with Acquires and loads gene and mutation data core-service#42

[jessept]

@dhimmel I've poked around the data retrieval sharing concept and I'm seeing a lot of code in cancer data that could be cleaned up and tweaked.

The end result: the cancer data repo SHOULD contain all data cleaning/downloading. This would allow both cognoml and core-service to use it as a submodule to enable code re-use. I can tackle this but need to understand more about the intended flow in core-service, make sure we are only pulling data from one source, and keep that source consistent through all of our projects.

[dhimmel]

The end result: the cancer data repo SHOULD contain all data cleaning/downloading.

So right now the cancer-data repo really just contains a data processing pipeline. Do you think it's the right choice to also include a package? This combined functionality didn't work out that well for machine-learning.

So do you think it makes more sense to make a separate repository for data download / upload? Currently, data download is in cognoml and data upload doesn't exist (data is manually uploaded to figshare). What would be awesome would be to have figshare upload automated. While it will most likely take more time to setup than it would save, this would be very state-of-the art.

[vasudevanv]

Should we just make a separate python interface for dealing with the figshare API residing either in cancer-data or cognoml?

[jessept]

I do think there should be a separate figshare-facing repo that both core-service and cognoml can use.

[dhimmel]

Okay I created https://github.com/cognoma/figshare and gave @jessept and @vasudevanv write access.

[vasudevanv]

Awesome. I started working on a figshare module with upload new, download and update existing article features. Can push something over the weekend.