[DEVX:123] Support Dataset Upload

[sainivedh]

Dataset Upload Functionality in SDK

What

This enhancement introduces dataset upload functionality to the Clarifai SDK, leveraging the existing Python utility data upload approach.

How

To utilize this new functionality, users can follow these steps:

from clarifai.client.app import App

app = App(app_id="", user_id="")
# List all datasets
all_datasets = app.list_datasets()
# Create a dataset in the Clarifai App
dataset = app.create_dataset(dataset_id="")
# Execute data upload to the Clarifai App dataset
dataset.upload_dataset(task='visual_segmentation', split="train", zoo_dataset='coco_segmentation')

Improvements

• Dataloader is changed to run time proto extraction instead of in-memory (more context)
• Added a base ClarifaiDataLoader class

Tests

• Manual data upload test for

  1. COCO det, seg
  2. xView det
  3. Cifar10, Food101 Image cls
  4. IMDB reviews text cls

[sanjaychelliah]

LGTM! Just minor comments..

[stmugisha]

Looking good.

Is the upload sub directory under datasets necessary? (clarifai.datasets.upload).
If not , I think we can reduce the nesting and move the files to under datasets.

[ackizilkale]

LGTM as long as the outstanding conversations are resolved.

[zeiler]

only the one comment on changing this notion of zoo to more of what it seems that folder is which is just different types of loaders of datasets. I didn't do a thorough code review as there are many eyes on it overall.

[stmugisha]

LGTM 👍