Migrate Image Training Endpoint to Django #1011

codingwithsurya · 2023-10-07T00:01:22Z

Migrate Image Training Endpoint to Django

Github Issue Number Here: Feature#993

What user problem are we solving?

With the ongoing migration from Flask Blueprint to Django for better scalability and maintainability, we need to ensure that the training endpoints for image datasets continue to function as expected. Users need a reliable endpoint to submit image training requests and this migration aims to improve the overall infrastructure.

What solution does this PR provide?

This PR introduces the /image route within the Django framework, replacing the old Flask Blueprint route. The new route handles image training requests, particularly focusing on classification tasks as regression is not supported. The structure of this endpoint is mirrored from the specified reference, with adjustments made to fit the Django framework. The old image train code served as inspiration for ensuring that the necessary features and behaviors are retained.

Testing Methodology

Testing yet to be done but to verify the functionality and integrity of the new /image route, but when testing begins, we can submit training requests using torch built-in datasets (MNIST, FashionMNIST) to ensure that models train as expected and the endpoint returns accurate results.

Also use postman for testing .

Any other considerations

training/training/core/dataset.py

training/training/routes/image/image.py

training/training/routes/image/schemas.py

training/training/routes/image/image.py

codingwithsurya · 2023-10-07T00:20:08Z

@karkir0003 @dwu359

took a stab at this. a couple questions i had was.

What should the correct path be in ImageDefaultDatasetCreator constructor (dataset.py)? Here is the code I am referring to:

self.train_set = datasets.__dict__[dataset_name](root='./backend/image_data_uploads', train=True, download=True, transform=self.train_transform)

self.test_set = datasets.__dict__[dataset_name](root='./backend/image_data_uploads', train=False, download=True, transform=self.test_transform)

With transition to django, is the path for data uploads still gonna be in backend/data_uploads?

Does my image.py look good in /routes/image? Is there any particular i have to add for image training?
Any additional schemas for image classification? I think i got the right ones.

If everything looks good here (no red flags in code), i can begin testing with our default datsets to make sure our trainspace flow is working as intended.

karkir0003 · 2023-10-07T00:23:38Z

for image data uploads, please dont use backend/ dir.

codingwithsurya · 2023-10-07T00:25:24Z

for image data uploads, please dont use backend/ dir.

sure, then what directory do i use? I dont see a similar directory in /training

karkir0003

left comments

training/training/core/dataset.py

training/training/routes/image/image.py

karkir0003

added more comments @codingwithsurya

training/training/core/dataset.py

karkir0003 · 2023-10-07T14:31:26Z

i think u can test the endpoint out @codingwithsurya

training/training/core/dataset.py

karkir0003 · 2023-10-07T18:56:42Z

for image data uploads, please dont use backend/ dir.

sure, then what directory do i use? I dont see a similar directory in /training

should be clarified

codingwithsurya · 2023-10-07T18:59:08Z

for image data uploads, please dont use backend/ dir.

sure, then what directory do i use? I dont see a similar directory in /training

should be clarified

yea looks good. i just commited what we talked about. and i'll test on colab as well soon when i get a chance.

farisdurrani · 2023-10-07T19:35:32Z

training/training/routes/image/image.py

+            )
+            for epoch_result in trainer:
+                print(epoch_result)
+            print(trainer.labels_last_epoch, trainer.y_pred_last_epoch)


Are all these print statements necessary?

saw this in the original tabular.py as well. if they're not necessary, i can remove them.

@farisdurrani @karkir0003 any clarification on what to do here. Are all the print statements necessary in grand scheme of things?

did we resolve this @codingwithsurya

training/training/core/dataset.py

training/training/routes/image/__init__.py

karkir0003

added minor comments on static typing python

training/training/routes/image/__init__.py

codingwithsurya · 2023-10-13T00:41:40Z

added minor comments on static typing python

addressed all these

codingwithsurya · 2023-10-13T02:55:49Z

TO-DO: find viable nnSequential PyTorch MNIST Architecture for testing

codingwithsurya · 2023-10-27T02:44:34Z

TODO: Cross-Entropy Loss not computing correctly. Figure out correct param for FLATTEN in current architecture.

Otherwise, we can make our own

github-actions · 2023-11-18T23:47:32Z

backend/tests/test_model.py

-        for module in my_model.model.modules()
-        if not isinstance(module, nn.Sequential)
-    ] == input_list
+import pytest


🚫 [pyright] _{reported by reviewdog 🐶}
Import "pytest" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:32Z

backend/tests/test_model.py

@@ -21,0 +1,20 @@
+import pytest
+import torch.nn as nn


🚫 [pyright] _{reported by reviewdog 🐶}
Import "torch.nn" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:33Z

backend/tests/test_model_parser.py

-)
-def test_parse_user_architecture(user_model, expected):
-    assert [i == j for i, j in zip(parse_deep_user_architecture(user_model), expected)]
+import pytest


🚫 [pyright] _{reported by reviewdog 🐶}
Import "pytest" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:33Z

backend/tests/test_model_parser.py

@@ -30,0 +1,29 @@
+import pytest
+import torch.nn as nn


🚫 [pyright] _{reported by reviewdog 🐶}
Import "torch.nn" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:33Z

training/tests/test_loss_function.py

-    loss_function_name = "BCELOSS"
-    computed_loss = compute_loss(loss_function_name, output, labels)
-    assert pytest.approx(expected_number) == computed_loss
+import pytest


🚫 [pyright] _{reported by reviewdog 🐶}
Import "pytest" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:34Z

training/training/core/dl_model.py

@@ -1,8 +1,11 @@
 from typing import Any, Callable
+from ninja import Schema


🚫 [pyright] _{reported by reviewdog 🐶}
Import "ninja" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:34Z

training/training/routes/tabular/tabular.py

-            for epoch_result in trainer:
-                print(epoch_result)
+from typing import Literal, Optional
+from django.http import HttpRequest


🚫 [pyright] _{reported by reviewdog 🐶}
Import "django.http" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:34Z

training/training/routes/tabular/tabular.py

@@ -66,0 +1,65 @@
+from typing import Literal, Optional
+from django.http import HttpRequest
+from ninja import Router, Schema


🚫 [pyright] _{reported by reviewdog 🐶}
Import "ninja" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:35Z

training/training/routes/tabular/tabular.py

+from training.core.criterion import getCriterionHandler
+from training.core.dataset import SklearnDatasetCreator
+from training.core.dl_model import DLModel
+from torch.utils.data import DataLoader


🚫 [pyright] _{reported by reviewdog 🐶}
Import "torch.utils.data" could not be resolved (reportMissingImports)

github-actions · 2023-11-18T23:47:35Z

training/training/routes/tabular/tabular.py

+            drop_last=True,
+        )
+
+        model = DLModel.fromLayerParamsList(tabularParams.user_arch)


🚫 [pyright] _{reported by reviewdog 🐶}
Argument of type "list[LayerParams]" cannot be assigned to parameter "layer_params_list" of type "list[LayerParams]" in function "fromLayerParamsList"
  "builtins.list" is incompatible with "builtins.list"
    Type parameter "_T@list" is invariant, but "training.training.routes.tabular.schemas.LayerParams" is not the same as "training.training.routes.image.schemas.LayerParams"
    Consider switching from "list" to "Sequence" which is covariant (reportGeneralTypeIssues)

codingwithsurya · 2023-11-19T00:51:36Z

Update: MNIST dataset testing works!

TODO: Train/test transforms as parameter to endpoint. CIFAR10 Testing.

sonarcloud · 2023-11-19T22:11:31Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
6 Code Smells

No Coverage information
0.0% Duplication

adding image training endpoint

c3d8938

codingwithsurya added the backend backend tasks label Oct 7, 2023

codingwithsurya self-assigned this Oct 7, 2023

github-actions bot reviewed Oct 7, 2023

View reviewed changes

codingwithsurya linked an issue Oct 7, 2023 that may be closed by this pull request

[FEATURE]: Migrate Image Training Endpoint #993

Closed

🎨 Format Python code with psf/black

91f56bf

github-actions bot reviewed Oct 7, 2023

View reviewed changes

training/training/routes/image/image.py Show resolved Hide resolved

karkir0003 reviewed Oct 7, 2023

View reviewed changes

codingwithsurya added 3 commits October 6, 2023 20:38

importing image for dl_model.py

d7ce98d

Merge remote-tracking branch 'origin/feature-993' into feature-993

2edb68d

fixing return typing and typos

cd8f850

karkir0003 reviewed Oct 7, 2023

View reviewed changes

training/training/core/dataset.py Outdated Show resolved Hide resolved

optimizing imports and adding cifar dataset

a5d0c0b

github-actions bot reviewed Oct 7, 2023

View reviewed changes

training/training/core/dataset.py Show resolved Hide resolved

deleting local downloaded data

9e03ea0

farisdurrani reviewed Oct 7, 2023

View reviewed changes

karkir0003 reviewed Oct 8, 2023

View reviewed changes

training/training/core/dataset.py Show resolved Hide resolved

karkir0003 reviewed Oct 8, 2023

View reviewed changes

training/training/core/dataset.py Show resolved Hide resolved

karkir0003 reviewed Oct 8, 2023

View reviewed changes

training/training/routes/image/__init__.py Outdated Show resolved Hide resolved

karkir0003 reviewed Oct 8, 2023

View reviewed changes

adding return typing

b3e82b9

codingwithsurya force-pushed the feature-993 branch from 7f9ab66 to b3e82b9 Compare October 13, 2023 00:22

github-actions bot reviewed Oct 13, 2023

View reviewed changes

training/training/routes/image/__init__.py Outdated Show resolved Hide resolved

codingwithsurya and others added 5 commits October 13, 2023 00:23

🎨 Format Python code with psf/black

f9f1edb

fixing comment

ce4351f

pulling remote branch

ac4cc8e

adding dummy keyword arg

64645d6

🎨 Format Python code with psf/black

5185cda

DSGT-DLP deleted a comment from karkir0003 Oct 13, 2023

codingwithsurya added 3 commits October 12, 2023 20:53

added logging

30f5dd2

added logging

ce571e4

testing endpoint

5cbbaf0

codingwithsurya force-pushed the feature-993 branch from 9ab1b12 to 5cbbaf0 Compare October 13, 2023 02:55

adding maxpool2d and flatten

6a35554

codingwithsurya force-pushed the feature-993 branch from 5fadf7f to 6a35554 Compare October 27, 2023 01:45

updating criterion

2922549

github-actions bot reviewed Nov 18, 2023

View reviewed changes

removing print statements

dbc97d1

codingwithsurya force-pushed the feature-993 branch from 07a70f3 to dbc97d1 Compare November 19, 2023 00:46

fixing cifar typo

5308884

codingwithsurya marked this pull request as ready for review November 19, 2023 20:48

codingwithsurya requested a review from a team as a code owner November 19, 2023 20:48

karkir0003 force-pushed the feature-993 branch from eab06b8 to 07a70f3 Compare November 19, 2023 21:35

removed unused code

3af1d65

codingwithsurya force-pushed the feature-993 branch from 36501e3 to 3af1d65 Compare November 19, 2023 22:00

solving nits

f2d6f5b

codingwithsurya closed this Nov 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate Image Training Endpoint to Django #1011

Migrate Image Training Endpoint to Django #1011

codingwithsurya commented Oct 7, 2023 •

edited

Loading

codingwithsurya commented Oct 7, 2023 •

edited

Loading

karkir0003 commented Oct 7, 2023

codingwithsurya commented Oct 7, 2023

karkir0003 left a comment

karkir0003 left a comment

karkir0003 commented Oct 7, 2023

karkir0003 commented Oct 7, 2023

codingwithsurya commented Oct 7, 2023

farisdurrani Oct 7, 2023

codingwithsurya Oct 7, 2023

codingwithsurya Oct 13, 2023

karkir0003 Nov 19, 2023

karkir0003 left a comment

codingwithsurya commented Oct 13, 2023

codingwithsurya commented Oct 13, 2023 •

edited

Loading

codingwithsurya commented Oct 27, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

github-actions bot Nov 18, 2023

codingwithsurya commented Nov 19, 2023 •

edited

Loading

sonarcloud bot commented Nov 19, 2023

		@@ -1,8 +1,11 @@
		from typing import Any, Callable
		from ninja import Schema

Migrate Image Training Endpoint to Django #1011

Migrate Image Training Endpoint to Django #1011

Conversation

codingwithsurya commented Oct 7, 2023 • edited Loading

Migrate Image Training Endpoint to Django

codingwithsurya commented Oct 7, 2023 • edited Loading

karkir0003 commented Oct 7, 2023

codingwithsurya commented Oct 7, 2023

karkir0003 left a comment

Choose a reason for hiding this comment

karkir0003 left a comment

Choose a reason for hiding this comment

karkir0003 commented Oct 7, 2023

karkir0003 commented Oct 7, 2023

codingwithsurya commented Oct 7, 2023

farisdurrani Oct 7, 2023

Choose a reason for hiding this comment

codingwithsurya Oct 7, 2023

Choose a reason for hiding this comment

codingwithsurya Oct 13, 2023

Choose a reason for hiding this comment

karkir0003 Nov 19, 2023

Choose a reason for hiding this comment

karkir0003 left a comment

Choose a reason for hiding this comment

codingwithsurya commented Oct 13, 2023

codingwithsurya commented Oct 13, 2023 • edited Loading

codingwithsurya commented Oct 27, 2023

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

github-actions bot Nov 18, 2023

Choose a reason for hiding this comment

codingwithsurya commented Nov 19, 2023 • edited Loading

sonarcloud bot commented Nov 19, 2023

codingwithsurya commented Oct 7, 2023 •

edited

Loading

codingwithsurya commented Oct 7, 2023 •

edited

Loading

codingwithsurya commented Oct 13, 2023 •

edited

Loading

codingwithsurya commented Nov 19, 2023 •

edited

Loading