Enabling EDGE_ONLY option (#87)

* first pass?

* fix typo

* Automatically reformatting code with black and isort

* abstract duplicate code, reorganize

* Automatically reformatting code with black and isort

* readme update

* add to yaml file

* update readme again

* move helper function down and make private

* runtimeerror instead

* rework logic (more similar to previous)

* consistency

* Automatically reformatting code with black and isort

* readme change

* improve logic ordering

* Automatically reformatting code with black and isort

* mention motion detection in readme

* change edge only flag to 0/1

* small

* add note to readme

---------

Co-authored-by: Auto-format Bot <[email protected]>

README.md

@@ -49,6 +49,20 @@ print(f"The answer is {image_query.result}")
 
 See the [SDK's getting started guide](https://code.groundlight.ai/python-sdk/docs/getting-started) for more info.
 
+### Experimental: getting only edge model answers
+If you only want to receive answers from the edge model, you can set the `EDGE_ONLY` environment variable in [the edge deployment YAML file](/edge-endpoint/deploy/k3s/edge_deployment/edge_deployment.yaml) like so:
+```
+- name: EDGE_ONLY
+    value: "ENABLED"
+```
+Then, if you make requests to the edge endpoint, you will only receive answers from the edge model (regardless of the confidence). Additionally, note that no image queries submitted this way will show up in the web app or be used to train the model. This option should therefore only be used if you don't need the model to improve and only want fast answers from the edge model.
+
+If this flag is enabled and the edge inference model for a detector is not available, attempting to send image queries to that detector will return a 500 error response.
+
+This feature is currently not fully compatible with motion detection. If motion detection is enabled, some image queries may still be sent to the cloud API.
+
+This is an experimental feature and may be modified or removed in the future. `EDGE_ONLY` is disabled by default.
+
 ## Development and Internal Architecture
 
 This section describes the various components that comprise the Groundlight Edge Endpoint, and how they interoperate.

app/api/routes/image_queries.py

@@ -1,4 +1,5 @@
 import logging
+import os
 from io import BytesIO
 from typing import Optional
 
@@ -108,10 +109,12 @@ async def post_image_query(
         },
     )
 
+    edge_only = bool(int(os.environ.get("EDGE_ONLY", 0)))
+
     # TODO: instead of just forwarding want_async calls to the cloud, facilitate partial
     #       processing of the async request on the edge before escalating to the cloud.
     _want_async = want_async is not None and want_async.lower() == "true"
-    if _want_async:
+    if _want_async and not edge_only:  # If edge-only mode is enabled, we don't want to make cloud API calls
         return safe_call_api(
             gl.submit_image_query,
             detector=detector_id,
@@ -131,6 +134,7 @@ async def post_image_query(
 
     if not require_human_review and motion_detection_manager.motion_detection_is_available(detector_id=detector_id):
         motion_detected = motion_detection_manager.run_motion_detection(detector_id=detector_id, new_img=img_numpy)
+        # TODO motion detection logic will likely need to be altered to work with the EDGE_ONLY flag
         if not motion_detected:
             # Try improving the cached image query response's confidence
             # (if the cached response has low confidence)
@@ -162,12 +166,15 @@ async def post_image_query(
         results = edge_inference_manager.run_inference(detector_id=detector_id, img_numpy=img_numpy)
         confidence = results["confidence"]
 
-        if _is_confident_enough(
+        if edge_only or _is_confident_enough(
             confidence=confidence,
             detector_metadata=get_detector_metadata(detector_id=detector_id, gl=gl),
             confidence_threshold=confidence_threshold,
         ):
-            logger.info("Edge detector confidence is high enough to return")
+            if edge_only:
+                logger.info("EDGE_ONLY is enabled. The edge model's answer will be returned regardless of confidence.")
+            else:
+                logger.info("Edge detector confidence is high enough to return")
 
             if patience_time is None:
                 patience_time = constants.DEFAULT_PATIENCE_TIME  # Default patience time
@@ -202,6 +209,10 @@ async def post_image_query(
             }
         )
 
+        # Fail if edge inference is not available and edge-only mode is enabled
+        if edge_only:
+            raise RuntimeError("EDGE_ONLY is set to ENABLED, but edge inference is not available.")
+
     # Finally, fall back to submitting the image to the cloud
     if not image_query:
         logger.debug("Submitting image query to cloud API server")

deploy/README.md

@@ -36,7 +36,7 @@ a Groundlight API token in the [Groundlight web app](https://app.groundlight.ai/
 export GROUNDLIGHT_API_TOKEN="api_xxxxxx"
 
 # Choose an inference flavor, either CPU or (default) GPU.
-# Note that appropriate setup for GPU must be done separately.
+# Note that appropriate setup for GPU may need to be done separately.
 export INFERENCE_FLAVOR="CPU"
 # export INFERENCE_FLAVOR = "GPU"
 ```

deploy/k3s/edge_deployment/edge_deployment.yaml

@@ -75,6 +75,10 @@ spec:
         # TODO: Once we have kubernetes-based tests, we can remove this feature flag.
         - name: DEPLOY_DETECTOR_LEVEL_INFERENCE
           value: "1"
+        # When EDGE_ONLY is set to 1, requests to the edge endpoint will only return the edge model's answer, regardless
+        # of the confidence.
+        - name: EDGE_ONLY
+          value: "0" # disabled by default
         volumeMounts:
         - name: edge-config-volume
           mountPath: /etc/groundlight/edge-config