Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix port already in use when running inference #2064

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

7174Andy
Copy link
Collaborator

@7174Andy 7174Andy commented Dec 18, 2024

Description

This PR solves the port bug mentioned in the Issue #2056.

The cause of the bug is that the port after checking if it is free becomes TIME_WAIT state where it remains bound after being recently closed. When binding to the TIME_STATE ports, there will be an port already in use error. To solve this issue, I added a wait for 0.1 seconds in order to wait until the port is out of TIME_WAIT state after the port is bound to check if it is free to bind because I cannot directly add the SO_REUSEADDR that binds to other addresses without conflicts to the socket options (Source).

Types of changes

  • Bugfix
  • New feature
  • Refactor / Code style update (no logical changes)
  • Build / CI changes
  • Documentation Update
  • Other (explain)

Does this address any currently open issues?

#2056

Outside contributors checklist

  • Review the guidelines for contributing to this repository
  • Read and sign the CLA and add yourself to the authors list
  • Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
  • Add tests that prove your fix is effective or that your feature works
  • Add necessary documentation (if appropriate)

Thank you for contributing to SLEAP!

❤️

Summary by CodeRabbit

  • New Features

    • Enhanced interactivity and responsiveness of the GUI for real-time updates during training.
    • Improved handling of log scale and outlier settings in the LossViewer and LossPlot.
    • Introduced a new function to find a free port for binding, enhancing port management.
  • Bug Fixes

    • Corrected y-axis scaling for log transformations to prevent issues with values close to zero.
  • Refactor

    • Streamlined logic for updating the plot based on training data and improved efficiency of the plotting process.
    • Updated method signatures for log scale and ignore outliers in the LossViewer class.

@7174Andy 7174Andy linked an issue Dec 18, 2024 that may be closed by this pull request
4 tasks
@7174Andy 7174Andy self-assigned this Dec 18, 2024
Copy link

coderabbitai bot commented Dec 18, 2024

Walkthrough

The changes in the monitor.py file focus on enhancing the LossViewer and LossPlot classes within the SLEAP GUI. Key modifications include adjustments to y-axis scaling for log scale functionality and improved outlier handling. A function for finding free ports has been introduced in the utils.py module, which replaces the previous port management logic in the LossViewer class. Additionally, methods for updating training information and rendering plots have been refined for better responsiveness and accuracy.

Changes

File Change Summary
sleap/gui/widgets/monitor.py - Removed find_free_port method from LossViewer class.
- Updated log_scale and ignore_outliers method signatures to include type hints.
- Modified LossPlot y-axis scaling logic to set minimum to 0.001 for log scale.
- Improved handling of log_scale and ignore_outliers properties in LossViewer.
- Refined _check_messages method for better training info updates.
sleap/gui/utils.py - Added find_free_port function for port management.
- Introduced sleep delay of 0.1 seconds in is_port_free function after socket operations.

Poem

🐰 In the realm of learning's plot,
Where losses dance and metrics trot,
A rabbit's code, with care so neat,
Makes graphs that make the training sweet!
Log scales and outliers now align,
With precision that makes science shine! 🚀


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@7174Andy 7174Andy changed the title add wait Fix port already in use when running inference Dec 18, 2024
Copy link

codecov bot commented Dec 18, 2024

Codecov Report

Attention: Patch coverage is 66.66667% with 4 lines in your changes missing coverage. Please review.

Project coverage is 76.14%. Comparing base (7991f14) to head (8097348).
Report is 96 commits behind head on develop.

Files with missing lines Patch % Lines
sleap/gui/utils.py 63.63% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2064      +/-   ##
===========================================
+ Coverage    75.43%   76.14%   +0.71%     
===========================================
  Files          134      134              
  Lines        24749    24783      +34     
===========================================
+ Hits         18670    18872     +202     
+ Misses        6079     5911     -168     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@7174Andy 7174Andy marked this pull request as ready for review December 18, 2024 18:19
@7174Andy 7174Andy requested a review from roomrys December 18, 2024 18:19
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
sleap/gui/widgets/monitor.py (2)

Line range hint 780-830: Refactor _setup_zmq for better separation of concerns.

The method handles multiple responsibilities: context management, port selection, socket setup, and timer initialization. Consider breaking it down into smaller, focused methods.

Suggested refactoring:

def _setup_zmq(self, zmq_context: Optional[zmq.Context] = None):
    """Connect to ZMQ ports that listen to commands and updates."""
    self._initialize_context(zmq_context)
    self._setup_subscriber()
    if self.show_controller:
        self._setup_controller()
    self._setup_message_timer()

def _initialize_context(self, zmq_context: Optional[zmq.Context] = None):
    """Initialize ZMQ context."""
    self.ctx_given = zmq_context is not None
    self.ctx = zmq.Context() if zmq_context is None else zmq_context

def _setup_subscriber(self):
    """Set up the subscriber socket."""
    self.sub = self.ctx.socket(zmq.SUB)
    self.sub.subscribe("")
    publish_port = self._find_free_port(self.zmq_ports["publish_port"])
    self.zmq_ports["publish_port"] = publish_port
    publish_address = f"tcp://127.0.0.1:{publish_port}"
    self._bind_with_retry(self.sub, publish_address)

def _setup_controller(self):
    """Set up the controller socket."""
    self.zmq_ctrl = self.ctx.socket(zmq.PUB)
    controller_port = self._find_free_port(self.zmq_ports["controller_port"])
    self.zmq_ports["controller_port"] = controller_port
    controller_address = f"tcp://127.0.0.1:{controller_port}"
    self.zmq_ctrl.bind(controller_address)

def _setup_message_timer(self):
    """Set up the message polling timer."""
    self.timer = QtCore.QTimer()
    self.timer.timeout.connect(self._check_messages)
    self.timer.start(self.message_poll_time_ms)

Line range hint 1012-1031: Enhance socket cleanup robustness.

Consider implementing the context manager protocol and adding error handling for socket cleanup operations.

Suggested implementation:

+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self._unbind()
+
     def _unbind(self):
         """Disconnect from all ZMQ sockets."""
         if self.sub is not None:
-            self.sub.unbind(self.sub.LAST_ENDPOINT)
-            self.sub.close()
+            try:
+                if self.sub.LAST_ENDPOINT:
+                    self.sub.unbind(self.sub.LAST_ENDPOINT)
+                self.sub.close()
+            except zmq.ZMQError as e:
+                logger.warning(f"Error during subscriber socket cleanup: {e}")
             self.sub = None

         if self.zmq_ctrl is not None:
-            url = self.zmq_ctrl.LAST_ENDPOINT
-            self.zmq_ctrl.unbind(url)
-            self.zmq_ctrl.close()
+            try:
+                if self.zmq_ctrl.LAST_ENDPOINT:
+                    self.zmq_ctrl.unbind(self.zmq_ctrl.LAST_ENDPOINT)
+                self.zmq_ctrl.close()
+            except zmq.ZMQError as e:
+                logger.warning(f"Error during controller socket cleanup: {e}")
             self.zmq_ctrl = None
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1eff33d and 20d0bb7.

📒 Files selected for processing (1)
  • sleap/gui/widgets/monitor.py (2 hunks)

sleap/gui/widgets/monitor.py Outdated Show resolved Hide resolved
@7174Andy 7174Andy added bug Something isn't working and removed bug Something isn't working labels Dec 18, 2024
Copy link
Collaborator

@roomrys roomrys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is a race condition issue, and it can this tie-up can last several minutes

TIME_WAIT is the state that typically ties up the port for several minutes after the process has completed.

, then an automated test that quickly makes back-to-back calls to the code we use for finding a free port should be able to consistently catch this issue.

I know that the code for find_free_port is a nested function at the moment, but perhaps it is worth breaking out the logic inside _setup_zmq to a more test-able file-level function.

@7174Andy 7174Andy marked this pull request as draft December 18, 2024 20:16
@7174Andy 7174Andy marked this pull request as ready for review December 18, 2024 21:32
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
sleap/gui/utils.py (2)

33-42: Add return type hint and improve documentation

The function's type hints and documentation could be enhanced:

  1. Add return type hint
  2. Document possible exceptions
  3. Add example usage
-def find_free_port(port: int, zmq_context: zmq.Context):
+def find_free_port(port: int, zmq_context: zmq.Context) -> int:
     """Find free port to bind to.
 
     Args:
         port: The port to start searching from.
         zmq_context: The ZMQ context to use.
 
     Returns:
         The free port.
+
+    Raises:
+        RuntimeError: If no free port is found after max_attempts.
+        ValueError: If port number is invalid.
+
+    Example:
+        >>> ctx = zmq.Context()
+        >>> free_port = find_free_port(5555, ctx)
     """

33-55: Consider implementing port reservation mechanism

The current implementation has a potential race condition: another process could take the port between the time we check it's free and when it's actually used. Consider these architectural improvements:

  1. Implement a port reservation system using file locks
  2. Keep the socket bound until actual use
  3. Use a dedicated port range for the application

This would provide more reliable port allocation in multi-process scenarios.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e57a720 and baf7612.

📒 Files selected for processing (2)
  • sleap/gui/utils.py (3 hunks)
  • sleap/gui/widgets/monitor.py (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • sleap/gui/widgets/monitor.py
🔇 Additional comments (1)
sleap/gui/utils.py (1)

4-4: LGTM: Import added correctly

The time import is necessary for the new sleep functionality and follows PEP 8 guidelines.

sleap/gui/utils.py Show resolved Hide resolved
Comment on lines +43 to +54
attempts = 0
max_attempts = 10
while not is_port_free(port=port, zmq_context=zmq_context):
if attempts >= max_attempts:
raise RuntimeError(
f"Could not find free port to display training progress after "
f"{max_attempts} attempts. Please check your network settings "
"or use the CLI `sleap-train` command."
)
port = select_zmq_port(zmq_context=zmq_context)
attempts += 1

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add input validation and improve error handling

The function should validate the port number and provide more specific error messages about TIME_WAIT states.

+    if not isinstance(port, int) or port < 1 or port > 65535:
+        raise ValueError(f"Invalid port number: {port}. Must be between 1 and 65535.")
+
     attempts = 0
     max_attempts = 10
     while not is_port_free(port=port, zmq_context=zmq_context):
         if attempts >= max_attempts:
             raise RuntimeError(
-                f"Could not find free port to display training progress after "
-                f"{max_attempts} attempts. Please check your network settings "
-                "or use the CLI `sleap-train` command."
+                f"Could not find free port after {max_attempts} attempts. "
+                "This might be due to ports in TIME_WAIT state. "
+                "Please wait a few minutes and try again, or use a different "
+                "port range. Alternatively, use the CLI `sleap-train` command."
             )
         port = select_zmq_port(zmq_context=zmq_context)
         attempts += 1
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
attempts = 0
max_attempts = 10
while not is_port_free(port=port, zmq_context=zmq_context):
if attempts >= max_attempts:
raise RuntimeError(
f"Could not find free port to display training progress after "
f"{max_attempts} attempts. Please check your network settings "
"or use the CLI `sleap-train` command."
)
port = select_zmq_port(zmq_context=zmq_context)
attempts += 1
if not isinstance(port, int) or port < 1 or port > 65535:
raise ValueError(f"Invalid port number: {port}. Must be between 1 and 65535.")
attempts = 0
max_attempts = 10
while not is_port_free(port=port, zmq_context=zmq_context):
if attempts >= max_attempts:
raise RuntimeError(
f"Could not find free port after {max_attempts} attempts. "
"This might be due to ports in TIME_WAIT state. "
"Please wait a few minutes and try again, or use a different "
"port range. Alternatively, use the CLI `sleap-train` command."
)
port = select_zmq_port(zmq_context=zmq_context)
attempts += 1

@7174Andy 7174Andy linked an issue Dec 18, 2024 that may be closed by this pull request
4 tasks
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between baf7612 and 73a015b.

📒 Files selected for processing (2)
  • sleap/gui/utils.py (3 hunks)
  • sleap/gui/widgets/monitor.py (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • sleap/gui/utils.py
🧰 Additional context used
🪛 Ruff (0.8.2)
sleap/gui/widgets/monitor.py

1093-1093: Undefined name time

(F821)


1100-1100: Undefined name time

(F821)

🔇 Additional comments (1)
sleap/gui/widgets/monitor.py (1)

1092-1094: LGTM: Socket cleanup delay addresses port reuse issue.

The added sleep delays after unbinding sockets allow proper cleanup and address the TIME_WAIT state issue mentioned in the PR objectives.

Also applies to: 1099-1101

🧰 Tools
🪛 Ruff (0.8.2)

1093-1093: Undefined name time

(F821)

@@ -1114,12 +1090,14 @@ def _unbind(self):
if self.sub is not None:
self.sub.unbind(self.sub.LAST_ENDPOINT)
self.sub.close()
time.sleep(0.1)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Add missing import for the time module.

The time module is used but not imported.

Add this import at the beginning of the file:

from time import perf_counter
+import time

Also applies to: 1100-1100

🧰 Tools
🪛 Ruff (0.8.2)

1093-1093: Undefined name time

(F821)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🔭 Outside diff range comments (1)
sleap/gui/widgets/monitor.py (1)

Line range hint 447-448: Fix inconsistency in log scale minimum value.

The comment mentions setting minimum to 0.001 but the code uses 1e-8. This could lead to numerical stability issues with very small values.

-            y_min = max(y_min, 1e-8)
+            y_min = max(y_min, 0.001)  # Minimum value for log scale to maintain numerical stability
🧹 Nitpick comments (1)
sleap/gui/widgets/monitor.py (1)

Line range hint 1052-1059: Add missing type hints and improve documentation.

Consider adding return type hints and improving parameter documentation for better code maintainability.

-    def _toggle_ignore_outliers(self):
+    def _toggle_ignore_outliers(self) -> None:
         """Toggles whether to ignore outliers in chart scaling."""
 
-    def _toggle_log_scale(self):
+    def _toggle_log_scale(self) -> None:
         """Toggle whether to use log-scaled y-axis."""
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 73a015b and c911010.

📒 Files selected for processing (2)
  • sleap/gui/utils.py (3 hunks)
  • sleap/gui/widgets/monitor.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • sleap/gui/utils.py
🔇 Additional comments (1)
sleap/gui/widgets/monitor.py (1)

Line range hint 1-1124: Overall changes look good!

The implementation effectively addresses the port usage issue while also improving the plotting functionality and code quality. The changes are well-documented and maintain good separation of concerns.

@@ -12,7 +12,7 @@
import matplotlib.transforms as mtransforms
from qtpy import QtCore, QtWidgets

from sleap.gui.utils import is_port_free, select_zmq_port
from sleap.gui.utils import find_free_port
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Consider adding retry logic for port binding.

While using find_free_port helps, there's still a potential race condition between finding a free port and binding to it. Another process could take the port in this window.

Consider implementing a retry mechanism:

 def _setup_zmq(self, zmq_context: Optional[zmq.Context] = None):
     # ... existing setup code ...
     
     # Find a free port and bind to it.
     self.zmq_ports["publish_port"] = find_free_port(
         port=self.zmq_ports["publish_port"], zmq_context=self.ctx
     )
     publish_address = f"tcp://127.0.0.1:{self.zmq_ports['publish_port']}"
-    self.sub.bind(publish_address)
+    max_retries = 3
+    for attempt in range(max_retries):
+        try:
+            self.sub.bind(publish_address)
+            break
+        except zmq.error.ZMQError as e:
+            if attempt == max_retries - 1:
+                raise RuntimeError(f"Failed to bind to {publish_address} after {max_retries} attempts: {e}")
+            self.zmq_ports["publish_port"] = find_free_port(
+                port=self.zmq_ports["publish_port"], zmq_context=self.ctx
+            )
+            publish_address = f"tcp://127.0.0.1:{self.zmq_ports['publish_port']}"

Also applies to: 1093-1100

@7174Andy 7174Andy requested a review from roomrys December 19, 2024 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ports occupied (intermittently) when running inference
2 participants