Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

23.8.0 breaks websocket and task container #14885

Closed
6 of 11 tasks
adpavlov opened this issue Feb 15, 2024 · 1 comment
Closed
6 of 11 tasks

23.8.0 breaks websocket and task container #14885

adpavlov opened this issue Feb 15, 2024 · 1 comment
Labels
community component:api component:ui needs_triage type:bug

Comments

@adpavlov
Copy link

Please confirm the following

  • I agree to follow this project's code of conduct.
  • I have checked the current issues for duplicates.
  • I understand that AWX is open source software provided for free and that I might not receive a timely response.
  • I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

After upgrade to 23.8.0 websocket is rejecting authentication and therefore kills task container.

2024-02-15 19:20:57,043 WARNING  [-] awx.main.wsrelay Connection from awx-task-d5465549d-h8jbq to 10.42.0.104 failed for unknown reason: '403, message='Invalid response status', url=URL('http://10.42.0.104:8052/websocket/relay/')'.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/wsrelay.py", line 89, in connect
    async with session.ws_connect(uri, ssl=self.verify_ssl, heartbeat=20) as websocket:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/aiohttp/client.py", line 795, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 403, message='Invalid response status', url=URL('http://10.42.0.104:8052/websocket/relay/')
2024-02-15 19:21:07,033 INFO     [-] awx.main.wsrelay Removing {'awx-web-699868b696-ttzqs'} from websocket broadcast list
/usr/lib64/python3.9/asyncio/events.py:80: RuntimeWarning: coroutine 'WebSocketRelayManager.cleanup_offline_host' was never awaited
  self._context.run(self._callback, *self._args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 175, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/management/commands/run_wsrelay.py", line 171, in handle
    asyncio.run(websocket_relay_manager.run())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/wsrelay.py", line 342, in run
    await asyncio.gather(self.cleanup_offline_host(h) for h in deleted_remote_hosts)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/wsrelay.py", line 342, in <genexpr>
    await asyncio.gather(self.cleanup_offline_host(h) for h in deleted_remote_hosts)
RuntimeError: Task got bad yield: <coroutine object WebSocketRelayManager.cleanup_offline_host at 0x7efe7a8f3040>
2024-02-15 21:21:07,665 WARN exited: wsrelay (exit status 1; not expected)
2024-02-15 21:21:07,665 WARN exited: wsrelay (exit status 1; not expected)
2024-02-15 21:21:10,669 INFO spawned: 'wsrelay' with pid 293
2024-02-15 21:21:10,669 INFO spawned: 'wsrelay' with pid 293
2024-02-15 19:21:13,115 INFO     [-] awx.main.wsrelay Active instance with hostname awx-task-d5465549d-h8jbq is registered.
2024-02-15 19:21:23,129 INFO     [-] awx.main.wsrelay Adding {'awx-web-699868b696-ttzqs'} to websocket broadcast list
2024-02-15 19:21:23,146 WARNING  [-] awx.main.wsrelay Connection from awx-task-d5465549d-h8jbq to 10.42.0.104 failed for unknown reason: '403, message='Invalid response status', url=URL('http://10.42.0.104:8052/websocket/relay/')'.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/wsrelay.py", line 89, in connect
    async with session.ws_connect(uri, ssl=self.verify_ssl, heartbeat=20) as websocket:
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/aiohttp/client.py", line 795, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 403, message='Invalid response status', url=URL('http://10.42.0.104:8052/websocket/relay/')
2024-02-15 19:21:33,136 INFO     [-] awx.main.wsrelay Removing {'awx-web-699868b696-ttzqs'} from websocket broadcast list
/usr/lib64/python3.9/asyncio/events.py:80: RuntimeWarning: coroutine 'WebSocketRelayManager.cleanup_offline_host' was never awaited
  self._context.run(self._callback, *self._args)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 175, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/management/commands/run_wsrelay.py", line 171, in handle
    asyncio.run(websocket_relay_manager.run())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/wsrelay.py", line 342, in run
    await asyncio.gather(self.cleanup_offline_host(h) for h in deleted_remote_hosts)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/wsrelay.py", line 342, in <genexpr>
    await asyncio.gather(self.cleanup_offline_host(h) for h in deleted_remote_hosts)
RuntimeError: Task got bad yield: <coroutine object WebSocketRelayManager.cleanup_offline_host at 0x7f3dd437f040>
2024-02-15 21:21:33,771 WARN exited: wsrelay (exit status 1; not expected)
2024-02-15 21:21:33,771 WARN exited: wsrelay (exit status 1; not expected)
2024-02-15 21:21:34,773 INFO gave up: wsrelay entered FATAL state, too many start retries too quickly
2024-02-15 21:21:34,773 INFO gave up: wsrelay entered FATAL state, too many start retries too quickly
Processing Event: ver:3.0 server:supervisor serial:0 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:64
2024-02-15 21:21:35,774 WARN received SIGQUIT indicating exit request
2024-02-15 21:21:35,774 WARN received SIGQUIT indicating exit request
2024-02-15 21:21:35,775 INFO waiting for superwatcher, dispatcher, callback-receiver to die
2024-02-15 21:21:35,775 INFO waiting for superwatcher, dispatcher, callback-receiver to die
2024-02-15 19:21:35,775 WARNING  [-] awx.main.dispatch received SIGTERM, stopping
2024-02-15 19:21:35,775 WARNING  [-] awx.main.commands.run_callback_receiver received SIGTERM, stopping
2024-02-15 19:21:35,802 WARNING  [-] awx.main.tasks.system Normal shutdown signal for instance awx-task-d5465549d-h8jbq, removed self from capacity pool.
2024-02-15 21:21:36,502 INFO stopped: callback-receiver (exit status 0)
2024-02-15 21:21:36,502 INFO stopped: callback-receiver (exit status 0)
2024-02-15 21:21:36,529 INFO stopped: dispatcher (exit status 0)
2024-02-15 21:21:36,529 INFO stopped: dispatcher (exit status 0)
2024-02-15 21:21:36,530 WARN stopped: superwatcher (terminated by SIGTERM)
2024-02-15 21:21:36,530 WARN stopped: superwatcher (terminated by SIGTERM)

AWX version

23.8.0

Select the relevant components

  • UI
  • UI (tech preview)
  • API
  • Docs
  • Collection
  • CLI
  • Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Upgrade to AWX 23.8.0

Expected results

enhancements

Actual results

  • task container failing
  • websocket not working

Additional information

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community component:api component:ui needs_triage type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@adpavlov