Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_cloud_events_player_example fails waiting on ksvc. #264

Closed
orfeas-k opened this issue Dec 10, 2024 · 5 comments · Fixed by #267
Closed

test_cloud_events_player_example fails waiting on ksvc. #264

orfeas-k opened this issue Dec 10, 2024 · 5 comments · Fixed by #267
Labels
bug Something isn't working

Comments

@orfeas-k
Copy link
Contributor

orfeas-k commented Dec 10, 2024 •
edited by mvlassis
Loading

Bug Description

After bumping the dependencies in #259, the aforementioned test fails waiting on the ksvc https://github.com/canonical/knative-operators/actions/runs/12257700684/job/34195774643

INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
FAILED

To Reproduce

rerun the PR's CI

Environment

juju 3.6.0, microk8s 1.29

Relevant Log Output

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-8.3.4, pluggy-1.5.0 -- /home/runner/work/knative-operators/knative-operators/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /home/runner/work/knative-operators/knative-operators
configfile: pyproject.toml
plugins: operator-0.38.0, asyncio-0.21.2, anyio-4.5.2
asyncio: mode=strict
collecting ... collected 7 items

tests/test_bundle.py::test_build_deploy_knative_charms 
-------------------------------- live log setup --------------------------------
INFO     pytest_operator.plugin:plugin.py:766 Connecting to existing model github-pr-c4238-microk8s:knative-test on unspecified cloud
-------------------------------- live log call ---------------------------------
INFO     pytest_operator.plugin:plugin.py:583 Using tmp_path: /home/runner/work/knative-operators/knative-operators/.tox/integration/tmp/pytest/knative-test0
INFO     pytest_operator.plugin:plugin.py:1139 Building charm knative-operator
INFO     pytest_operator.plugin:plugin.py:583 Using tmp_path: /home/runner/work/knative-operators/knative-operators/.tox/integration/tmp/pytest/knative-test0
INFO     pytest_operator.plugin:plugin.py:1139 Building charm knative-serving
INFO     pytest_operator.plugin:plugin.py:583 Using tmp_path: /home/runner/work/knative-operators/knative-operators/.tox/integration/tmp/pytest/knative-test0
INFO     pytest_operator.plugin:plugin.py:1139 Building charm knative-eventing
INFO     pytest_operator.plugin:plugin.py:1144 Built charm knative-serving in 1492.52s
INFO     pytest_operator.plugin:plugin.py:1144 Built charm knative-eventing in 1501.63s
INFO     pytest_operator.plugin:plugin.py:1144 Built charm knative-operator in 1562.73s
INFO     juju.model:model.py:2284 Deploying ch:amd64/focal/istio-pilot-1246
INFO     juju.model:model.py:2284 Deploying ch:amd64/focal/istio-gateway-1298
WARNING  juju.model:model.py:1666 relate is deprecated and will be removed. Use integrate instead.
INFO     juju.model:model.py:3211 Waiting for model:
  istio-pilot/0 [allocating] waiting: installing agent
  istio-ingressgateway/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:3211 Waiting for model:
  istio-pilot/0 [executing] active: 
  istio-ingressgateway/0 [executing] waiting: List of <ops.model.Relation istio-pilot:1> versions not found for apps: istio-pilot
INFO     juju.model:model.py:2284 Deploying local:knative-operator-0
INFO     juju.model:model.py:3211 Waiting for model:
  knative-operator/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:3211 Waiting for model:
  knative-operator/0 [idle] active: 
INFO     juju.model:model.py:2284 Deploying local:knative-serving-0
INFO     juju.model:model.py:2284 Deploying local:knative-eventing-0
INFO     juju.model:model.py:3211 Waiting for model:
  knative-serving/0 [idle] waiting: agent initialising
  knative-eventing/0 [allocating] waiting: installing agent
INFO     juju.model:model.py:3211 Waiting for model:
  knative-eventing/0 [idle] active: 
PASSED
tests/test_bundle.py::test_cloud_events_player_example 
-------------------------------- live log setup --------------------------------
INFO     test_bundle:test_bundle.py:146 Waiting on ksvc to exist
INFO     test_bundle:test_bundle.py:149 Checking for ksvc CRD
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/services.serving.knative.dev "HTTP/1.1 200 OK"
-------------------------------- live log call ---------------------------------
INFO     httpx:_client.py:1038 HTTP Request: POST https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services "HTTP/1.1 201 Created"
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:198 Waiting on ksvc for status to be available
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
INFO     test_bundle:test_bundle.py:191 Checking ksvc status
INFO     httpx:_client.py:1038 HTTP Request: GET https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"
INFO     test_bundle:test_bundle.py:202 Waiting on ksvc with conditions [] to be Ready
FAILED
------------------------------ live log teardown -------------------------------
INFO     httpx:_client.py:1038 HTTP Request: DELETE https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/cloudevents-player "HTTP/1.1 200 OK"

tests/test_bundle.py::test_eventing_custom_image 
-------------------------------- live log setup --------------------------------
WARNING  juju.client.connection:connection.py:490 Receiver: Connection closed, reconnecting
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-3308' coro=<Connection.reconnect() done, defined at /home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:697> exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 709, in reconnect
    res = await connector(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 800, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 743, in _connect
    result = await task
  File "/usr/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 729, in _try_endpoint
    return await self._open(endpoint, cacert)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 330, in _open
    await websockets.connect(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/client.py", line 654, in __await_impl__
    _transport, protocol = await self._create_connection()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1066, in _create_connection_transport
    sock.setblocking(False)
OSError: [Errno 9] Bad file descriptor
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
ERROR    asyncio:base_events.py:1707 Task exception was never retrieved
future: <Task finished name='Task-3311' coro=<Connection.reconnect() done, defined at /home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:697> exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 709, in reconnect
    res = await connector(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 800, in _connect_with_login
    await self._connect(endpoints)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 743, in _connect
    result = await task
  File "/usr/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 729, in _try_endpoint
    return await self._open(endpoint, cacert)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 330, in _open
    await websockets.connect(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/client.py", line 654, in __await_impl__
    _transport, protocol = await self._create_connection()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1066, in _create_connection_transport
    sock.setblocking(False)
OSError: [Errno 9] Bad file descriptor
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
ERROR
tests/test_bundle.py::test_serving_custom_image 
-------------------------------- live log setup --------------------------------
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
ERROR
tests/test_bundle.py::test_ksvc_deployment_configs 
-------------------------------- live log call ---------------------------------
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
FAILED
------------------------------ live log teardown -------------------------------
INFO     httpx:_client.py:1038 HTTP Request: DELETE https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/helloworld "HTTP/1.1 404 Not Found"

tests/test_bundle.py::test_ksvc_deployment_configs ERROR
tests/test_bundle.py::test_queue_sidecar_image_config 
-------------------------------- live log setup --------------------------------
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
ERROR
tests/test_bundle.py::test_serving_proxy_config 
-------------------------------- live log call ---------------------------------
WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed
FAILED
------------------------------ live log teardown -------------------------------
INFO     pytest_operator.plugin:plugin.py:903 Model status:

Model         Controller                Cloud/Region        Version  SLA          Timestamp
knative-test  github-pr-c4238-microk8s  microk8s/localhost  3.6.0    unsupported  14:40:19Z

App                   Version  Status       Scale  Charm             Channel       Rev  Address         Exposed  Message
istio-ingressgateway           active           1  istio-gateway     latest/edge  1298  10.152.183.85   no       
istio-pilot                    maintenance    0/1  istio-pilot       latest/edge  1246  10.152.183.155  no       
knative-eventing               active           1  knative-eventing                  0  10.152.183.153  no       
knative-operator               maintenance    0/1  knative-operator                  0  10.152.183.31   no       
knative-serving                active           1  knative-serving                   0  10.152.183.66   no       

Unit                     Workload  Agent  Address      Ports  Message
istio-ingressgateway/0*  active    idle   10.1.139.75         
istio-pilot/0            unknown   lost                       agent lost, see 'juju show-status-log istio-pilot/0'
knative-eventing/0*      active    idle   10.1.139.80         
knative-operator/0       unknown   lost                       agent lost, see 'juju show-status-log knative-operator/0'
knative-serving/0*       active    idle   10.1.139.79         

INFO     pytest_operator.plugin:plugin.py:909 Juju error logs:


INFO     pytest_operator.plugin:plugin.py:991 Forgetting model main...


==================================== ERRORS ====================================
_________________ ERROR at setup of test_eventing_custom_image _________________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 953, in transfer_data
    message = await self.read_message()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1023, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1098, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1155, in read_frame
    frame = await Frame.read(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/framing.py", line 70, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 840, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 875, in pytest_runtest_setup
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 160, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 514, in setup
    col.setup()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 697, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 532, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 617, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1091, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1140, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 898, in call_fixture_func
    fixture_result = fixturefunc(**kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 302, in _asyncgen_fixture_wrapper
    result = event_loop.run_until_complete(setup())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 284, in setup
    res = await gen_obj.__anext__()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 242, in restore_eventing_custom_image_settings
    custom_image_config = (await ops_test.model.applications["knative-eventing"].get_config())[
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/application.py", line 465, in get_config
    return (await app_facade.Get(application=self.name)).config
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client20.py", line 381, in Get
    reply = await self.rpc(msg)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 662, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 574, in rpc
    await self._ws.send(outgoing)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 628, in send
    await self.ensure_open()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 938, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
_________________ ERROR at setup of test_serving_custom_image __________________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 953, in transfer_data
    message = await self.read_message()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1023, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1098, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1155, in read_frame
    frame = await Frame.read(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/framing.py", line 70, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 840, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 875, in pytest_runtest_setup
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 160, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 514, in setup
    col.setup()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 697, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 532, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 617, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1091, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1140, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 898, in call_fixture_func
    fixture_result = fixturefunc(**kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 302, in _asyncgen_fixture_wrapper
    result = event_loop.run_until_complete(setup())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 284, in setup
    res = await gen_obj.__anext__()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 286, in restore_serving_custom_image_settings
    custom_image_config = (await ops_test.model.applications["knative-serving"].get_config())[
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/application.py", line 465, in get_config
    return (await app_facade.Get(application=self.name)).config
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client20.py", line 381, in Get
    reply = await self.rpc(msg)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 662, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 574, in rpc
    await self._ws.send(outgoing)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 628, in send
    await self.ensure_open()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 929, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
______________ ERROR at teardown of test_ksvc_deployment_configs _______________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/lightkube/core/generic_client.py", line 235, in raise_for_status
    resp.raise_for_status()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/httpx/_models.py", line 763, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.1.0.101:16443/apis/serving.knative.dev/v1/namespaces/knative-test/services/helloworld'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 174, in remove_helloworld_example
    lightkube_client.delete(KSVC, "helloworld", namespace=ops_test.model_name)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/lightkube/core/client.py", line 143, in delete
    return self._client.request(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/lightkube/core/generic_client.py", line 311, in request
    return self.handle_response(method, resp, br)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/lightkube/core/generic_client.py", line 251, in handle_response
    self.raise_for_status(resp)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/lightkube/core/generic_client.py", line 237, in raise_for_status
    raise transform_exception(e)
lightkube.core.exceptions.ApiError: services.serving.knative.dev "helloworld" not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 97, in pytest_runtest_teardown
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 100, in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 853, in pytest_runtest_teardown
    yield from self._runtest_for(item, "teardown")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 885, in pytest_runtest_teardown
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 189, in pytest_runtest_teardown
    item.session._setupstate.teardown_exact(nextitem)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 557, in teardown_exact
    raise exceptions[0]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 546, in teardown_exact
    fin()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1032, in finish
    raise exceptions[0]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1021, in finish
    fin()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 907, in _teardown_yield_fixture
    next(it)
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 177, in remove_helloworld_example
    if e.code == 404:
AttributeError: 'ApiError' object has no attribute 'code'
______________ ERROR at setup of test_queue_sidecar_image_config _______________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 953, in transfer_data
    message = await self.read_message()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1023, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1098, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1155, in read_frame
    frame = await Frame.read(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/framing.py", line 70, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_setup
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 840, in pytest_runtest_setup
    yield from self._runtest_for(item, "setup")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 875, in pytest_runtest_setup
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_setup
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 160, in pytest_runtest_setup
    item.session._setupstate.setup(item)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 514, in setup
    col.setup()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1630, in setup
    self._request._fillfixtures()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 697, in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 532, in getfixturevalue
    fixturedef = self._get_active_fixturedef(argname)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 617, in _get_active_fixturedef
    fixturedef.execute(request=subrequest)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1091, in execute
    result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 167, in _multicall
    teardown.throw(outcome._exception)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 1140, in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/fixtures.py", line 898, in call_fixture_func
    fixture_result = fixturefunc(**kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 302, in _asyncgen_fixture_wrapper
    result = event_loop.run_until_complete(setup())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 284, in setup
    res = await gen_obj.__anext__()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 399, in restore_queue_sidecar_image_setting
    await ops_test.model.applications["knative-serving"].get_config()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/application.py", line 465, in get_config
    return (await app_facade.Get(application=self.name)).config
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client20.py", line 381, in Get
    reply = await self.rpc(msg)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 662, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 574, in rpc
    await self._ws.send(outgoing)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 628, in send
    await self.ensure_open()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 929, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
=================================== FAILURES ===================================
_______________________ test_cloud_events_player_example _______________________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 880, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 217, in test_cloud_events_player_example
    wait_for_ready(resource=KSVC, name="cloudevents-player", namespace=ops_test.model_name)
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 189, in wait_for_ready
    for attempt in RETRY_FOR_THREE_MINUTES:
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/tenacity/__init__.py", line 443, in __iter__
    do = self.iter(retry_state=retry_state)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 206, in wait_for_ready
    raise timeout_error
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 206, in wait_for_ready
    raise timeout_error
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 206, in wait_for_ready
    raise timeout_error
  [Previous line repeated 33 more times]
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 199, in wait_for_ready
    raise timeout_error
TimeoutError: Timed out waiting for ksvc to be ready
_________________________ test_ksvc_deployment_configs _________________________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 953, in transfer_data
    message = await self.read_message()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1023, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1098, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1155, in read_frame
    frame = await Frame.read(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/framing.py", line 70, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 880, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
Task exception was never retrieved
future: <Task finished name='Task-3310' coro=<Connection._connect.<locals>._try_endpoint() done, defined at /home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:724> exception=OSError(9, 'Bad file descriptor')>
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 729, in _try_endpoint
    return await self._open(endpoint, cacert)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 330, in _open
    await websockets.connect(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/client.py", line 654, in __await_impl__
    _transport, protocol = await self._create_connection()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 1066, in _create_connection_transport
    sock.setblocking(False)
OSError: [Errno 9] Bad file descriptor
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 336, in test_ksvc_deployment_configs
    await ops_test.model.applications["knative-serving"].set_config(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/application.py", line 692, in set_config
    return await app_facade.SetConfigs(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client20.py", line 676, in SetConfigs
    reply = await self.rpc(msg)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 662, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 574, in rpc
    await self._ws.send(outgoing)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 628, in send
    await self.ensure_open()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 929, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
__________________________ test_serving_proxy_config ___________________________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 953, in transfer_data
    message = await self.read_message()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1023, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1098, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 1155, in read_frame
    frame = await Frame.read(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/framing.py", line 70, in read
    data = await reader(2)
  File "/usr/lib/python3.8/asyncio/streams.py", line 723, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.8/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 880, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 464, in test_serving_proxy_config
    await ops_test.model.applications["knative-serving"].set_config(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/application.py", line 692, in set_config
    return await app_facade.SetConfigs(
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client20.py", line 676, in SetConfigs
    reply = await self.rpc(msg)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 662, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 574, in rpc
    await self._ws.send(outgoing)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 628, in send
    await self.ensure_open()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/websockets/legacy/protocol.py", line 929, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
=========================== short test summary info ============================
FAILED tests/test_bundle.py::test_cloud_events_player_example - TimeoutError: Timed out waiting for ksvc to be ready
FAILED tests/test_bundle.py::test_ksvc_deployment_configs - websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
FAILED tests/test_bundle.py::test_serving_proxy_config - websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
ERROR tests/test_bundle.py::test_eventing_custom_image - websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
ERROR tests/test_bundle.py::test_serving_custom_image - websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
ERROR tests/test_bundle.py::test_ksvc_deployment_configs - AttributeError: 'ApiError' object has no attribute 'code'
ERROR tests/test_bundle.py::test_queue_sidecar_image_config - websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
============== 3 failed, 1 passed, 4 errors in 2040.62s (0:34:00) ==============
Task was destroyed but it is pending!
task: <Task pending name='Task-3301' coro=<Connection._pinger.<locals>._do_ping() running at /home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py:514> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f66568fc1c0>()]> cb=[create_task_with_handler.<locals>._task_result_exp_handler(task_name='tmp', logger=<Logger juju....ion (WARNING)>)() at /home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/jasyncio.py:101]>
Exception ignored in: <coroutine object Connection._pinger.<locals>._do_ping at 0x7f66571724c0>
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 514, in _do_ping
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 483, in wrapper
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/_client1.py", line 410, in Ping
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/facade.py", line 662, in rpc
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 589, in rpc
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/client/connection.py", line 382, in _recv
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/juju/utils.py", line 113, in get
  File "/usr/lib/python3.8/asyncio/queues.py", line 165, in get
  File "/usr/lib/python3.8/asyncio/base_events.py", line 719, in call_soon
  File "/usr/lib/python3.8/asyncio/base_events.py", line 508, in _check_closed
RuntimeError: Event loop is closed
integration: 2059615 C exit 1 (2041.50 seconds) /home/runner/work/knative-operators/knative-operators> pytest --show-capture=no --log-cli-level=INFO -vvs --tb=native --model knative-test tests/test_bundle.py pid=16220 [tox/execute/api.py:286]
  integration: FAIL code 1 (2059.24=setup[17.75]+cmd[2041.50] seconds)
  evaluation failed :( (2059.42 seconds)

The logs of the action can be found here

Additional Context

No response
@orfeas-k orfeas-k added the bug Something isn't working label Dec 10, 2024
@syncronize-issues-to-jira Syncronize issues to Jira
Copy link

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6666.

This message was autogenerated

@mvlassis mvlassis mentioned this issue Dec 17, 2024
Open
@NohaIhab
Copy link
Contributor

This is currently blocking the CI in #265 and #259 , and we need to fix it before the 1.10 release

@misohu
Copy link
Member

misohu commented Jan 8, 2025

Tested locally and on aws with success no problems the ksvc is created ... the issue occurs only in the github runner.

I have sshed into runner after the test failure. This is the situation with charms

runner@fv-az2025-713:~/work/knative-operators/knative-operators$ juju status 
Model         Controller                Cloud/Region        Version  SLA          Timestamp
knative-test  github-pr-36518-microk8s  microk8s/localhost  3.6.1    unsupported  11:00:52Z

App                   Version  Status       Scale  Charm             Channel       Rev  Address         Exposed  Message
istio-ingressgateway           active           1  istio-gateway     latest/edge  1298  10.152.183.188  no       
istio-pilot                    maintenance    0/1  istio-pilot       latest/edge  1246  10.152.183.17   no       Upgrading Istio control plane.
knative-eventing               maintenance    0/1  knative-eventing                  0  10.152.183.215  no       
knative-operator               maintenance    0/1  knative-operator                  0  10.152.183.49   no       
knative-serving                maintenance    0/1  knative-serving                   0  10.152.183.202  no       

Unit                     Workload  Agent  Address       Ports  Message
istio-ingressgateway/0*  active    idle   10.1.181.139         
istio-pilot/0            unknown   lost                        agent lost, see 'juju show-status-log istio-pilot/0'
knative-eventing/0       unknown   lost                        agent lost, see 'juju show-status-log knative-eventing/0'
knative-operator/0       unknown   lost                        agent lost, see 'juju show-status-log knative-operator/0'
knative-serving/0        unknown   lost                        agent lost, see 'juju show-status-log knative-serving/0'

Although they are in maintenance I can see all needed crds in cluster for ksvc.

runner@fv-az2025-713:~/work/knative-operators/knative-operators$ kubectl get crd | grep knative
knativeeventings.operator.knative.dev                 2025-01-08T10:40:31Z
knativeservings.operator.knative.dev                  2025-01-08T10:40:31Z
images.caching.internal.knative.dev                   2025-01-08T10:41:31Z
certificates.networking.internal.knative.dev          2025-01-08T10:41:29Z
configurations.serving.knative.dev                    2025-01-08T10:41:29Z
clusterdomainclaims.networking.internal.knative.dev   2025-01-08T10:41:29Z
domainmappings.serving.knative.dev                    2025-01-08T10:41:29Z
ingresses.networking.internal.knative.dev             2025-01-08T10:41:30Z
metrics.autoscaling.internal.knative.dev              2025-01-08T10:41:30Z
podautoscalers.autoscaling.internal.knative.dev       2025-01-08T10:41:30Z
revisions.serving.knative.dev                         2025-01-08T10:41:30Z
routes.serving.knative.dev                            2025-01-08T10:41:30Z
serverlessservices.networking.internal.knative.dev    2025-01-08T10:41:31Z
services.serving.knative.dev                          2025-01-08T10:41:31Z
apiserversources.sources.knative.dev                  2025-01-08T10:41:54Z
brokers.eventing.knative.dev                          2025-01-08T10:41:55Z
channels.messaging.knative.dev                        2025-01-08T10:41:55Z
containersources.sources.knative.dev                  2025-01-08T10:41:55Z
parallels.flows.knative.dev                           2025-01-08T10:41:55Z
sequences.flows.knative.dev                           2025-01-08T10:41:56Z
sinkbindings.sources.knative.dev                      2025-01-08T10:41:56Z
subscriptions.messaging.knative.dev                   2025-01-08T10:41:56Z
triggers.eventing.knative.dev                         2025-01-08T10:41:56Z
pingsources.sources.knative.dev                       2025-01-08T10:41:56Z
eventtypes.eventing.knative.dev                       2025-01-08T10:41:55Z
inmemorychannels.messaging.knative.dev                2025-01-08T10:42:05Z

I have tried to manually create a ksvc from the test manifests.

kubectl apply -f examples/cloudevents-player.yaml

The object is created but goes to state

   Last Transition Time:        2025-01-08T10:55:46Z
    Message:                     Configuration "cloudevents-player" does not have any ready Revision.
    Reason:                      RevisionMissing
    Status:                      False
    Type:                        RoutesReady

This is caused by pod not being scheduled on nodes

runner@fv-az2025-713:~/work/knative-operators/knative-operators$ kubectl get po -n knative-test
NAME                                                   READY   STATUS    RESTARTS   AGE
modeloperator-78c7c9b9c8-p7m54                         1/1     Running   0          45m
istio-ingressgateway-0                                 1/1     Running   0          17m
istiod-568789f896-zcxwp                                1/1     Running   0          17m
istio-ingressgateway-workload-86475c4848-c682t         1/1     Running   0          17m
knative-serving-0                                      1/1     Running   0          15m
knative-eventing-0                                     1/1     Running   0          15m
istio-pilot-0                                          0/1     Pending   0          8m21s
knative-operator-0                                     0/3     Pending   0          8m10s
cloudevents-player-00001-deployment-6759c796b5-8tj89   0/2     Pending   0          59s

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  82s   default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not 
helpful for scheduling..
  Normal   Scheduled         7s    default-scheduler  Successfully assigned knative-test/cloudevents-player-00001-deployment-6759c796b5-8tj89 to fv-az2025-713
  Normal   Pulling           7s    kubelet            Pulling image "index.docker.io/ruromero/cloudevents-player@sha256:f16639652efb372a17a08e415cace1593596d132fea8c7e89f0fe9549f4028a8"
  Normal   Pulled            1s    kubelet            Successfully pulled image "index.docker.io/ruromero/cloudevents-player@sha256:f16639652efb372a17a08e415cace1593596d132fea8c7e89f0fe9549f4028a8" in 5.343
188192s (5.3433315s including waiting)
  Normal   Created           1s    kubelet            Created container user-container
  Normal   Started           1s    kubelet            Started container user-container

This is caused by insufficient space on runner

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       76026616 74552092   1458140  99% /
devtmpfs         8183144        0   8183144   0% /dev
tmpfs            8187660        4   8187656   1% /dev/shm
tmpfs            1637532     3036   1634496   1% /run
tmpfs               5120        0      5120   0% /run/lock

After waiting 10 more minutes the pod is created

runner@fv-az2025-713:~/work/knative-operators/knative-operators$ kubectl get po -n knative-test
NAME                                                   READY   STATUS                   RESTARTS   AGE
istiod-568789f896-zcxwp                                1/1     Running                  0          26m
istio-ingressgateway-workload-86475c4848-c682t         1/1     Running                  0          26m
cloudevents-player-00001-deployment-6759c796b5-8tj89   2/2     Running                  0          10m
knative-serving-0                                      1/1     Running                  0          9m9s
knative-eventing-0                                     1/1     Running                  0          8m32s
modeloperator-78c7c9b9c8-nwqr2                         0/1     Pending                  0          2m46s
modeloperator-78c7c9b9c8-p7m54                         0/1     ContainerStatusUnknown   1          54m
istio-ingressgateway-0                                 0/1     Pending                  0          99s
istio-pilot-0                                          1/1     Running                  0          9m3s
knative-operator-0                                     0/3     Pending                  0          64s

@misohu
Copy link
Member

misohu commented Jan 8, 2025

I managed to make the test pass by using the action to maximise github space

- name: Maximise GH runner space
  uses: jlumbroso/[email protected]

Now the http proxy test is falling with

=================================== FAILURES ===================================
__________________________ test_serving_proxy_config ___________________________
Traceback (most recent call last):
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/capture.py", line 880, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/knative-operators/knative-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/knative-operators/knative-operators/tests/test_bundle.py", line 505, in test_serving_proxy_config
    assert http_proxy_env == test_http_proxy
AssertionError: assert None == 'my_http_proxy'
=========================== short test summary info ============================
FAILED tests/test_bundle.py::test_serving_proxy_config - AssertionError: assert None == 'my_http_proxy'
=================== 1 failed, 6 passed in 2167.50s (0:36:07) ===================
integration: 218[567](https://github.com/canonical/knative-operators/actions/runs/12669424106/job/35306913700?pr=259#step:5:568)1 C exit 1 (2168.30 seconds) /home/runner/work/knative-operators/knative-operators> pytest --show-capture=no --log-cli-level=INFO -vvs --tb=native --model knative-test tests/test_bundle.py pid=152790 [tox/execute/api.py:286]
  integration: FAIL code 1 (2185.42=setup[17.12]+cmd[2168.30] seconds)
  evaluation failed :( (2185.50 seconds)

After sshing into the runner the controller looks like it has the correct environment

- name: CONFIG_LOGGING_NAME
         value: config-logging
       - name: CONFIG_OBSERVABILITY_NAME
         value: config-observability
       - name: METRICS_DOMAIN
         value: knative.dev/internal/serving
       - name: HTTP_PROXY
         value: my_http_proxy
       - name: HTTPS_PROXY
         value: my_https_proxy
       - name: NO_PROXY
         value: no_proxy

@misohu
Copy link
Member

misohu commented Jan 9, 2025

Managed to resolve the http problem by adding retry for it looks like the check was faster then the controller after clearing resources. all changes are part of the PR #259

@misohu misohu mentioned this issue Jan 9, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add maximise space action + add retry for http test canonical/knative-operators
3 participants
@misohu @orfeas-k @NohaIhab