[BUG] Missing background refresh of OIDC access_token (for /app/dashboards)

[briend]

What is the bug?

Most of our auth issues went away with #1966 in opensearch 2.15.0, however if instead of the discover (/app/data-explorer) or visualize (/app/visualize) UI, you use the dashboards UI (/app/dashboards), you should see the same refresh issue with tokens expiring. We use gitlab for OIDC which has a default 2 minutes lifetime, that is currently not easily configurable: https://gitlab.com/gitlab-org/gitlab/-/issues/377654. This short lifetime might be making the problem more obvious.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. set OIDC token expiration to 2 minutes
2. visit an opensearch dashboard (/app/dashboards) (not the discover or visualize interfaces)
3. wait about 5-10 minutes and you may be redirected to the login page to log in again via oidc when trying to interact with the page.

What is the expected behavior?
Viewing a dashboard for more than 10-20 minutes should not let the token expire.

What is your host/environment?

• OS: kubernetes/gitlab
• Version 2.15.0

Do you have any additional context?

error message:

Error: Response Error: 400 Bad Request
    at internals.Client._shortcut (/usr/share/opensearch-dashboards/plugins/securityDashboards/node_modules/@hapi/wreck/lib/index.js:569:15)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at callTokenEndpoint (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/helper.ts:88:25)
    at OpenIdAuthentication.isValidCookie (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/openid/openid_auth.ts:287:38)
    at /usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/types/authentication_type.ts:145:24
    at Object.interceptAuth [as authenticate] (/usr/share/opensearch-dashboards/src/core/server/http/lifecycle/auth.js:116:22)
    at exports.Manager.execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at module.exports.internals.Auth._authenticate (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/auth.js:273:30)
    at Request._lifecycle (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/usr/share/opensearch-dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)

'www-authenticate': 'Bearer realm="Doorkeeper", error="invalid_grant", error_description="The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."',

If you open two browser tabs/windows, one with discover (app/data-explorer) and one with dashboards, the discover window/tab will keep the token refreshed and you won't have the problem with dashboards

[cwperks]

[Triage] Thank you for filing this issue with detailed steps how to reproduce! Marking this as triaged.