Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock UI improvements #25130

Open
zmb3 opened this issue Apr 24, 2023 · 2 comments
Open

Lock UI improvements #25130

zmb3 opened this issue Apr 24, 2023 · 2 comments
Labels
feature-request Used for new features in Teleport, improvements to current should be #enhancements igs ui

Comments

@zmb3
Copy link
Collaborator

zmb3 commented Apr 24, 2023

A few suggestions for improving locks, now that they're more usable in the UI:

  1. Display the node name instead of UUID.

image

The node name is not part of the lock resource, so we'd have to make a separate call for this, similar to what we do for access requests.

  1. Improve the display of the session.rejected event in the audit log. Large amounts of text inside [ ] is hard to read. (We should probably remove all traces of these square brackets and find another way to emphasize certain text.

image

  1. Make it clear that the session was rejected due to a lock. The web UI just says "disconnected" and you need to check the audit log or Teleport logs to see what went wrong.

image

Device trust is able to show nice errors when sessions are rejected due to untrusted devices, see if we can do the same thing here.

@zmb3 zmb3 added the feature-request Used for new features in Teleport, improvements to current should be #enhancements label Apr 24, 2023
@zmb3
Copy link
Collaborator Author

zmb3 commented May 1, 2023

Item 3 may be fixed by #25342.

Edit, the PR above does not fix it. It looks like the frontend tries to upgrade to websocket immediately, but the backend fails with an error before completing the websocket handshake, so the UI just observes a disconnect.

@zmb3 zmb3 added the ui label May 10, 2023
@zmb3 zmb3 added the igs label Jan 17, 2025
@zmb3
Copy link
Collaborator Author

zmb3 commented Jan 17, 2025

Did a little more digging into this one and it looks like the UX for locks could use a lot of improvements and not just UI updates.

In addition to the generic "disconnected" error that you see when an existing SSH session is terminated due to lock, we have a number of other issues.

Here's a snippet of my logs (auth, proxy, node all in one process).

2025-01-14T22:50:28.345-07:00 INFO  emitting audit event event_type:lock.created fields:map[cluster_name:zac code:TLK00I ei:0 event:lock.created expires:2025-01-15T06:50:28.33695Z lock:map[target:map[server_id:b507c22c-685e-45bd-a796-7a5bcd63e3e6]] name:a84dcded-b0fa-46ed-a9cb-03266240d648 target:map[server_id:b507c22c-685e-45bd-a796-7a5bcd63e3e6] time:2025-01-15T05:50:28.345Z trace.component:audit uid:1778868e-78be-4029-9e1b-9da7ad68d91a updated_by:zac user:zac user_kind:1] events/emitter.go:287
2025-01-14T22:50:29.051-07:00 INFO [SESSION:N] Closing party party_id:ad280382-5ff1-4f44-b224-27a72e42fcce srv/sess.go:2242
2025-01-14T22:50:29.051-07:00 INFO [SESSION:N] Removing party from session. session_id:7ddda71b-b7aa-4426-9c5c-8d101cdf4a27 party.id:2facbd26-ff62-4916-9251-cd77782dc342 party.remote_addr:127.0.0.1:52165 srv/sess.go:1634
2025-01-14T22:50:29.051-07:00 ERRO [NODE]      Failed to close connection local_addr:127.0.0.1:3080 remote_addr:127.0.0.1:52165 login:zmb teleport_user:zac id:1 error:"[
ERROR REPORT:
Original Error: *net.OpError close tcp 127.0.0.1:3022->127.0.0.1:52208: use of closed network connection
Stack Trace:
   github.com/gravitational/teleport/lib/limiter/listener.go:99 github.com/gravitational/teleport/lib/limiter.(*wrappedConn).Close
   github.com/gravitational/teleport/lib/utils/timeout.go:77 github.com/gravitational/teleport/lib/utils.(*timeoutConn).Close
   golang.org/x/[email protected]/ssh/connection.go:97 golang.org/x/crypto/ssh.(*connection).Close
   github.com/gravitational/teleport/lib/srv/monitor.go:435 github.com/gravitational/teleport/lib/srv.(*Monitor).disconnectClient
   github.com/gravitational/teleport/lib/srv/monitor.go:476 github.com/gravitational/teleport/lib/srv.(*Monitor).handleLockInForce
   github.com/gravitational/teleport/lib/srv/monitor.go:400 github.com/gravitational/teleport/lib/srv.(*Monitor).start
   github.com/gravitational/teleport/lib/srv/monitor.go:299 github.com/gravitational/teleport/lib/srv.StartMonitor.func1
   runtime/asm_arm64.s:1223 runtime.goexit
User Message: close tcp 127.0.0.1:3022->127.0.0.1:52208: use of closed network connection]" srv/monitor.go:436

2025-01-14T22:50:29.052-07:00 WARN [AUTH:1] access denied pid:58164.1 error:"[
ERROR REPORT:
Original Error: *trace.AccessDeniedError lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force
Fields:
lock-in-force: Kind:"lock" Version:"v2" Metadata:<Name:"a84dcded-b0fa-46ed-a9cb-03266240d648" Namespace:"default" Revision:"292b152f-fee6-4455-88d7-715a362fc1ee" > Spec:<Target:<ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" > Expires:<seconds:1736923828 nanos:336950000 > CreatedAt:<seconds:1736920228 nanos:341867000 > CreatedBy:"zac" >
Stack Trace:
github.com/gravitational/teleport/lib/services/lock.go:39 github.com/gravitational/teleport/lib/services.LockInForceAccessDenied
github.com/gravitational/teleport/lib/services/watcher.go:998 github.com/gravitational/teleport/lib/services.(*lockCollector).CheckLockInForce
github.com/gravitational/teleport/lib/authz/permissions.go:433 github.com/gravitational/teleport/lib/authz.(*authorizer).Authorize
github.com/gravitational/teleport/lib/auth/grpcserver.go:5565 github.com/gravitational/teleport/lib/auth.(*GRPCServer).authenticate
github.com/gravitational/teleport/lib/auth/grpcserver.go:4368 github.com/gravitational/teleport/lib/auth.(*GRPCServer).UpdateSessionTracker
github.com/gravitational/teleport/[email protected]/client/proto/authservice.pb.go:22402 github.com/gravitational/teleport/api/client/proto._AuthService_UpdateSessionTracker_Handler.func1
github.com/gravitational/teleport/lib/auth/middleware.go:567 github.com/gravitational/teleport/lib/auth.(*Middleware).withAuthenticatedUserUnaryInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/lib/limiter/limiter.go:152 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*Limiter).UnaryServerInterceptorWithCustomRate.func1
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/[email protected]/metadata/metadata.go:76 github.com/gravitational/teleport/api/metadata.UnaryServerInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/[email protected]/utils/grpc/interceptors/errors.go:76 github.com/gravitational/teleport/api/utils/grpc/interceptors.GRPCServerUnaryErrorInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func2
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:316 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
google.golang.org/[email protected]/server.go:1202 google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
github.com/gravitational/teleport/[email protected]/client/proto/authservice.pb.go:22404 github.com/gravitational/teleport/api/client/proto._AuthService_UpdateSessionTracker_Handler
google.golang.org/[email protected]/server.go:1394 google.golang.org/grpc.(*Server).processUnaryRPC
google.golang.org/[email protected]/server.go:1805 google.golang.org/grpc.(*Server).handleStream
google.golang.org/[email protected]/server.go:1029 google.golang.org/grpc.(*Server).serveStreams.func2.1
runtime/asm_arm64.s:1223 runtime.goexit
User Message: lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force]" authz/permissions.go:617
2025-01-14T22:50:29.051-07:00 INFO emitting audit event event_type:client.disconnect fields:map[addr.local:127.0.0.1:3080 addr.remote:127.0.0.1:52165 code:T3006I ei:0 event:client.disconnect login:zmb reason:lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force server_id:b507c22c-685e-45bd-a796-7a5bcd63e3e6 server_version:18.0.0-dev time:2025-01-15T05:50:29.051Z trace.component:audit uid:df9449f7-e7a9-4763-a537-bbaf86b884fc user:zac] events/emitter.go:287
2025-01-14T22:50:29.053-07:00 WARN [AUTH:1] access denied pid:58164.1 error:"[
ERROR REPORT:
Original Error: *trace.AccessDeniedError lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force
Fields:
lock-in-force: Kind:"lock" Version:"v2" Metadata:<Name:"a84dcded-b0fa-46ed-a9cb-03266240d648" Namespace:"default" Revision:"292b152f-fee6-4455-88d7-715a362fc1ee" > Spec:<Target:<ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" > Expires:<seconds:1736923828 nanos:336950000 > CreatedAt:<seconds:1736920228 nanos:341867000 > CreatedBy:"zac" >
Stack Trace:
github.com/gravitational/teleport/lib/services/lock.go:39 github.com/gravitational/teleport/lib/services.LockInForceAccessDenied
github.com/gravitational/teleport/lib/services/watcher.go:998 github.com/gravitational/teleport/lib/services.(*lockCollector).CheckLockInForce
github.com/gravitational/teleport/lib/authz/permissions.go:433 github.com/gravitational/teleport/lib/authz.(*authorizer).Authorize
github.com/gravitational/teleport/lib/auth/grpcserver.go:5565 github.com/gravitational/teleport/lib/auth.(*GRPCServer).authenticate
github.com/gravitational/teleport/lib/auth/grpcserver.go:224 github.com/gravitational/teleport/lib/auth.(*GRPCServer).EmitAuditEvent
github.com/gravitational/teleport/[email protected]/client/proto/authservice.pb.go:23114 github.com/gravitational/teleport/api/client/proto._AuthService_EmitAuditEvent_Handler.func1
github.com/gravitational/teleport/lib/auth/middleware.go:567 github.com/gravitational/teleport/lib/auth.(*Middleware).withAuthenticatedUserUnaryInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/lib/limiter/limiter.go:152 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*Limiter).UnaryServerInterceptorWithCustomRate.func1
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/[email protected]/metadata/metadata.go:76 github.com/gravitational/teleport/api/metadata.UnaryServerInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/[email protected]/utils/grpc/interceptors/errors.go:76 github.com/gravitational/teleport/api/utils/grpc/interceptors.GRPCServerUnaryErrorInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func2
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:316 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
google.golang.org/[email protected]/server.go:1202 google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
github.com/gravitational/teleport/[email protected]/client/proto/authservice.pb.go:23116 github.com/gravitational/teleport/api/client/proto._AuthService_EmitAuditEvent_Handler
google.golang.org/[email protected]/server.go:1394 google.golang.org/grpc.(*Server).processUnaryRPC
google.golang.org/[email protected]/server.go:1805 google.golang.org/grpc.(*Server).handleStream
google.golang.org/[email protected]/server.go:1029 google.golang.org/grpc.(*Server).serveStreams.func2.1
runtime/asm_arm64.s:1223 runtime.goexit
User Message: lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force]" authz/permissions.go:617
2025-01-14T22:50:29.054-07:00 ERRO Failed to emit audit event of type event_type:client.disconnect error:"[
ERROR REPORT:
Original Error: trace.aggregate access denied
Stack Trace:
github.com/gravitational/teleport/lib/events/emitter.go:314 github.com/gravitational/teleport/lib/events.(*MultiEmitter).EmitAuditEvent
github.com/gravitational/teleport/lib/events/emitter.go:178 github.com/gravitational/teleport/lib/events.(*CheckingEmitter).EmitAuditEvent
github.com/gravitational/teleport/lib/events/emitter.go:99 github.com/gravitational/teleport/lib/events.(*AsyncEmitter).forward
runtime/asm_arm64.s:1223 runtime.goexit
User Message: access denied]" events/emitter.go:180
2025-01-14T22:50:29.054-07:00 ERRO Failed to emit audit event. error:"[
ERROR REPORT:
Original Error: trace.aggregate access denied
Stack Trace:
github.com/gravitational/teleport/lib/events/emitter.go:314 github.com/gravitational/teleport/lib/events.(*MultiEmitter).EmitAuditEvent
github.com/gravitational/teleport/lib/events/emitter.go:178 github.com/gravitational/teleport/lib/events.(*CheckingEmitter).EmitAuditEvent
github.com/gravitational/teleport/lib/events/emitter.go:99 github.com/gravitational/teleport/lib/events.(*AsyncEmitter).forward
runtime/asm_arm64.s:1223 runtime.goexit
User Message: access denied]" events/emitter.go:104
2025-01-14T22:50:29.054-07:00 INFO emitting audit event event_type:client.disconnect fields:map[addr.local:127.0.0.1:3080 addr.remote:127.0.0.1:52165 code:T3006I ei:0 event:client.disconnect login:zmb reason:lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force server_id:b507c22c-685e-45bd-a796-7a5bcd63e3e6 server_version:18.0.0-dev time:2025-01-15T05:50:29.055Z trace.component:audit uid:a9689ed2-f900-44dc-a985-a1c3096b3b3a user:zac] events/emitter.go:287
2025-01-14T22:50:29.060-07:00 WARN [AUTH:1] access denied pid:58164.1 error:"[
ERROR REPORT:
Original Error: *trace.AccessDeniedError lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force
Fields:
lock-in-force: Kind:"lock" Version:"v2" Metadata:<Name:"a84dcded-b0fa-46ed-a9cb-03266240d648" Namespace:"default" Revision:"292b152f-fee6-4455-88d7-715a362fc1ee" > Spec:<Target:<ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" > Expires:<seconds:1736923828 nanos:336950000 > CreatedAt:<seconds:1736920228 nanos:341867000 > CreatedBy:"zac" >
Stack Trace:
github.com/gravitational/teleport/lib/services/lock.go:39 github.com/gravitational/teleport/lib/services.LockInForceAccessDenied
github.com/gravitational/teleport/lib/services/watcher.go:998 github.com/gravitational/teleport/lib/services.(*lockCollector).CheckLockInForce
github.com/gravitational/teleport/lib/authz/permissions.go:433 github.com/gravitational/teleport/lib/authz.(*authorizer).Authorize
github.com/gravitational/teleport/lib/auth/grpcserver.go:5565 github.com/gravitational/teleport/lib/auth.(*GRPCServer).authenticate
github.com/gravitational/teleport/lib/auth/grpcserver.go:224 github.com/gravitational/teleport/lib/auth.(*GRPCServer).EmitAuditEvent
github.com/gravitational/teleport/[email protected]/client/proto/authservice.pb.go:23114 github.com/gravitational/teleport/api/client/proto._AuthService_EmitAuditEvent_Handler.func1
github.com/gravitational/teleport/lib/auth/middleware.go:567 github.com/gravitational/teleport/lib/auth.(*Middleware).withAuthenticatedUserUnaryInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/lib/limiter/limiter.go:152 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*Limiter).UnaryServerInterceptorWithCustomRate.func1
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/[email protected]/metadata/metadata.go:76 github.com/gravitational/teleport/api/metadata.UnaryServerInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/gravitational/teleport/[email protected]/utils/grpc/interceptors/errors.go:76 github.com/gravitational/teleport/api/utils/grpc/interceptors.GRPCServerUnaryErrorInterceptor
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/server.go:22 github.com/gravitational/teleport/lib/auth.(*Middleware).UnaryInterceptors.(*ServerMetrics).UnaryServerInterceptor.UnaryServerInterceptor.func2
google.golang.org/[email protected]/server.go:1211 google.golang.org/grpc.getChainUnaryHandler.func1
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:316 go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
google.golang.org/[email protected]/server.go:1202 google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
github.com/gravitational/teleport/[email protected]/client/proto/authservice.pb.go:23116 github.com/gravitational/teleport/api/client/proto._AuthService_EmitAuditEvent_Handler
google.golang.org/[email protected]/server.go:1394 google.golang.org/grpc.(*Server).processUnaryRPC
google.golang.org/[email protected]/server.go:1805 google.golang.org/grpc.(*Server).handleStream
google.golang.org/[email protected]/server.go:1029 google.golang.org/grpc.(*Server).serveStreams.func2.1
runtime/asm_arm64.s:1223 runtime.goexit
User Message: lock targeting ServerID:"b507c22c-685e-45bd-a796-7a5bcd63e3e6" is in force]" authz/permissions.go:617

In these logs we see:

  • log's cluttered with error messages from the node trying to call UpdateSessionTracker and failing to do so (because it's cert is no longer good) - we should probably suppress these or at least make it more clear that it's an expected failure due to the lock
  • the node tries to emit a client.disconnect event to the audit log, but fails (again, because it's cert is no longer valid) - maybe we can't rely on the node to emit the disconnect event

It's also interesting that the node is trying to emit a client.disconnect event, but the original audit event from when I originally reported this issue (for desktop session locks) is a session.rejected event instead.

In summary, let's expand the scope here to include:

  • cleaning up log spam when creating a lock
  • ensuring that a proper audit trail exists when sessions are terminated due to locks
  • documenting what audit events users can expect to see when sessions are terminated due to locks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request Used for new features in Teleport, improvements to current should be #enhancements igs ui
Projects
None yet
Development

No branches or pull requests

1 participant