Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[omdb] Add disks, historical VMMs to omdb db instance show #6935

Merged
merged 4 commits into from
Oct 26, 2024

Conversation

hawkw
Copy link
Member

@hawkw hawkw commented Oct 25, 2024

Inspired in part by what would have been nice to have while @augustuswm
and I were debugging failed instances on the colo rack a few days ago,
this branch adds additional data to the omdb db instance show command.

Now, we fetch and display a list of all disks attached to the instance:

Output with attached disks:
root@oxz_switch1:~# /var/tmp/omdb-eliza-test-7 db instance show e97e9fb9-62c3-4745-9d91-b0b6fa2baeba
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
WARN: found schema version 110.0.0, expected 111.0.0
It's possible the database is running a version that's different from what this
tool understands.  This may result in errors or incorrect output.

== INSTANCE ====================================================================
                        ID: e97e9fb9-62c3-4745-9d91-b0b6fa2baeba
                project ID: 5e49b6de-cb2d-438d-83af-95c415bbb901
                      name: mongodb-xfs-bs4096-secondary2
               description: mongodb cluster
                created at: 2024-03-30 05:40:49.049297 UTC
          last modified at: 2024-03-30 05:40:49.049297 UTC

== CONFIGURATION ===============================================================
                     vCPUs: 4
                    memory: 8 GiB
                  hostname: secondary2
                 boot disk: None
              auto-restart:
                  InstanceAutoRestart {
                      policy: None,
                      cooldown: None,
                  }

== RUNTIME STATE ===============================================================
               nexus state: Vmm
(i)     external API state: Starting
           last updated at: 2024-10-22T15:54:39.323895Z (generation 249)
       needs reincarnation: false
             karmic status: bound to saṃsāra
      last reincarnated at: Some(2024-10-22T15:54:45.258508Z)
             active VMM ID: Some(fa525216-9f81-4a8f-8fca-f7858994dce2)
             target VMM ID: None
              migration ID: None
              updater lock: UNLOCKED at generation: 49

         active VMM record:
             Vmm {
                 id: fa525216-9f81-4a8f-8fca-f7858994dce2,
                 time_created: 2024-10-24T20:51:43.102557Z,
                 time_deleted: None,
                 instance_id: e97e9fb9-62c3-4745-9d91-b0b6fa2baeba,
                 sled_id: 0c7011f7-a4bf-4daf-90cc-1c2410103300,
                 propolis_ip: V6(
                     Ipv6Network {
                         addr: fd00:1122:3344:104::1:41c,
                         prefix: 128,
                     },
                 ),
                 propolis_port: SqlU16(
                     12400,
                 ),
                 runtime: VmmRuntimeState {
                     time_state_updated: 2024-10-24T20:51:54.106560Z,
                     gen: Generation(
                         Generation(
                             3,
                         ),
                     ),
                     state: Starting,
                 },
             }

== ATTACHED DISKS===============================================================

SLOT NAME                          ID                                   SIZE   STATE
1    dbdata-xfs-bs4096-secondary2  258172d2-e89a-46bb-9509-ef2b0df8d8e5 80 GiB attached
0    mongodb-xfs-bs4096-secondary2 5e98b4b7-b539-4849-9953-55341a1c3772 30 GiB attached

Additionally, the command now accepts an optional --history flag, which will
print a list of all VMMs that have ever been associated with the instance. This
is useful for debugging instances which have migrated and/or been restarted ---
it's often useful to figure out what has happened to VMMs that previously
incarnated an instance, and determine their IDs and sled to find zone bundles
etc. The list of past migrations is now enabled or disabled by the --history
flag, as well.

Output with VMM history:
root@oxz_switch1:~# /var/tmp/omdb-eliza-test-7 db instance show e97e9fb9-62c3-4745-9d91-b0b6fa2baeba --history
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
WARN: found schema version 110.0.0, expected 111.0.0
It's possible the database is running a version that's different from what this
tool understands.  This may result in errors or incorrect output.

== INSTANCE ====================================================================
                        ID: e97e9fb9-62c3-4745-9d91-b0b6fa2baeba
                project ID: 5e49b6de-cb2d-438d-83af-95c415bbb901
                      name: mongodb-xfs-bs4096-secondary2
               description: mongodb cluster
                created at: 2024-03-30 05:40:49.049297 UTC
          last modified at: 2024-03-30 05:40:49.049297 UTC

# ... snipped out for conciseness ...

== ATTACHED DISKS===============================================================

SLOT NAME                          ID                                   SIZE   STATE
1    dbdata-xfs-bs4096-secondary2  258172d2-e89a-46bb-9509-ef2b0df8d8e5 80 GiB attached
0    mongodb-xfs-bs4096-secondary2 5e98b4b7-b539-4849-9953-55341a1c3772 30 GiB attached

== VMM HISTORY==================================================================

ID                                   STATE        GEN SLED_ID                              TIME_CREATED             TIME_DELETED
fa525216-9f81-4a8f-8fca-f7858994dce2 starting     3   0c7011f7-a4bf-4daf-90cc-1c2410103300 2024-10-24T20:51:43.102Z -
40b7a075-9b7f-4fb9-93b4-87107334243c saga_unwound 2   7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c 2024-10-22T15:54:45.174Z 2024-10-22T15:56:37.058Z
09814c36-302b-43ea-8f43-0e2d977fef39 failed       5   0c7011f7-a4bf-4daf-90cc-1c2410103300 2024-10-11T00:16:33.410Z 2024-10-22T15:54:43.103Z
36887bb1-24c2-47f7-9258-1dada23e33c6 failed       6   7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c 2024-10-09T04:32:44.359Z 2024-10-11T00:16:12.527Z
c3e29a0b-0c8a-4977-8cdc-4b9b6e355a7e failed       6   b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 2024-10-08T04:42:01.560Z 2024-10-09T04:32:35.413Z
5a62b22b-069f-4744-9303-b5cc1f0b1db8 failed       5   2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa 2024-10-07T05:42:59.516Z 2024-10-08T04:41:51.168Z
f05702ad-cbaa-4b9c-9263-ce0c7a156104 failed       5   5f6720b8-8a31-45f8-8c94-8e699218f28b 2024-10-02T18:18:39.008Z 2024-10-07T05:41:31.394Z
9b63d1b9-dfe8-4c78-a302-c7cdc9fc0970 failed       5   2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa 2024-09-26T21:53:30.820Z 2024-09-29T22:13:57.503Z
a37ee475-d158-4a4e-b396-33ceea78deb8 failed       6   5f6720b8-8a31-45f8-8c94-8e699218f28b 2024-09-19T00:32:09.839Z 2024-09-26T20:50:59.138Z
9febc78d-1aa8-4f40-9990-33c6d109e414 destroyed    8   b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 2024-09-12T18:07:06.294Z 2024-09-18T20:43:25.976Z
e1aecd78-69b6-40f1-9d9b-8d2801d5b511 destroyed    8   2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa 2024-08-31T04:28:47.870Z 2024-09-12T15:05:17.665Z
89726097-ef24-4b2d-812e-b968072cac24 destroyed    8   dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd 2024-08-31T04:08:52.567Z 2024-08-31T04:23:34.536Z
2d4a3e34-54a0-4ced-9ff6-0630797c5cb5 destroyed    8   0c7011f7-a4bf-4daf-90cc-1c2410103300 2024-08-31T03:26:07.477Z 2024-08-31T04:04:48.122Z
82cf4fa0-56a3-4e18-a461-e4ce581b8623 destroyed    8   71def415-55ad-46b4-ba88-3ca55d7fb287 2024-08-28T14:41:03.838Z 2024-08-31T01:47:23.120Z
11301fac-d145-4bad-8a33-3b02f57e0e24 destroyed    8   b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 2024-08-27T02:19:37.580Z 2024-08-27T18:37:20.671Z
1f7cb49f-d039-4028-b484-10d1c83e43eb destroyed    8   b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 2024-08-21T21:27:04.618Z 2024-08-25T18:37:26.915Z
5809e44e-1d90-4526-9c58-8fb4e393ab46 destroyed    8   f15774c1-b8e5-434f-a493-ec43f96cba06 2024-08-20T20:47:56.695Z 2024-08-21T19:10:29.366Z
9131d4d0-8ce4-40ed-b931-614f52469a82 destroyed    8   71def415-55ad-46b4-ba88-3ca55d7fb287 2024-08-14T03:24:27.896Z 2024-08-15T21:20:07.082Z
d8be34b6-f59c-4de4-9852-e60eead5682e destroyed    8   5f6720b8-8a31-45f8-8c94-8e699218f28b 2024-08-01T21:43:21.657Z 2024-08-14T01:19:21.004Z
0a0b6743-0f9c-45c6-9b31-dd83163d6fe8 destroyed    7   2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa 2024-08-01T00:17:06.056Z 2024-08-01T19:07:00.617Z
83e0672d-d1f8-440b-a4c2-5f3e659dc910 destroyed    7   f15774c1-b8e5-434f-a493-ec43f96cba06 2024-07-25T23:41:18.279Z 2024-07-26T16:03:54.205Z
ead4e634-50f8-4d58-bae1-568df37c8ff7 destroyed    7   0c7011f7-a4bf-4daf-90cc-1c2410103300 2024-07-18T22:49:01.899Z 2024-07-25T05:43:53.592Z
436dff2f-8392-4814-b045-98f8eecbbcad destroyed    8   5f6720b8-8a31-45f8-8c94-8e699218f28b 2024-07-13T21:18:07.784Z 2024-07-18T14:43:34.656Z
fe28c172-a9d5-45ac-8a82-b94f51e1f8c1 saga_unwound 2   f15774c1-b8e5-434f-a493-ec43f96cba06 2024-07-13T21:09:49.175Z 2024-09-12T18:00:18.441Z
5eb042a2-5ea4-4aa0-90e1-bd90e0e31af8 destroyed    99  87c2c4fc-b0c7-4fef-a305-78f0ed265bbc 2024-07-13T03:57:18.576Z 2024-07-13T21:07:14.735Z
500811e4-7a5a-4857-98ef-f20ad73438a5 destroyed    7   f15774c1-b8e5-434f-a493-ec43f96cba06 2024-07-12T05:28:37.432Z 2024-07-13T01:04:46.536Z
620e6014-0707-4b46-94ba-ef0c1ce9b5d2 destroyed    7   f15774c1-b8e5-434f-a493-ec43f96cba06 2024-07-11T00:55:48.868Z 2024-07-11T22:55:41.933Z
8f6b2953-1b72-4325-9fb9-08cc1b52d0a5 destroyed    7   71def415-55ad-46b4-ba88-3ca55d7fb287 2024-07-08T21:35:02.137Z 2024-07-10T23:18:21.667Z
eedbe3ee-1d8e-46ae-bd48-393e6cd00368 destroyed    7   2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa 2024-07-07T20:24:54.814Z 2024-07-08T19:53:29.134Z
7650d95c-a9e3-4087-8a1c-959a64f18742 destroyed    7   dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd 2024-07-03T17:35:10.428Z 2024-07-07T17:58:16.378Z
613d7e11-4f49-4a07-b0ba-a7c2617eaaef destroyed    8   71def415-55ad-46b4-ba88-3ca55d7fb287 2024-06-28T23:59:56.326Z 2024-07-01T16:56:39.303Z
01651e7a-88fe-4b09-a812-7d1e42d3cee3 destroyed    8   dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd 2024-06-21T18:09:48.050Z 2024-06-27T17:07:01.677Z
4058a07d-9ceb-42a0-99a9-2abfd280f75a destroyed    8   0c7011f7-a4bf-4daf-90cc-1c2410103300 2024-06-20T19:23:53.106Z 2024-06-20T23:57:03.869Z
2c2619dc-7920-4aa0-bd31-56753dffed2d destroyed    8   b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88 2024-06-14T19:15:06.049Z 2024-06-20T17:26:09.012Z
a4b51b6f-373f-496e-9c59-2c7b2965893c destroyed    8   5f6720b8-8a31-45f8-8c94-8e699218f28b 2024-06-11T23:36:11.780Z 2024-06-13T14:09:03.529Z
15ecaa92-81b9-42d7-84e1-6e21e35c46cf destroyed    29  dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd 2024-06-11T18:55:51.705Z 2024-06-11T20:26:44.169Z
3ab65681-7850-43fe-9f35-6d10a6fb7f7d destroyed    8   dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd 2024-06-08T04:39:12.921Z 2024-06-11T18:31:58.065Z
83c196cb-4410-470d-9c9e-1b428ea2f913 destroyed    8   bd96ef7c-4941-4729-b6f7-5f47feecbc4b 2024-06-07T05:53:04.289Z 2024-06-08T04:38:28.372Z
97303add-944c-4c34-9e15-7a6077e8909a destroyed    8   7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c 2024-05-31T03:44:48.196Z 2024-06-06T22:43:42.122Z
1ad7b454-907a-404c-8bf2-619f6c7ccc60 destroyed    8   87c2c4fc-b0c7-4fef-a305-78f0ed265bbc 2024-05-29T19:27:32.798Z 2024-05-31T00:18:43.729Z
root@oxz_switch1:~#

Depends on #6933
Closes #6929
Also addresses some of #6931, but not the networking part.

Currently, `omdb` formats `chrono::DateTime`s using the default
`fmt::Display` format. This uses a bunch of characters of line width,
which becomes important for `omdb` commands which output tabular data
with a large number of columns --- avoiding line wrapping wherever
possible makes it substantially easier to read such output. @iximeow
pointed out to me that we could save several characters of line width
by using a more concise timestamp format.

This commit changes tabular output in `omdb db` commands to format
timestamps using `chrono`'s RFC 3339 formatter with millisecond
precision, rather than nanosecond precision. This saves us a few
characters. RFC 3339 permits the use of `Z` rather than `+00:00` (the
UTC offset) for UTC timestamps, and since all our timestamps are UTC,
that saves us another four characters (also relative to the default
format, which uses ` UTC ` for UTC timestamps). Together, this makes
timestamp columns substantially shorter in `omdb db` output.

For example, consider `omdb db region-replacement list`:

```console
root@oxz_switch1:~# omdb db region-replacement list
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
WARN: found schema version 110.0.0, expected 111.0.0
It's possible the database is running a version that's different from what this
tool understands.  This may result in errors or incorrect output.
Region replacement requests
ID                                   REQUEST_TIME                   REPLACEMENT_STATE
07a289e6-af50-42f1-8888-062229547888 2024-08-14 19:43:13.318246 UTC Complete
08c449a6-25c1-46f8-b070-9223cb924cb0 2024-08-14 19:43:13.245003 UTC Complete
1ed5e8e1-1d98-4420-bef0-135cba510238 2024-08-14 19:43:13.227734 UTC Complete
20811f9d-f6ed-40fe-8403-305edc4b3e8c 2024-08-14 19:43:13.291419 UTC Complete
52bd1034-2a77-4a17-8871-77e535b509d5 2024-08-14 19:43:13.259033 UTC Complete
5c43d936-d5c4-4d4e-b1ca-890aabeeae87 2024-08-14 19:43:13.357786 UTC Complete
915f8726-a882-4960-9e5c-dfe359f54911 2024-08-14 19:43:13.279 UTC    Complete
a4e697b6-5f02-4f75-ad41-f5bc3e906f0b 2024-08-14 19:43:13.344742 UTC Complete
c2b5ef9d-a89a-46c8-b57e-1eebb5487804 2024-08-14 19:43:13.331527 UTC Complete
f0e009b5-1575-42f0-bca5-7df6db846ef0 2024-08-14 19:43:13.304655 UTC Complete
root@oxz_switch1:~# /var/tmp/omdb-eliza-test-concise db region-replacement list
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
WARN: found schema version 110.0.0, expected 111.0.0
It's possible the database is running a version that's different from what this
tool understands.  This may result in errors or incorrect output.
Region replacement requests
ID                                   REQUEST_TIME             REPLACEMENT_STATE
07a289e6-af50-42f1-8888-062229547888 2024-08-14T19:43:13.318Z Complete
08c449a6-25c1-46f8-b070-9223cb924cb0 2024-08-14T19:43:13.245Z Complete
1ed5e8e1-1d98-4420-bef0-135cba510238 2024-08-14T19:43:13.227Z Complete
20811f9d-f6ed-40fe-8403-305edc4b3e8c 2024-08-14T19:43:13.291Z Complete
52bd1034-2a77-4a17-8871-77e535b509d5 2024-08-14T19:43:13.259Z Complete
5c43d936-d5c4-4d4e-b1ca-890aabeeae87 2024-08-14T19:43:13.357Z Complete
915f8726-a882-4960-9e5c-dfe359f54911 2024-08-14T19:43:13.279Z Complete
a4e697b6-5f02-4f75-ad41-f5bc3e906f0b 2024-08-14T19:43:13.344Z Complete
c2b5ef9d-a89a-46c8-b57e-1eebb5487804 2024-08-14T19:43:13.331Z Complete
f0e009b5-1575-42f0-bca5-7df6db846ef0 2024-08-14T19:43:13.304Z Complete
root@oxz_switch1:~#
```

Although the output from the `region-replacement list` command is not
*that* wide, I'm presently working on an instance VMM history command
that displays three timestamps, and the width saving becomes quite
significant there.
@augustuswm
Copy link
Contributor

This is super helpful, I am very commonly querying the vmm table in crdb when trying to understand an instance's behavior

Copy link
Contributor

@gjcolombo gjcolombo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great--thanks for putting it together!

@hawkw hawkw enabled auto-merge (squash) October 25, 2024 21:53
@leftwo
Copy link
Contributor

leftwo commented Oct 25, 2024

Yeah, the changes look fine here.

For the instance output:

...
== ATTACHED DISKS===============================================================

SLOT NAME                          ID                                   SIZE   STATE
1    dbdata-xfs-bs4096-secondary2  258172d2-e89a-46bb-9509-ef2b0df8d8e5 80 GiB attached
0    mongodb-xfs-bs4096-secondary2 5e98b4b7-b539-4849-9953-55341a1c3772 30 GiB attached

Woudl we ever expect the STATE to be anything other than attached here? I know you have made a generic disk printer and that's fine, I was just wondering.

@hawkw
Copy link
Member Author

hawkw commented Oct 26, 2024

@leftwo I also wondered about this, and considered removing it. I thought maybe it was useful for debugging — if you see a disk that has an instance ID set but isn't Attached, that could mean something?

I don't really know what the disk's state machine in the control plane looks like, but can look?

@hawkw hawkw merged commit 38a34e1 into main Oct 26, 2024
16 checks passed
@hawkw hawkw deleted the eliza/instance-history branch October 26, 2024 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

want omdb db instance show to show all VMMs ever associated with an instance
4 participants