pageserver: wait for lsn lease duration after transition into AttachedSingle

[yliang412]

Part of #7497, closes #8890.

Problem

Since leases are in-memory objects, we need to take special care of them after pageserver restarts and while doing a live migration. The approach we took for pageserver restart is to wait for at least lease duration before doing first GC. We want to do the same for live migration. Since we do not do any GC when a tenant is in AttachedStale or AttachedMulti mode, only the transition from AttachedMulti to AttachedSingle requires this treatment.

Summary of changes

• Added lsn_lease_deadline field in GcBlock::reasons: the tenant is temporarily blocked from GC until we reach the deadline. This information does not persist to S3.
• In GCBlock::start, skip the GC iteration if we are blocked by the lsn lease deadline.
• In TenantManager::upsert_location, set the lsn_lease_deadline to Instant::now() + lsn_lease_length so the granted leases have a chance to be renewed before we run GC for the first time after transitioned from AttachedMulti to AttachedSingle.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

4968 tests run: 4804 passed, 0 failed, 164 skipped (full report)

---

Flaky tests (5)

Postgres 17

• test_ondemand_wal_download_in_replication_slot_funcs: release-x86-64

Postgres 16

• test_slots_and_branching: release-arm64
• test_scrubber_physical_gc[4]: release-arm64

Postgres 15

• test_subscriber_restart: release-x86-64

Postgres 14

• test_scrubber_physical_gc[4]: release-x86-64

Code coverage* (full report)

• functions: 31.9% (7425 of 23298 functions)
• lines: 49.9% (59745 of 119838 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
d1b2bb1 at 2024-09-19T15:58:13.456Z :recycle:}

[problame]

(Removing myself from this review)

[koivunej]

Thanks, I think this is looking good.

I am surprised how many gc sensitive tests we have ... but which were not sensitive to lsn lease initial wait before? I'll try to understand these more and possibly chat with you...