✨ Improve logging & error handling

[sbueringer]

What this PR does / why we need it:
Final PR to close out #2076. I reviewed the entire code in the controller package and all log statements. I also made some improvements to error handling and some smaller bug fixes by opportunity.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2076

[codecov]

Codecov Report

Attention: 400 lines in your changes are missing coverage. Please review.

Comparison is base (89f4029) 63.73% compared to head (3f27366) 63.45%.
Report is 10 commits behind head on main.

Files	Patch %	Lines
pkg/services/govmomi/util.go	16.94%	49 Missing ⚠️
pkg/services/govmomi/service.go	11.32%	47 Missing ⚠️
controllers/vspherecluster_reconciler.go	27.08%	33 Missing and 2 partials ⚠️
controllers/serviceaccount_controller.go	58.97%	27 Missing and 5 partials ⚠️
controllers/vspheremachine_controller.go	69.79%	26 Missing and 3 partials ⚠️
controllers/vspherevm_controller.go	52.08%	20 Missing and 3 partials ⚠️
controllers/servicediscovery_controller.go	75.64%	19 Missing ⚠️
pkg/session/session.go	56.41%	17 Missing ⚠️
pkg/util/machines.go	0.00%	15 Missing ⚠️
controllers/vspheredeploymentzone_controller.go	65.85%	14 Missing ⚠️
... and 21 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2527      +/-   ##
==========================================
- Coverage   63.73%   63.45%   -0.28%     
==========================================
  Files         123      120       -3     
  Lines        8812     8685     -127     
==========================================
- Hits         5616     5511     -105     
+ Misses       2773     2747      -26     
- Partials      423      427       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sbueringer]

/test pull-cluster-api-provider-vsphere-e2e-full-main

[sbueringer]

/assign @fabriziopandini @chrischdi @adityabhatia Should be ready for review
(let's do a log walkthrough early next week though)

Would be good if we can hold other bigger changes to avoid rebasing this PR. I also spent a few hours reviewing this PR myself so it should be in pretty good shape at the moment.

[fabriziopandini]

just a couple of nits from my side

[chrischdi]

A great PR which is a huge improvement to logging and helps in future to debug things!

[sbueringer]

@adityabhatia @fabriziopandini @chrischdi Really appreciate the quick reviews. I know it's a huge PR.

fixed/answered PTAL :)

[sbueringer]

/test pull-cluster-api-provider-vsphere-e2e-full-main

[chrischdi]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 30bef98f0738cd88409a519aef75f7e400413127

[sbueringer]

Squashed to make the PR mergable. Review fix commit can be seen here: b5286e5

[sbueringer]

/test pull-cluster-api-provider-vsphere-e2e-full-main

[k8s-ci-robot]

@sbueringer: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-vsphere-apidiff-main	3f27366	link	false	/test pull-cluster-api-provider-vsphere-apidiff-main

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[fabriziopandini]

Great work!
/lgtm

[sbueringer]

Going to merge to unblock other PRs given lgtm's above and Tide will wait for e2e-full to finish successfully

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sbueringer]

/override codecov/patch

(running since 1h10m)

[k8s-ci-robot]

@sbueringer: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

• codecov/patch

Only the following failed contexts/checkruns were expected:

• EasyCLA
• lint
• pull-cluster-api-provider-vsphere-apidiff-main
• pull-cluster-api-provider-vsphere-e2e-full-main
• pull-cluster-api-provider-vsphere-e2e-main
• pull-cluster-api-provider-vsphere-test-integration-main
• pull-cluster-api-provider-vsphere-test-main
• pull-cluster-api-provider-vsphere-verify-main
• tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

In response to this:

/override codecov/patch

(running since 1h10m)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.