[MAINT]: Add GitHub Actions workflow for integration testing

[stevehipwell]

Describe the need

When pushing a change to the code I'd like the integration tests to be able to be run. This doesn't have to be automatic as there are a number of models which could be applied to stop integration credentials being abused.

• Auto-run all actions (DON'T DO THIS)
• Auto-run actions from repeat contributors (DON'T DO THIS)
• Auto-run actions from organisation members
  • Put integration test credentials behind an environment and require approval to run

I'd be happy to contribute the code to do this.

SDK Version

No response

API Version

No response

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[nickfloyd]

@stevehipwell Thanks for the thoughts and input here! ❤

If you're up for it, this idea would be really great to see in code so that we can get a more clear picture of your thoughts here. I've also added the up for grabs label just incase others are up for working on this as well.

[stevehipwell]

@nickfloyd I was working from a memory that there weren't automated tests in GH Actions and from the fact that my PRs never get automated tests. However I just checked out what's there and I can see automated tests for branches with a test/ prefix and for PRs labeled in a specific way; but they seem to be broken? Do you have more context on this?

[stevehipwell]

OK, I can see what the issue is. I'll have a go at fixing this.

[stevehipwell]

@nickfloyd @kfcampbell I've got #2476 running and would expect the anonamous tests to work as is. Do you have a list of the currently broken acceptance tests as that'll help me figure out what should be working? Based on running some of the tests it looks like there are tests being run in anonymous mode which wont work.

[kfcampbell]

Do you have a list of the currently broken acceptance tests

We don't have a comprehensive list of broken acceptance tests unfortunately. As the suite hasn't been run front-to-back (maybe ever?), this is not readily available information. Also, it depends heavily on a really organized test setup that we just don't have (or have documented) at the moment: enterprise, organizations, test accounts, GitHub Apps, Enterprise Server instances, EMUs, Actions runners, various GitHub plans, etc., and many of those resources cost money in addition to the overhead in maintaining the test setup. #1414 is an attempt to quantify some of that setup, but hasn't taken off due to a lack of funding.

Based on running some of the tests it looks like there are tests being run in anonymous mode which wont work.

Agreed that we cannot run all tests in anonymous mode. Running any tests with anonymous mode would be a step up.

However, the test setup is stale, as you've noticed in your PR. I just checked and I'm an admin on github.com/terraformtesting, which is cool, but I have no idea where these https://terraformtesting-ghe.eastus.cloudapp.azure.com/ resources are configured (or if indeed they still exist at all).

In order for this to be successful long-term, we'll need organizational buy-in from GitHub to support time and money into testing the provider so that we can get resources deployed and maintained for testing purposes. I unfortunately have been unsuccessful in my quest to sell this as a valid use of company resources thus far; hence my ad-hoc testing before merges with my own testing org/accounts.

[stevehipwell]

@kfcampbell could you provide the list of tests that pass locally in your manual testing for each type (anon, user, org) when targeting github.com? Once I have these I'll look at skipping the currently failing tests conditionally based on type. I aim to have the tests where anonymous is valid running in GitHub Actions ASAP.

Could you please add dotcom & ghes environments but don't add protection yet? Could you also add a acctest (or equivalent) label? With these I can finish testing the test triggering logic and any credentials added can be gated.

RE GHES - the cron trigger works for anonymous tests. For now I'll disable automatic tests for GHES but we can revisit later. Ignore that, they're all being skipped.

After the above we will need to work on the authenticated tests. I suspect that as well as a test user we need a free organisation, a team organisation and an enterprise organisation for the testing to be valid, does that ring true? The process here will be to have the credentials stored as environment secrets on the environments created above (dotcom for now) and to add required approvers to the environment.

[stevehipwell]

I've captured the following issues based on the work to complete the acceptance test refactoring, which has now been completed.

• [BUG]: The github_repository resource churns on vulnerability_alerts if not explicitly set #2489
• [DOCS]: The codespaces organisation APIs don't currently work on free orgs #2490
• [BUG]: The github_organization_teams data source doesn't populate parent.id correctly #2491
• [BUG]: The github_organization_teams data source doesn't populate privacy correctly #2492
• [BUG]: The github_branch_protection_v3 resource churns when using checks or contexts in required_status_checks #2493
• [BUG]: Resources using the removed legacy projects API are still present and not deprecated in the provider #2494
• [BUG]: The github_repository resource churns if pages are enabled and build_type is set to workflow #2495
• [BUG]: The github_team data source repositories_detailed does not directly correlate repo name with role #2496

Other than #2490 which is a docs issue, the issues have code fixes in #2476 and I'm planning on opening individual PRs to resolve them.