Improved templating audits (#238)

* Include workflow name in cache key

* Update pyproject.toml

* Refactor StubLoader into SpyEnvironment, SpyContext, and SpyFileSystemLoader

* Distinguish between required and optional context

* Default details to open when errors occur

* Fix get_source auditing

* Rename Spy to Audit

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename spy contextmanager to audit

* Initialize test suite

* Add test_dump_summary

* Parametrize TemplateState tests

* Add initial test for template_file

* Initialize test suite

* Install all dependencies

* Upload coverage

* Add badges

* Update README.md

* Update test_template_files.py

* Stringify

* Standardize Python

* Enable dependabot updates for pip

* Add analyze step with automated error reports

* Minor cleanup

* Update existing failures

* Catch IsADirectoryError

* Typing

* Pytest ids

* Update README

* Add GHA test

* Fix stub audit

* Comment template-files summary

* Update tests

* Use a sticky comment instead

* Formatting

* Use fine-grained token

* Improve messaging

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jannis Leidel <[email protected]>

.github/workflows/tests.yml

@@ -84,9 +84,55 @@ jobs:
           assert '''${{ steps.json.outputs.content }}''' == '''${{ steps.yaml.outputs.content }}'''
           assert '''${{ fromJSON(steps.json.outputs.content)['foo'] }}''' == '''${{ fromJSON(steps.yaml.outputs.content)['foo'] }}'''
 
+  template-files:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Source
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Template Success
+        id: templates-success
+        uses: ./template-files
+        with:
+          config: ./template-files/data/success.yml
+          stubs: ./template-files/data/templates
+
+      - name: Template Error
+        id: templates-error
+        continue-on-error: true
+        uses: ./template-files
+        with:
+          config: ./template-files/data/error.yml
+          stubs: ./template-files/data/templates
+
+      - name: Run Tests
+        run: diff --recursive .github_cache/template-files template-files/data/expected
+
+      - name: Filter Changes
+        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+        id: filter
+        with:
+          filters: |
+            code:
+              - 'template-files/**'
+      - name: Comment on PR
+        if: github.event_name == 'pull_request' && steps.filter.outputs.code == 'true'
+        uses: marocchino/sticky-pull-request-comment@331f8f5b4215f0445d3c07b4967662a32a2d3e31  # v2.9.0
+        with:
+          message: |
+            ### Template Success
+            > [!WARNING]
+            > This is what the audit looks like when the templating has no errors.
+            ${{ steps.templates-success.outputs.summary }}
+            ### Template Error
+            > [!WARNING]
+            > This is what the audit looks like when templating results in errors.
+            ${{ steps.templates-error.outputs.summary }}
+          GITHUB_TOKEN: ${{ secrets.SANDBOX_TEMPLATE_TOKEN }}
+
   # required check
   analyze:
-    needs: [pytest, read-file]
+    needs: [pytest, read-file, template-files]
     if: '!cancelled()'
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -1,3 +1,5 @@
+.github_cache/
+
 # Created by https://www.toptal.com/developers/gitignore/api/node,python
 # Edit at https://www.toptal.com/developers/gitignore?templates=node,python
 

template-files/README.md

@@ -3,27 +3,39 @@
 A composite GitHub Action to template (or copy) files from other repositories and
 commits them to the specified PR.
 
-## GitHub Action Usage
+## Action Inputs
+
+| Name | Description | Default |
+|------|-------------|---------|
+| `config` | Configuration path defining what files to template/copy. | `.github/template-files/config.yml` |
+| `stubs` | Path to where stub files are located in the current repository. | `.github/template-files/templates/` |
+| `token` | GitHub token to fetch remote files from repositories (no extra permissions are needed to access public repositories). | `${{ github.token }}` |
+
+## Action Outputs
+
+| Name | Description |
+|------|-------------|
+| `summary` | Summary of the files that were templated/copied. |
+
+## Sample Workflows
 
 In your GitHub repository include this action in your workflows:
 
 ```yaml
-- uses: conda/actions/template-files
-  with:
-    # [optional]
-    # the path to the configuration file
-    config: .github/template-files/config.yml
-
-    # [optional]
-    # the path to the template stubs
-    stubs: .github/template-files/templates/
-
-    # [optional]
-    # the GitHub token with API access
-    token: ${{ github.token }}
+name: Template Files
+
+on:
+  workflow_dispatch:
+
+jobs:
+  template:
+    steps:
+      - uses: conda/actions/template-files
+        with:
+          token: ...
 ```
 
-Define what files to template in a configuration file, e.g., `.github/templates/config.yml`:
+## Sample Config (e.g., `.github/templates/config.yml`)
 
 ```yaml
 user/repo:

template-files/action.yml

@@ -10,13 +10,12 @@ inputs:
     description: Configuration path defining what files to template/copy.
     default: .github/template-files/config.yml
   stubs:
-    description: >-
-      Path to where stub files are located in the current repository.
+    description: Path to where stub files are located in the current repository.
     default: .github/template-files/templates/
   token:
     description: >-
-      A token with ability to comment, label, and modify the commit status
-      (`pull_request: write` and `statuses: write` for fine-grained PAT; `repo` for classic PAT)
+      GitHub token to fetch remote files from repositories
+      (no extra permissions are needed to access public repositories).
     default: ${{ github.token }}
 outputs:
   summary:
@@ -55,6 +54,9 @@ runs:
     - name: Template Files
       id: template
       shell: bash
-      run: python ${{ github.action_path }}/template_files.py --config ${{ inputs.config }} --stubs ${{ inputs.stubs }}
+      run: >
+        python ${{ github.action_path }}/template_files.py
+        --config ${{ inputs.config }}
+        --stubs ${{ inputs.stubs }}
       env:
         GITHUB_TOKEN: ${{ github.token }}

template-files/data/configs/inconsistent/no_dst.yml

@@ -0,0 +1,4 @@
+org/repo:
+  - remove: true
+  - with:
+      variable: value

template-files/data/configs/inconsistent/no_src.yml

@@ -0,0 +1,5 @@
+org/repo:
+  - dst: file.txt
+  - remove: true
+  - with:
+      variable: value

template-files/data/configs/invalid/dst_bool.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - dst: true

template-files/data/configs/invalid/dst_number.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - dst: 1

template-files/data/configs/invalid/remove_number.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - remove: 1

template-files/data/configs/invalid/remove_string.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - remove: string

template-files/data/configs/invalid/src_bool.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - src: true

template-files/data/configs/invalid/src_number.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - src: 1

template-files/data/configs/invalid/with_scalar.yml

@@ -0,0 +1,2 @@
+org/repo:
+  - with: string

template-files/data/configs/valid/complex.yml

@@ -0,0 +1,23 @@
+org/repo:
+  # paths are copied from org/repo to current repo
+  - file.txt
+  - path/to/file.txt
+
+org/repo2:
+  # paths are copied from org/repo2 to current repo
+  - src: file2.txt
+    dst: path/to/file2.txt
+  # paths are copied from org/repo2 to current repo with templating
+  - src: file3.txt
+    with:
+      variable: value
+  - src: file4.txt
+    dst: path/to/file4.txt
+    with:
+      variable: value
+  # remove dst paths from current repo
+  - dst: file5.txt
+    remove: true
+  # remove src paths from current repo
+  - src: file6.txt
+    remove: true

template-files/data/configs/valid/simple.yml

@@ -0,0 +1,3 @@
+org/repo:
+  - file.txt
+  - path/to/file.txt

template-files/data/error.yml

@@ -0,0 +1,5 @@
+./templates:
+  - src: file1.txt
+    dst: .github_cache/template-files/error/file1.txt
+  - src: file2.txt
+    dst: .github_cache/template-files/error/file2.txt

template-files/data/expected/error/file1.txt

@@ -0,0 +1,21 @@
+# file1
+
+This is a template and only uses the default context.
+
+| context | value |
+|---|---|
+| repo.html_url | https://github.com/conda/actions |
+| repo.full_name | conda/actions |
+| source.user | <local> |
+
+Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque aliquet justo ipsum, nec dapibus ex congue ac. Fusce non vulputate libero. Nulla fermentum nisl finibus turpis tempor, non porttitor massa rutrum. Sed suscipit metus arcu, eu blandit ante condimentum in. Proin vehicula various risus eget fermentum. Duis in sollicitudin ante. Curabitur aliquam elit vel dictum sagittis. Proin nec ultrices sem, ac sodales leo. Duis vehicula, metus nec tristique vestibulum, massa mi faucibus urna, quis pulvinar neque neque congue felis. Nulla lobortis ullamcorper imperdiet. Fusce justo arcu, tristique nec tincidunt commodo, porttitor sit amet neque. Nullam tincidunt orci a enim fringilla sodales. Morbi ut dui vitae justo interdum ullamcorper.
+
+# stub1
+
+Nullam blandit nisi non elit efficitur porttitor quis a justo. Sed venenatis, eros quis malesuada aliquet, ligula purus cursus velit, eget efficitur libero ante id ipsum. Aliquam et neque id justo tempus egestas quis ut augue. Curabitur eu sem velit. Phasellus sed nunc ipsum. Sed rhoncus luctus turpis, eu rutrum ipsum dictum nec. Curabitur vulputate dolor non ipsum ultrices, ut ultricies nibh vestibulum. Donec molestie scelerisque interdum. Vestibulum et finibus purus, rhoncus porttitor ligula.
+
+# stub2
+
+Curabitur metus libero, lacinia nec magna euismod, pellentesque lacinia sem. Nullam tempor vitae ex ut tincidunt. Etiam ullamcorper et nisi at cursus. Praesent ac mauris vel mauris ornare consequat sit amet ut magna. Integer luctus, ante at rhoncus cursus, est erat aliquam magna, et pretium urna felis eget lorem. Etiam tristique, justo viverra cursus congue, quam odio sagittis dolor, quis dictum risus tortor et turpis. Donec arcu urna, dapibus eget metus ac, consequat feugiat velit. Integer aliquet velit quis sollicitudin luctus. Maecenas aliquam arcu non urna maximus hendrerit. Integer porta efficitur blandit. Donec interdum nec metus at iaculis. Donec pellentesque enim turpis. Nulla facilisi. In hac habitasse platea dictumst.
+
+EOF

template-files/data/expected/error/file2.txt

@@ -0,0 +1,16 @@
+# file2
+
+This is a template and expects additional context.
+
+| context | value |
+|---|---|
+| variable |  |
+| optional | 12 |
+
+Name finibus augue ac efficitur sodales. Donec facilisis lacus sit amet metus congue aliquam. Proin mollis, sapien sit amet iaculis congue, libero erat volutpat neque, in bibendum lacus massa eget eros. Mauris fringilla turpis id pulvinar ultricies. Cras dictum condimentum mauris sit amet lobortis. Maecenas ultrices in lectus eget tristique. Etiam sollicitudin tempus dolor sit amet tincidunt. Nulla nec mi eget est gravida elementum vitae vitae risus. Nullam eget urna dignissim, sollicitudin massa at, cursus urna.
+
+# stub2
+
+Curabitur metus libero, lacinia nec magna euismod, pellentesque lacinia sem. Nullam tempor vitae ex ut tincidunt. Etiam ullamcorper et nisi at cursus. Praesent ac mauris vel mauris ornare consequat sit amet ut magna. Integer luctus, ante at rhoncus cursus, est erat aliquam magna, et pretium urna felis eget lorem. Etiam tristique, justo viverra cursus congue, quam odio sagittis dolor, quis dictum risus tortor et turpis. Donec arcu urna, dapibus eget metus ac, consequat feugiat velit. Integer aliquet velit quis sollicitudin luctus. Maecenas aliquam arcu non urna maximus hendrerit. Integer porta efficitur blandit. Donec interdum nec metus at iaculis. Donec pellentesque enim turpis. Nulla facilisi. In hac habitasse platea dictumst.
+
+EOF

template-files/data/expected/success/file1.txt

@@ -0,0 +1,21 @@
+# file1
+
+This is a template and only uses the default context.
+
+| context | value |
+|---|---|
+| repo.html_url | https://github.com/conda/actions |
+| repo.full_name | conda/actions |
+| source.user | <local> |
+
+Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque aliquet justo ipsum, nec dapibus ex congue ac. Fusce non vulputate libero. Nulla fermentum nisl finibus turpis tempor, non porttitor massa rutrum. Sed suscipit metus arcu, eu blandit ante condimentum in. Proin vehicula various risus eget fermentum. Duis in sollicitudin ante. Curabitur aliquam elit vel dictum sagittis. Proin nec ultrices sem, ac sodales leo. Duis vehicula, metus nec tristique vestibulum, massa mi faucibus urna, quis pulvinar neque neque congue felis. Nulla lobortis ullamcorper imperdiet. Fusce justo arcu, tristique nec tincidunt commodo, porttitor sit amet neque. Nullam tincidunt orci a enim fringilla sodales. Morbi ut dui vitae justo interdum ullamcorper.
+
+# stub1
+
+Nullam blandit nisi non elit efficitur porttitor quis a justo. Sed venenatis, eros quis malesuada aliquet, ligula purus cursus velit, eget efficitur libero ante id ipsum. Aliquam et neque id justo tempus egestas quis ut augue. Curabitur eu sem velit. Phasellus sed nunc ipsum. Sed rhoncus luctus turpis, eu rutrum ipsum dictum nec. Curabitur vulputate dolor non ipsum ultrices, ut ultricies nibh vestibulum. Donec molestie scelerisque interdum. Vestibulum et finibus purus, rhoncus porttitor ligula.
+
+# stub2
+
+Curabitur metus libero, lacinia nec magna euismod, pellentesque lacinia sem. Nullam tempor vitae ex ut tincidunt. Etiam ullamcorper et nisi at cursus. Praesent ac mauris vel mauris ornare consequat sit amet ut magna. Integer luctus, ante at rhoncus cursus, est erat aliquam magna, et pretium urna felis eget lorem. Etiam tristique, justo viverra cursus congue, quam odio sagittis dolor, quis dictum risus tortor et turpis. Donec arcu urna, dapibus eget metus ac, consequat feugiat velit. Integer aliquet velit quis sollicitudin luctus. Maecenas aliquam arcu non urna maximus hendrerit. Integer porta efficitur blandit. Donec interdum nec metus at iaculis. Donec pellentesque enim turpis. Nulla facilisi. In hac habitasse platea dictumst.
+
+EOF

template-files/data/expected/success/file2.txt

@@ -0,0 +1,16 @@
+# file2
+
+This is a template and expects additional context.
+
+| context | value |
+|---|---|
+| variable | value |
+| optional | 12 |
+
+Name finibus augue ac efficitur sodales. Donec facilisis lacus sit amet metus congue aliquam. Proin mollis, sapien sit amet iaculis congue, libero erat volutpat neque, in bibendum lacus massa eget eros. Mauris fringilla turpis id pulvinar ultricies. Cras dictum condimentum mauris sit amet lobortis. Maecenas ultrices in lectus eget tristique. Etiam sollicitudin tempus dolor sit amet tincidunt. Nulla nec mi eget est gravida elementum vitae vitae risus. Nullam eget urna dignissim, sollicitudin massa at, cursus urna.
+
+# stub2
+
+Curabitur metus libero, lacinia nec magna euismod, pellentesque lacinia sem. Nullam tempor vitae ex ut tincidunt. Etiam ullamcorper et nisi at cursus. Praesent ac mauris vel mauris ornare consequat sit amet ut magna. Integer luctus, ante at rhoncus cursus, est erat aliquam magna, et pretium urna felis eget lorem. Etiam tristique, justo viverra cursus congue, quam odio sagittis dolor, quis dictum risus tortor et turpis. Donec arcu urna, dapibus eget metus ac, consequat feugiat velit. Integer aliquet velit quis sollicitudin luctus. Maecenas aliquam arcu non urna maximus hendrerit. Integer porta efficitur blandit. Donec interdum nec metus at iaculis. Donec pellentesque enim turpis. Nulla facilisi. In hac habitasse platea dictumst.
+
+EOF

template-files/data/success.yml

@@ -0,0 +1,7 @@
+./templates:
+  - src: file1.txt
+    dst: .github_cache/template-files/success/file1.txt
+  - src: file2.txt
+    dst: .github_cache/template-files/success/file2.txt
+    with:
+      variable: value

template-files/data/templates/file1.txt

@@ -0,0 +1,15 @@
+# file1
+
+This is a template and only uses the default context.
+
+| context | value |
+|---|---|
+| repo.html_url | [[ repo.html_url ]] |
+| repo.full_name | [[ repo.full_name ]] |
+| source.user | [[ source.user ]] |
+
+Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque aliquet justo ipsum, nec dapibus ex congue ac. Fusce non vulputate libero. Nulla fermentum nisl finibus turpis tempor, non porttitor massa rutrum. Sed suscipit metus arcu, eu blandit ante condimentum in. Proin vehicula various risus eget fermentum. Duis in sollicitudin ante. Curabitur aliquam elit vel dictum sagittis. Proin nec ultrices sem, ac sodales leo. Duis vehicula, metus nec tristique vestibulum, massa mi faucibus urna, quis pulvinar neque neque congue felis. Nulla lobortis ullamcorper imperdiet. Fusce justo arcu, tristique nec tincidunt commodo, porttitor sit amet neque. Nullam tincidunt orci a enim fringilla sodales. Morbi ut dui vitae justo interdum ullamcorper.
+
+[% include 'stub1.txt' ignore missing -%]
+
+EOF

template-files/data/templates/file2.txt

@@ -0,0 +1,14 @@
+# file2
+
+This is a template and expects additional context.
+
+| context | value |
+|---|---|
+| variable | [[ variable ]] |
+| optional | [[ optional or 12 ]] |
+
+Name finibus augue ac efficitur sodales. Donec facilisis lacus sit amet metus congue aliquam. Proin mollis, sapien sit amet iaculis congue, libero erat volutpat neque, in bibendum lacus massa eget eros. Mauris fringilla turpis id pulvinar ultricies. Cras dictum condimentum mauris sit amet lobortis. Maecenas ultrices in lectus eget tristique. Etiam sollicitudin tempus dolor sit amet tincidunt. Nulla nec mi eget est gravida elementum vitae vitae risus. Nullam eget urna dignissim, sollicitudin massa at, cursus urna.
+
+[% include 'stub2.txt' ignore missing -%]
+
+EOF

template-files/data/templates/stub1.txt

@@ -0,0 +1,5 @@
+# stub1
+
+Nullam blandit nisi non elit efficitur porttitor quis a justo. Sed venenatis, eros quis malesuada aliquet, ligula purus cursus velit, eget efficitur libero ante id ipsum. Aliquam et neque id justo tempus egestas quis ut augue. Curabitur eu sem velit. Phasellus sed nunc ipsum. Sed rhoncus luctus turpis, eu rutrum ipsum dictum nec. Curabitur vulputate dolor non ipsum ultrices, ut ultricies nibh vestibulum. Donec molestie scelerisque interdum. Vestibulum et finibus purus, rhoncus porttitor ligula.
+
+[% include 'stub2.txt' ignore missing -%]

template-files/data/templates/stub2.txt

@@ -0,0 +1,5 @@
+# stub2
+
+Curabitur metus libero, lacinia nec magna euismod, pellentesque lacinia sem. Nullam tempor vitae ex ut tincidunt. Etiam ullamcorper et nisi at cursus. Praesent ac mauris vel mauris ornare consequat sit amet ut magna. Integer luctus, ante at rhoncus cursus, est erat aliquam magna, et pretium urna felis eget lorem. Etiam tristique, justo viverra cursus congue, quam odio sagittis dolor, quis dictum risus tortor et turpis. Donec arcu urna, dapibus eget metus ac, consequat feugiat velit. Integer aliquet velit quis sollicitudin luctus. Maecenas aliquam arcu non urna maximus hendrerit. Integer porta efficitur blandit. Donec interdum nec metus at iaculis. Donec pellentesque enim turpis. Nulla facilisi. In hac habitasse platea dictumst.
+
+[% include 'stub3.txt' ignore missing -%]

template-files/data/upstream/python_error

@@ -0,0 +1 @@
+{{ 1/0 }}

template-files/data/upstream/stub

@@ -0,0 +1 @@
+This is a stub.

template-files/data/upstream/success

@@ -0,0 +1,6 @@
+Sample Template
+
+Destination repository: {{ dst.full_name }}
+Source repository: {{ src.full_name }}
+
+This will cause error if context is undefined: {{ variable }}

template-files/data/upstream/template_error

@@ -0,0 +1 @@
+{% include 'missing.stub' %}

template-files/requirements.txt

@@ -3,3 +3,4 @@ jsonschema
 pygithub
 pyyaml
 rich
+wrapt