Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand test harness's reflection approach to include "Exercisers" #568

Open
4 tasks
idearat opened this issue Dec 12, 2021 · 0 comments
Open
4 tasks

Expand test harness's reflection approach to include "Exercisers" #568

idearat opened this issue Dec 12, 2021 · 0 comments
Assignees
Labels
Authoring authoring enhancements / simplifications LIMIT not quite a bug... but frustrating/limiting TEST author tests and/or manually test

Comments

@idearat
Copy link
Contributor

idearat commented Dec 12, 2021

Our current approach relies too heavily on manual authoring of common tests. We need to leverage reflection and comments.

Introduce "exercisers", TP.test.Exerciser types which focus on exercising a particular set of objects using automated logic.

An exerciser should take in a reflection filter that lets it find the list of target objects (if not provided directly).

The exerciser then runs its exercise* methods (in random order) against that list.

Each exercise* method can be coded to reflect on the target object to get information such as input contract (from docs if a function), "default response" etc. so it can automate execution and evaluation of the function in question.

Objects being exercised should be queried via reflection (including new APIs injected by the exerciser(s) as needed) such that they can individually respond to I/O, mocking hooks, etc. during the exercise process.

Examples:

  • "is" tests (find all TP.is* methods... ask them their default response... then pass them one of everything in the system)
  • "as" tests (find all "TP.as* methods.... follow pattern above...
  • expansion tests... find all tag types and run their expand process using provided input and output xhtml strings.
  • binding tests... find all tag types, inject them with one or more binds to standard data sets, verify results
@idearat idearat added Authoring authoring enhancements / simplifications TEST author tests and/or manually test LIMIT not quite a bug... but frustrating/limiting labels Dec 12, 2021
@idearat idearat self-assigned this Dec 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Authoring authoring enhancements / simplifications LIMIT not quite a bug... but frustrating/limiting TEST author tests and/or manually test
Projects
None yet
Development

No branches or pull requests

1 participant