Replacing Matchstick tests with queries

[EmperorOrokuSaki]

The Graph's official documentation recommends developers use the Matchstick testing framework to test their subgraphs. However, this framework only allows us to do mainly three things:

1. Update the store by manipulating events
2. Mock contract calls
3. Mock IPFS files

In my experience, keeping the Matchstick tests updated with the newest changes to the contract ABI and the subgraph is a time-consuming task that does not provide much value over using actual queries and testing them directly by using clients such as Apollo or the GraphClient.

How they differ from one another

E2E testing

With Matchstick tests, we are directly creating events and manipulating them. In my opinion, that is something we should avoid since events are supposed to be emitted by functions after performing specific actions. We can mock contract calls with the framework but in order to use that feature, a live version of the contract is needed.

Currently, all Matchstick tests in the project are literally just creating events, handling them by calling the handler function in the mapping file, and then doing assertions on their metadata. The only part of the subgraph that is tested with the current implementation is the mapping file and its handlers. With queries, we can perform E2E tests that rely on the main live version of the Subgraph that is going to be called by our front-end and other services that might integrate our subgraph into their stack.

Efficiency

The process of writing the function that creates the events based on the ABI can become unnecessarily time-consuming (examples). A simple mistake in the order of the parameters can send you down a rabbit hole. The compiler always gives ambiguous errors. These functions will not be needed for query testing the Subgraph. In that case, the queries and the testing environment are all that's needed.

Queries based on actual needs

Queries let us test the actual functionalities we need from the Subgraph by specifying the return values and filters. On the other hand, Matchstick does not even allow access to a certain entity without passing its id. Considering how important retrieving information based on different fields and conditions is for our project, I believe this is a great disadvantage.

My proposal

A deployed instance of the Subgraph is needed with queries, whereas Matchstick tests can be run locally without deploying the Subgraph to the Graph Network or the Hosted Service. But this is not a disadvantage of using queries because what matters the most is consistency between tests and the live version of the Subgraph. This consistency can be achieved by deploying the newest version of the Subgraph with every release using a GitHub custom workflow and then running the tests.

Using GitHub workflows to make automatic deployments of the Subgraph to the Hosted Service has been the plan since the first stages of the Subgraph development. So, this will only build on what we have already considered being a part of our plan for the NFA's Subgraph.

The workflow can first, update the test Subgraph instance and then if the tests pass, deploy the new version to the main Subgraph instance on the Hosted Service.

We can use other testing frameworks to create a simple testing environment. The GraphClient package can be utilized to perform queries on the Subgraph, like a real application that has integrated the Subgraph into the stack.

[jsonsivar]

Saving this for later in case but it might be irrelevant if we remove matchsticks. #126.