Improvements: Test Control

[strichmo]

Use this issue to gather all ideas to improve or highlight long-term issues with any of the following:

• Makefiles
• Directory structure
• Simulation control
• Test/regression definitions

Add comments for all ideas to this issue. Please try to refrain from deeper discussion of inidividual items. A separate issue should be opened to enable and record that discussion.

[strichmo]

Test relocatable results directory (e.g. _results) across all common Makefiles and targets)

[strichmo]

Add configuration (CFG=) as a path in all output artifacts (e.g. _results//)

[strichmo]

Consider making <RUN_INDEX> permanent part of all tests. In other words, any test could be run for multiple iterations with different seeds without requiring the _<RUN_INDEX> in the test.yaml file. For default simulations, the RUN_INDEX is 0. For example running the standard hello-world would put outputs in _results/hello-world_0

[MikeOpenHWGroup]

Here's another one: make the test-program generator (e.g. corev-dv, force-riscv) and toolchain components of the UVM environment. This would have several benefits:

• simply the compile and run-flow into a single step.
• make program generation and/or toolchain aware of the Core configuration inputs (boot_addr_i, 'mtvec_addr_i`, etc.) and thus controllable by a UVM testcase.
• state-aware instruction generation.

[MikeOpenHWGroup]

This Issue is really a discussion (which has gone stale) and has, for the most part, been implemented in the CORE-V-VERIF scriptware. I am going to close it.

Watch for a Discussion thread on this topic.

[MikeOpenHWGroup]

In Issue #2171, @murattokez asked the following:

I don't see any “default_cfg:” configuration in the test yaml files in the current repo.
Is it executable with the following command, including the one I added myself?

make test TEST=pulp_hardware_loop

No, you need to specify the configuration on the make command-line using the CFG variable. This is documented in mk/README. If you add your own build-configuration, the scriptware will assume it is located in the directory.

@murattokez also asked, what is the way to understand which “CFG” is needed for which test?

This is not as well documented as it should be. If you look in the cv32e40p/regress directory you will see a number of yaml files that are intended to be used as input control to the cv_regress script. This is the best place to see the relationship between specific test programs (as selected with the TEST variable) and build configurations (as selected by the CFG variable). Instructions for running the regression script for the README of the cv32e40p/regress directory.

Also (and you probably already know this), to use the configuration of the CV32E40P that enables the PULP and F ISAs, you need to be using the cv32e40p/dev branch of CORE-V-VERIF.