Datasets preprocessing

[fabio-ivona]

main idea

Following the ideas and neets stated in #469 and #319, I think we could work on a dataset preprocessing feature that will allow to manipulate a dataset before using it in a specific test

the main sintax could be:

test('run on a dataset sample', function($postcode){

    $cityFinder = new CityFinder($postcode);
    expect($cityFinder->find())->not->toBeNull();

})->with(dataset: 'cities', before: function($citiesDataset){
    return Collection::make($citiesDataset)->random(10)->only('postcode')->toArray();
});

this will result on the test being executed on a subset of 10 elements of the 'cities' dataset, extracting the 'postcode' key of each entry

implementation

this could be achieved by storing the preprocessing closure along with the dataset inside DatasetRepository and applying it inside DatasetRepository::processDatasets

note

this would target Pest V2, as it will result in a breaking change as now ->with(..) can accept a variadic argument to define a dataset matrix, with this feature it won't anymore

the dataset matrix feature will still be available by calling ->with() multiple times:

test('...', function(){
    //test 
})->with('dataset1', preprocess: function(){...}
  ->with('dataset2);

[lukeraymonddowning]

I like it. Although think that logic should be deferred to the dataset. How about if any params to 'with' after the first are passed to the dataset? That way, you can handle it however you want?

[fabio-ivona]

hmm, this will work only if your dataset is generated by a closure, right?

[lukeraymonddowning]

Yes, but I think that's fine. It wouldn't be hard to change to a closure where required and would remove duplication.

[fabio-ivona]

I like this approach, this way you can also pass parameters for dataset generation

@nunomaduro what do you think?

[lukeraymonddowning]

To externalise the conversation we've had internally, it would be nice to support collection style named parameters for common processes:

->with('users', pluck: 'email')
->with(Status::cases(), except: Status::completed())

[fabio-ivona]

I'll just add some more detail to what @lukeraymonddowning reported:

given this dataset:

dataset('my dataset', function(int $count = 3, bool $uppercase = false){
    $dataset = Collection::make([
        'key1' => ['name' => 'foo', 'title' => 'mr'],
        'key2' => ['name' => 'bar', 'title' => 'mrs', 'age' => 21],
        'key3' => ['name' => 'baz', 'title' => 'ms'],
    ]);

    if($uppercase){
           $dataset = dataset->map(function($value){
                  $vaue['name'] = strtoupper($value['name']);
          });
    }

    return $dataset->random($count);
});

with the implementation me and @lukeraymonddowning are planning, this feature will allow you to:

->with('my dataset', 2)      // [
                                 ['name' => 'baz', 'title' => 'ms'],
                                 ['name' => 'foo', 'title' => 'mr'],
                             ]


->with('my dataset', uppercase: true) // [
                                 ['name' => 'foo', 'title' => 'mr'],
                                 ['name' => 'bar', 'title' => 'mrs'],
                                 ['name' => 'baz', 'title' => 'ms'],
                            ]

// plus some common processes


->with('my dataset', only: ['key1', 'key2']) // [
                                 ['name' => 'foo', 'title' => 'mr'],
                                 ['name' => 'bar', 'title' => 'mrs'],
                              ]



->with('my dataset', except: 'key2') // [
                                ['name' => 'foo', 'title' => 'mr'],
                                ['name' => 'baz', 'title' => 'ms'],
                              ]

->with('my dataset', pluck: 'name') // [
                                 'foo'
                                 'bar'
                                 'baz'
                               ]

->with('my dataset', map: fn($element) => count($element)) // [
                                                                  2,
                                                                  3,
                                                                  2
                                                                ]

[nunomaduro]

Does this work without named arguments?

[fabio-ivona]

@nunomaduro can you elaborate?

[nunomaduro]

Can I do this?

->with('cars', fn ($car) => $car->name);

[fabio-ivona]

it does work without named arguments, (the preprocess: named argument in my example was put there only to clarify the example)

but the initial idea was to get the entire dataset in the closure, so your example should be

->with('cars', fn($cars) => array_map($cars, fn($car) => $car->name))

if we instead apply the closure to each of the dataset entry, you could not use this feature to extract n samples from your dataset