Beyond Data aka laravel-data v4

[rubenvanassche]

We've started working on laravel-data v4, this will like v2 be a big one and reshape the package quite a bit:

What will be added or changed

Arrays, Collections and Paginators are now valid DataCollections

With v1 of this package we introduced DataCollection, PaginatedDataCollection and CursorPaginatedDataCollection. Though these classes will still exist, they are extremely handy to return a collection of data objects where you want to include certain properties.

As from v4, you're allowed to use arrays, collections, ... within your data objects and store data objects within them. Magic from's, includes, excludes and all other features the package now provides will still keep working.

Addition of the collect method

Each data object will now have a collect method, this method will look a lot like the collection method we're having now. It creates data objects from the items in the collection passed to it. But there will be one difference, it will return the same collection type as provided.

So providing an array of arrays to the collect method returns an array of data objects, same with Laravel collections and paginators. You'll even be able to convert collections, for example, an array to a DataCollection.

Removal of the collection method

With the addition of the collect method, the collection method becomes something strange in the new data objects, it will be a relic from the past and that's why we decided to remove it. You'll have to refactor your code to the new collect method. or you could include a trait which adds the collection back for an easy refactor, this trait will be removed in v5 so don't wait to long with refactoring your code!

Magic data collections

We had already magic from* methods on data objects which magically created data objects, now we're also adding support to define magic collect* methods which allow you to define how a collection of data objects will be constructed.

Rewritten partial trees resolving

Partial trees allow you to include, exclude, except and only certain properties in a tree of data objects. The original system dates back to v1 of the package where each data object would store what their partial trees were. This meant that including, for example, on nested data objects would be ignored. Since only the initial include would be used when transforming.

In v4, a global partial tree would be constructed without touching the individual data objects, this is a much more robust system which should have a lot of benefits in the future.

Dto and Resource

You've always defined data objects by extending from Data. From now on you will also be able to extend from Dto a minimal data object which allows you to create and validate data objects and nothing more. It is basically a more light version of Data.

We'll also introduce Resource, a data object which doesn't do validation. And can be used perfectly for resources sen't to your frontend.

The original Data class will still exists and provides all functionality, but sometimes a more light object is required.

Smaller changes:

• Custom attributes will now be included when creating data from a model
• You'll be able to define defaults in a special static method
• Calling from without parameters will be possible if you've already defined all the defaults

The maybe list

These ideas could be implemented, I'm not entirely sure if we want to though:

• Support multiple types of data objects in DataCollections, arrays, ...
• Add a prepareForValidation method which would be called in the normalizers (Add Request validation data pipe #381)
• Lazy streaming, allow some properties to be streamed when an initial payload is already sent out

Todo list

• Benchmarks show a slowdown of 50%, let's see how we can reduce this
• Split the progress of normalizing and pipelines (Add Request validation data pipe #381)
• Rewriting the docs + adding new docs
• Add a lot of new tests
• exclude, exclude_with rules not working #432

Other features?

If you have any ideas for cool new features, or things which needs to look different. Then add them to the comments, we can't guarantee these things will end up in this release but

Testing

The beyond-data branch is already available, it is very work in progress so expect things to break. But you're always welcome to test drive it!

Expected release

At the moment I have no idea, it will probably take a few more months before release. We'll keep on supporting and merging PR's to v3 for the moment, these then will be ported to v4.

PR's with breaking changes will be send to v4 so that we're having one big release with a lot of new functionality.

[mlshvdv]

@rubenvanassche what do you think about introducing the "scope" attribute (as tags for each property)?

class InvoiceDataObject extends DataObject
{
    public function __construct(
        #[WithScope(['nano'])]
        public string $id,

        #[WithScope(['nano', 'micro'])]
        public string $description,

        #[WithScope(['micro'])]
        public \DateTime $expiresAt,
    )
    {}
}

Then for transformation:

$dataObject = InvoiceDataObject::from(...)->transform(scope: ['nano'])
// or
$dataObject = InvoiceDataObject::from(...)->scope(['nano'])->transform()

The result will be:

[
    "id" => "...",
    "description" => "..."
]

[patrickomeara]

I'm only a few hours deep into the package so take this with a grain of salt.

I'm currently trying to migrate from JsonResource

I currently have different levels of the same resource that all inherit from the former one. eg. UserSummaryResource, UserResource, UserDetailResource.

I had planned on putting methods on the Data class to handle this data separation ie. UserData::from($user)->summary() where the separation method would use only or except.

The scopes mentioned above sound to me like another solution to this problem.

I guess it's a trade off between stringy arrays, and potentially messy attributes.

[genesiscz]

@patrickomeara Did you somehow manage to implement this in your project? This is sooo good!
@rubenvanassche Please, we need this! <3 Also, if you have a big model, lot of use cases, it is kinda annoying creating the fromModel method which just copies most of the fields while creating the lazy ones. This would help it quite a lot! Maybe I missed something, but how would you implement the fromModel in a way that you just re-use most of the attributes (id, name, email) but make the relations lazy/and or add a virtual field? Maybe there's a way but I haven't found a better way than this:

    public static function fromModel(FileThumbnail $fileThumbnail): self
	{
		$data = self::factory()->withoutMagicalCreation()->from($fileThumbnail);
		$data->full_url = $fileThumbnail->getUrl();

		return $data;
	}

[jameshulse]

I reckon you could implement this in your own code with a custom trait and attribute. Add the attribute like #[WithScope(...)] in the original example. Then add a custom trait to your data class. The trait could have a __call function. Use the name of the incoming function e.g. "nano" from MyObject::from($user)->nano() to search the constructor properties for attributes, and form an "include" array; then call $this->include($keysFromConstructorAttributes).

[genesiscz]

That's actually so clever! Thanks buddy!

[genesiscz]

Also a bit of offtopic, but still kinda relevant to the Lazy. When I create a job which has the Data object in it which has Lazy::whenLoaded() created in fromModel(), which in turn tries to serialize when I dispatch the job, I get an error that says I can't serialize closures. Any clues how to overcome that other than having |null in the Data object instead of just Lazy|DataCollection or Lazy|SomeModel?

[mlshvdv]

It will be useful to support the customization of paginated collection transformation (by adapter or something like it) because at present it is hardcoded:

protected function wrapPaginatedArray(array $paginated): array
{
    $wrapKey = $this->wrap->getKey() ?? 'data';

    return [
        $wrapKey => $paginated['data'],
        'links' => $paginated['links'] ?? [],
        'meta' => Arr::except($paginated, [
            'data',
            'links',
        ]),
    ];
}

For example, in our API we have this response structure at this moment. This is a little pain to transform laravel-data transform() result to our structure:

{
	"success": true,
	"data": [...],
	"meta": {
		"pagination": {
			"total": 129,
			"count": 100,
			"per_page": 100,
			"current_page": 1,
			"total_pages": 2,
			"links": {
				"next": "https://.../v1/feed/events?page=2"
			}
		}
	}
}

[rubenvanassche]

In the next version you will be able to pass on your own paginators which can generate this output. I don't think this package should actually care about how paginators are sent to the front (sadly enough in the current version this inevitable due to the PaginatedDataCollections which hook into the paginators).

[ejunker]

When writing tests I think it would be nice if there was an easy way to get a faked version of a DTO. For example, if you are creating DTOs from the Request then you would want a way to generate a DTO with faker data that you could then use in the test.

Maybe it would look something like this package for creating fake FormRequests https://github.com/worksome/request-factories

[rubenvanassche]

Like the idea! But I think this is whole separate package next to laravel-data

[jameshulse]

It would be great to get support for read-only fields. My use-case is using a JSON column to store a blob of data - but when it is read, hydrating it into a useful DTO.

This DTO could have methods to manipulate it's state, but to ensure that the state stays valid, it would be great to be able to support read-only properties.

Maybe this could just be an attribute on the constructor fields which throws an exception when a write is detected to a field.

[rubenvanassche]

Could you provide me a code sample how such a thing would look? I'm not completely following

[jameshulse]

I should have mentioned that the concept I'm after is a value object.

A simple example might be a Money type:

class Money extends Data
{
    public function __construct(
        public int amountCents,
        public string $currency
    ) {}
}

Currently with laravel-data I could use this on an Eloquent model with the $casts feature to save and retrieve the model from a JSON database field. It should save to the database as e.g total = { "amountCents": 10000, "currency": "NZD" }. But when retrieved from the DB I can access those fields directly:

$invoice = Invoice::first()
dd($invoice->total->amountCents); // returns "10000"

Now, I may want to only ever edit the invoice total by adding other Money amounts like so:

$invoice = Invoice::first();
$invoice->update([
    'total' => $invoice->total->add(new Money(100, "AUD")),
])

But currently, someone could directly edit the public amountCents or currency field, which would break the contract of the object:

$invoice = Invoice::first()
$invoice->total->amountCents += 100; // Shouldn't be possible

But with something like #[Readonly] on the class:

#[Readonly] // Because I don't think the PHP 8 'readonly' feature will work with the Data sub-class
public class Money {
    ...

Maybe the base class has something like:

public function __set($name, mixed $value): void {
    // pseudo-code
    if ($this->isReadonly) {
        throw new AttemptedToSetReadonlyPropertyException(...);
    }
}

Hope that helps. I'm not sure if I'm kind of misusing this library by attempting this, but the ease at which you can use these objects in casts or responses or whatever makes it a great candidate to build out this kind of functionality I think.

[rubenvanassche]

Something to think about: #470

[alexrififi]

It lacks the ability to add additional fields to the collection output.

That is, use the AppendableData trait in the collection.

For example:

class SongController
{
    public function index()
    {
        return SongData::collection(Song::paginate())->additional(['key' => 'value']);
    }
}

The JSON will now look like this:

{
    "data" : [
        {
            "name" : "Never Gonna Give You Up",
            "artist" : "Rick Astley"
        },
        {
            "name" : "Giving Up on Love",
            "artist" : "Rick Astley"
        }
    ],
    "key" : "value"
}

[woodspire]

This would also be nice.

#516

[innocenzi]

Benchmarks show a slowdown of 50%, let's see how we can reduce this

I think this is super important. In some situations, when dealing with a lot of data, Laravel Data (v3) has been a bottleneck so hearing it will be worse with v4 doesn't sound like good news :(

Here's a recent example, there's a 4s difference when using DataCollection instead of json when casting with Eloquent. The associated controller also takes ~700ms to instanciate the DataCollection after paginating, while only paginating takes ~150ms.

[rubenvanassche]

Data will always be slower than regular arrays, that being said a 4x increase of time is not good. I'm going to take a look why this is the case but don't have any idea right now.

[innocenzi]

Thank you! Feel free to ping me if you need any help.

[jameshulse]

I don't want to just pile on, but we had incredibly slow responses (~5-7 seconds) when using DataCollection on a large dataset (~200 items). By replacing this with plain old PHP objects in an array this came down to ~100ms. I would accept that using an advanced package for handling this could slow things down, but we couldn't justify using laravel-data any longer on these endpoints as it was just too slow. Maybe there are features we can opt out of to speed things up if we aren't using them?

An idea: it looks like DataCollection just loops over each item and runs the same transformation. Because the transformation would be identical for each iteration it could be useful to pre-calculate how the data will be transformed and apply that to each item.

[rubenvanassche]

It is out, sadly enough not with all features we've wanted to implement but that's something for another release 😄

Blogpost: https://rubenvanassche.com/hi-there-laravel-data-4/

[jameshulse]

I'm so excited to upgrade!! Amazing work getting it out @rubenvanassche.

[puyt]

I stumbled on this package because of a showcase video on YouTube, I'm mostly liking the typescript transform feature, and I'm looking for more info about the exported routes as types as shown here?

Is this something that has missed the v4 release?

[rubenvanassche]

That's more typescript-transformer territory, version 3 is almost done but it's the 5% that takes the most work. I haven't had enough time in the past months to spend on typescript-transformer due to laravel-data v4. Since that has been released, I'm going to start working on typescript-transformer 3 again.