NEW Validate DBFields

[emteknetnz]

Issue #11403

• Adds the new FieldValiator class and multple subclasses
• Adds a mechanism for adding FieldValidators to DBFields and FormFields, and calling them when appropriate
• Updates some setValue() methods on DBField subclasses to convert values e.g. string integers to integers. This is mainly done to soften some of the upgrade pain now that just about everything is being validated.
• Adds new DBField types - DBEmail, DBIp, DBUrl
• Adds a $value param to ValidationResult. This is mainly to make debugging easier.

Note that while this PR adds support to FormField for FieldValidators, I haven't actually added the FieldValidators to the FormField's in this PR, as too much would be changing at once and many things would break in CI. Instead I've split off a separate issue for that to be done after this has been merged Have removed FormField changes

There is some type conversion going on in DBField::setValue(), which happens before validation, though I'm trying to not be too permissive as this PR is essentially adding strict typing to DBFields. Some examples:

• DBBoolean allows a wide range of values for backwards compatibility e.g. 't' will get converted to true
• DBInt will convert numeric ints to int. One reason for allow this is because HTTP form field submissions are strings, so it's required when doing things like $form->saveInto($dataObject);
• DBDecimal will convert int to float, because not allowing $obj->MyDecimal = 5; is pointlessly annoying as we're not loosing any precision. Note the opposite is not allowed i.e. there is no conversion when going $obj->MyInt = 5.1, as this would truncate the decimal place. This will trigger a validation exception instead.

A few things that we might want to consider changing (easiest done as follow up cards):

• DBBoolean should possibly stop accepting values such as 't' will retain backward compatibility
• DBDate currently accepts a range of dates and attempts to fix things. Changes were made in this PR, though one not unexpected and not great side effect is that a value that previously threw an error during DBDate::setValue() "03-03-04" now gets turned into "2003-03-04". The existing behaviour of turning "0" into "1970-01-01" remain. Perhaps we should be stricter and only allow 'Y-m-d' style inputs. Have reverted changes, now behaves like CMS 5
• DBTime::parseTime() uses strtotime() so will parse things like "23.02.56" and "11:02 pm". We should consider forcing "H:i:s" format for all inputted strings
• DBString subclasses e.g. DBVarchar will not convert int/float to string. The intention here is a user may mistakenly be assigning ints/floats to a Varchar field and then later attempt numeric operation e.g. adding, subtraction. Argument could be made that we should convert here. (opposite of above points, i.e. be "more helpful" rather than "more strict")

Also to answer the question "why are we even doing this when you can just get the database to do this?". It's because we cannot give any guarantees is MySQL is set in strict mode (will complain and not update value when invalid) vs not in strict mode (will truncate and insert). Also, FieldValidator is an abstract class that will be shared with FormField validation, which doesn't even touch the database. Also, things like EmailFieldValidator can't be validated by MySQL.

[GuySartorelli]

There's a bunch of stuff still not addressed.
I've decided to let most of them slide, but the following at the very least need to be responded to:

• NEW Validate DBFields #11408 (comment)
• NEW Validate DBFields #11408 (comment)
• NEW Validate DBFields #11408 (comment)

[emteknetnz]

@GuySartorelli after rebasing in the template work I needed to make a small change to a test in CastingServiceTest.php as DBCurrency extends DBDecimal and getValue() now returns a float, previously the test was expecting a string

[GuySartorelli]

Looks mostly good, but found a few anomolies while testing

[GuySartorelli]

In testing I'm able to enter values like "this is not a boolean" and instead of getting a validation error, the value just gets turned into 1.

This seems to be related to setting the value from forms specifically - setting the value programatically results in the appropriate ValidationException.

[emteknetnz]

That's behaving as expected due to prepValueForDB() being called before DBField->validate()

public function prepValueForDB(mixed $value): array|int|null
    {
        $bool = $this->convertBooleanLikeValue($value);
        // Ensure a tiny int is returned no matter what e.g. value is an
        return $bool ? 1 : 0;
    }

Once we put on FormField validation, these validation message will be surfaced in the CMS

[GuySartorelli]

That's behaving as expected due to prepValueForDB() being called before DBField->validate()

Seems to me like those are being called the wrong way around. Validation should be performed on the value that is currently stored. prepValueForDB() should then operate on values which have been validated and are known to not be garbage and is only used to (as the method name says) prepare the valid value for storage in the database. It should not be used to transform invalid values into valid ones because that goes against the whole purpose of the validation.

tl;dr "this is not a boolean" is not a boolean value, so setting that value in a DBBoolean field in a DataObject record and then calling write() on that record should result in a validation exception being thrown.

[emteknetnz]

Apologies, confused myself, DBField->validate() is called BEFORE DataObject::prepValueForDB()

They're both called from DataObject::write() - order below with the call at the top being first

DataObject::write()
DataObject::preWrite()
- DataObject::validateWrite()
- DataObject::validate()
- DBField::validate()
DataObject::writeBaseRecord() / DataObject::writeManipulation()
- DataObject::prepareManipulationTable()
- DBField::writeToManipulation()
- DBField::prepValueForDB()

values such as 't' are converted to the appropriate bool in DBBoolean::setValue() BEFORE validation which retains backwards compatibility

In testing I'm able to enter values like "this is not a boolean" and instead of getting a validation error, the value just gets turned into 1. .. This seems to be related to setting the value from forms specifically

Just ignore any 'setting via forms' for now as that's out of scope for this card, and is being handled in a follow up card. We only need to be concerned with setting programmatically for now

[GuySartorelli]

We only need to be concerned with setting programmatically for now

In that case I'm going to need to add some ACs to the formfield card (which we didn't refine or else we would have probably caught this at the time)

[GuySartorelli]

As discussed in person, the concern here is when someone uses a form field other than the one that's scaffolded. e.g. in the case above I was using a TextField. The value from that text field was definitely 100% not a valid boolean, and the form field validation wouldn't catch that.

We need to find out why the value was being transformed, and make sure it instead gets correctly validated by the DBField validation.

You mentioned in our discussion you probably want to do it separately to this PR - in that case please open a new card for it.

[emteknetnz]

That particular instance is is happening in DBBoolean::saveInto(), which only happens as part of saving a form in the CMS - $dataObject->__set($fieldName, (bool) $this->value);

I'll open a third card for one as I think it doesn't quite fit on the FormField one, as this is a DBField interacting Forms issue which is a little different