Why Smarty is dropping support for the use of unregistered callables (such as PHP functions) in templates

[wisskid]

Starting from Smarty v5.0.0, it is no longer possible to use unregistereded callables (such as PHP functions) in templates as modifiers or as functions in expressions. To use the callables, you need to register them first. Preferably in an Extension or in the old-fashioned way by using $smarty->registerPlugin("modifier", "strrev", "strrev").

To assist users in making the transition to Smarty 5, Smarty 4.5 has introduced deprecation notices to warn users when they are using unregistereded callables in their templates. Deprecation notices are not errors and they don't mean your code is broken (yet). They are helpful to assist you in making the transition to Smarty 5, should you ever wish to do so. Read more about dealing with deprecations if you want to learn more.

But why?

Supporting changes in PHP

Some people have expressed frustrations about the decision to drop support for unregistered callables and wonder why Smarty is doing this. This is discussed in length in Issues and PRs, such as #813. To summarize: many users have filed bug reports to "fix" compatibility issues in modifiers that were never part of the Smarty language. For example: in PHP's join or implode it used to be possible to switch separator and array. When using join or implode as a modifier, it makes sense to switch them for readability. However, since PHP8.0 this is no longer allowed. Is this a bug in Smarty now?

Security and separation of concerns

Not relevant for everybody, but supporting each and every callable can be a nightmare when you need a strict separation between front-end developers (possibly even anonymous users) and PHP-developers. Allowing callables means you can do funny stuff like:

{"pwd"|passthru}
{"whoami"|passthru}

And yes, Smarty has security features that are meant to block stuff like this, but we felt it was better to stop this at the beginning.

Stability

One of the goals of Smarty is to provide a stable interface. This means we try to have automated testing for every language element for every supported version of PHP. If we do not somehow limit the amount of functions template authors can use, this is impossible.

Other template engines, such as Twig for example, follow the same principle.

A dirty fix

If writing a proper Extension feels as too much work, we have an easy fix for Smarty 5.

class WildcardExtension extends \Smarty\Extension\Base {

	public function getModifierCallback(string $modifierName) {
		if (is_callable($modifierName)) {
			return $modifierName;
		}
		return null;
	}

}

and

$smarty->addExtension(new WildcardExtension());

It's not pretty, but at least this makes it rather clear who is responsible for supporting the callables.

[olelis]

Thank you for starting this as a discussion!

If I understand correctly, then the main reasons are:

1. Some user are complaining that when some function in the php changed order/logic, and now Smarty is not working(even if it is not related to smarty at all)
2. Security point - so there will be some way of limiting what functions frontend developers can use

Both are valid points, but I do afraid that this solution raises more problems that it solves.

For example, when this update will be implemented, then it will mean that there will be even more complains from users that now Smarty is not working at all (and this time it will be Smarty, not PHP version change). Ok, there is a workaround, but it means that every developer in the world will have to find this workaround and copypaste it to the code. Is it worth it?

Also, I am not sure that the main behavior is that only limited list of functions is allowed. During the years (I still have my project where I have used Smarty around 2010) , developers are got used to be able to use native/userland functions directly in the Smarty code. Is it good/bad ? Hard to say.

Also, it will be quite hard to find all functions used, especially for the old, large codebase, which means that wildcard solution would be de facto standard for the old code.

Other platforms

I have checked Laravel Blade, and looks it actually supports native/userland functions out of the box. For example:
{{mt_rand(1,200)}} {{file_get_contents('test.txt')}}
is converted to
<?php echo e(mt_rand(1,200)); ?><?php echo e(file_get_contents('test.txt')); ?>

Looks like Twig does not natively support it, correct me if I am wrong.

Proposed solution

Should this new behavior be optional, similar how muteUndefinedOrNullWarnings was implemented?
So if I want to use "secure" way, then I can opt-in for it. Otherwise, I will be able to use all functions .
This will not help with the issue 1, you raised, but it will allow to have sandboxed Smarty, which is good

This solution is kinda like implementing "dirty solution" you have above, but inside Smarty itself. This will simplify life of many developers in the world.

[wisskid]

It's not that hard to find them. Several applications have alreay done so. Someone has even developed a regex that you can use in your IDE to search. Also, this is where the depreciation notices come in handy.

And if all of that is not sufficient for your usecase, you can even create a wildcard extension that throws warnings.

[olelis]

@wisskid, it Is still extra work, that should be repeated for all existing project. And this means that every developer will have to find workaround OR find all such functions. I still don't see why you force all developers to do this work.

However, based on this and other discussions, looks like your decision is final and you really think that this is the only option. In other words, looks like there is no points of discussion.,

[lkppo]

that should be repeated for all existing project. And this means that every developer will have to find workaround OR find all such functions. I still don't see why you force all developers to do this work.

However, based on this and other discussions, looks like your decision is final and you really think that this is the only option. In other words, looks like there is no points of discussion.,

Do you say the same thing when the authors of PHP introduce incompatible modifications with each new version, each year? To a technical decision you respond with feelings. Place your arguments at the same level if you want a chance to debate.

These modifications are justified, documented and announced. It's great work.

[olelis]

To a technical decision you respond with feelings.

Actually not. My comment about that there is no room on to discussion is based on reading other issue, for example #813 (I did found it between my first and second post). In the issue, similar concerns/questions were raised in Jan-Jun 2023, without affecting anything.

Also, there are already tickets that that are based on the fact that Smarty don't allow native/userland functions by default. For example #896, #959, #961. And I do believe, that there will be more of them (well, strictly speaking, this sentence is feeling, not a fact, I can't predict the future).

Do you say the same thing when the authors of PHP introduce incompatible modifications with each new version, each year?
Sometimes I don't like the changes, but at least I can always find solid reasoning why the change is done and long discussion with voting about pros and cons. This is especially true if the change actually affects a lot of people.

In addition to that, I see that quite a lot of PHP changes are rejected based on the "backward compatibility" issue. PHP developers actually try to make sure that the old code works without major changes, if this even possible. I would say that in the since php 7.4, the only major change that actually broke functionality (for me) was about strict string comparisons, other changes didn't break anything/much.

[Ninjinka]

First af all thanx for the hard work, we use smarty since the first release and followed the development and sometimes contribuited too (before the current team take the lead).

This i think is a major issue, we use smarty in around 200 projects.
Commonly 1 time every year we update our main library for all the projects, this includes every composer plugin.
Of course we cannot follow every single library decision, we "hope" they mantein retro-compatibility or made only small changes every time so we can adjust everything without giving big downtime.

This huge changes broke completely every single application so we should stay on old version since we cannot use 3-4 weeks time following every single change and also making single modifier for every single php function.

We will wait some time but if retro-compatibility disappear we should search for a new library or develop our own or maybe fork the latest working release, and i think many will do same..

Sorry to have bothered with this reflection..

[ssigwart]

That's good news.

In our case, assign cannot be used as a direct replacement of assignByRef. What we do is something like this (with much cleaner and safer code) in many places:

$originalFormData = [
	'field1' => 'abc'
];
$tpl->assignByRef('originalFormData', $originalFormData);

if (isHttpPost())
{
	$originalFormData['field1'] = $_POST['field1'];
}

By doing this, if there's a form error and we don't redirect the user, the TPL file will refill the template with the data the user entered.

These are fairly easy to fix by switching to assign and putting that below the POST handling, but still requires tons of testing.

Our problem is that we have other places where we assign array class properties by reference. So anywhere in the class or subclasses we want to add a field to that array, it will automatically be available in the TPL file. So those aren't easy fixes. I'll give you that it's not ideal code, but we've been using Smarty for 15 years, so we've got some stuff that we wouldn't do today, but trying to find and clean up the old code just isn't feasible.

[wisskid]

I see. That would complicate things a bit indeed. Maybe you could try using an ArrayObject instead? That would behave like an object but be passed by reference by default.

Meanwhile, I think #964 should be fixed by #985.

[ssigwart]

That's a good suggestion, but didn't seem to work when I tried this:

$this->tpl->assign('myArray', new ArrayObject($this->myArray));

It doesn't seem to be by reference, so the TPL file doesn't see changes after that.

I could potential change the type of the class parameter to ArrayObject, but that has it's own set of issues in that I still need to update too many spots that use it including function type hints.

One workable solution might be to create my own ArrayObjectByRef class, but we also use it for nullable fields and non-array fields too.

[wisskid]

This is what I had in mind and what it outputs:

class DummySmarty  {
	private $data = [];
	public function assign(string $varname, $value)	{ $this->data[$varname] = $value; }
	public function getVariable(string $varname) { return $this->data[$varname]; }
}

$smarty = new DummySmarty();
$formDataArray = ['a' => 'A', 'b' => 'B'];
$formDataObject = new ArrayObject($formDataArray);
$smarty->assign('formdata', $formDataObject);

// pre-change
var_dump($smarty->getVariable('formdata'));

$formDataObject['b'] = 'C';

// post-change
var_dump($smarty->getVariable('formdata'));

output

object(ArrayObject)#2 (1) {
  ["storage":"ArrayObject":private]=>
  array(2) {
    ["a"]=>
    string(1) "A"
    ["b"]=>
    string(1) "B"
  }
}
object(ArrayObject)#2 (1) {
  ["storage":"ArrayObject":private]=>
  array(2) {
    ["a"]=>
    string(1) "A"
    ["b"]=>
    string(1) "C"
  }
}

[ssigwart]

Yes, that would work if we could swap the array for ArrayObject, but that's not really viable for us. We use type hints extensively, so if we swapped out our array property for an ArrayObject and then tried calling, function doSomething(array $arr): void, we'd get a TypeError.

[marclaporte]

we use smarty in around 200 projects

Wow, this is huge!

[KarelWintersky]

However, since PHP8.0 this is no longer allowed.

Why? What change in PHP 8 prohibits the use of this mechanism? I don’t see any prohibitions in the changelog:

https://php.watch/versions/8.0

[KarelWintersky]

However, if these changes allow you to quickly implement other PHP 8 features that I requested - well, say that “this is the price of future changes” and I will accept that the inconvenience in one place will outweigh the convenience in another.

I'm talking about:
#864

[Ninjinka]

Hello!
You are limiting the use of php function, why in the world you should test for example nl2br or stripslashes or array_pop, and why i should not be allowed to use 'em? And why should i use a "dirty fix" if the engine need a fix release it...

If you want to "limiting the amount logic that can be used in templates" you should block the creation of new variables and data modification too (so the = symbol)

[KarelWintersky]

@Ninjinka I have a guess what the true reasons for such changes may be, but they lie in the field of psychology and go beyond our professional sphere. Not being a professional in the field of psychoanalysis, I will not divulge them. I hope I'm wrong and this isn't the reason.

[wisskid]

This isn't helpful. Lets try to keep it nice here.

[KarelWintersky]

@wisskid , sorry. Is my arguments valid ?

[KarelWintersky]

we have an easy fix for Smarty 5.

Is there a similar solution for smarty 4?

[wisskid]

Exactly, thank you. 🙏
This is what deprecations are for. Everyone uses them, including PHP itself. This is also why I included a link to https://stitcher.io/blog/dealing-with-deprecations above for more backgroud.

[KarelWintersky]

@wisskid

But we can't disable deprecation notices only for smarty (yes, knowing that the functionality is deprecated and agreeing to it), but leave these notifications for other packages.

This setting is global for the entire interpreter.

// $SMARTY->registerPlugin(Smarty::PLUGIN_MODIFIER, 'count', 'count');

$SMARTY->setErrorReporting(E_ALL & ~E_NOTICE & ~E_DEPRECATED);

Does not work with message

Using unregistered function "count" in a template is deprecated and will be removed in a future release. Use Smarty::registerPlugin to explicitly register a custom modifier.

P.S. Smarty version 4.5.2

[olelis]

@KarelWintersky , this is the same issue I had yesterday, please, add & ~E_USER_DEPRECATED as in my example above.

[KarelWintersky]

@olelis , @wisskid , absolutely not obvious :-(

[olelis]

@KarelWintersky You should blame PHP for this, not Smarty 😊
For example, pure php code:

error_reporting(E_ALL & ~E_DEPRECATED & ~E_NOTICE);
trigger_error('Deprecated',E_DEPRECATED);
trigger_error('Deprecated 2',E_USER_DEPRECATED);

First one will hidden, while second one not. I do understand some reasoning for this, but yes, it is counter-intuitive.
Nevertheless, it is not Smarty issue.

[Amaury]

I have two related questions.
From the extension documentation, I understand that the addPluginsDir() method is deprecated. Why? It is really useful to just have to put a file in a directory to be able to use a modifier (for example). Having to declare each extension forces us to modify our framework. (I know there is the BCPluginsAdapter extension, but it doesn't seem to be a sustainable way to solve the problem)

The Smarty 5 documentation still show the count modifier although it is no longer available. Maybe it should be updated.
But at the same time, the count function exists; why is it not available as a modifier?

[wisskid]

PHP has been moving away from file_exists and require_once for a while now to using autoloading, opcode caching and dependency injection. Also, the core code becomes much more elegant (in my opinion) if it can query an Extension directly instead of scanning directories. But that's more an opinion than a fact.

The |count modifier should be available. It's in src/Extension/DefaultExtension.php:57 and there are unit tests using it. What makes you think it isn't available?

[Amaury]

I tested the count modifier again (after deleting the compiled files) and it worked fine… My mistake, sorry for the trouble on that.

Regarding the removal of directory scanning, I might agree if there was an autoloading-based system so that we didn't need to declare every extension we wanted to use.
In the Temma framework, we always allowed developers to use Smarty extensions (modifiers, tags…) in their projects, simply by adding the appropriate files to a folder, without having to declare them one by one.

[dipeshsukhia]

Hi @wisskid

As per discussion #967, Smarty 4.5 has introduced deprecation notices to warn users about the usage of unregistered callables in templates.

To prevent these warnings from cluttering the logs, we've implemented a fix that suppresses these deprecation notices by utilizing the muteUndefinedOrNullWarnings function for Smarty 4.5.

This change ensures that deprecated notices for unregistered callables are properly handled, preventing unnecessary interruptions
in Smarty 4.5.

Please review PR #1067 for the same.

[Sarke]

Hi, I want to add our frustration to this too. We have 550+ sites (through some no longer active) that we've been slowly upgrading from Smarty 3 as needed. I get the concerns raised by the maintainers, but I think it needs to be easier to not have to deal with these deprecation notices.

As a basic example, we have a function called empty_date() that acts basically like empty() but checks from things like 0000-00-00 too. Either way, it could be a built in PHP function too, doesn't matter. Here's how we could use it:

{if empty_date($date)}
    (no date)
{/if}

So, let's say we have a list of these functions:

$safe = ['empty_date', 'ltrim', 'whatever']; // there's a bunch, but as an example

What DOESN'T work:

$security = new \Smarty_Security($smarty);
$security->php_functions = $safe;
$smarty->enableSecurity($security);

Why doesn't that work?

As an aside, $security->php_functions is used by $security->isTrustedPhpFunction() which is itself also deprecated, but it used to raise the above deprecation notice. Kinda ironic.

This DOESN'T work either:

foreach ($safe as $function)
{
    if (!function_exists($function))
        continue;
	
    $smarty->registerPlugin(\Smarty::PLUGIN_FUNCTION, $function, $function);
}

Why doesn't that work?

So after some not inconsiderable frustration and digging through the Smarty code, we found that this DOES work:

foreach ($safe as $function)
{
    if (!function_exists($function))
        continue;
	
    $smarty->registerPlugin(\Smarty::PLUGIN_MODIFIER, $function, $function);
}

Why?

smarty/libs/sysplugins/smarty_internal_templatecompilerbase.php

Lines 658 to 666 in c11676e

	if (
	!$this->smarty->loadPlugin('smarty_modifiercompiler_' . $name)
	&& !isset($this->smarty->registered_plugins[Smarty::PLUGIN_MODIFIER][$name])
	&& !in_array($name, ['time', 'join', 'is_array', 'in_array', 'count'])
	) {
	trigger_error('Using unregistered function "' . $name . '" in a template is deprecated and will be ' .
	'removed in a future release. Use Smarty::registerPlugin to explicitly register ' .
	'a custom modifier.', E_USER_DEPRECATED);
	}

Why does registering these FUNCTIONs that are use as FUNCTIONs need to be registered as MODIFIERs?

I get that if we were using them like {$date|empty_date} it should be a modifier, but we're not.

Of the reasons given for this change, there is still no justification for this amount of BC and frustration, and no reason why there shouldn't be a simple AND DOCUMENTED method of whitelisting functions.

$smart->registerFunctions(['empty_date', 'ltrim', 'whatever']);

That's all it should take.

We had a similar issue with static classes, but it could be solved with $smarty->registerClass().

[wisskid]

Please see the section "But why?" in my intro for this discussion for reasons.

[wisskid]

And also see "A dirty fix" for a workaround if you still don't agree.

[Sarke]

I know the reason why, and I don't disagree with it. In fact, we used the Security features before to only allow certain functions. My issue is with the pain in allowing them again. We don't want to allow everything. We didn't before. It's a good move but poorly executed, and it seems we're being punished unnecessarily.

[Sarke]

And my most recent issue above was with the fundamental differences between functions and modifiers, and why in order to use a function it needs to be registered as a modifier even though we don't want to use it as a modifier but as a function.

[wisskid]

There is no fundamental difference between a modifier and a function. They are two types of syntax for the same thing. So you won't have to reimplement both as they are one and the same.