microsoft-psrp securestring jinja2.exceptions.TemplateAssertionError

[noderat]

Apache Airflow Provider(s)

microsoft-psrp

Versions of Apache Airflow Providers

apache-airflow-providers-microsoft-psrp==2.8.0

Apache Airflow version

2.10.2

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Deployment details

Using official helm chart and the following slightly extended Dockerfile:

FROM apache/airflow:2.10.2
RUN pip install --no-cache-dir "apache-airflow[microsoft-psrp]==${AIRFLOW_VERSION}" \
    --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-3.9.txt"

What happened

jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'

It appears the securestring template filter is not working as intended.

Summarized Error:

ERROR - Exception rendering Jinja template for task 'Write-Output', field 'arguments'. Template: ["{{ 'foo' | securestring }}"]
  ( ... lengthy stack trace redacted ... )
jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'.

Full execution log including the redacted stack trace:

[2024-11-13, 16:11:36 MST] {taskinstance.py:2865} INFO - Starting attempt 1 of 1
[2024-11-13, 16:11:36 MST] {taskinstance.py:2888} INFO - Executing <Task(PsrpOperator): Write-Output> on 2024-11-13 23:11:35.867453+00:00
[2024-11-13, 16:11:36 MST] {warnings.py:112} WARNING - /home/airflow/.local/lib/python3.12/site-packages/airflow/task/task_runner/standard_task_runner.py:70: DeprecationWarning: This process (pid=14633) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()
[2024-11-13, 16:11:36 MST] {standard_task_runner.py:72} INFO - Started process 14644 to run task
[2024-11-13, 16:11:36 MST] {standard_task_runner.py:104} INFO - Running: ['airflow', 'tasks', 'run', 'psrpoperator_securestring_error_reproduction', 'Write-Output', 'manual__2024-11-13T23:11:35.867453+00:00', '--job-id', '802', '--raw', '--subdir', 'DAGS_FOLDER/psrp_operator_securestring_error_repoduction.py', '--cfg-path', '/tmp/tmpwrdenr76']
[2024-11-13, 16:11:36 MST] {standard_task_runner.py:105} INFO - Job 802: Subtask Write-Output
[2024-11-13, 16:11:37 MST] {task_command.py:467} INFO - Running <TaskInstance: psrpoperator_securestring_error_reproduction.Write-Output manual__2024-11-13T23:11:35.867453+00:00 [running]> on host experimental-temp-airflow-worker-0.experimental-temp-airflow-worker.experimental-temp-airflow.svc.cluster.local
[2024-11-13, 16:11:37 MST] {abstractoperator.py:778} ERROR - Exception rendering Jinja template for task 'Write-Output', field 'arguments'. Template: ["{{ 'foo' | securestring }}"]
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/abstractoperator.py", line 770, in _do_render_template_fields
    rendered_content = self.render_template(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 183, in render_template
    return [self.render_template(element, context, jinja_env, oids) for element in value]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 170, in render_template
    template = jinja_env.from_string(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 1108, in from_string
    return cls.from_code(self, self.compile(source), gs, None)
                               ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 768, in compile
    self.handle_exception(source=source_hint)
  File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'.
[2024-11-13, 16:11:37 MST] {taskinstance.py:3310} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 273, in _run_raw_task
    TaskInstance._execute_task_with_callbacks(
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3114, in _execute_task_with_callbacks
    task_orig = self.render_templates(context=context, jinja_env=jinja_env)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3533, in render_templates
    original_task.render_template_fields(context, jinja_env)
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 1419, in render_template_fields
    self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/abstractoperator.py", line 770, in _do_render_template_fields
    rendered_content = self.render_template(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 183, in render_template
    return [self.render_template(element, context, jinja_env, oids) for element in value]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/template/templater.py", line 170, in render_template
    template = jinja_env.from_string(value)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 1108, in from_string
    return cls.from_code(self, self.compile(source), gs, None)
                               ^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 768, in compile
    self.handle_exception(source=source_hint)
  File "/home/airflow/.local/lib/python3.12/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'.
[2024-11-13, 16:11:37 MST] {taskinstance.py:1225} INFO - Marking task as FAILED. dag_id=psrpoperator_securestring_error_reproduction, task_id=Write-Output, run_id=manual__2024-11-13T23:11:35.867453+00:00, execution_date=20241113T231135, start_date=20241113T231136, end_date=20241113T231137

What you think should happen instead

The minimal example provided should return through XCom the ToString representation of a powershell SecureString which would be "System.Security.SecureString"

How to reproduce

The following DAG can be used to reproduce the issue and does not require a remote connection as the task fails before attempting any connection:

import datetime
from airflow import DAG
from airflow.providers.microsoft.psrp.operators.psrp import PsrpOperator

with DAG(
    dag_id="psrpoperator_securestring_error_reproduction",
    start_date=None,
    schedule=None,
    render_template_as_native_obj=True,
):
    PsrpOperator(
        task_id = "psrp_task",
        psrp_conn_id = "SomeConnection",
        cmdlet = "Write-Output", 
        arguments = [ "{{ 'foo' | securestring }}" ] # Throws jinja2.exceptions.TemplateAssertionError: No filter named 'securestring'.
    )

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

I think you misunderstood how it works. Jinja templates that you mention are resolved on the worker (which is Linux - based) not windows. PSRP operator will execute PowerShell code that is result of this Jinja template processing.

[noderat]

I can't find a situation where the get_template_env method() is ever called in my rudimentary testing because render_template_fields() has a if not jinja_env: jinja_env = self.get_template_env() conditional seen at

airflow/airflow/models/baseoperator.py

Lines 759 to 760 in 47cdb84

	if not jinja_env:
	jinja_env = self.get_template_env()

and at least in my testing jinja_env is always passed in and therefore truthy.

The only other case of an operator adding extra jinja filters I could find was the sql to slack webhook and it instead extends render_template_fields() as seen in

airflow/providers/src/airflow/providers/slack/transfers/sql_to_slack_webhook.py

Lines 137 to 152 in 47cdb84

	def render_template_fields(self, context, jinja_env=None) -> None:
	# If this is the first render of the template fields, exclude slack_message from rendering since
	# the SQL results haven't been retrieved yet.
	if self.times_rendered == 0:
	fields_to_render: Iterable[str] = (x for x in self.template_fields if x != "slack_message")
	else:
	fields_to_render = self.template_fields
	if not jinja_env:
	jinja_env = self.get_template_env()
	# Add the tabulate library into the JINJA environment
	jinja_env.filters["tabulate"] = tabulate
	self._do_render_template_fields(self, fields_to_render, context, jinja_env, set())
	self.times_rendered += 1

Can this be converted back into an issue, please?

[noderat]

@potiuk @malthe I posted a suggested solution because I do believe this is a bug and that the securestring jinja filter, at least in newer versions of airflow, will never be made available as written.

[malthe]

What's weird is that the automated tests are working fine (I presume).

Note that it's not possible to convert back to an issue – at least not yet.

[malthe]

@noderat looking at the code, I don't see a single call to get_template_env where the jinja_env is provided. If it's non-trivial in your case, what code passed a value there?

[noderat]

@malthe My code and use-case is almost identical to the minimal example I provided in the original post. I'm entirely new to Apache Airflow and I was trying to use this PsrpOperator in my first experimental DAG to call a powershell cmdlet on a remote machine and pass it an API key as a SecureString just like it says can be done using the jinja template filter in the PsrpOperator's documentation. It's darn near identical to my minimal example aside from the string being converted being different. As far as I'm aware I am trying to use the PsrpOperator in the most standard possible way way. If it were not for trying to solve this bug I wouldn't have even known that get_template_env method exists, so I'm not even sure when or why one might ever use that method themselves directly if they are just trying to use the securestring jinja filter - get_template_env is not something that is even mentioned in the Airflow tutorial on using Jinja templates.

[noderat]

I'm pretty sure this is a bug and that this operator extends the wrong method on BaseOperator to add the template filter. I tossed this together and it looks like it fixes things, but since I'm not all that familiar with Airflow and this is my first attempt at doing any kind of Airflow work I'll leave this snippet to the experts to decide if it is indeed a bug and if this is the correct resolution:

    def render_template_fields(
            self,
            context: Context,
            jinja_env: jinja2.Environment | None = None,
        ) -> None:

            if not jinja_env:
                jinja_env = self.get_template_env()

            native = isinstance(jinja_env, NativeEnvironment)
            def securestring(value: str):
                if not native:
                    raise AirflowException(
                        "Filter 'securestring' not applicable to non-native templating environment"
                    )
                return TaggedValue("SS", value)
            jinja_env.filters["securestring"] = securestring

            self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())