Metaprogramming

Mix.install([
  {:jason, "~> 1.4"},
  {:kino, "~> 0.9", override: true},
  {:youtube, github: "brooklinjazz/youtube"},
  {:hidden_cell, github: "brooklinjazz/hidden_cell"}
])

Navigation

Home

Report An Issue

Advanced Score Tracker

MetaMath

Review Questions

Upon completing this lesson, a student should be able to answer the following questions.

What does the use keyword do?
What are the differences between a macro and a function?

Overview

Compile-time Vs Runtime

In programming, compile-time refers to the period of time during which a program is being compiled from source code into executable machine code, and runtime refers to the period of time during which a program is executing.

OTP

The Open Telecom Platform (OTP) is a collection of Erlang libraries and tools for building concurrent, fault-tolerant, and distributed systems in the Erlang programming language.

The BEAM (Erlang Virtual Machine)

The BEAM is the runtime environment for the Erlang and Elixir. It is a virtual machine that runs on a wide variety of platforms, including Unix-like operating systems, Windows, and MacOS. We often use the terms Erlang Virtual Machine and BEAM interchangeably.

The Elixir compiler compiles Elixir source code into bytecode .beam files that are executed on the BEAM.

The BEAM is part of the Erlang Run-Time System (ERTS).

Erlang Run-time System

The Erlang Run-Time System (ERTS) is a component of the Open Telecom Platform (OTP) and is responsible for executing Erlang and Elixir programs on the Erlang Virtual Machine (VM). The BEAM virtual machine is part of the Erlang Run-Time System.

Abstract Syntax Tree (AST)

The Abstract Syntax Tree (AST) is a representation of the structure of a program's source code as a tree-like data structure. The Elixir AST is used by the Elixir compiler to transform Elixir code into bytecode that can be executed by the BEAM virtual machine. The Elixir compiler converts the Elixir code into the AST, and then uses the AST to generate the bytecode.

The AST provides a convenient intermediate representation of the source code that can be used to perform various optimizations and transformations on the code before it is translated into bytecode.

Metaprogramming

Metaprogramming is the process of writing code that generates code.

In Elixir, we use macros for code generation. Macros are a powerful tool for generating code. However, they can lead to additional complexity and should be used with care.

Even though Elixir attempts its best to provide a safe environment for macros, the major responsibility of writing clean code with macros falls on developers. Macros are harder to write than ordinary Elixir functions and it’s considered to be bad style to use them when they’re not necessary. So write macros responsibly.

elixir-lang.org

We're going to learn how to leverage metaprogramming to extend the Elixir language and minimize boilerplate code.

While we're not likely to use metaprogramming in our day-to-day programming (depending on what you're building), we're going to use metaprogramming to gain a better understanding of the inner-workings of Elixir.

Quote

Under the hood, Elixir represents expressions as three-element tuples. We call this representation the AST (abstract syntax tree). Elixir lets us inspect the AST representation of expressions using the quote macro.

quote do
  2 + 2
end

The three-element tuple above is often called a quoted expression.

The first element in the tuple is the function name. The second element is a keyword list containing metadata, and the third element is a list of arguments.

{function, metadata, arguments}

So 2 + 2 as a quoted expression is

function: :+
metadata: [context: Elixir, import: Kernel]
arguments: [2, 2]

The function name is :+, which refers to the Kernel.+/2 function. + is simply a convenient syntax for calling this function.

Kernel.+(2, 2) == 2 + 2

The metadata includes information about the environment. By default, the :context is Elixir because we are in the top-level scope.

The context changes if we use quote in a module. Now the context will be the name of the module.

defmodule MyModule do
  def example do
    quote do
      2 + 2
    end
  end
end

MyModule.example()

We can also call quote on a single line.

quote do: 1 - 1

The AST represents primitive data types as themselves rather than three-element tuples.

quote do: 2

All other expressions will be three-element tuples—even non-primitive data types such as maps.

quote do: %{key: "value"}

Here's an anonymous function as a quoted expression.

sum = fn int1, int2, int3 -> int1 + int2 + int3 end

quote do: sum(1, 2, 3)

Here's a named function as a quoted expression.

defmodule Math do
  def sum(int1, int2, int3) do
    int1 + int2 + int3
  end
end

quote do: Math.sum(1, 2, 3)

Arguments in the three-element tuple can themselves be three-element tuples.

quote do: sum(1, 2, sum(1, 2, 3))

Your Turn

Use the quote macro to discover the AST representation of the following expression. You may also choose to experiment with quote with other Elixir expressions to see their quoted expression representation.

2 + 2 + 2

Unquote

unquote injects code into the quote macro.

We can use unquote to inject some computed value into a quote block.

For example, the following unquote(1 + 1) evaluates to 2 inside of the quote block.

quote do
  2 + unquote(1 + 1)
end

The above quote expression is equivalent to 2 + 2, because we injected the result of 1 + 1 into the quote expression using unquote.

quote do
  2 + 2
end

Notice this is not the same AST as 2 + 1 + 1, which breaks down into multiple three-element tuples, because it is actually two separate addition expressions.

quote do
  2 + 1 + 1
end

Variables outside the quote block will not be available within the quote block. So we can use unquote to inject their evaluated value.

my_variable = 5

quote do
  2 + unquote(my_variable)
end

This creates the same quoted expression as 2 + 5.

quote do
  2 + 5
end

Macro And Code Modules

The Elixir Code module is a module in the Elixir standard library that provides functions for working with code at runtime. The Code module provides several functions for evaluating code, such as Code.eval_string/2 and Code.eval_quoted/2, which allow you to dynamically generate and execute code based on runtime conditions.

The Code module also provides functions for working with the Abstract Syntax Tree (AST) of Elixir code, such as Code.string_to_quoted/1 which allow you to convert between Elixir code as a string and the AST representation of the code.

The Macro modules contains functions for manipulating the AST and implementing macros. The Macro module provides the Macro.to_string/1 function.

The Code module is often used in conjunction with metaprogramming techniques in Elixir, as it allows you to manipulate and execute code at runtime. However, it is important to use these functions carefully, as dynamically generating and executing code can be complex and difficult to understand, and can have unintended consequences if not used correctly.

Reading AST

We can use Macro.to_string/1 to convert a quoted expression into an Elixir expression (in a string).

quoted = quote do: 2 + 2

Macro.to_string(quoted)

We can provide the AST directly as a three-element tuple.

ast = {:+, [context: Elixir, imports: [{1, Kernel}, {2, Kernel}]], [2, 5]}

Macro.to_string(ast)

Evaluating Strings As Source Code

We can evaluate a string as Elixir code using the Code.eval_string/3 function.

Code.eval_string("2 + 5")

Variables in the string can be provided by providing a keyword list of bindings as the second argument to the function.

Code.eval_string("2 + a", a: 2)

Evaluating AST

We can use the Code.eval_quoted/2 function to evaluate the AST. Here, we take a quoted expression and evaluate it to find the result.

quoted =
  quote do
    2 + 2
  end

Code.eval_quoted(quoted)

Quoted expressions to not have access to bound variables. Below we'll see that a is not defined in the quoted expression.

a = 2

quoted =
  quote do
    a + 2
  end

Code.eval_quoted(quoted)

To gain access to the variable, we can use unquote to evalate it into the quoted expression.

a = 2

quoted =
  quote do
    unquote(a) + 2
  end

Code.eval_quoted(quoted)

Alternatively, we can provide the variable bindings in a keyword list as the second argument to the Code.eval_quoted/2 function.

quoted =
  quote do
    unquote(a) + 2
  end

Code.eval_quoted(quoted, a: 2)

Macros

We can use macros to extend the Elixir language or create DSLs (Domain-specific Languages). For example, every time you use test and assert in ExUnit, you use ExUnit macros.

ExUnit.start(auto_run: false)

defmodule Test do
  use ExUnit.Case

  test "example" do
    assert 1 == 2
  end
end

ExUnit.run()

It's often idiomatic to use macros without round brackets. However, there's nothing preventing you from using brackets with macros. do end blocks are actually just an alternative syntax for writing a keyword list as an argument.

ExUnit.start(auto_run: false)

defmodule BracketExample do
  use ExUnit.Case

  test("example", do: assert(1 == 1))
end

ExUnit.run()

Much of Elixir syntax is implemented using macros. Keywords such as def are actually just macros on the Kernel module, and do end blocks are actually just keyword lists being provided as an argument to the macro. 🤯

Kernel.defmodule(Mind, do: Kernel.def(blown, do: "🤯"))

Mind.blown()

Writing Our Own Macro

Unlike functions, macros are expanded at compile-time, so they retain knowledge of the values they are called with.

Notice that ExUnit can determine the operator and values used in the assertion assert 1 == 2 to provide test feedback.

"""
Assertion with `==` failed.

left: 1
right: 2
"""

To understand how ExUnit leverages the power of macros to provide better test feedback, we're going to create our own assert macro.

The assert macro will accept a truthy expression and print a message with feedback. Notice that we cannot accomplish this with a function. Functions accept the evaluated result of an expressions an argument. We lose the context about the operator and values.

inspect_argument = fn expression ->
  IO.inspect(expression, label: "Evaluated Result Of Expression")
end

inspect_argument.(1 == 2)

We use defmacro to define a macro.

The AST representation of an expression knows the function and the arguments the macro was called with.

defmodule ASTInspector do
  defmacro inspect(ast) do
    IO.inspect(ast, label: "ast")
  end
end

To use a macro, we need to require it.

require ASTInspector

ASTInspector.inspect(2 == 1)

We have everything we need in this AST tuple to get the operator, the left side of the expression, and the right side of the expression.

defmodule ExpressionInspector do
  defmacro inspect({operator, _meta, [left, right]}) do
    IO.inspect(operator, label: "operator")
    IO.inspect(left, label: "left")
    IO.inspect(right, label: "right")
  end
end

require ExpressionInspector

ExpressionInspector.inspect(2 == 2)

We want to verify if the left and right values are equal. We'll make an Assertion.Test module that uses pattern matching to return a success message, or failed assertion message depending on if the left and right sides are equal.

defmodule Assertion.Test do
  def assert(:==, left, right) when left == right do
    "Success!"
  end

  def assert(:==, left, right) do
    """
    Assertion with == failed.
    left: #{left}
    right: #{right}
    """
  end
end

Assertion.Test.assert(:==, 1, 2) |> IO.puts()

Then we can use a Macro to get the operator, left, and right values to use with our Assertion.Test module.

A macro generates code, so generally it should return an AST expression, not a return value. We use quote to create the AST representation of our function call. We'll use the Assertion.Test.assert function inside our quoted expression. We also need to use unquote to use bound variables inside the quote block. The same is true for parameters. So we need to use unquote to inject their evaluated value into the quote block.

defmodule Assertion do
  defmacro assert({operator, _meta, [left, right]}) do
    quote do
      Assertion.Test.assert(unquote(operator), unquote(left), unquote(right))
    end
  end
end

When we call the assert macro below it compiles into Assertion.Test.assert(:==, 1, 2) which then evaluates during runtime.

require Assertion

Assertion.assert(1 == 2) |> IO.puts()

Alternatively, we can use bind_quoted to bind multiple values to the quoted expression without unquote. This is just a syntax sugar to avoid using unquoted multiple times.

defmodule AssertionWithBindQuoted do
  defmacro assert({operator, _meta, [left, right]}) do
    quote bind_quoted: [operator: operator, left: left, right: right] do
      Assertion.Test.assert(operator, left, right)
    end
  end
end

The macro continues to work as expected.

require AssertionWithBindQuoted

AssertionWithBindQuoted.assert(1 == 2) |> IO.puts()

Use And using

While you may not write macros often, you are likely to use them daily. For example, we have already relied on macros with the use keyword.

When we use GenServer, a macro generates the necessary boilerplate code to make a GenServer.

defmodule Server do
  use GenServer

  def init(state) do
    {:ok, state}
  end
end

We can use the __info__/2 function on the Server module to gain insight into code generated under the hood. Here we see it defines several functions.

Server.__info__(:functions)

The use keyword provides a clean and controlled interface for working with macros. Under the hood, the use keyword calls a __using__ macro in the specified module.

defmodule Template do
  defmacro __using__(_opts) do
    quote do
      def template_function do
        "hello"
      end
    end
  end
end

Conceptually, it may help think of modules that define macros as templates or common patterns that we can reuse throughout a program. For example, GenServer is a common pattern we want to extend and reuse.

defmodule ExtendedTemplate do
  use Template

  def extended_function() do
    template_function() <> " world"
  end
end

Once we define our pattern, we can reuse it throughout our program and extend its functionality.

ExtendedTemplate.template_function()

ExtendedTemplate.extended_function()

For a real-world example, it's common to create custom ExUnit cases for common test scenarios. The example below is only a small example of what's possible. Here we create an IOCase module which automatically imports the ExUnit.CaptureIO module that provides the capture_io/1 for testing if we print a message using IO.

defmodule IOCase do
  # Use the module
  defmacro __using__(_opts) do
    quote do
      use ExUnit.Case
      import ExUnit.CaptureIO
    end
  end
end

ExUnit.start(auto_run: false)

defmodule IOTest do
  use IOCase

  test "capure io" do
    capture_io(fn -> IO.puts("hello") end) =~ "hello"
  end
end

ExUnit.run()

Your Turn

Create a Greetings module with a __using__ macro. define a hello/0 function inside of the __using__ macro. You may choose to experiment with defining other functions or module attributes.

def hello do
  "hello"
end

Create a Usage module that uses the use keyword to call the __using__ macro in the Greetings module.

defmodule Greetings do
end

defmodule Usage do
end

Call Usage.hello() to ensure your solution works correctly.

Usage.hello()

Commit Your Progress

DockYard Academy now recommends you use the latest Release rather than forking or cloning our repository.

Run git status to ensure there are no undesirable changes. Then run the following in your command line from the curriculum folder to commit your progress.

$ git add .
$ git commit -m "finish Metaprogramming reading"
$ git push

We're proud to offer our open-source curriculum free of charge for anyone to learn from at their own pace.

We also offer a paid course where you can learn from an instructor alongside a cohort of your peers. We will accept applications for the June-August 2023 cohort soon.

Navigation

Home

Report An Issue

Advanced Score Tracker

MetaMath

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metaprogramming.livemd

metaprogramming.livemd

Metaprogramming

Navigation

Review Questions

Overview

Compile-time Vs Runtime

OTP

The BEAM (Erlang Virtual Machine)

Erlang Run-time System

Abstract Syntax Tree (AST)

Metaprogramming

Quote

Your Turn

Unquote

Macro And Code Modules

Reading AST

Evaluating Strings As Source Code

Evaluating AST

Macros

Writing Our Own Macro

Use And using

Your Turn

Further Reading

Commit Your Progress

Navigation

Files

metaprogramming.livemd

Latest commit

History

metaprogramming.livemd

File metadata and controls

Metaprogramming

Navigation

Review Questions

Overview

Compile-time Vs Runtime

OTP

The BEAM (Erlang Virtual Machine)

Erlang Run-time System

Abstract Syntax Tree (AST)

Metaprogramming

Quote

Your Turn

Unquote

Macro And Code Modules

Reading AST

Evaluating Strings As Source Code

Evaluating AST

Macros

Writing Our Own Macro

Use And using

Your Turn

Further Reading

Commit Your Progress

Navigation