BUILTIN.adoc.jam

= Writing Built-In Macros for Jamal
{%@comment This is the ASCIIDOC version of the BUILTIN document.
The source of the file is the BUILTIN.adoc.jam.
This file is processed by the test javax0.jamal.documentation.TestConvertReadme in the module jamal-snippet.

%}{%@import res:jamal.jim%}\
{%@snip:collect from="jamal-test/src/test/java/javax0/jamal/test/examples"%}\
{%@snip:collect from="jamal-test/src/main/java/"%}\
{%@snip:collect from="jamal-api/src/main/java/"%}\
{%@snip:xml pom=pom.xml%}\
{%#define VERSION={%pom /project/version/text()%}%}\
{%@define MACRO_INTERFACE={%@java:class (classFormat=`$name`) javax0.jamal.api.Macro%}%}
{%@define INPUT_INTERFACE={%@java:class (classFormat=`$name`) javax0.jamal.api.Input%}%}
{%@define java($snippet)=[source,java,title=`$snippet` from {%#file (format=`$simpleName`) {%@snip:file $snippet%}%}]
----
{%#trimLines
{%@snip $snippet%}%}
----%}
In this document we will describe how you can write built-in macros for your specific application.

Built-in macros in Jamal are the macros, which are implemented using Java code.
The document assumes that you are familiar with Jamal macrostructure, and you know

* the difference between built-in and user defined macros in the Jamal file,

* how to define user defined macros

* what it means when a built-in macro uses the `@` character or the `#` in front of the macro name

* what is verbatim

* the macro evaluation order and how it can be modified using

* the `ident` and `eval` macros, and also the `!` and the backtick characters in front of the macros

* what are the different macro scopes

* how to export user defined macros to a higher scope

* how user defined macro arguments are separated and parsed

* generally, how to use Jamal.

[[helloworld]]
=== Creating a "Hello World" Macro

A "Hello, World!" example built-in macro is extremely simple.
You need the following dependency in your project:

[source,xml]
----
<dependency>
    <groupId>com.javax0.jamal</groupId>
    <artifactId>jamal-api</artifactId>
    <version>{%VERSION%}</version>
</dependency>
----


In later versions, when we will do a bit more in our macros we will also need the dependency:

[source,xml]
----
<dependency>
    <groupId>com.javax0.jamal</groupId>
    <artifactId>jamal-tools</artifactId>
    <version>{%VERSION%}</version>
</dependency>
----

This library has a transitive dependency to the library `jamal.api` so there is no need to specify both.
If you plan to use Java Platform Module System (JPMS) then you also have to add the

[source,java]
----
    requires jamal.api;
    requires jamal.tools;
----

directives to your `module-info.java` file.
Having the project set up that way we can create our first built-in macro.
The whole class is included in this document:

{%java HelloWorld%}

{%@define classFormat=`$simpleName`%}{%@define methodFormat=`$name()`%}
The class has to implement the interface {%MACRO_INTERFACE%}.
This interface, as well as the other interfaces and classes imported are defined in the package {%@java:class (classFormat=`$packageName`) javax0.jamal.api.Macro%}.
This interface requires us to create one method {%@java:method javax0.jamal.api.Macro::evaluate%}.
This method gets the input text of the macro, a reference to the processor executing Jamal and it has to return a `String` that will be the content of the macro after evaluation.
In the example case we do not read the input and we do not use any service the processor can provide.
We will see in later chapters how to read and parse the input and also how to use the processor to access the executing environment.

The last step is to create a unit test.

The simplest way is using the classes from the `jamal-testsupport` library.
This library contains test utilities that mock a Jamal environment for the macro and also provide useful test mocks.
It is recommended to use this package instead of trying to invoke the macro method directly and creating mocks using some general library.
It is to note that technically these tests are not unit tests, because they use the actual Jamal implementation and not a mock.
However, they test against a certain release, which is supposed to be bug free and any bug discovered should signal an error in the macro  implementation.
These tests are more readable than a clean unit tests, they are fast, reproducible, and exhibit all the important aspects of a regular unit tests.

To use the test support library you have to add the dependency to your `pom.xml` file:

[source,xml]
----
<dependency>
    <groupId>com.javax0.jamal</groupId>
    <artifactId>jamal-testsupport</artifactId>
    <version>{%VERSION%}</version>
    <scope>test</scope>
</dependency>
----

Note that the scope of this dependency is `test` because these tools are needed only during unit test running.
You may also need to add a JVM command line parameter via the surefire plugin configuration, like the followings:

[source,xml]
----
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version> ... actual version of the surefire plugin ... </version>
    <configuration>
        <argLine>
            --add-opens yourModuleName/packageName=ALL-UNNAMED
        </argLine>
    </configuration>
</plugin>
----

This is needed if your library uses JPMS.
When the tests are running they need access to your code using reflection, and therefore you need to "opens" the package.
When you run the tests in the IDE it may ignore the JPMS settings and run your code as a plain library.
In that case the fact that the package is not open for the reflective access is not important.

[NOTE]
====
JPMS is complex and needs much more explanation than what can fit here.
Here I describe a few hints only, but it does not intend to teach you JPMS.
If you do not understand how JPMS works and how to use it you can get into trouble using it.
In that case you have two choices.

. Ignore the possibility to use JPMS, and create your macros as plan old classes in a plain old library, which will be available for Jamal from the `classpath`.

. Learn JPMS from excellent book of Nicolai: https://www.manning.com/books/the-java-module-system

I recommend the second.
====

Our test will be the following:


{%java TestHelloWorld_1%}

With this we are essentially ready with the hello world macro application.
There is one more topic, though, which is worth discussing here.

In the tests code we had to declare the class in the Jamal file as a macro to be used.
This is one of the three possibilities to make a Java class available for the Jamal code.
The second is to register the class for the standard Java service loader.

When a Jamal processor object is created it calls the Java service loader to find all the classes, which implement the {%MACRO_INTERFACE%} interface.
The returned list of instances are registered into the Jamal global macro registry and are available to be used for the Jamal processing.

The Java service loader can find a class if it is

* declared in the `module-info.java` module descriptor file as one providing the {%#java:class {%@define classFormat=`$name`%}javax0.jamal.api.Macro%} interface, and/or

{%@define METAINFSERVICEFILE={%#file (format=`$name` root=jamal-snippet)/src/main/resources/META-INF/services/javax0.jamal.api.Macro%}%}
* the full class name is listed in the file {%METAINFSERVICEFILE%}

I recommend that you do both in case you use JPMS, because it will help test running inside the IDE, which may not use JPMS.
Having the class names listed in the {%METAINFSERVICEFILE%} file may also help applications that use your library as a normal JAR file and not as a module.

The module file will look something like this:

{%java module_declaration%}

Our module needs the `jamal.api` module, so we `require` it, and we provide the {%MACRO_INTERFACE%} implementation.
After this out unit test will be the following:

{%java TestHelloWorld_2%}

Now we do not need to declare the class in the Jamal file, it is available in the global scope.

There is a third option to register a macro in the Jamal processor.
The processor has an API and it is possible to register a user defined or a built-in macro programmatically.

=== Name of a Built-In Macro

In the <<helloworld,Creating a "Hello World" Macro>> chapter we did not discuss how the name of the macro is created.
We just created a class implementing an interface and then magically it was usable in the Jamal source in the unit test with a reasonable name.
There is no magic.

The name of the macro can be defined in the macro `use` when a macro class is explicitly declared for use.
The syntax of the `use` macro is

    use [global] fully_class_name [ as macroname]

The parts between `[` and `]` are optional.

When the macro is registered via the service loader this is not a possibility.
In this case, adn in cases when the `use` macro does not have the optional `as macroname` part the name of the macro will be the string, which is returned by the method {%@java:method javax0.jamal.api.Macro#getId%}.
This method is also part of the {%MACRO_INTERFACE%} interface, and it has a `default` implementations.

{%java getId%}

In our case the name of the class was `HelloWorld` which converts to `helloworld` all lower case as a macro name.
You are free to override the implementation of the default method, and there are real examples for that.
For example the `jamal-snippet` library macros `trimLines`, or `killLines` override the method {%@java:method javax0.jamal.api.Macro#getId%}.

Starting with the version 1.9.0 a macro can define multiple names to register itself.
The different names will serve as equal aliases for the same macro.
To do that the macro can define the method {%@java:method javax0.jamal.api.Macro#getIds%}.
(note that the name is the plural of the previous `getId()`)
The default implementation of this method simply calls {%@java:method javax0.jamal.api.Macro#getId%} and returns a one element string array with the single name.

{%java getIds%}

When the method is overridden all the strings will be registered.
It was first used to register `assert:equals`, `assert:lessOrEquals` and so on, with the 3rd person `equals` ending as well as with the simple `equal` ending.
Modesty and discipline is recommended when defining multiple names for a macro.

=== Handling the Input of the Macro

In the `HelloWorld` macro we totally ignored the input of the macro.
There are some built-in macros, like `comment` or `block` which deliberately do this.
It is usually not something we can do.
Macros usually need their input to work with.
Even macros ignoring the input are encouraged to check that there are no extra characters following the macro name.

If we write another test, we can see that the macro really ignores its input.


{%java TestHelloWorld_3%}

==== Hello, Me Macro

The next macro we will write is one that will not simply greet the whole word, but rather the person, who we tell it to.
The code of the macro `Hello` will be the following:

{%java Hello%}

It will use the `input`, convert it to string and cutting off the spaces from the start, and from the end of the string it uses it as a name for the greeting.
The test is also straightforward and shows the direct use of the macro:

{%java TestHello_1%}

We are handling the simplest possible way  in this example.
We use it as it is, as a whole string, only cutting off the strings from the start and the end.
In the next chapter we will look at an example that handles the input in a more complex way.

=== Working with the Input: Example: Spacer Macro

Most of the macros use their input, and they use it in a more complex way.
To do that macros can parse, split up the input into smaller pieces that the code can afterwards work with.
To do that there are many possibilities.

First of all, the interface {%INPUT_INTERFACE%} extends the Java JDK `CharSequence` interface.
You can use all the methods defined there.
The characters in the underlying structure are stored in a `StringBuilder`, and you can get direct access to that calling {%@java:method javax0.jamal.api.Input#getSB%}.

Built-in macros, however, rarely use these methods directly.
They use the static methods implemented in the {%@java:class javax0.jamal.tools.InputHandler%} instead.

The `Input` object is essentially a character sequence, which also keeps track of the file name, and location the characters came from.
If you directly access the underlying `StringBuilder` and modify it then you may lose track of the line number and column position.

The class {%@java:class javax0.jamal.tools.InputHandler%} defined methods that are safe to use for parsing the input.
The definite reference is the up-to-date JavaDoc.
In the following examples we will look at how to use some of these methods.

The following macro takes the input of the macro and inserts spaces between the characters.
That way it will convert

[source,text]
----
{@spacer this is
some text
}
----

to

[source,text]
----
t h i s   i s
s o m e   t e x t
----

The implementation of the macro is the following:

{%java Spacer%}

The very first thing the macro does is that it skips the white spaces.
It is customary to skip these spaces because one or more space has to be there after the id of the macro and they usually only separate the macro name and the content.
Some macros skip spaces only to the end of the line and in case there are more spaces, but on the next line then they are taken into account.
In this case all white spaces including new lines are skipped at the start of the input.
It is important to understand that the skipping process also takes care of the line number and the column position of the actual character.

The input keeps track of the file name, the line number and the column position of the character at the start of the character sequence.
These three things make a `Position` object.
The current position of an `Input` can be queried using the {%@java:method javax0.jamal.api.Input#getPosition%} method.

If the input contained only spaces then we skipped them all and in that case we simply return the empty string.
If there are characters in the input then we go through them one by one and we insert a space in front of each of them unless the character is at the start of a line.
To do this we create a new `Input` object, which is empty at the start and inherits the position of the original input.
Because `Input` is also a `CharacterSequence` we can easily get any character at a certain position calling `charAt()`.
We can also `move` characters from one input to the other.
The moving deletes the character from the `Input` `in` and it also modifies the current `Position` of the input.

Finally, the `result` is converted to `String` and is returned.

This macro interpreted the input as an array of characters.
Many times macros want to work with individual parameters.
In the next chapter we will look at an example how we should do that.

=== Splitting the Input

If you look at the core built-in macro `if` then you can see that it does not have a special syntax.
It just has three parameters and in case the first parameter is true, then it returns the second parameter, otherwise the third.
In case there are just two parameters then it results empty string in case the first parameter is false.
The syntax of the macro is:

[source]
----
{@if 'sep' condition 'sep' then result [ 'sep'else result] }
----

Here the `'sep'` is some kind of separator.
It can be a space, some non-alphanumeric character or some complex separator.
These three cases are handled by the method {%@java:method javax0.jamal.tools.InputHandler#getParts%}.
This method is defined in the class {%@java:class javax0.jamal.tools.InputHandler%}.

This method skips the white spaces at the start of the input and then looks at the first character.
If it is a back-tick, then it fetches more characters until it finds a pairing back-tick character.
The string it fetches is used as a regular expression to split up the rest of the input.

If the first non-space character on the input is not a back-tick, but still a non-alphanumeric character then this character will be used as separator to split up the input.

Last, but not least if the first non-space character is alphanumeric then the input will be split up along the spaces.

The following example uses this method to implement a macro that can fetch one string from many based on an index.
For example

{%#snip:define TestArray_test1Formatted=
{%#replace |{%@snip TestArray_test1%}|"||,aaa||,|%}
%}

[source]
----
{%@snip TestArray_test1Formatted
        {@array /1/x/aaa/z}

%}
----

will select the second element, that is `aaa` from the array of `[ "x", "aaa", "z"]`.
The code of the macro is the following:

{%java Array%}

The macro calls the method {%@java:method javax0.jamal.tools.InputHandler#getParts%} passing only the input as one argument.
There is another version of the method that limits the number of the arguments.
Calling that the last element of the returned array will contain the rest of the string even if it could be split up more.
The macro implementation checks that there are enough number of parts and then converts the first part to integer.
This will be the index, the rest of the parts array are the values to choose from.
The code also checks the array bounds and throws exception in case there is an error.

When implementing a macro and there is an error the code has to detect it and it can throw a {%@java:class javax0.jamal.api.BadSyntax%} exception.
It is also declared in the interface.
The exception {%@java:class javax0.jamal.api.BadSyntaxAt%} is an extension of {%@java:class javax0.jamal.api.BadSyntax%}.
This second exception also contains the reference to the input location.

If the location of the error is not interesting inside the macro then it is good enough to throw a simple {%@java:class javax0.jamal.api.BadSyntax%} exception.
The processor catches that exception and converts it to a {%@java:class javax0.jamal.api.BadSyntaxAt%} exception that will reference the character at the very start of the macro.

== General Structure of the {%@define methodFormat=`$name()`%}{%@java:method javax0.jamal.api.Macro::evaluate%} Method

== Macros that are {%@java:class javax0.jamal.api.InnerScopeDependent%}

The macro evaluation order is detailed in the link:{%@file README.adoc%}[README] of Jamal.
When Jamal sees a built-in macro that starts with a `#` character at the start then it evaluates the content of it before invoking the macro itself.
For example

[source,text]
----
{#trimLines {@define margin=1}
{@snip sampleText}
}
----

will first evaluate the `define` macro resulting `margin` to become a user defined macro with the value `1`.
After that the `snip` macro will be evaluated and that way replaced with the snippet named `sampleText`.
Only when it is done starts the execution of the macro `trimLines` that will shift the lines left or right with spaces so that there will exactly be one space on the leftmost line.

The macro `margin` is defined in a local scope.
The scope starts with the opening `{` character of the macro `trimLines` and ends with the closing `}`.
If the implementation of the macro `snip` would query the macro register, it could see the value of the macro `margin` as `1`.

The question is whether the macro execution `trimMacro` sees `margin` as defined in itself or not.
Is the scope already closed when the execution of `trimLines` starts?

It depends.

If the `Macro` implementing class also implements the {%@java:class javax0.jamal.api.InnerScopeDependent%} interface then the scope is open.
If it does not then Jamal closes the scope before starting the execution of the macro.

The macro `trimLines` implements this interface because it uses parameters.
Implementing this interface is simply adding the name of the interface after the `implements` keyword.
There are no abstract methods in this interface to implement in the class.
The first few lines of the method {%@java:method javax0.jamal.snippet.TrimLines#evaluate%} are the followings:
{%@snip:collect from="jamal-snippet/src/main/java/javax0/jamal/snippet/TrimLines.java"%}\

{%java trimLinesStart%}

The macro class implements the interface {%@java:class (format=`$canonicalName`) javax0.jamal.tools.Scanner$FirstLine%}.
The method {%@java:method javax0.jamal.tools.Scanner$FirstLine#newScanner%} is inherited from this interface and it creates a scanner object that is able to parse the first line of the macro.

In this case it parses only the first line and scans for the parameters `margin`, `trimVertical` and so on.
If the macro fetches the input from between `()` characters then the class has to implement the {%@java:class (format=`$canonicalName`) javax0.jamal.tools.Scanner%} interface.
There are interfaces for the core classes using the `[]` characters and for the macros that use the whole input.

When a parameter is not defined in the macro, then the class tries to use the value of the macro with the same name.
Thus, the value of the variable `margin` will be a configuration parameter holding the integer value 1.

[NOTE]
====
In earlier version of Jamal there was no utility class to support the parsing of the parameters.
The first approach to configure a macro was to define a user defined macro without any parameter of a given name.
Later the {%@java:class javax0.jamal.tools.Params%} was developed, and it kept the functionality to fall back to macro definitions in case the parameter was not defined.

This backward compatibility can also be useful when there is a sense to define the parameter globally and not only for the macro invocation.
====

The macros created before the class {%@java:class javax0.jamal.tools.Params%} had no other choice but use macros for configuration.
These macros supported the local scope of the macro implementing the signal interface {%@java:class javax0.jamal.api.InnerScopeDependent%}.
With the availability of parameter parsing there is no need to define a configuration user defined macro inside the build.in macro body.
Instead, you can simply use the configuration parameters in the macro body.
Newer macros developed after parameter parsing do not implement the interface {%@java:class javax0.jamal.api.InnerScopeDependent%}.

There is still a use defining a parameter as a macro though.
It is the case when the parameter should be defined for a larger scope, and you do not want to copy the parameter `key=value` to each use of the macro.
In that case you can write `{@define key=value}` before the first use of the built-in macro.

The parameter parsing allows the use of aliases.
The example macro above uses both `verticalTrimOnly`, and `vtrimOnly`.
Any of them can be used to define that the trimming is vertical only.
They are aliases.
However, only the first one, `verticalTrimOnly`, is considered as a macro name when the parameter is not defined.

Some built-in macros list the names of the parameters starting with `null`.
It means that the parameter has no name, only aliases.
Such parameters cannot be defined using a user defined macro.

[NOTE]
====
Boolean parameters cannot be defined using user defined macros.
They always have a default value of `false` if not defined in the macro body.
The default value can be altered if they are defined in an `options` macro.
If you say `{@options trimVertical}` then the default value of `trimVertical` is changed to true.

Technically the options are stored in the same (identifier,value) store where the user defined macros.
The consequence is that you cannot use the same name for an option and for a user defined macro.
The options, however, are not user defined macros.
====

Macros that rely on user defined macros or options as parameters defined _inside_ should implement the interface {%@java:class javax0.jamal.api.InnerScopeDependent%}.
It is recommended not to implement this interface anymore.

== Creating User Defined Macros

You can easily create user defined macros using the `define` macro.
However, user defined macros can also be created programmatically.
This chapter will describe the latter.

== Creating Your Own User Defined Macro Implementation

Programmatically created user defined macros can define their own evaluation.


== Strategies to Register Built-In Macros

In this chapter I will explain the advantages, and the disadvantages of the two strategies that you can follow to register your built-in macros.
It is a more theoretical chapter with less example code.
You can skip this section and return to it later.