-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
6 changed files
with
455 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
81 changes: 81 additions & 0 deletions
81
priv/posts/testing-tricks-in-elixir-2-parallel-test-execution.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
--- | ||
author: "Juan Carlos Martinez de la Torre" | ||
date: 2024-03-01 | ||
linktitle: testing-tricks-in-elixir-2-parallel-test-execution | ||
title: Testing tricks in Elixir - Parallel test execution | ||
intro: hello hello | ||
# toc: true | ||
--- | ||
|
||
|
||
## Concurrent test execution | ||
|
||
Enabling `async: true` in Elixir tests is a powerful feature, fostering parallelization in the test execution and faster feedback loops. You can activate it by simply adding the attribute to the `use` statement: | ||
|
||
```elixir | ||
defmodule MyApp.Test do | ||
use ExUnit.Case, async: true | ||
...... | ||
end | ||
``` | ||
However, nuanced considerations come into play, and it's crucial to be aware of these subtleties: | ||
|
||
### Limited concurrency within modules | ||
|
||
While `async: true` promotes concurrency, it's essential to note that tests within an individual module are still executed sequentially. This means that tests within the same file will run one after the other nevermind if `async` was set to `true` or `false`. | ||
|
||
### Database support for concurrent tests | ||
|
||
Elixir projects often integrate database operations into Unit tests. When utilizing `async: true`, be cautious of the fact that not all databases support concurrent transactional tests. For instance, MySQL poses challenges due to its transaction implementation, potentially leading to deadlocks. PostgreSQL is recommended in such scenarios, as it supports concurrent tests within the SQL sandbox. Elixir documenation clearly states this: | ||
|
||
> While both postgresql and mysql support sql sandbox, only postgresql supports concurrent tests while running the sql sandbox. therefore, do not run concurrent tests with mysql as you may run into deadlocks due to its transaction implementation. | ||
This consideration reinforces the preference for PostgreSQL in Phoenix projects. | ||
|
||
### Challenges with Mox and Process Isolation | ||
|
||
When using [Mox](https://github.com/dashbitco/mox) for mocking in Elixir tests, complexities may arise when processes other than the test process access mocks. By default, mocks are confined to the test process, which basically means that we have to rethink our strategy to keep our test concurrent in scenarios where different processess are involved. | ||
|
||
One potential solution is the use of [explicit allowances](https://hexdocs.pm/mox/mox.html), allowing controlled access to mocks. However, in certain cases, identifying the process's PID, especially under supervision, might be challenging. | ||
|
||
An alternative workaround is employing the [global mode](https://hexdocs.pm/mox/mox.html#module-global-mode), granting all processes access to the mock. Unfortunetely, this approach isn't compatible with the `async: true` option. | ||
|
||
These considerations highlights the need for a thoughtful approach when adopting `async: true`. While it unlocks parallelization benefits, understanding its limitations ensures a smooth testing experience and avoids potential pitfalls in concurrent testing environments. | ||
|
||
|
||
## Parallelizing the test suite with partitions | ||
|
||
In addition to the option of running tests concurrently within different modules using `async: true`, other possibility for parallelizing test execution is to leverage multiple Elixir instances. | ||
|
||
To achieve this, the [--partitions](https://hexdocs.pm/mix/1.12/Mix.Tasks.Test.html) argument on `mix test` comes into play, allowing us to specify the partition being executed using the `MIX_TEST_PARTITION` environment variable. Suppose we aim to distribute the tests across three different instances: | ||
|
||
|
||
```bash | ||
MIX_TEST_PARTITION=1 mix test --partitions 1 | ||
MIX_TEST_PARTITION=2 mix test --partitions 2 | ||
MIX_TEST_PARTITION=3 mix test --partitions 3 | ||
``` | ||
|
||
This alternative is specially interesting in cases where there are global resources that make not possible to use the `async: true` approach. | ||
|
||
**Using mix `test --partitions` with MySQL** | ||
|
||
In the case of having a project that uses MySQL where it is not possible to use `async: true`, using this approach is a great way to improve the time that requires to run the test suite in the CI pipeline. | ||
|
||
The main challenge related with this approach is that we need to provide a different database to each running Elixir instance. We can use the same MySQL server for that purpose, but initializing different databases through the Elixir configuration. | ||
|
||
At `config/runtime.test.exs` we can read this `MIX_TEST_PARTITION` environment variable, and we can inject this value into the database name. | ||
|
||
```elixir | ||
db_suffix = | ||
case System.get_env("MIX_TEST_PARTITION") do | ||
"" -> "" | ||
partition -> "_#{partition}" | ||
end | ||
|
||
# Configure your database | ||
config :your_app, YourApp.Repo, | ||
database: "#{System.get_env("POSTGRES_DB", "your_app_test")}#{db_suffix}", | ||
...... | ||
``` | ||
|
263 changes: 263 additions & 0 deletions
263
priv/posts/testing-tricks-in-elixir-3-setup-customization.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,263 @@ | ||
--- | ||
author: "Juan Carlos Martinez de la Torre" | ||
date: 2024-04-01 | ||
linktitle: testing-tricks-in-elixir-3-setup-customization | ||
title: Testing tricks in Elixir - Setup customization | ||
intro: hello hello | ||
toc: false | ||
--- | ||
|
||
|
||
## Setup blocks | ||
|
||
As explained on the introduction, it is important to keep the tests as **concise** as possible, which means that we should avoid any boilerplate code that is not really adding value for understanding the test. [ExUnit](https://hexdocs.pm/ex_unit/1.12.3/ExUnit.html) provides different setup blocks that are executed at the beginning of the tests, allowing us to share some initialization between tests. | ||
|
||
Let's say we need to create one `user` in all our tests, we could do something like: | ||
|
||
```elixir | ||
defmodule AssertionTest do | ||
use ExUnit.Case, async: true | ||
|
||
# "setup" is called before each test | ||
setup do | ||
[user: create_user()] | ||
end | ||
|
||
test "test1", %{user: _user} do | ||
assert true | ||
end | ||
end | ||
``` | ||
|
||
It is interesting to note that this `setup` will be executed once per test. The keyword list returned at setup block will be accessible on each test, as what is commonly named as test `context`. | ||
|
||
When using `describe` blocks, it is possible to have different setup blocks like: | ||
|
||
```elixir | ||
defmodule AssertionTest do | ||
use ExUnit.Case, async: true | ||
|
||
describe "func1/0" do | ||
setup do | ||
[user: create_user()] | ||
end | ||
....... | ||
end | ||
|
||
describe "func2/0" do | ||
setup do | ||
[company: create_company(:company)] | ||
end | ||
....... | ||
end | ||
end | ||
``` | ||
|
||
### Other setup block alternatives | ||
|
||
On my experience this basic `setup` block per describe block or per file is the most usual way for definning common boilerplate in Elixir tests. However, ExUnit provides more options for sharing common setup code between tests. | ||
|
||
**Use setup_all/1 ** | ||
|
||
The [setup_all/1](https://hexdocs.pm/ex_unit/1.12.3/ExUnit.Callbacks.html#setup_all/1) callbacks are invoked only once per module, before any test is run. In contrast with `setup/1` blocks, this `setup_all/1` is executed in a different process that the one executing the test. | ||
|
||
|
||
```elixir | ||
setup_all do | ||
[conn: Plug.Conn.build_conn()] | ||
end | ||
``` | ||
|
||
**Invoke functions through `setup`** | ||
|
||
It is possible to call several times the `setup` macro with different functions for example: | ||
|
||
```elixir | ||
defmodule Test do | ||
use ExUnit.Case, async: true | ||
setup build_connection() | ||
setup build_user() | ||
|
||
defp build_connection(), do: [conn: Plug.Conn.build_conn()] | ||
defp create_company(), do: [company: insert(:company)] | ||
|
||
test "my test", _context = %{conn: conn, company: company} do | ||
.... | ||
end | ||
end | ||
``` | ||
|
||
We will end up with a `setup` context where the `map()` returned in each one of the functions is merged. This allows us to create a kind of composable setup, depending on the functions that are called. However, it may be not very maintenable in the long term since it could make difficult to identify where each element was setup. | ||
|
||
In any case, **it is preferable to keep the setup of the tests simple and easy to follow. Ideally just using the `setup` block**. When start adding too many places where the setup is happening this will make much harder to follow the whole test flow, which can be quite frustrating for the developers. | ||
|
||
|
||
### Setup customization through `@tags` | ||
|
||
In some scenarios we may want to use a setup block, but we would also like to customize some of the parameters of the setup. | ||
|
||
Let's imagine we have a function called `list_papers/2`, where the papers that are readable by each user depdends on the role of the user and other query parameters. In Elixir, it is recommended to use a `describe` block to group all the tests related to the same function. We could have a test suite like this: | ||
|
||
|
||
```elixir | ||
describe "list_papers/2" do | ||
test "an admin user can see all papers" do | ||
user = create_user(role: :admin) | ||
# test body | ||
end | ||
|
||
test "a regular user can see published papers" do | ||
user = create_user(role: :regular) | ||
# test body | ||
end | ||
|
||
test "a regular user can see draft papers wrote by him" do | ||
user = create_user(role: :regular) | ||
# test body | ||
end | ||
end | ||
``` | ||
|
||
We can see how the initialization of the users is quite repetitive. We could think about creating a `setup` block for sharing this initial setup, but we would need to customize the `role` for each role depending on the test. For that purpose we can use the `@tag` property | ||
|
||
## Setup blocks | ||
|
||
As explained on the introduction, it is important to keep the tests as **concise** as possible, which means that we should avoid any boilerplate code that is not really adding value for understanding the test. [ExUnit](https://hexdocs.pm/ex_unit/1.12.3/ExUnit.html) provides different setup blocks that are executed at the beginning of the tests, allowing us to share some initialization between tests. | ||
|
||
Let's say we need to create one `user` in all our tests, we could do something like: | ||
|
||
```elixir | ||
defmodule AssertionTest do | ||
use ExUnit.Case, async: true | ||
|
||
# "setup" is called before each test | ||
setup do | ||
[user: create_user()] | ||
end | ||
|
||
test "test1", %{user: _user} do | ||
assert true | ||
end | ||
end | ||
``` | ||
|
||
It is interesting to note that this `setup` will be executed once per test. The keyword list returned at setup block will be accessible on each test, as what is commonly named as test `context`. | ||
|
||
When using `describe` blocks, it is possible to have different setup blocks like: | ||
|
||
```elixir | ||
defmodule AssertionTest do | ||
use ExUnit.Case, async: true | ||
|
||
describe "func1/0" do | ||
setup do | ||
[user: create_user()] | ||
end | ||
....... | ||
end | ||
|
||
describe "func2/0" do | ||
setup do | ||
[company: create_company(:company)] | ||
end | ||
....... | ||
end | ||
end | ||
``` | ||
|
||
### Other setup block alternatives | ||
|
||
On my experience this basic `setup` block per describe block or per file is the most usual way for definning common boilerplate in Elixir tests. However, ExUnit provides more options for sharing common setup code between tests. | ||
|
||
**Use setup_all/1 ** | ||
|
||
The [setup_all/1](https://hexdocs.pm/ex_unit/1.12.3/ExUnit.Callbacks.html#setup_all/1) callbacks are invoked only once per module, before any test is run. In contrast with `setup/1` blocks, this `setup_all/1` is executed in a different process that the one executing the test. | ||
|
||
|
||
```elixir | ||
setup_all do | ||
[conn: Plug.Conn.build_conn()] | ||
end | ||
``` | ||
|
||
**Invoke functions through `setup`** | ||
|
||
It is possible to call several times the `setup` macro with different functions for example: | ||
|
||
```elixir | ||
defmodule Test do | ||
use ExUnit.Case, async: true | ||
setup build_connection() | ||
setup build_user() | ||
|
||
defp build_connection(), do: [conn: Plug.Conn.build_conn()] | ||
defp create_company(), do: [company: insert(:company)] | ||
|
||
test "my test", _context = %{conn: conn, company: company} do | ||
.... | ||
end | ||
end | ||
``` | ||
|
||
We will end up with a `setup` context where the `map()` returned in each one of the functions is merged. This allows us to create a kind of composable setup, depending on the functions that are called. However, it may be not very maintenable in the long term since it could make difficult to identify where each element was setup. | ||
|
||
In any case, **it is preferable to keep the setup of the tests simple and easy to follow. Ideally just using the `setup` block**. When start adding too many places where the setup is happening this will make much harder to follow the whole test flow, which can be quite frustrating for the developers. | ||
|
||
|
||
### Setup customization through `@tags` | ||
|
||
In some scenarios we may want to use a setup block, but we would also like to customize some of the parameters of the setup. | ||
|
||
Let's imagine we have a function called `list_papers/2`, where the papers that are readable by each user depdends on the role of the user and other query parameters. In Elixir, it is recommended to use a `describe` block to group all the tests related to the same function. We could have a test suite like this: | ||
|
||
|
||
```elixir | ||
describe "list_papers/2" do | ||
test "an admin user can see all papers" do | ||
user = create_user(role: :admin) | ||
# test body | ||
end | ||
|
||
test "a regular user can see published papers" do | ||
user = create_user(role: :regular) | ||
# test body | ||
end | ||
|
||
test "a regular user can see draft papers wrote by him" do | ||
user = create_user(role: :regular) | ||
# test body | ||
end | ||
end | ||
``` | ||
|
||
We can see how the initialization of the users is quite repetitive. We could think about creating a `setup` block for sharing this initial setup, but we would need to customize the `role` for each role depending on the test. For that purpose we can use the `@tag` property: | ||
|
||
|
||
```elixir | ||
describe "list_papers/2" do | ||
setup context do | ||
user_role = Map.get(context, :role, :regular) | ||
{:ok, user: create_user(role: user_role)} | ||
end | ||
|
||
@tag role: :admin | ||
test "an admin user can see all papers", %{user: user} do | ||
# test body | ||
end | ||
|
||
@tag role: :regular | ||
test "a regular user can see public papers", %{user: user} do | ||
# test body | ||
end | ||
|
||
@tag role: :regular | ||
test "a regular user can see draft papers wrote by him", %{user: user} do | ||
# test body | ||
end | ||
end | ||
``` | ||
|
||
This is just a very simple case where this `@tag` attibute is applicable. I would be also perfectly fine to keep the `create_user()` function af each individual test, since there is not a lot of overhead beind handled on the setup block. Normally, this `@tag` would make more sense when having a "heavy" setup block that need to be cusomized in one of the first statements. | ||
|
||
|
||
|
Oops, something went wrong.