OCI is a framework for continuous integration and benchmarks. At its heart it is a container manager and at the top a tool that allows to compile, test, and compare compilations and runs of inter-dependent git repositories.
Simply run
opam pin add oci --kind=git "https://github.com/bobot/oci.git#master"For more precise [installation instructions](INSTALL.md).
The goal of the tutorial is to introduce the user to the predefined rules of OCI. It doesn’t describe the framework part of OCI.
|
Note
|
The server uses advanced features of linux that need to be activated in some distributions (e.g. Debian but not Ubuntu) before starting the server (once per boot): sudo sysctl kernel.unprivileged_userns_clone=1 |
One interacts with OCI using clients that connect to a server. Starting the server with the default configuration is done by:
oci-monitor|
Note
|
By default the server keeps its data in the oci-master-tools cleanA copy of this binary is kept in `$prefix/var`for convenience. |
At first we will use the default OCI client oci-default-client.
The first step is to install a rootfs, a small distribution that would be executed inside a container. You could create your own image but the linuxcontainer project (LXC) maintains a set of prebuilt images. You can see the list with:
oci-default-client list-download-rootfsNow we are going to download an image (debian, jessie, amd64)`. The
image is about 100MB. It can be long, but you have to do it
only once:
oci-default-client download-rootfs --distrib debian --release jessie \
--arch amd64The output is
2016-04-12 10:44:08.497949+02:00 Info Download the index.
2016-04-12 10:44:09.378128+02:00 Info Index downloading done.
2016-04-12 10:44:09.379359+02:00 Info Downloading rootfs.
[10:47:20] Extract meta archive: [...]/.oci_tmp/meta.tar.xz
[10:47:20] Extract rootfs archive: [...]/.oci_tmp/rootfs.tar.xz
[10:47:26] Create artefact
[10:47:33] New rootfs created
[Result] ((id 0)
(info
((distribution debian) (release jessie) (arch amd64) (packages ())
(comment default)))
(rootfs 0))The (id 0) indicates the number of the rootfs. Since the creation of
a rootfs depends on the outside (internet) we can’t refer it with
debian,testing,amd64.
Now we are installing packages needed for the compilation of the repository we are going to use in this tutorial:
oci-default-client add-package --rootfs 0 autotools-dev binutils-dev \
libiberty-dev libncurses5-dev pkg-config zlib1g-dev git gcc \
build-essential m4 autoconf[10:51:54] Runner started
[10:51:54] dispatch runner Oci_Cmd_Runner_Api.copy_to
[10:51:54] Copy artefact 0 to /
[10:52:02] result received
[10:52:02] dispatch runner Oci_Cmd_Runner_Api.get_internet
[10:52:02] Get internet
[10:52:02] result received
[10:52:02] Update Apt Database
[10:52:02] dispatch runner Oci_Cmd_Runner_Api.run
[10:52:02] apt-get update, --option, APT::Sandbox::User=root, --option, Acquire::Retries=3
[...]
[10:52:06] apt-get install, --yes, --option, Apt::Install-Recommends=false, --option, APT::Sandbox::User=root, --option, Acquire::Retries=3, autotools-dev, binutils-dev, libiberty-dev, libncurses5-dev, pkg-config, zlib1g-dev, git, gcc, build-essential, m4, autoconf
[...]
[10:52:42] Create artefact /
[10:52:54] result received
[10:52:54] New rootfs created
[Result] ((id 1)
(info
((distribution debian) (release jessie) (arch amd64)
(packages
(autotools-dev binutils-dev libiberty-dev libncurses5-dev pkg-config
zlib1g-dev git gcc build-essential m4 autoconf))
(comment default)))
(rootfs 1))At the end we see that this rootfs will have the number 1 ((id 1)).
|
Caution
|
If the command fails because of problems to download a file, just try again. |
|
Tip
|
Instead of relying on the package of the distribution, we could add all these packages as a repository of OCI, but since we don’t want to test or benchmark with different versions of these packages, it is simpler like that. |
Now we can compile our first repository. In the usage of OCI, nothing is specific to OCaml. However, the rules for some OCaml-related repositories are predefined. The compilation of OCaml can be run with the following command, the first step of OCI is to download the OCaml git repository (it can take some time):
oci-default-client run --rootfs 1 ocaml2016-04-12 11:00:10.919814+02:00 Info Check the revspecs:
2016-04-12 11:06:13.563312+02:00 Info configuration: --rootfs 1 --ocaml bdf3b0fac7dd2c93f80475c9f7774b62295860c1
[11:06:13] dispatch runner Oci_Generic_Masters.compile_git_repo_runner
[11:06:13] Link Artefacts
[11:06:13] Link artefact 1 to /
[11:06:13] mount -t, tmpfs, tmpfs, /checkout
[11:06:13] Clone repository at bdf3b0fac7dd2c93f80475c9f7774b62295860c1
[11:06:13] Git clone https://github.com/ocaml/ocaml.git in /oci/git_clone/0
[11:06:13] git -C, /checkout, -c, advice.detachedHead=false, checkout, --detach, bdf3b0fac7dd2c93f80475c9f7774b62295860c1
[11:06:14] HEAD is now at bdf3b0f... increment version number after tagging 4.02.3
[11:06:14] ./configure
[...]
[11:09:21] Create artefact /
[Result] New artefact 2 created
[11:09:22] umount /checkoutThe git commit number bdf3b0fac7dd2c93f80475c9f7774b62295860c1 is
the default one (corresponds to 4.02.3). The command line option
--ocaml 4.03 can be added for using the current tip of the 4.03
branch of ocaml.
Let’s suppose that we have a very nice tool that sorts lists, oci-sort,
that is developed in
oci-repository-for-tutorial.
We want to test it continuously and we want to benchmark it. We just have to
create an ml file oci_sort_client.ml with the following content that
describes how to find, compile and test our repository.
open Core.Std
open Async.Std
open Oci_Client.Git
open Oci_Client.Cmdline
let oci_sort_url = "https://github.com/bobot/oci-repository-for-tutorial.git"
let oci_sort,oci_sort_revspec = mk_repo
"oci-sort"
~url:oci_sort_url
~revspec:"master"
~deps:Oci_Client.Cmdline.Predefined.[ocaml;ocamlbuild;ocamlfind]
~cmds:[
run "autoconf" [];
run "./configure" [];
make [];
make ["install"];
]
~tests:[
make ["tests"];
]
let () =
don't_wait_for (Oci_Client.Cmdline.default_cmdline
~doc:"Oci client for oci-sort"
~version:Oci_Client.oci_version
"oci_sort_client");
never_returns (Scheduler.go ())This file is compiled with:
ocamlfind ocamlopt -thread -linkpkg -package oci.client \
oci_sort_client.ml -o oci-sort-clientThe obtained command oci-sort-client has the same subcommands and
options than oci-default-client with the addition of the subcommand
run:
* the positional argument oci-sort for requesting the compilation
and testing of oci-sort
* the optional argument --oci-sort that specifies the revision of
oci-sort to use
To compile oci-sort with the current state of the default branch
master.
./oci-sort-client run --rootfs 1 oci-sort2016-04-12 11:16:08.818239+02:00 Info Check the revspecs:
2016-04-12 11:16:14.147274+02:00 Info configuration: --rootfs 1 --ocaml bdf3b0fac7dd2c93f80475c9f7774b62295860c1 --ocamlbuild 93681343df7f42e4621f6c81d5f4d3678f7af1e4 --ocamlfind f902fbd26fba3de09c1ce475c676ef27500a1f2a --oci-sort f5df503023e461dae8a6a98e37b3038219963295
[11:16:14] dispatch master Oci_Generic_Masters.compile_git_repo ocaml
[11:16:14] dispatch master Oci_Generic_Masters.compile_git_repo ocamlbuild
[11:16:14] dispatch master Oci_Generic_Masters.compile_git_repo ocamlfind
[11:16:14] Dependency ocaml done
[11:16:20] Dependency ocamlfind done
[11:16:21] Dependency ocamlbuild done
[11:16:21] dispatch runner Oci_Generic_Masters.compile_git_repo_runner
[11:16:21] Link Artefacts
[11:16:21] Link artefact 1 to /
[11:16:21] Link artefact 3 to /
[11:16:21] Link artefact 4 to /
[11:16:21] Link artefact 2 to /
[11:16:21] mount -t, tmpfs, tmpfs, /checkout
[11:16:21] Clone repository at f5df503023e461dae8a6a98e37b3038219963295
[11:16:21] Git clone https://github.com/bobot/oci-repository-for-tutorial.git in /oci/git_clone/0
[11:16:21] git -C, /checkout, -c, advice.detachedHead=false, checkout, --detach, f5df503023e461dae8a6a98e37b3038219963295
[11:16:21] HEAD is now at f5df503... Optimize comparison function!
[11:16:21] autoconf
[Result] Ok in {kernel:8ms; user:200ms; wall:230.331ms}
[11:16:22] ./configure
[11:16:22] configure: creating ./config.status
[11:16:22] config.status: creating .config
[Result] Ok in {kernel:16ms; user:44ms; wall:214.345ms}
[11:16:22] make --jobs=1
[11:16:22] Generating Merlin file
[11:16:22] ocamlbuild -no-sanitize -no-links -tag debug -use-ocamlfind -cflags -w,+a-4-9-18-41-30-42-44-40 -cflags -warn-error,+5+10+8+12+20+11 -cflag -bin-annot -j 8 -tag thread -tag principal -I src src/sort.native
[11:16:22] ocamlfind ocamldep -modules src/sort.ml > src/sort.ml.depends
[11:16:22] ocamlfind ocamlc -c -w +a-4-9-18-41-30-42-44-40 -warn-error +5+10+8+12+20+11 -bin-annot -g -principal -thread -I src -o src/sort.cmo src/sort.ml
[11:16:22] ocamlfind ocamlopt -c -w +a-4-9-18-41-30-42-44-40 -warn-error +5+10+8+12+20+11 -bin-annot -g -principal -thread -I src -o src/sort.cmx src/sort.ml
[11:16:22] + ocamlfind ocamlopt -c -w +a-4-9-18-41-30-42-44-40 -warn-error +5+10+8+12+20+11 -bin-annot -g -principal -thread -I src -o src/sort.cmx src/sort.ml
[11:16:22] findlib: [WARNING] Interface sort.cmi occurs in several directories: /usr/local/lib/ocaml, src
[11:16:22] ocamlfind ocamlopt -linkpkg -g -thread src/sort.cmx -o src/sort.native
[11:16:22] # No parallelism done
[Result] Ok in {kernel:4ms; user:60ms; wall:110.469ms}
[11:16:22] make --jobs=1, install
[11:16:22] install bin/sort.native "/usr/local/bin"/oci-sort
[Result] Ok in {kernel:0s; user:0s; wall:4.73595ms}
[11:16:22] Create artefact /
[Result] New artefact 5 created
[11:16:22] make --jobs=1, tests
[11:16:22] ocamlbuild -no-sanitize -no-links -tag debug -use-ocamlfind -cflags -w,+a-4-9-18-41-30-42-44-40 -cflags -warn-error,+5+10+8+12+20+11 -cflag -bin-annot -j 8 -tag thread -tag principal -I src src/sort.native
[11:16:22] # No parallelism done
[11:16:22] DEBUG_OCI_SORT=yes bin/sort.native tests/simple_example.sort
[11:16:22] [2;0;4;6;2;3;8;2;4;7;8;3;8;5;9;8;8;5;6;8;7;0;2;1;8;8;8;1;1;0;0;1;9;8;4;8;7;5;
[11:16:22] 2;7;9;0;1;6;4;2;0;5;3;1;0;6;8;1;4;2;5;9;8;8;3;4;5;6;4;8;2;4;6;8;4;2;3;7;0;1;
[11:16:22] 8;4;8;1;2;9;4;1;5;3;4;3;7;7;4;4;1;9;1;3;4;1;5;3;]
[11:16:22] [0;0;0;0;0;0;0;0;1;1;1;1;1;1;1;1;1;1;1;1;1;2;2;2;2;2;2;2;2;2;2;3;3;3;3;3;3;3;
[11:16:22] 3;3;4;4;4;4;4;4;4;4;4;4;4;4;4;4;4;5;5;5;5;5;5;5;5;6;6;6;6;6;6;7;7;7;7;7;7;7;
[11:16:22] 8;8;8;8;8;8;8;8;8;8;8;8;8;8;8;8;8;8;9;9;9;9;9;9;]
[11:16:22] sum_before: 443
[11:16:22] sum_after: 443
[Result] Ok in {kernel:0s; user:12ms; wall:25.8262ms}
[11:16:22] umount /checkoutThe dependencies of oci-sort, ocaml, ocamlfind, ocamlbuild,
are compiled automatically. The results of their installation, the
artefacts, are hardlinked (Link artefact).
The following command allows to see the saved log, even if the master
of oci-sort, ocamlbuild` or ocamlfind changes. You can replace
the last oci-sort by ocamlfind or ocamlbuild to see the log of
their compilation.
./oci-sort-client run --rootfs 1 \
--ocaml bdf3b0fac7dd2c93f80475c9f7774b62295860c1 \
--ocamlbuild 93681343df7f42e4621f6c81d5f4d3678f7af1e4 \
--ocamlfind f902fbd26fba3de09c1ce475c676ef27500a1f2a \
--oci-sort f5df503023e461dae8a6a98e37b3038219963295 \
oci-sortThe compilation of oci-sort can be tested with different versions of
ocaml
./oci-sort-client run --rootfs 1 --ocaml 4.03 oci-sort./oci-sort-client run --rootfs 1 --ocaml trunk oci-sortNow we want to benchmark different
versions
of oci-sort.
So we had to oci_sort_client.ml before the last let () = the following code:
let () = mk_compare
~deps:[oci_sort]
~x_of_sexp:Oci_Common.Commit.t_of_sexp
~sexp_of_x:Oci_Common.Commit.sexp_of_t
~y_of_sexp:Oci_Filename.t_of_sexp
~sexp_of_y:Oci_Filename.sexp_of_t
~cmds:(fun conn revspecs x y ->
let revspecs = WP.ParamValue.set revspecs
oci_sort_revspec (Oci_Common.Commit.to_string x) in
commit_of_revspec conn ~url:oci_sort_url ~revspec:"master"
>>= fun master ->
return
(revspecs,
[Oci_Client.Git.git_copy_file ~url:oci_sort_url ~src:y
~dst:(Oci_Filename.basename y)
(Option.value_exn ~here:[%here] master)],
(run
~memlimit:(Byte_units.create `Megabytes 500.)
~timelimit:(Time.Span.create ~sec:10 ())
"oci-sort" [Oci_Filename.basename y])))
~analyse:(fun _ timed ->
Some (Time.Span.to_sec timed.Oci_Common.Timed.cpu_user))
"oci-sort"After recompilation, oci-sort-client gains for the subcommand
compare the positional option oci-sort.
We create two files that configure the benchmark:
master
master~1
master~2The following command compiles the needed version of oci-sort, runs
the benchmarks and show thes resulting graphics in a new window
(gnuplot required in the host computer)
oci-sort-client compare --rootfs 1 oci-sort \
--x-input tests_oci_sort1.commits \
--y-input tests_oci_sort1.bench \
--show-qt --output-png tests_oci_sort1_1.png
The comparison can be done at another version of the dependencies
(here the ocaml version is set to the branch 4.03):
oci-sort-client compare --rootfs 1 oci-sort \
--x-input tests_oci_sort1.commits \
--y-input tests_oci_sort1.bench \
--show-qt --ocaml 4.03Now, we would like to benchmark using the new flambda optimisation
pass which is activated through a configure option of ocaml. The
predefined OCI rule for ocaml adds an option for that:
--ocaml-configure. So we just have to run:
oci-sort-client compare --rootfs 1 oci-sort \
--x-input tests_oci_sort1.commits \
--y-input tests_oci_sort1.bench \
--show-qt --ocaml 4.03 --ocaml-configure=-flambdaSince OCaml has not yet been compiled with this particular configuration, it is automatically done. You can see it in another terminal by running:
oci-sort-client run --rootfs 1 ocaml --ocaml 4.03 --ocaml-configure=-flambdaIf we want to compare with and without flambda, we need to change the
format of the --x-input. Currently it takes only the oci-sort
version, now we want to add at least ocaml-configure.
Oci_Client.Cmdline defines WP.ParamValue.t which can store all the
arguments that configure repositories (ocaml, --ocaml-configure,
oci-sort, …) and a sexpr is provided for it. So we can replace in
oci_sort_client.ml the let () = mk_compare … by the following:
let () = mk_compare
~deps:[oci_sort]
~x_of_sexp:WP.ParamValue.t_of_sexp
~sexp_of_x:WP.ParamValue.sexp_of_t
~y_of_sexp:Oci_Filename.t_of_sexp
~sexp_of_y:Oci_Filename.sexp_of_t
~cmds:(fun conn revspecs x y ->
let revspecs = WP.ParamValue.replace_by revspecs x in
commit_of_revspec conn ~url:oci_sort_url ~revspec:"master"
>>= fun master ->
return
(revspecs,
[Oci_Client.Git.git_copy_file ~url:oci_sort_url ~src:y
~dst:(Oci_Filename.basename y)
(Option.value_exn ~here:[%here] master)],
(run
~memlimit:(Byte_units.create `Megabytes 500.)
~timelimit:(Time.Span.create ~sec:10 ())
"oci-sort" [Oci_Filename.basename y]))
)
~analyse:(fun _ timed ->
Some (Time.Span.to_sec timed.Oci_Common.Timed.cpu_user))
"oci-sort"Moreover we need to be able to give the option -O2 or -O3 to
ocamlopt during the compilation of oci-sort. We will do that by
setting the variable OCAMLPARAM. For that purpose we will add a new
option, and parameterize the rule for building oci-sort.
We replace `let oci_sort, … ` by:
open Cmdliner
let oci_sort_ocamlparam =
WP.mk_param ~default:None "oci-sort-ocamlparam"
~sexp_of:[%sexp_of: string option]
~of_sexp:[%of_sexp: string option]
~cmdliner:Arg.(opt (some string) None)
~docv:"ARG"
~doc:"Determine the argument to give to ocaml \
OCAMLPARAM"
~to_option_hum:(function None -> "" | Some s -> "--oci-sort-ocamlparam="^s)
let oci_sort_revspec =
mk_revspec_param ~url:oci_sort_url "oci-sort"
let oci_sort =
add_repo_with_param "oci-sort"
WP.(const (fun commit ocamlparam ->
Oci_Client.Git.repo
~deps:Oci_Client.Cmdline.Predefined.[ocaml;ocamlbuild;
ocamlfind]
~cmds:[
Oci_Client.Git.git_clone ~url:oci_sort_url commit;
run "autoconf" [];
run "./configure" [];
make ?env:(match ocamlparam with
| None -> None
| Some v -> Some (`Extend ["OCAMLPARAM", v])) [];
make ["install"];
]
~tests:[
make ["tests"];
]
())
$? oci_sort_revspec
$? oci_sort_ocamlparam);
"oci-sort"And we are going to use the following tests_oci_sort2.commits:
((oci-sort-ocamlparam (Some "_,O3=")))
((oci-sort-ocamlparam (Some "_,O2=")))
()After recompilation, the benchmark is done with:
./oci-sort-client compare --rootfs 1 oci-sort \
--x-input tests_oci_sort2.commits \
--y-input tests_oci_sort1.bench \
--show-qt --ocaml 4.03 --ocaml-configure=-flambda \
--oci-sort "master~2"
Currently the result of the comparison has been drawn using the default. But other options exist:
- --summation-by-sort
-
For a given time compute the maximal number of runs that could be run sequentially in the given time. It is the default.
- --summation-by-timeout
-
For a given time compute the number of runs that finish before that time. It is simpler than --summation-by-sort but the end of the curve depends less of the time taken by fast runs. --compare-two Compare each run individually
The option --compare-two works only if there are only two x-inputs.
((oci-sort master~2)(oci-sort-ocamlparam (Some "_,O3="))(ocaml-configure (-flambda)))
((oci-sort master~2)(ocaml-configure (-flambda)))There is one point for each bench. Inside the two green lines, the difference is too small to really matter.
./oci-sort-client compare --rootfs 1 oci-sort \
--x-input tests_oci_sort3.commits \
--y-input tests_oci_sort1.bench \
--show-qt --ocaml 4.03 --compare-two
For more precise cgroup one can ask OCI to use cgroup for placing the
runners (the compilation, tests, …) on different cpus. For that you
need cgmanager. Run inside the terminal that will run oci_monitor:
sudo cgm create all oci
sudo cgm chown all oci $(id -u) $(id -g)
cgm movepid all oci $PPIDAnd run oci-monitor with the option --cgroup "."
oci-monitor --cgroup "."