Build instructions

[NicoHood]

I try to build arduino-builder 1.5.1 for archlinux. It looks like the build system and folder structure changed again. I got the following error when building:

==> Starting build()...
# github.com/jstemmer/go-junit-report
loadinternal: cannot find runtime/cgo
src/github.com/arduino/arduino-builder/main.go:47:2: cannot find package "github.com/arduino/arduino-cli/arduino/cores" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/arduino/cores (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/arduino/cores (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:48:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:49:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder/grpc" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder/grpc (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder/grpc (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:50:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder/i18n" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder/i18n (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder/i18n (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:51:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder/types" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder/types (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder/types (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:55:2: cannot find package "github.com/sirupsen/logrus" in any of:
	/usr/lib/go/src/github.com/sirupsen/logrus (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/sirupsen/logrus (from $GOPATH)
==> ERROR: A failure occurred in build().

Is the readme build instruction up to date? Do you know anything I have missed to build it properly? I guess it must be something obvious that I missed.

[cmaglie]

The structure it's not changed from 1.5.0, but the README is indeed outdated.

To build you just need the latest version of go and run go build, no need to setup GOPATH or something like that, this is no more needed once we migrated to go modules.

Here a transcript from a clean build:

:~/tmp$ git clone [email protected]:arduino/arduino-builder.git
Clone in 'arduino-builder' in corso...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 6260 (delta 19), reused 29 (delta 11), pack-reused 6222
Ricezione degli oggetti: 100% (6260/6260), 1.48 MiB | 3.20 MiB/s, fatto.
Risoluzione dei delta: 100% (4071/4071), fatto.
:~/tmp$ cd arduino-builder/
:~/tmp/arduino-builder$ go build
:~/tmp/arduino-builder$ file arduino-builder 
arduino-builder: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped

[NicoHood]

Okay, thanks for that hint.

Just to understand better:
Why are the additional steps not required anymore, and what is the benefit? I am asking, because I want to package it with the best quality for the distribution and users. I've managed to also compile it by adding another go get github.com/arduino/arduino-cli.

Where is the hardware folder gone? I normally edit the platforms.txt files so they point to the distribution provided arduino-ctags version, but it seems it has moved somewhere. But where? :-)

[cmaglie]

Why are the additional steps not required anymore, and what is the benefit?

The benefit is that the dependent packages are listed here:
https://github.com/arduino/arduino-builder/blob/master/go.mod

and each package is listed with a specific git revision, so the build is reproducible. When you run go build with go modules enabled you're sure that the project is built with the correct dependencies.

If you run go get github.com/arduino/arduino-cli you're, instead, downloading and building with the current master of the arduino-cli package (probably, in this exact moment, it corresponds to the correct version, but if you re-run it again in a month, or even tomorrow, it may be not, depending on when the next commit comes in in the arduino-cli project).

The hardware folder is gone because it just contained the old general platform.txt that is now integrated in the new builder, we don't need it anymore.

[NicoHood]

@cmaglie Thank you for your explanation.

I've seen in the PR, that arduino-builder is now deprecated. It looks like a lot has changed recently, and I am not up to date on how to properly package arduino. I kindly ask you, if you can give me a few hints what I need to know about the recent changes.

From my understanding arduino-builder is no longer needed for the current ide, and I should rather package arduino-cli? How does that integrate with the new IDE? You also mentioned that platform.txt no longer exists, but how would I reference the avr-gcc binaries or the arduino-ctags binary?

It would help me a lot to provide a quality good arduino package for archlinux, which is quite outdated for a while (I am so sorry, that was my fault!).

[matthijskooijman]

From my understanding arduino-builder is no longer needed for the current ide,

AFAIU, the current IDE still uses arduino-builder, but the main code for that has all moved to the arduino-cli repo and development happens there. Possibly Arduino create has switched to calling arduino-cli internally already (not sure) and AFAIU this should happen for the java IDE in the future as well. Until then, this repo still serves to build the arduino-builder binaries.

You also mentioned that platform.txt no longer exists, but how would I reference the avr-gcc binaries or the arduino-ctags binary?

I believe this only referred to the platform.txt that was previously shipped along arduino-builder and contained some default values (mostly for compatibility with cores that were created before arduino-builder exists, I suspect). You will still need a platform.txt in the hardware / core directories for your cores.

As for packaging: I think that the arduino/Arduino repo is still the starting point. That has an antfile that will download/collect all needed components (though you probably want to refer to existing packages rather than bundle them, but the antfile should at least show you what's needed for a normal IDE build).

[cmaglie]

@NicoHood don't worry, just package as always, the Arduino IDE still uses arduino-builder (we are phasing it out, but it's not done yet, it will take some time).

There are no major changes in the way the build is done, even if the arduino-builder now is a wrapper of arduino-cli from the outside almost nothing has changed.

[NicoHood]

The following test failed:

go: github.com/nats-io/nats-server/server tested by
	github.com/nats-io/nats-server/server.test imports
	github.com/nats-io/go-nats: github.com/nats-io/[email protected]: parsing go.mod:
	module declares its path as: github.com/nats-io/nats.go
	        but was required as: github.com/nats-io/go-nats

Can that be fixed somehow?

[NicoHood]

You will still need a platform.txt in the hardware / core directories for your cores.

Looks like you are correct:

Picked up JAVA_TOOL_OPTIONS: 
Set log4j store directory /home/user/.arduino15
java.io.FileNotFoundException: /usr/share/arduino/hardware/platform.keys.rewrite.txt (No such file or directory)
WARNING: Error loading hardware folder /usr/share/arduino/hardware/archlinux-arduino
  No valid hardware definitions found in folder archlinux-arduino.
java.io.FileNotFoundException: /usr/share/arduino/hardware/platform.keys.rewrite.txt (No such file or directory)
java.io.FileNotFoundException: /usr/share/arduino/hardware/platform.keys.rewrite.txt (No such file or directory)
java.lang.IllegalStateException: Menu has no enabled items
	at processing.app.Base.selectFirstEnabledMenuItem(Base.java:1665)
	at processing.app.Base.rebuildBoardsMenu(Base.java:1504)
	at processing.app.Base.<init>(Base.java:277)
	at processing.app.Base.main(Base.java:150)

But where can I find those platform.txt files? I mean I could use those "old" files from previous releases, but I expect those files to be hosted somewhere. If they are completely removed I suggest adding them back to this arduino-builder repository.

And what about the java version? Do I need to force java8 or will arduino update/support newer java versions? arduino/Arduino#8903

[cmaglie]

github.com/nats-io/nats-server/server

uh... how did you run tests on this library?

It seems that you are running tests on a (deep) dependency of arduino-builder, but this is not a common practice, I don't think this is what you want.

There are basically no more direct tests on this repository for arduino-builder because we moved them into arduino-cli. Maybe you can try to run tests just on the package github-com/arduino/arduino-cli that is now used by arduino-builder, but it's still a bit weird (becuase then: why just arduino-cli and not all the dependencies...?)

IMHO you can safely skip the test here.

[cmaglie]

But where can I find those platform.txt files? I mean I could use those "old" files from previous releases, but I expect those files to be hosted somewhere. If they are completely removed I suggest adding them back to this arduino-builder repository.

Let me check this one.

[cmaglie]

It seems you're right @NicoHood, I'll check this for the next release of the arudino-builder. In the meantime I've readded the file in the git repo, is this sufficient for you to complete the build?

[NicoHood]

I ran the tests using go test -timeout 60m ..., maybe that is the problem. I will try to fix that.

Adding the hardware folder is a very nice idea. I think you should also add the platform.txt, not only the "rewrite" file. You can tag a new point release or I can add them manually to the package. Depends on if you want to tag a new release or not.

[cmaglie]

I think you should also add the platform.txt

that one is not needed (I'm sure because it's not even in the official build of IDE :-))

You can tag a new point release or I can add them manually to the package

Let me update the tag...

[cmaglie]

Tag updated.

[NicoHood]

that one is not needed (I'm sure because it's not even in the official build of IDE :-))

But that file existed in the previous release:
https://github.com/arduino/arduino-builder/blob/1.4.1/hardware/platform.txt#L10

Because now I get the following error when compiling a sketch:

/usr/share/arduino/arduino-builder -dump-prefs -logger=machine -hardware /usr/share/arduino/hardware -hardware /home/user/.arduino15/packages -tools /usr/share/arduino/tools-builder -tools /home/user/.arduino15/packages -libraries /home/user/Arduino/libraries -fqbn=arduino:avr:nano:cpu=atmega328 -ide-version=10811 -build-path /tmp/arduino_build_527230 -warnings=all -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.avr-gcc.path=/home/user/.arduino15/packages/arduino/tools/avr-gcc/4.9.2-atmel3.5.4-arduino2 -prefs=runtime.tools.avr-gcc-4.9.2-atmel3.5.4-arduino2.path=/home/user/.arduino15/packages/arduino/tools/avr-gcc/4.9.2-atmel3.5.4-arduino2 -prefs=runtime.tools.arduinoOTA.path=/home/user/.arduino15/packages/arduino/tools/arduinoOTA/1.1.1 -prefs=runtime.tools.arduinoOTA-1.1.1.path=/home/user/.arduino15/packages/arduino/tools/arduinoOTA/1.1.1 -prefs=runtime.tools.avrdude.path=/home/user/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino9 -prefs=runtime.tools.avrdude-6.3.0-arduino9.path=/home/user/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino9 -verbose /home/user/hackallthethings/avr/projects/Klingel/klingel_arduino/klingel_arduino.ino
/usr/share/arduino/arduino-builder -compile -logger=machine -hardware /usr/share/arduino/hardware -hardware /home/user/.arduino15/packages -tools /usr/share/arduino/tools-builder -tools /home/user/.arduino15/packages -libraries /home/user/Arduino/libraries -fqbn=arduino:avr:nano:cpu=atmega328 -ide-version=10811 -build-path /tmp/arduino_build_527230 -warnings=all -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.avr-gcc.path=/home/user/.arduino15/packages/arduino/tools/avr-gcc/4.9.2-atmel3.5.4-arduino2 -prefs=runtime.tools.avr-gcc-4.9.2-atmel3.5.4-arduino2.path=/home/user/.arduino15/packages/arduino/tools/avr-gcc/4.9.2-atmel3.5.4-arduino2 -prefs=runtime.tools.arduinoOTA.path=/home/user/.arduino15/packages/arduino/tools/arduinoOTA/1.1.1 -prefs=runtime.tools.arduinoOTA-1.1.1.path=/home/user/.arduino15/packages/arduino/tools/arduinoOTA/1.1.1 -prefs=runtime.tools.avrdude.path=/home/user/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino9 -prefs=runtime.tools.avrdude-6.3.0-arduino9.path=/home/user/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino9 -verbose /home/user/hackallthethings/avr/projects/Klingel/klingel_arduino/klingel_arduino.ino
Using board 'nano' from platform in folder: /home/user/.arduino15/packages/arduino/hardware/avr/1.6.20
Using core 'arduino' from platform in folder: /home/user/.arduino15/packages/arduino/hardware/avr/1.6.20
Detecting libraries used...
/home/user/.arduino15/packages/arduino/tools/avr-gcc/4.9.2-atmel3.5.4-arduino2/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10811 -DARDUINO_AVR_NANO -DARDUINO_ARCH_AVR -I/home/user/.arduino15/packages/arduino/hardware/avr/1.6.20/cores/arduino -I/home/user/.arduino15/packages/arduino/hardware/avr/1.6.20/variants/eightanaloginputs /tmp/arduino_build_527230/sketch/klingel_arduino.ino.cpp -o /dev/null
Generating function prototypes...
/home/user/.arduino15/packages/arduino/tools/avr-gcc/4.9.2-atmel3.5.4-arduino2/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -flto -w -x c++ -E -CC -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10811 -DARDUINO_AVR_NANO -DARDUINO_ARCH_AVR -I/home/user/.arduino15/packages/arduino/hardware/avr/1.6.20/cores/arduino -I/home/user/.arduino15/packages/arduino/hardware/avr/1.6.20/variants/eightanaloginputs /tmp/arduino_build_527230/sketch/klingel_arduino.ino.cpp -o /tmp/arduino_build_527230/preproc/ctags_target_for_gcc_minus_e.cpp
{runtime.tools.ctags.path}/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives /tmp/arduino_build_527230/preproc/ctags_target_for_gcc_minus_e.cpp
fork/exec {runtime.tools.ctags.path}/ctags: no such file or directory
Error compiling for board Arduino Nano.

[cmaglie]

We do add ctags in our arduino-builder dist files:

$ tar tfv arduino-builder-linux64-1.5.1.tar.bz2
-rwxrwxr-x jenkins/jenkins 18746673 2020-01-23 16:42 arduino-builder
drwxrwxr-x jenkins/jenkins        0 2020-01-23 16:44 hardware/
-rw-rw-r-- jenkins/jenkins     7194 2020-01-23 16:44 hardware/platform.keys.rewrite.txt
drwxrwxr-x jenkins/jenkins        0 2020-01-23 16:43 tools/
drwxrwxr-x jenkins/jenkins        0 2020-01-23 16:43 tools/ctags/
drwxrwxr-x jenkins/jenkins        0 2020-01-23 16:43 tools/ctags/5.8-arduino11/
-rwxr-xr-x jenkins/jenkins   268147 2016-11-23 11:30 tools/ctags/5.8-arduino11/ctags
drwxrwxr-x jenkins/jenkins        0 2020-01-23 16:43 tools/arduino-preprocessor/
drwxrwxr-x jenkins/jenkins        0 2020-01-23 16:43 tools/arduino-preprocessor/0.1.5/
-rwxrwxr-x jenkins/jenkins 19254168 2018-01-17 17:41 tools/arduino-preprocessor/0.1.5/arduino-preprocessor

and when we build the IDE the content of the tools folder in the dist file is copied inside the Arduino IDE's tools-builder folder (well we also add arduino-preprocessor but it's not strictly needed for the Arduino IDE).

arduino-builder is able to find ctags there because the Arduino IDE invoke it with the command line flag -tools /usr/share/arduino/tools-builder.

But now I'm wondering... how did it worked before? I mean, you provided ctags in some other way?

[NicoHood]

Before I used the provided platform.txt file and patched it using sed to point to our packaged version:

# Fix platform.txt for arch arduino-ctags
cd "${srcdir}/build/src/github.com/arduino/arduino-builder/hardware"
sed -i 's#^tools.ctags.path=.*#tools.ctags.path=/usr/bin#' platform.txt
sed -i 's#^tools.ctags.cmd.path=.*#tools.ctags.cmd.path={path}/arduino-ctags#' platform.txt

https://git.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/arduino-builder&id=f54efa9dd18e18d0ffae65c7bf329153b77ae51e#n54

[cmaglie]

I see, for now you can:

• add a top-level platform.txt just with your tools.ctags.path and tools.ctags.cmd.path

or

• maybe symlinking {ide-root}/tools-builder/ctags/5.8-arduino11/ctags to the real ctags would work?

This should be enough for the current release.
BTW we should think a way to properly handle bundled tools, I foresee that this is going to create some problems once we deeply integrate arduino-cli.
/cc @rsora @masci

[NicoHood]

So dont we need the platform.txt file in the arduino-builder repo? It would be nice to have it there, as it seems to be still required.

[cmaglie]

It's not required, in the official distribution of the IDE is not present anymore, but it's still parsed for backward compatibility. In your case you can use it to force ctags path.

Anyway I'd like to provide a better way for package maintaners to do that.

(we are going to ger rid also of the platform.rewrites.tx, see #351)

[cmaglie]

In your case you can use it to force ctags path

I mean you can generate a platform.txt with just two lines:

tools.ctags.path=.....
tools.ctags.cmd.path=.....

[NicoHood]

Thank you, I got it running now. Would it be possible to collect such build changes somewhere, so we can package arduino easier in the future?

I also have another question. The latest arduino release 1.8.12 says: "Reverted the bundled Java JRE to the latest freely redistributable Oracle JRE for all platforms except:". Does that mean, we can use the OS Java runtime? If yes, which version? Is java 8 still preferred or does arduino support newer versions. What are the plans to update to newer versions?

[cmaglie]

Thank you, I got it running now. Would it be possible to collect such build changes somewhere, so we can package arduino easier in the future?

Sure, I'll notify you when such change will happen.

Does that mean, we can use the OS Java runtime? If yes, which version? Is java 8 still preferred or does arduino support newer versions. What are the plans to update to newer versions?

Theoretically any JRE 8.0 should be fine... but the reality is that the Oracle JRE has still something, I don't know what, that makes it work better than the other open alternatives, in particular regatd to font rendering and 2d graphics. We tried to ship the latest OpenJDK JRE built from AdoptOpenJDK, and we had some weird problems (mostly related to font/graphics and unexplicable crash/freeze). We are keeping the complete list under the "OpenJDK" label in the Arduino IDE repository if you want to take a look.

We didn't tested JRE 11, again theoretically it should be fine, but we never know until we try it for real.