Unit test failures for Ubuntu Xenial build on Travis-CI

[dwcaress]

For some time one of the test builds on Travis CI has been failing seven of the unit tests while all of the other builds succeed. The problem instance is build 5 on Ubuntu Xenial. An example is at
https://travis-ci.com/github/dwcaress/MB-System/builds/166606866

The failures are for mbbackangle, mbgrid, mbmosaic, mbprocess, mbsegygrid, mbsegypsd, mbsslayout, which is also a list of all the c++ programs that call the functions that in turn call GMT functions to read and write GMT grids.

I have been unable to figure out how to get the actual shell output of the tests for these programs into logs that I can see, and so I don't know what the failures actually are. I have been unable to produce a similar build outside Travis-CI, which would presumably show the problem. The tests that fail are simply comparing the shell output of invoking these programs without data input to the expected text. All other Ubuntu builds in the testing succeed.

I'm hoping for advice from @schwehr or @carueda

[schwehr]

check-TESTS: $(check_SCRIPTS)
	log_list=`for i in $$bases; do echo $$i.log; done`; \
	trs_list=`for i in $$bases; do echo $$i.trs; done`; \
	log_list=`echo $$log_list`; trs_list=`echo $$trs_list`; \
	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list"; \
	exit $$?;

Perhaps add a cat test/utilities/test-suite.log to it? Untested...

check-TESTS: $(check_SCRIPTS)
	log_list=`for i in $$bases; do echo $$i.log; done`; \
	trs_list=`for i in $$bases; do echo $$i.trs; done`; \
	log_list=`echo $$log_list`; trs_list=`echo $$trs_list`; \
	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list"; \
	result=$$?; cat "$$log_list"; exit $$?;

[dwcaress]

Where is this code? It's not in test/utilities/Makefile.am or test/Makefile.am or Makefile.am...

[schwehr]

https://tecnocode.co.uk/tag/automake/

In the build script, sorry, on my phone... I. The Travis for maybe?

make V=1 VERBOSE=1

[carueda]

Not really much I can add, other than just pointing to the files/lines

MB-System/test/utilities/Makefile.in

Lines 723 to 732 in 9f95efa

	check-TESTS: $(check_SCRIPTS)
	@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
	@list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
	@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
	@set +e; $(am__set_TESTS_bases); \
	log_list=`for i in $$bases; do echo $$i.log; done`; \
	trs_list=`for i in $$bases; do echo $$i.trs; done`; \
	log_list=`echo $$log_list`; trs_list=`echo $$trs_list`; \
	$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list"; \
	exit $$?;

and

MB-System/ci/travis/build_linux/script.sh

Line 10 in 9d9cf51

docker exec -e CC="${CC}" -e CXX="${CXX}" -e CFLAGS="${CFLAGS}" -t ${BUILD_CONTAINER_NAME} bash -c "./configure ${MBSYSTEM_CONFIGURE_ARGS} --enable-test && make && make check"

that are being suggested to edit for some extra output ...

[dwcaress]

I was able to modify test/utilities/Makefile.in to cat out the relevant log files when I run make check on my laptop (brute force, as below). However, I still can't get this to work when it runs within Travis-CI - the overall log file comes out rather different, and doesn't have the text from the additional commands.

check-TESTS: $(check_SCRIPTS)
@list='$(RECHECK_LOGS)'; test -z "$$list" || rm -f $$list
@list='$(RECHECK_LOGS:.log=.trs)'; test -z "$$list" || rm -f $$list
@test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG)
@set +e; $(am__set_TESTS_bases);
log_list=for i in $$bases; do echo $$i.log; done;
trs_list=for i in $$bases; do echo $$i.trs; done;
log_list=echo $$log_list; trs_list=echo $$trs_list;
$(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list";
result=$$?; pwd; ls; cat mbbackangle_test.log mbgrid_test.log mbprocess_test.log; exit $$?;

My next approach will be to try a run a docker instance identical to the failing build, and see what the output of the problem programs is.