Test Suite Failure with GHC 8.2.2

[TikhonJelvis]

I'm getting a test suite failure building ghc-exactprint with GHC 8.2.2:

Running 1 test suites...
Test suite test: RUNNING...
GHC82
### Failure in: 1:Round-trip tests:0:ghc710:20:Control.hs
tests/Test/Common.hs:123
Control.hs
### Failure in: 1:Round-trip tests:0:ghc710:77:Internals.hs
tests/Test/Common.hs:123
Internals.hs
Cases: 1764  Tried: 1764  Errors: 0  Failures: 2
Counts {cases = 1764, tried = 1764, errors = 0, failures = 2}
Test suite test: FAIL
Test suite logged to: dist/test/ghc-exactprint-0.5.6.1-test.log
0 of 1 test suites (0 of 1 test cases) passed.

I've got this failure with the master branch and several versions from hackage:

• 0.5.6.1
• 0.5.6.0
• 0.5.5.0

[TikhonJelvis]

I mostly got these results building with Nix, so it's possible that there is some misconfiguration on the Nixpkgs side. However, I also got the same failure building a clone of the repo directly with cabal.

[alanz]

It passes CI, and I just tried

$ cabal unpack ghc-exactprint
$ cd ghc-exactprint-0.5.6.1/
$ cabal install --dependencies-only --enable-tests --with-compiler=ghc-8.2.2
$ cabal configure --enable-tests --with-compiler=ghc-8.2.2
$ cabal test

and got

...
Linking dist/build/test/test ...
Running 1 test suites...
Test suite test: RUNNING...
Test suite test: PASS
Test suite logged to: dist/test/ghc-exactprint-0.5.6.1-test.log
1 of 1 test suites (1 of 1 test cases) passed.

[TikhonJelvis]

Interesting. This means there's something about the Nix setup causing the tests to fail, but I don't know what that could be. I'll poke around a bit more to see if I can find the issue.

Do you have any hunches?

[TikhonJelvis]

Is there some way to get more detailed output about what caused the test case to fail?

[TikhonJelvis]

Looks like it might be a macOS issue of some sort. One of my colleagues followed the same build steps as I did on Linux and did not encounter the test failure.

[alanz]

To see the failure, should be simple, but we re-use some result files.

So edit https://github.com/alanz/ghc-exactprint/blob/ghc-8.6/tests/Test.hs#L135 to comment out the last test category, i.e.

  return $ TestList [
                      internalTests,
                      roundTripTests
                    ,
                      transformTests
                    , failingTests
                    , noAnnotationTests
                    -- ,
                    --   prettyRoundTripTests
                    ]

Then when the tests fail, you can compare https://github.com/alanz/ghc-exactprint/blob/ghc-8.6/tests/examples/ghc710/Control.hs to a file in the same directory called Control.hs.out

The first part of the .out file should be a verbatim copy of the original, if there is a diff then that is where the error lies.

The rest of the file is the AST, and annotations found.

[TikhonJelvis]

Just took a look at Control.hs.out and the behavior is really weird. The part of the file between module GHC.Event.Control and #include "EventConfig.h" has lots of duplicated text mixed in with the correct text.

Here's the broken part:

module GHC.Event.Control
module(GHC.Event.Control
    (-- * Managing the IO manager
    -- * Managing the IO managerSignal
    , SignalControlMessage(..)
    , ControlMessageControl(..)
    , ControlnewControl
    , newControlcloseControl
    ,-- ** Control message reception
    ,-- ** Control message receptionreadControlMessage
    ,-- *** File descriptors
    ,-- *** File descriptorscontrolReadFd
    , controlReadFdcontrolWriteFd
    , controlWriteFdwakeupReadFd
    ,-- ** Control message sending
    ,-- ** Control message sendingsendWakeup
    , sendWakeupsendDie
    ,-- * Utilities
    ,-- * UtilitiessetNonBlockingFD
    , setNonBlockingFDwhere
    ) where
#include "EventConfig.h"

Here's what it's supposed to look like:

module GHC.Event.Control
    (
    -- * Managing the IO manager
      Signal
    , ControlMessage(..)
    , Control
    , newControl
    , closeControl
    -- ** Control message reception
    , readControlMessage
    -- *** File descriptors
    , controlReadFd
    , controlWriteFd
    , wakeupReadFd
    -- ** Control message sending
    , sendWakeup
    , sendDie
    -- * Utilities
    , setNonBlockingFD
    ) where

#include "EventConfig.h"

I had some rough theories before looking at this, but none of them are consistent with this behavior.

[alanz]

That's odd. Normally anything with a #include fails the tests. So I am not sure why it passes on linux, tbh.

[alanz]

Does it pass if you delete that line from the file? The #include.

CPP must die for haskell.

[TikhonJelvis]

Commenting out or deleting the #include resulted in the whole module getting messed up in the same way. The version with an #include was only messed up until the #include, after which it was totally fine.

Off-hand, this looks a bit like two streams (ie STDOUT and STDERR) getting interleaved in the .out file. When the #include is there, it's possible one stream failed and the other didn't, leaving the rest of the module untouched. Is there anything in the test runner that could cause this?

[alanz]

Can you make the .out file that you get (together with the original if you have changed it) available via a gist or some such?

[TikhonJelvis]

Yes, I'm just rerunning the tests to do exactly that :).

[alanz]

BTW, there is a function tt' in Tests.hs that I normally edit to run specific tests, then do it in the repl.

That way you can run a single test at a time.

[TikhonJelvis]

Okay, here's a Gist with both outputs: https://gist.github.com/TikhonJelvis/fb2b4ec9dbc4e73eee060883c33f9fda

The only change I made to Control.hs was deleting the #include "EventConfig.h" line for one of the runs.

[TikhonJelvis]

Oh yeah, here's the output for the one other failing test case (Internals.hs). I didn't make any changes to the original.

https://gist.github.com/TikhonJelvis/4f7049926cf8e984692f41540917cde6

[jwiegley]

@alanz I'm seeing the same test failures as @TikhonJelvis is, building with Nix, using either GHC 8.2 and 8.4.

[alanz]

I just re-read this thread, and it struck me that there is something about the Mac CPP that might be causing an issue. But I forget what it is.

I guess we need to set up some mac CI, however that is done.

[TikhonJelvis]

Does ghc-exactprint depend on the system CPP?

[alanz]

It depends on whatever GHC uses, so in that sense yes.

[TikhonJelvis]

Makes sense, thanks.

Is there some way to get GHC to use a different CPP implementation? I believe there's one written in Haskell. At the very least this would let me rerun the tests locally to see if it makes a difference.

[jwiegley]

@alanz Are you using Travis for CI now?

[alanz]

For ghc-exactprint, yes. Will gladly accept a PR to add macos to the matrix