can't reify muuntaja.format.core/Decode in lein uberjar

[joerupen]

I have tried to create a custom encoder/decoder for xml. When I run lein uberjar from the docker image clojure I get a compile error:

Caused by: clojure.lang.ExceptionInfo: Invalid format "application/xml" for type :muuntaja/decode. It should satisfy muuntaja.format.core.Decode {:format "application/xml", :type :muuntaja/decode, :spec [#object[application.middleware$decoder 0x63fe055 "application.middleware$decoder@63fe055"]], :coder #object[application.middleware$decoder$reify__4217 0xab90b8c "application.middleware$decoder$reify__4217@ab90b8c"], :protocol {:on muuntaja.format.core.Decode, :on-interface muuntaja.format.core.Decode, :sigs {:decode {:name decode, :arglists ([this data charset]), :doc nil}},

The decoder looks like this:

(defn decoder [options]
  (reify muuntaja.format.core/Decode
    (decode [this data charset]
      .....)))))

What's weird is that it works on my local machine (sounds familiar ;). This only occurs in the docker image. I can't see what should make a difference.

Essentially this expression turns out to be false inside the container when lein uberjar executes.

(satisfies? muuntaja.format.core/Decode
  (reify muuntaja.format.core/Decode
    (decode [this data charset]
      .....))
=> false

For other protocols it works:

(defprotocol Qux 
  (decode [this data charset]))

(satisfies? Qux
  (reify Qux
    (decode [this data charset]
      .....))
=> true

This occurs when I aot-compile the whole application. What is not clear to me:

• does muuntaja support AOT compilation or is it not compatible with?
• what is special about the namespace muuntaja.format.core
• why does it work on my local machine but not on docker?
• is this a clojure bug or a muuntaja bug? (I take docker out of the mix, don't think it's related to it. I suppose this bug should be reproducible on linux.)

I use this dockerfile

FROM clojure
COPY . /usr/src/app
WORKDIR /usr/src/app

RUN lein uberjar

Then execute docker build .

• lein version 2.8.1
• java 1.8.0_111

[ikitommi]

This is a Clojure AOT feature. We only AOT the entry point into our apps, which dymanically loads all the things. But slower to start thou. All our libs are deployed as source code to avoid these problems.

The case of failure is that for some reason, there are two versions of the Decode protocol in the final classpath, a other AOTd, and another not - and the reification fails. I would ask on Clojurians Slack how to debug this, if you would like full AOT.

[ikitommi]

A minimal repro would help.

[ikitommi]

One thing to try: lein do clean, uberjar to get rid of cached compiled classes.

[joerupen]

Here a minimal thing to reproduce:

Create a small app using muuntaja

lein new luminus protocol-bug

Here I import the muuntaja namespaces and implement the decoder/encoder. Also I add gen-class

cat <<EOF > src/clj/protocol-bug/middleware.clj
(ns protocol-bug.middleware
  (:require [protocol-bug.env :refer [defaults]]
            [cheshire.generate :as cheshire]
            [cognitect.transit :as transit]
            [clojure.tools.logging :as log]
            [protocol-bug.layout :refer [error-page]]
            [ring.middleware.anti-forgery :refer [wrap-anti-forgery]]
            [ring.middleware.webjars :refer [wrap-webjars]]
            [protocol-bug.middleware.formats :as formats]
            [muuntaja.middleware :refer [wrap-format wrap-params]]
            [protocol-bug.config :refer [env]]
            [ring.middleware.flash :refer [wrap-flash]]
            [immutant.web.middleware :refer [wrap-session]]
            [ring.middleware.defaults :refer [site-defaults wrap-defaults]]
            [muuntaja.format.core :as cf]
            [muuntaja.core :as m]
            [muuntaja.middleware :as mw])
  (:import  [org.joda.time ReadableInstant])
  (:gen-class))

(defn wrap-internal-error [handler]
  (fn [req]
    (try
      (handler req)
      (catch Throwable t
        (log/error t (.getMessage t))
        (error-page {:status 500
                     :title "Something very bad has happened!"
                     :message "We've dispatched a team of highly trained gnomes to take care of the problem."})))))

(defn wrap-csrf [handler]
  (wrap-anti-forgery
    handler
    {:error-response
     (error-page
       {:status 403
        :title "Invalid anti-forgery token"})}))


(defn wrap-formats [handler]
  (let [wrapped (-> handler wrap-params (wrap-format formats/instance))]
    (fn [request]
      ;; disable wrap-formats for websockets
      ;; since they're not compatible with this middleware
      ((if (:websocket? request) handler wrapped) request))))

(defn wrap-base [handler]
  (-> ((:middleware defaults) handler)
      wrap-webjars
      wrap-flash
      (wrap-session {:cookie-attrs {:http-only true}})
      (wrap-defaults
        (-> site-defaults
            (assoc-in [:security :anti-forgery] false)
            (dissoc :session)))
      wrap-internal-error))

(defn encoder [_]
  (reify
    cf/EncodeToBytes
    (encode-to-bytes [_ data charset]
      nil)
    cf/EncodeToOutputStream
    (encode-to-output-stream [_ data charset]
      nil)))


(defn decoder [options]
  (reify cf/Decode
    (decode [this data charset]
      nil)))


(def options
  (m/create (-> m/default-options
                (assoc-in [:formats "application/xml"]
                          {
                           :encoder [encoder]
                           :decoder [decoder]
                           })
               )))
EOF

Modify the dockerfile

cat <<EOF > Dockerfile
FROM clojure
COPY . /usr/src/app
WORKDIR /usr/src/app
RUN lein uberjar
CMD ["java", "-jar", "/usr/app/protocol-bug.jar"]
EOF

And here goes the error.

docker build .

[joerupen]

My temporary workaround is:

Instead of

(def options
  (m/create (-> m/default-options
                (assoc-in [:formats "application/xml"]
                          {
                           :encoder [encoder]
                           :decoder [decoder]
                           })
               )))

I simply inline the call in the wrap- function, so it's not executed at compile time.

(defn wrap-format [handler]
  (mw/wrap-format handler (m/create (-> m/default-options
                                        (assoc-in [:formats "application/xml"]
                                                  {
                                                   :encoder [encoder]
                                                   :decoder [decoder]
                                                   })
                                        ))))

[ikitommi]

Hi. I can reproduce this. I think this is a Leiningen / Clojure bug: the namespaces are not compiled in right order (Leiningen) and double-compile fails currently (https://dev.clojure.org/jira/browse/CLJ-1741). A workaround is to define the aot-oder manually:

:aot [protocol-bug.env protocol-bug.layout protocol-bug.middleware.formats protocol-bug.nrepl protocol-bug.middleware protocol-bug.core protocol-bug.handler protocol-bug.routes.home protocol-bug.config]

, which is really bad. Will try to resolve this.

[ikitommi]

locally, I get a working order:

Compiling protocol-bug.env
Compiling protocol-bug.layout
Compiling protocol-bug.middleware.formats
Compiling protocol-bug.nrepl
Compiling protocol-bug.middleware
Compiling protocol-bug.core
Compiling protocol-bug.handler
Compiling protocol-bug.routes.home
Compiling protocol-bug.config

On docker, its:

Compiling protocol-bug.env
Compiling protocol-bug.core
Compiling protocol-bug.nrepl
Compiling protocol-bug.routes.home
Compiling protocol-bug.layout
Compiling protocol-bug.config
Compiling protocol-bug.middleware.formats
Compiling protocol-bug.handler
Compiling protocol-bug.middleware

[ikitommi]

More pointers

• https://dev.clojure.org/jira/browse/CLJ-2303
• Wrong compilation order (uberjar, check) (WAS: with linux x86-64) technomancy/leiningen#2508

[ikitommi]

Tested this with 2.9.1, it now fails on all environments. Asked to reopen the Leiningen issue.

Bad, bad, bad.