Internal error when clicking on the timeline button

[irevoire]

Hey!

I finally got everything set up correctly, wrote a small client to interact with rebenchDB that seems to work, and took a bunch of old benchmark runs I had (~900 commits and about 20 datapoints per commit, I would say).

Everything was accepted correctly from what I see, but in the end the view is really strange, I can't compare two commits, and the timeline button doesn't work and return an internal error.

I think the internal error is generated because of this message in the logs:

rebenchdb-app-1  | 
rebenchdb-app-1  |   TypeError: ejs:50
rebenchdb-app-1  |       48| <div class="container-fluid"><div class="row flex-xl-nowrap">
rebenchdb-app-1  |       49|   <nav class="compare">
rebenchdb-app-1  |    >> 50|   {% for (const b of it.benchmarks) {
rebenchdb-app-1  |       51|        for (const e of b.exec) { %}
rebenchdb-app-1  |       52|       <nav><span>{%= e.execName %}</span>
rebenchdb-app-1  |       53|       <a href="#se-{%= b.suiteId %}-{%= e.execId %}">{%= b.suiteName %}</a>
rebenchdb-app-1  |   
rebenchdb-app-1  |   it.benchmarks is not iterable
rebenchdb-app-1  |       at eval (eval at compile (/project/node_modules/ejs/lib/ejs.js:673:12), <anonymous>:36:27)
rebenchdb-app-1  |       at anonymous (/project/node_modules/ejs/lib/ejs.js:703:17)
rebenchdb-app-1  |       at renderTimeline (/project/src/backend/timeline/timeline.ts:52:16)
rebenchdb-app-1  |       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
rebenchdb-app-1  | 

---

Here's the view I get right before clicking on the timeline button:

As you can see I can't compare two commits since I only have one column of commits.

---

And if it can help, here is a subset of what I sent to rebench:

Toggle me!

{
// ...
    {
      "d": [
        {
          "in": 1,
          "it": 10,
          "m": [
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            },
            {
              "c": 0,
              "v": 37195962.712608315
            }
          ]
        }
      ],
      "runId": {
        "benchmark": {
          "name": "smol-songs.csv: words/the black saint and the sinner lady and the good doggo ",
          "suite": {
            "name": "search_songs_main_70c906d4/smol-songs.csv: words/the black saint and the sinner lady and the good doggo ",
            "desc": null,
            "executor": {
              "name": "Bench",
              "desc": null
            }
          },
          "runDetails": {
            "maxInvocationTime": 0,
            "minIterationTime": 0,
            "warmup": null
          },
          "desc": null
        },
        "cmdline": "cargo bench --bench search_songs -- smol-songs.csv: words/the black saint and the sinner lady and the good doggo ",
        "location": "smol-songs.csv_ words/the black saint and the sinner lady and the good doggo ",
        "varValue": "the black saint and the sinner lady and the good doggo ",
        "cores": null,
        "inputSize": null,
        "extraArgs": null
      }
    },
    {
      "d": [
        {
          "in": 1,
          "it": 10,
          "m": [
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            },
            {
              "c": 0,
              "v": 55297151.1253019
            }
          ]
        }
      ],
      "runId": {
        "benchmark": {
          "name": "smol-songs.csv: words/whathavenotnsuchforth and a good amount of words to pop to match the first one ",
          "suite": {
            "name": "search_songs_main_70c906d4/smol-songs.csv: words/whathavenotnsuchforth and a good amount of words to pop to match the first one ",
            "desc": null,
            "executor": {
              "name": "Bench",
              "desc": null
            }
          },
          "runDetails": {
            "maxInvocationTime": 0,
            "minIterationTime": 0,
            "warmup": null
          },
          "desc": null
        },
        "cmdline": "cargo bench --bench search_songs -- smol-songs.csv: words/whathavenotnsuchforth and a good amount of words to pop to match the first one ",
        "location": "smol-songs.csv_ words/whathavenotnsuchforth and a good amount of words to pop to match",
        "varValue": "whathavenotnsuchforth and a good amount of words to pop to match the first one ",
        "cores": null,
        "inputSize": null,
        "extraArgs": null
      }
    }
  ],
  "criteria": [
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    },
    {
      "i": 0,
      "c": "total",
      "u": "ms"
    }
  ],
  "env": {
    "hostname": null,
    "cpu": "Bench",
    "clockSpeed": 0,
    "memory": 4194304,
    "osType": "Linux",
    "software": [],
    "userName": "Bench",
    "manualRun": false
  },
  "source": {
    "repoUrl": "http://github.com/meilisearch/meilisearch",
    "branchOrTag": "main",
    "commitId": "70c906d4",
    "commitMsg": "Merge #3576\n\n3576: Add boolean support for csv documents r=irevoire a=irevoire\n\nFixes https://github.com/meilisearch/meilisearch/issues/3572\r\n\r\n## What does this PR do?\r\nAdd support for the boolean types in csv documents.\r\nThe type definition is `boolean` and the possible values are\r\n- `true` for true\r\n- `false` for false\r\n- ` ` for null\r\n\r\nHere is an example:\r\n```csv\r\n#id,cute:boolean\r\n0,true\r\n1,false\r\n2,\r\n```\n\nCo-authored-by: Tamo <[email protected]>\n",
    "authorName": "bors[bot]",
    "authorEmail": "26634292+bors[bot]@users.noreply.github.com",
    "committerName": "GitHub",
    "committerEmail": "[email protected]"
  },
  "experimentName": "search_songs",
  "experimentDescription": null,
  "startTime": "2023-03-14T12:28:12Z",
  "endTime": null,
  "projectName": "Milli's benchmark"
}

Which, even if it's super ugly and with a lot of repetitions, was accepted into rebenchDB;

[/home/irevoire/rebenchdb-client/src/lib.rs:18:9] ret = Response[status: 201, status_text: Created, url: http://localhost:33333/rebenchdb/results]

[smarr]

Sorry, examining a PhD today. I'll try to look at it late this week. Could you possibly give me a SQL dump of the database? This way, I'd be sure to have the same data in the database for testing.

[irevoire]

Sorry, examining a PhD today. I'll try to look at it late this week.

No problem; hope they did well 😁

Could you possibly give me a SQL dump of the database? This way, I'd be sure to have the same data in the database for testing.

Sorry, I struggled to get one and, in the end, didn't know how to do it.
I tried a bunch of stuff with docker and volumes, but nothing worked, and I still can't run rebenchDB locally 😔

I wrote a reproducer, though; if you're ok with installing rust, maybe that should put you in the same situation as I am;

% curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # install rust, C/C from https://rustup.rs/
% git clone https://github.com/irevoire/meilisearch_upload_milli_benchmark_to_rebenchdb
% cd meilisearch_upload_milli_benchmark_to_rebenchdb
% cargo run --release

That should do it if rebenchDB can be accessed on localhost:33333. If you need to change the addr you can specify it at the end: cargo run --release -- http://addr.to.rebench.

Or, if you prefer, I also made a Linux binary that you should be able to execute as-is and should reproduce the issue as well:
reproducer.gz

I had to compress it in order to send it through GitHub thus, you'll need to decompress it first:

% wget https://github.com/smarr/ReBenchDB/files/14037721/reproducer.gz
% gunzip reproducer.gz
% chmod +x reproducer
% ./reproducer [addr to rebenchDB if needed]

I can also send you a binary for macOS aarch64 if you prefer.

[irevoire]

Hey, we tried a way simpler use case, and we still don't know how to make it work:pensive:
It should be easier to reproduce on your case because it's composed of only two jons files instead of a thousands.
rebench.json
rebench-other-commit.json

[smarr]

Sorry... Unfortunately this is really not my day job. So took a long while to get to look at this, but I believe I fixed this with this commit: 93929f3

There might be another underlying reason why no benchmarks were found for the project though. Not sure.

One of the reasons for the delay was that I finally wanted to get #100 out of the way before delving into the various other issues.
This is a major change of the database schema and has a bit of a bug tail...
And it also changed the API. To keep my own pain as minimal as possible, I didn't add backwards compatibility though. (the code for it is there, since I use it to convert the test data, it's just not hooked up) Details for the new API are on the expanded PR summary of #100.

[irevoire]

Hey, thanks for the update!

We had to finish this subject a few weeks ago and ended up making our own frontend in the meantime.
I’m still interested in rebenchDB, though, and I’ll let you know if I end up revisiting the project one day.

Thanks

[smarr]

Ah, nice. Yeah, sometimes a quickly hacked together custom solution is really much better suited for the needs :)