setSaveAfterEachStimulus() overwrites trials' existing on_finish callback

[jessestorbeck]

The current version of the setSaveAfterEachStimulus looks like this:

  setSaveAfterEachStimulus(stimuli) {
    return stimuli.map((s) => ({
      ...s,
      on_finish: this.saveStimulusResponse.bind(this),
    }));
  }

What seems to be happening is that any existing on_finish callback one includes in experiment.js gets overwritten here. The fix I tried to implement is this (see af3ab32):

  setSaveAfterEachStimulus(stimuli) {
    return stimuli.map((s) => ({
      ...s,
      // If s already has an on_finish, wrap it in a function that calls both
      // the original on_finish and the saveStimulusResponse function
      on_finish: (data) => {
        if (s.on_finish) s.on_finish(data); // If s already has an on_finish, call it
        this.saveStimulusResponse.bind(this);
      }
    }));
  }

I think I've followed the instructions here correctly to test my changes via yalc, but the data in the database look exactly the same as before. I'm not sure if my code does nothing or I didn't use yalc correctly. I think there may be some missing steps in the instructions, so here's what I did:

1. Cloned pushkin-client, made a branch, and edited the code.
2. I didn't originally run yarn install and yarn build in the root of pushkin-client when I tried this the first time, but it seems like that might be needed, so I did that. Then I ran yalc publish
3. In pushkin/front-end of my site folder, I ran yalc add pushkin-client. I can see the local version of pushkin-client in the package.json now.
4. The Dockerfile there looked like it was already set up to copy yalc files, so I left that alone.

From there, I was able to run prep and start, load the experiment, and inspect the data. So maybe my fix does nothing?

[jessestorbeck]

Aha! I think I need to yalc add in the experiment's web page folder too, since I see it's getting the npm version of pushkin-client.

[jessestorbeck]

Related to above, I'm now having a weird issue now with yalc and prep. After install site and install experiment, it seems there are a few places where I'd need to run yalc add pushkin-client. pushkin/front-end and 'experiments/[experiment]/web page' are the two I identified above. If I then run prep, I get an error that says:

Error: Command failed: yarn install
error Package "pushkin-client" refers to a non-existing file '"/Users/jessestorbeck/Desktop/clientTestSite/pushkin/front-end/.yalc/basicTest_web/.yalc/pushkin-client"'.

I believe it's install experiment that makes this folder [experiment]_web in pushkin/front-end. I attempt to fix the error by going to pushkin/front-end/.yalc/basicTest_web and running yalc add pushkin-client there. However, when I run prep again, the .yalc file that's created there disappears and I get the same error.

I also tried editing package.json in pushkin/front-end/.yalc/basicTest_web directly so that it points to pushkin-client in the parent folder instead of the disappearing .yalc folder, but prep seems to overwrite that, and I just get the same error again.

[jkhartshorne]

Related to above, I'm now having a weird issue now with yalc and prep. After install site and install experiment, it seems there are a few places where I'd need to run yalc add pushkin-client. pushkin/front-end and 'experiments/[experiment]/web page' are the two I identified above.

That's correct. You don't want to put the dev version of pushkin-client into a template. Since you are installing from a template, it'll install using the published version. You then go in and switch it to the .yalc version.

I believe it's install experiment that makes this folder [experiment]_web in pushkin/front-end.

I don't think so. I think that's made during prep. install should only put things in the /experiments folder. (I say this without rereading the code. But I'm pretty sure. You can check by running install experiment and see if that folder comes into existence.

I attempt to fix the error by going to pushkin/front-end/.yalc/basicTest_web and running yalc add pushkin-client there. However, when I run prep again, the .yalc file that's created there disappears and I get the same error.

You are right. That won't work. Basically, when you run yalc add [package] it puts that package in the .yalc folder. If there was already something there, it will be copied over. prep runs yalc add [experiment] for every experiment you've installed. The question is why, when it does that, the local package version of pushkin-client isn't coming along for the ride.

I'll see if there's a simple solution. Worst-case scenario, I think we can make prep look for any .yalc packages and copy them over manually.

[jkhartshorne]

OK, according to the yalc repository issues, this is an expected issue. What is going on is that our experiment templates specify which files to use when installing using the "files" option in package.json. It didn't say we should include ".yalc", so it didn't. What you need to do is update, for instance:

{
"name": "trystyle_web",
"version": "1.0.0",
"description": "",
"main": "build/index.js",
"files": [
"build/*",
".yalc"
],
"scripts": {
"test": "jest",
"build": "babel src -d build; cp -r src/assets build/"
},
"author": "",
"license": "MIT",
"dependencies": {
"@jspsych/plugin-html-keyboard-response": "^1.1.2",
"buffer": "^6.0.3",
"build-if-changed": "^1.5.5",
"js-yaml": "^4.1.0",
"jspsych": "^7.3.3",
"pushkin-client": "file:.yalc/pushkin-client",
"react": "^18.2.0",
"react-router-dom": "^5.2.0"
},
"devDependencies": {
"@babel/cli": "^7.10.5",
"@babel/core": "^7.10.5",
"@babel/preset-env": "^7.10.4",
"@babel/preset-react": "^7.10.4",
"babel-plugin-static-fs": "^3.0.0",
"jest": "^29.6.1",
"jest-environment-jsdom": "^29.6.1"
}
}

I checked, and the pushkin/front-end/package.json in the site template does not use the "files" option, so ... I think we don't need to do anything there?

Anyway, making this changed resolved the earlier error, but now I'm getting a different one:

Dockerfile:10

---

8 |
9 | WORKDIR /usr/src/app
10 | >>> RUN yarn install
11 | RUN apt-get update && apt-get install -y netcat
12 |

ERROR: failed to solve: process "/bin/sh -c yarn install" did not complete successfully: exit code: 1

If I were to hazard a guess, I'd say that there's a .yalc folder that isn't getting copied over to the Docker image. Though it's not immediately obvious to me why that would be.

It turns out I have two back-to-back meetings for the next two hours, so I won't be able to follow up on this until then.

[jessestorbeck]

I was able to add ".yalc" to package.json.files and now prep and start work for me! Unfortunately there is now zero data in the database after I run my toy experiment :(

So I guess I'll look back at what I added to setSaveAfterEachStimulus() and see if there's anything obviously wrong with what I wrote.

[jkhartshorne]

Huh. prep is still crashing on me.

[jkhartshorne]

Nevermind. It's working for me now. No idea why.

@jessestorbeck -- can you please update the docs to correctly explain how to test local versions? Thanks.

[jessestorbeck]

@jkhartshorne Can do. Will we push an update to the exp templates that adds .yalc to the files list? i.e. do I need to instruct people to make that change?

[jkhartshorne]

Will we push an update to the exp templates that adds .yalc to the files list? i.e. do I need to instruct people to make that change?

No. There won't always be a .yalc, and it will crash if it's not there. Easier to add it manually when you want to do local testing. You'll of course need to undo it before publishing.

---

Unfortunately there is now zero data in the database after I run my toy experiment :(

I just tried with the basic experiment template, and it still pushed to the database fine, even using the new version of pushkin-client. Not sure what that means.

I'm currently using commit b065705d281a0ecaa2d6f69a0258ae71d3c98951 of the basic experiment template.

[jessestorbeck]

Hmm I am using the latest basic template v5.0.1. I'm not sure why it didn't work with the published version, but I can try that version too (pushkin-consortium/pushkin-exptemplates-basic@b065705). The updated yalc instructions are now added to this PR: pushkin-consortium/pushkin#234.

Did you try to add any on_finish callbacks to the experiment? This was the test code I was using:

    const timeline = []

    var hello_trial_1 = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: 'Hello world!',
        choices: [' ']
    };

    var hello_trial_2 = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: 'Hello!',
        choices: [' '],
        data: { test: 'test' }
    };

    var hello_trial_3 = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: 'Hello world!',
        choices: [' '],
        data: { test: 'test' },
        on_finish: function (data) {
            if (data.test == 'test') {
                data.correct = true;
            } else {
                data.correct = false;
            }
        }
    };

    timeline.push(hello_trial_1, hello_trial_2, hello_trial_3);

[jkhartshorne]

I played around in console to see what code like this does:

setSaveAfterEachStimulus(stimuli) {
    return stimuli.map((s) => ({
      ...s,
      // If s already has an on_finish, wrap it in a function that calls both
      // the original on_finish and the saveStimulusResponse function
      on_finish: (data) => {
        if (s.on_finish) s.on_finish(data); // If s already has an on_finish, call it
        this.saveStimulusResponse.bind(this);
      }
    }));
  }

I think it actually sets up an infinite loop with on_finish calling itself over and over. So try in the console:

myjson = {a: () => console.log("nothing")}
myjson.a();
myjson.a = () => {myjson.a(); console.log("hi")}
myjson.a();

The final line should give you:

Uncaught RangeError: Maximum call stack size exceeded
    at myjson.a (REPL3:1:19)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)
    at myjson.a (REPL3:1:26)

I'm seeing if I can work out an alternative...

[jkhartshorne]

So the reason on_finish functions aren't triggering doesn't have to do with saveAfterEachStimulus(). I actually ran your example here without saveAfterEachStimulus() invoked, and it still didn't trigger the on_finish function. I'll need to look at on_finish in jsPsych 7 and see if we are doing something wrong.

[jodeleeuw]

Nothing changed from the user side with how on_finish works in v7 and I don't see any reason why the example code wouldn't trigger the on_finish function.

[jkhartshorne]

-----EDIT-----
I previously said that yarn build wasn't listed as a step in the gitbook. It was.

Looks like it does actually work! I confirmed by sticking an alert("hi") in Jesse's timeline:

export function createTimeline(jsPsych) {
    const timeline = []

    var hello_trial_1 = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: 'Hello world!',
        choices: [' '],
        on_finish: function (data) {
            console.log(`hello_trial_1`)
        }
    };

    var hello_trial_2 = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: 'Hello!',
        choices: [' '],
        data: { test: 'test' },
        on_finish: function(data) {
            console.log(`hello_trial_2`)
        }
    };

    var hello_trial_3 = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: 'Hello world!',
        choices: [' '],
        data: { test: 'test' },
        on_finish: function (data) {
            alert("hi!")
            if (data.test == 'test') {
                data.correct = true;
            } else {
                data.correct = false;
            }
        }
    };

    timeline.push(hello_trial_1, hello_trial_2, hello_trial_3);
    console.log(timeline)

    return timeline;
}

I think the issue was one or both of the following:

1. I forgot to run yalc build before yalc publish
2. pushkin kill isn't deleting the server for some reason, and pushkin prep wasn't reliably updating it. I'll need to look into that. We shouldn't have to delete everything to update it.

[jkhartshorne]

I have confirmed, however, that with this upgrade no data is now written to the database at all. So I'm going to look at that next.

[jkhartshorne]

OK, I think I figured it out. on_finish expects to have as a value a function, to which it passes data. In the old setSaveAfterEverySimulus, we wrote a new value for on_finish that was a function (saveStimulusResponse()). It gets called with the argument data. So far, so good.

HOWEVER, with our new code, we simply list the function, but doesn't give it any arguments:

setSaveAfterEachStimulus(stimuli) {
    return stimuli.map((s) => ({
      ...s,
      // If s already has an on_finish, wrap it in a function that calls both
      // the original on_finish and the saveStimulusResponse function
      on_finish: (data) => {
        if (s.on_finish) s.on_finish(data); // If s already has an on_finish, call it
        this.saveStimulusResponse.bind(this);
      }
    }));
  }

This can be solved with

this.saveStimulusResponse.bind(this)(data);

This works for the example we've been working with. So the basic template is back to working. I don't know yet if it'll address the problem with the other templates. But I'm going to put in a pull request on pushkin-client for at least this much.

[jodeleeuw]

FYI it might be easier and less intrusive to the user's timeline to use the on_data_update event in initJsPsych() instead of modifying the timeline like this. Since initJsPsych is called in the template you could just modify it directly instead of having to intercept the timeline and modify it.

[jodeleeuw]

Relevant docs: https://www.jspsych.org/7.3/overview/events/#on_data_update

[jkhartshorne]

@jodeleeuw Thanks for that. That does sound better. However, the new code is passing the tests. (I wrote it last night but hadn't finished writing the unit tests.) If it also works in end-to-end (which I don't have automated), I'll just stick with it for now. But I've added what you wrote as a comment in the relevant code in pushkin-client. If we run into issues in the future, no doubt whoever is debugging will see that comment :)

[jkhartshorne]

OK. Somehow it updated most of the on_finish's. For the SPR experiment, it now records the RT for the last word of the sentence. Which ... yay?

I'm going to try @jodeleeuw's suggestion.

[jkhartshorne]

Relevant docs: https://www.jspsych.org/7.3/overview/events/#on_data_update

I just wanted to confirm that this can only be modified during initJsPsych(). There's no good way to edit it afterwards, right?

[jessestorbeck]

I just tried with pushkin client at commit e3f0e8f. I can confirm that all the data looks good except for the SPR template. I'm getting nothing from the SPR plugin (should appear as self-paced-reading), but I am getting data from the comprehension questions, which were missing previously.

[jodeleeuw]

I just wanted to confirm that this can only be modified during initJsPsych(). There's no good way to edit it afterwards, right?

This isn't supported behavior, meaning it'll almost certainly break when we release v8.0, but you can do this:

const jsPsych = initJsPsych()

jsPsych.opts.on_data_update = (data) => { console.log(data) }

[jkhartshorne]

I just tried with pushkin client at commit e3f0e8f. I can confirm that all the data looks good except for the SPR template. I'm getting nothing from the SPR plugin (should appear as self-paced-reading), but I am getting data from the comprehension questions, which were missing previously.

Yah. Same here. I'm rewriting to use @jodeleeuw's suggestion. It is much more elegant, assuming it works. (I'm having a little trouble getting the syntax right for use with pushkin.saveStimulusResponse(). But I'll get there...

[jodeleeuw]

What about just adding it directly to the template instead of making a separate api thing for it?

https://github.com/pushkin-consortium/pushkin-exptemplates-basic/blob/028953ca034443c99987f9e381b417d7ddfcc8e7/web%20page/src/index.js#L39-L42

[jkhartshorne]

What about just adding it directly to the template instead of making a separate api thing for it?

https://github.com/pushkin-consortium/pushkin-exptemplates-basic/blob/028953ca034443c99987f9e381b417d7ddfcc8e7/web%20page/src/index.js#L39-L42

Yah, I decided that was equally easy. So I did that. But I think the issue with not recording data from the actual SPR trials is a jsPsych thing -- or at least, our failure to understand jsPsych. So on_data_update is NOT triggered by the self-paced reading trials themselves. The experiment is set up in

index.js

import React from 'react';
import pushkinClient from 'pushkin-client';
import { initJsPsych } from 'jspsych';
import { connect } from 'react-redux';
import { createTimeline } from './experiment';
import jsYaml from 'js-yaml';
const fs = require('fs');

//stylin'
import './assets/experiment.css';
import experimentConfig from './config';

const expConfig = jsYaml.load(fs.readFileSync('../config.yaml'), 'utf8');

const pushkin = new pushkinClient();

const mapStateToProps = state => {
  return {
    userID: state.userInfo.userID
  };
}

class quizComponent extends React.Component {

  constructor(props) {
    super(props);
    this.state = { loading: true };
  }

  componentDidMount() {
    this.startExperiment();
  }

  async startExperiment() {
    this.setState({ experimentStarted: true });

    await pushkin.connect(this.props.api);
    await pushkin.prepExperimentRun(this.props.userID);

    const jsPsych = initJsPsych({
      display_element: document.getElementById('jsPsychTarget'),
      on_finish: this.endExperiment.bind(this),
      on_data_update: pushkin.saveStimulusResponse(data),
    });

    jsPsych.data.addProperties({user_id: this.props.userID}); //See https://www.jspsych.org/core_library/jspsych-data/#jspsychdataaddproperties
    
    const timeline = pushkin.setSaveAfterEachStimulus(createTimeline(jsPsych));

    jsPsych.run(timeline);

    document.getElementById('jsPsychTarget').focus();
    
    // Settings from config file
    document.getElementById('jsPsychTarget').style.color = experimentConfig.fontColor;
    document.getElementById('jsPsychTarget').style.fontSize = experimentConfig.fontSize;
    document.getElementById('jsPsychTarget').style.fontFamily = experimentConfig.fontFamily;
    document.getElementById('jsPsychTarget').style.paddingTop = '15px';

    this.setState({ loading: false });
  }

  async endExperiment() {
    document.getElementById("jsPsychTarget").innerHTML = "Processing...";
    await pushkin.tabulateAndPostResults(this.props.userID, expConfig.experimentName)
    document.getElementById("jsPsychTarget").innerHTML = "Thank you for participating!";
  }

  render() {

    return (
      <div>
        {this.state.loading && <h1>Loading...</h1>}
        <div id="jsPsychTarget" />
      </div>
    );
  }
}

export default connect(mapStateToProps)(quizComponent);

The timeline can be found in experiment.js:

import jsPsychHtmlKeyboardResponse from '@jspsych/plugin-html-keyboard-response';
import jsPsychSelfPacedReading from '@jspsych-contrib/plugin-self-paced-reading';
import experimentConfig from './config';
import consent from './consent';
import stimArray from './stim';
import debrief from './debrief';

export function createTimeline(jsPsych) {
    // Construct the timeline inside this function just as you would in jsPsych v7.x
    const timeline = [];

    // Resize the jsPsychTarget div
    // This will help later with centering content
    // and making sure the SPR canvas doesn't change the page layout
    var resizejsPsychTarget = function (divHeight) {
        let jsPsychTarget = document.querySelector('#jsPsychTarget');
        jsPsychTarget.style.height = divHeight + 'px';
    };
    // This will set the height of the jsPsychTarget div to half the height of the browser window
    // You can play with the multiplier according to the needs of your experiment
    resizejsPsychTarget(window.innerHeight * 0.5);
    
    // Add correct answer information to stimArray for comprehension questions
    var correctKey = jsPsych.randomization.sampleWithReplacement(['f','j'], stimArray.length);
    for (let i = 0; i < stimArray.length; i++) {
        // Add the correct key to the end of the array holding the question info
        stimArray[i].comprehension.push(correctKey[i]);
    };
    
    // Welcome/consent page
    var welcome = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: consent + '<p>Press spacebar to continue.</p>',
        choices: [' ']
    };
    timeline.push(welcome);

    // Instruction page
    var instructions = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: function () {
            let standard_instructions = `<p>In this experiment, you'll read sentences one word at a time. Use the spacebar to advance to the next word.</p>`;
            let comprehension_instructions = `<p>After reading the sentence, you'll be asked a question about what you read. Use the keyboard to respond to the question.</p>`;
            if (experimentConfig.comprehension) {
                return standard_instructions + comprehension_instructions + '<p>Press the spacebar to continue to the experiment.</p>';
            } else {
                return standard_instructions + '<p>Press spacebar to continue to the experiment.</p>';
            }
        },
        choices: [' ']
    };
    timeline.push(instructions);

    // Set up feedback
    // This will later be integrated into the comprehension questions
    var feedback = {
        timeline: [
            {
                type: jsPsychHtmlKeyboardResponse,
                stimulus: function () {
                    let last_correct = jsPsych.data.getLastTrialData().values()[0].correct;
                    if (last_correct) {
                        return '<p class="correct"><strong>Correct</strong></p>';
                    } else {
                        return '<p class="incorrect"><strong>Incorrect</strong></p>';
                    }
                },
                choices: 'NO_KEYS',
                trial_duration: 2000
            },
        ],
        // This timeline is only executed if correctiveFeedback is set to true in config.js
        conditional_function: function () {
            return experimentConfig.correctiveFeedback;
        }
    };
    
    // Set up comprehension questions
    // This will later be integrated into the experiment trials
    var comprehension_questions = {
        timeline: [
            {
                type: jsPsychHtmlKeyboardResponse,
                data: {comprehension: jsPsych.timelineVariable('comprehension')},
                stimulus: function () {
                    let question = jsPsych.timelineVariable('comprehension')[0];
                    let choice_correct = jsPsych.timelineVariable('comprehension')[1];
                    let choice_incorrect = jsPsych.timelineVariable('comprehension')[2];
                    let correct_key = jsPsych.timelineVariable('comprehension')[3];
                    if (correct_key == 'f') {
                        return `<p>${question}</p><p><strong>F.</strong> ${choice_correct}&emsp;<strong>J.</strong> ${choice_incorrect}</p>`;
                    } else {
                        return `<p>${question}</p><p><strong>F.</strong> ${choice_incorrect}&emsp;<strong>J.</strong> ${choice_correct}</p>`;
                    }
                },
                choices: ['f','j'],
                // Check whether the response was correct
                // compareKeys() will return true if the response key matches the correct key from the timeline variable
                on_finish: function (data) {
                    data.correct = jsPsych.pluginAPI.compareKeys(jsPsych.timelineVariable('comprehension')[3], data.response);
                }
            },
            // Integrate the conditional timeline for feedback
            feedback
        ],
        // This timeline (including the sub-timeline for feedback) is only executed if comprehension is set to true in config.js
        conditional_function: function () {
            return experimentConfig.comprehension;
        }
    };

    // Experiment trials
    var experiment = {
        timeline: [
            {
                type: jsPsychHtmlKeyboardResponse,
                stimulus: 'Press spacebar when you are ready.',
                choices: [' ']
            },
            {
                type: jsPsychSelfPacedReading,
                // Learn more about the parameters available for this plugin here: https://github.com/jspsych/jspsych-contrib/blob/main/packages/plugin-self-paced-reading/docs/jspsych-self-paced-reading.md
                sentence: jsPsych.timelineVariable('sentence'),
                // Set the size of the SPR canvas to match the size of the jsPsych content area
                canvas_size: function () {
                    let width = document.querySelector('.jspsych-content-wrapper').offsetWidth;
                    let height = document.querySelector('.jspsych-content-wrapper').offsetHeight;
                    return [width, height];
                }
            },
            // Integrate the conditional timeline for comprehension questions
            comprehension_questions
        ],
        timeline_variables: stimArray,
        randomize_order: true
    };
    timeline.push(experiment);

    // A final feedback and debrief page
    var end = {
        type: jsPsychHtmlKeyboardResponse,
        stimulus: function () {
            // Calculate performance on comprehension questions
            let correct_questions = jsPsych.data.get().filter({ correct: true }).count();
            let total_questions = jsPsych.data.get().filterCustom(function (data) { return Array.isArray(data.comprehension) }).count();
            let mean_correct_rt = jsPsych.data.get().filter({ correct: true }).select('rt').mean();
            
            // Show results and debrief from debrief.js
            let results = `
                <p>You were correct on ${correct_questions} of ${total_questions} questions!
                Your average response time for these was ${Math.round(mean_correct_rt)} milliseconds.</p>
            `
            if (experimentConfig.comprehension) {
                return results + debrief + '<p>Press spacebar to finish.</p>'
            } else {
                return debrief + '<p>Press spacebar to finish.</p>'
            }
        },
        choices: [' ']
    };
    timeline.push(end);

    return timeline;
}

[jkhartshorne]

@jessestorbeck while we wait to see if @jodeleeuw can find something wrong with our code, do I remember correctly that there's a plain vanilla jsPsych version of this experiment? It presumably posts all the data at the end? Can you see if it posts the individual trial data?

[jkhartshorne]

@jodeleeuw @jessestorbeck OK pretty sure it's an issue with the contributed plugin. jspsych/jspsych-contrib#75

@jessestorbeck -- do you want to try putting together a minimal working example that doesn't use pushkin but shows this same error? Or a minimal working example that doesn't use pushkin and doesn't show the error. Either one would be helpful!

[jessestorbeck]

Here's what one run of the SPR procedure in vanilla jsPsych looks like:

	{
		"rt": 407,
		"stimulus": "Press spacebar when you are ready.",
		"response": " ",
		"trial_type": "html-keyboard-response",
		"trial_index": 2,
		"time_elapsed": 4653,
		"internal_node_id": "0.0-2.0-0.0"
	},

We get this pre-SPR readiness check in Pushkin, but then the SPR plugin takes over, and the following data doesn't make it to the database.

	{
		"rt_sentence": 202,
		"rt_word": 202,
		"word_number": 0,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 4862,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 388,
		"rt_word": 186,
		"word": "The",
		"word_number": 1,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 5047,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 550,
		"rt_word": 162,
		"word": "horse",
		"word_number": 2,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 5210,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 718,
		"rt_word": 168,
		"word": "raced",
		"word_number": 3,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 5377,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 888,
		"rt_word": 170,
		"word": "past",
		"word_number": 4,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 5547,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 1061,
		"rt_word": 173,
		"word": "the",
		"word_number": 5,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 5720,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 1263,
		"rt_word": 202,
		"word": "barn",
		"word_number": 6,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 5923,
		"internal_node_id": "0.0-2.0-1.0"
	},
	{
		"rt_sentence": 1534,
		"rt_word": 271,
		"word": "fell.",
		"word_number": 7,
		"sentence": "The horse raced past the barn fell.",
		"trial_type": "self-paced-reading",
		"trial_index": 3,
		"time_elapsed": 6193,
		"internal_node_id": "0.0-2.0-1.0"
	},

Now the SPR plugin is done, and we do get the following trials in the database.

	{
		"rt": 2421,
		"stimulus": "<p>What fell?</p><p><strong>F.</strong> The horse&emsp;<strong>J.</strong> The barn</p>",
		"response": "f",
		"comprehension": [
			"What fell?",
			"The horse",
			"The barn",
			"f"
		],
		"trial_type": "html-keyboard-response",
		"trial_index": 4,
		"time_elapsed": 8615,
		"internal_node_id": "0.0-2.0-2.0-0.0",
		"correct": true
	},
	{
		"rt": null,
		"stimulus": "<p class=\"correct\"><strong>Correct</strong></p>",
		"response": null,
		"trial_type": "html-keyboard-response",
		"trial_index": 5,
		"time_elapsed": 10625,
		"internal_node_id": "0.0-2.0-2.0-1.0-0.0"
	}

I am wondering if it has anything to do with the fact that the SPR plugin is a bit different in that the data it outputs looks like it's coming from distinct trials, but the trial_index value is the same for each one.

[jkhartshorne]

We get this pre-SPR readiness check in Pushkin, but then the SPR plugin takes over, and the following data doesn't make it to the database.

How did you get that printed out? Can you please post all the code somewhere?

I am wondering if it has anything to do with the fact that the SPR plugin is a bit different in that the data it outputs looks like it's coming from distinct trials, but the trial_index value is the same for each one.

That seems quite plausible.

[jessestorbeck]

@jkhartshorne the vanilla jsPsych code is now here.

[jessestorbeck]

To add to that, the data I pasted in above is what comes out after you finish the timeline as a vanilla jsPsych experiment. It comes from jsPsych.data.displayData() in the excerpt below:

    const jsPsych = initJsPsych({
      on_finish: function() {
        jsPsych.data.displayData();
      }
    });

[jkhartshorne]

@jkhartshorne the vanilla jsPsych code is now here.

How do you test it? If I try opening the HTML file in my browser, I get:

[jessestorbeck]

I use the Live Server extension for VSCode

[jkhartshorne]

So looking at the SPR plugin, I see that after every word, it writes data using jsPsych.data.write()

      if (response.rt_word > 0) {
        // valid rts
        response.word = words_concat[word_number];
        response.word_number = word_number + 1;
        if (trial.save_sentence) {
          response.sentence = sentence;
        }
        if (word_number < sentence_length - 1) {
          this.jsPsych.data.write(response);
        }
        // keep drawing until words in sentence complete
        word_number++;
        this.jsPsych.pluginAPI.setTimeout(function () {
          if (word_number < sentence_length) {
            clear_canvas();
            draw_mask();
            draw_word();
          } else {
            end_trial();
          }
        }, trial.inter_word_interval);
      } else {
        rts.pop(); // invalid rt possible when trial.inter_word_interval is > 0
      }

The jsPsych docs say this is a bad idea:

This method is used by jsPsych.finishTrial for writing data. You should probably not use it to add data. Instead use jsPsych.data.addProperties.

In the example of the code, we are warned

 // don't use this! data should only be written once per trial. use jsPsych.finishTrial to save data.

[jkhartshorne]

@jessestorbeck I have rewritten the self-paced-reading plugin: https://github.com/jkhartshorne/jspsych-contrib/tree/SRD-data

I'm not entirely sure how to test it locally. Your MWE pulls the code from an online library, and I assume we don't want to do that. @jodeleeuw may have some thoughts.

Anyway, please test it out and see if you can get it to work. If it's easier, you can try it from within Pushkin using yalc to import the module locally.

[jessestorbeck]

I couldn't figure out how to test it within a plain jsPsych experiment, but I was able to yalc add it to a Pushkin experiment in the process of testing for pushkin-consortium/pushkin-exptemplates-reading#3. I am pretty sure I did this correctly running yalc add @jspsych-contrib/plugin-self-paced-reading in the experiment's web page directory; however, there's still no data from the plugin being recorded in the database. I am using the updated client and api as well: 52d55ac and pushkin-consortium/pushkin-api@d5e61e5.

I noticed in the index.js for the reading template (pushkin-consortium/pushkin-exptemplates-reading@ebbd439), there's still a call to setSaveAfterEachStimulus:

const timeline = pushkin.setSaveAfterEachStimulus(createTimeline(jsPsych));

which I replaced with:

const timeline = createTimeline(jsPsych);

But that didn't fix the missing data issue.