Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nanny task ending run prematurely #202

Open
johnaohara opened this issue Jun 22, 2023 · 1 comment
Open

nanny task ending run prematurely #202

johnaohara opened this issue Jun 22, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@johnaohara
Copy link
Member

the nannytask will terminate a run if it finds scripts that are not suspended at a wait-for or a sh command that has not returned.

if the command is something else, e.g. sleep, the trun is terminated prematurely.

e.g. for a script that does;

...
script:
...
  capture-stats: 
    ...
            then:
            - set-state: RUN.${{SCOMBO}}.warm ${{=[...${{RUN.${{SCOMBO}}.warm}} , ${{WARM}}]}}
            - sleep: 2000
          - signal: ${{DCOMBO}}-DONE
...
    start-docker: 
    - for-each: APP ${{QUICKSTART_APPS}}
      then:
      - for-each: VARIANT ${{VARIANTS}}
        then:
        - set-state: VALUES ''
        - set-state: DCOMBO ${{APP.name}}-${{VARIANT.name}}-${{CPU.cores}}
        - for-each: CPU ${{CPUS}}
          then:
          - for-each: ITERATION ${{=[...Array(${{APP_STARTS}}).keys()]}}
            then:
            ...
            - wait-for: ${{DCOMBO}}-DONE

qDup will terminate the run if the only non wait-for script running is executing sleep

�[m�[37m05:24:12.033 ending phase with 3 active idle waiting scripts
[
  {
    "uid": 1045,
    "lastUpdate": 1687411450329,
    "name": "wait-for: config-quickstart-native-1-WRK-START",
    "host": "[email protected]:22",
    "contextId": "run-wrk:1030@[email protected]:22",
    "startTime": 1687411450329,
    "runTime": 1703,
    "idleTime": 1703,
    "script": "1030:script-cmd: run-wrk"
  },
  {
    "uid": 1124,
    "lastUpdate": 1687411450364,
    "name": "sleep: 2000",
    "host": "[email protected]:22",
    "contextId": "capture-stats:1032@[email protected]:22",
    "startTime": 1687411450364,
    "runTime": 1668,
    "idleTime": 1668,
    "script": "1032:script-cmd: capture-stats"
  },
  {
    "uid": 1111,
    "lastUpdate": 1687411405662,
    "name": "wait-for: config-quickstart-JVM-16-DONE",
    "host": "[email protected]:22",
    "contextId": "start-docker:1034@[email protected]:22",
    "startTime": 1687411405662,
    "runTime": 46370,
    "idleTime": 46370,
    "script": "1034:script-cmd: start-docker"
  }
]
@johnaohara johnaohara added the bug Something isn't working label Jun 22, 2023
@johnaohara johnaohara changed the title nanny task ending run prematurly nanny task ending run prematurely Jun 22, 2023
@johnaohara
Copy link
Member Author

Digging some more into this, the capture-stats script sleep is actually in a reapeat-until loop;

          - repeat-until: ${{DCOMBO}}-WRK-DONE
            then:
            - sh: sudo pmap -x ${{PID}} | grep total | awk '{print $4}' | sed 's/[^0-9]*//g'
              then:
              - set-state: WARM
            - set-state: RUN.${{SCOMBO}}.warm ${{=[...${{RUN.${{SCOMBO}}.warm}} , ${{WARM}}]}}
            - sleep: 2000
          - signal: ${{DCOMBO}}-DONE

the nannytask should walk the Cmd stack and find the repeat-until command, realise that the script is waiting for a signal (which may have already been fired) and not stop the run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant