Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] - cncli.sh - integration of full stakepool history load into epochdata table #1793

Merged
merged 23 commits into from
Sep 25, 2024

Commits on Jul 30, 2024

  1. added new function getProtocolParamsHist in env file

    added epochdata table load code in cncli.sh
    Manuel committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    a34741d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8fe2d58 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. added requested changes for epochdata

    Manuel committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    448aeb5 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. updated getConsensus function to use getProtocolParams on next_epoch …

    …instead of getProtocolParamsHist
    Manuel committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    fa5c98d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c5eb3fe View commit details
    Browse the repository at this point in the history
  3. small syntax change

    Manuel committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    bc9f59f View commit details
    Browse the repository at this point in the history
  4. cncli epochdata: updated sql query for EPOCHS variable:

    EPOCHS=$(sqlite3 "$BLOCKLOG_DB" "SELECT group_concat(epoch,' ') FROM (SELECT DISTINCT epoch FROM blocklog ORDER BY epoch);")
    Manuel committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    9fe6995 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2024

  1. Configuration menu
    Copy the full SHA
    af949fd View commit details
    Browse the repository at this point in the history
  2. removed --ledger-set param and replaced with --epoch in funcs: runCur…

    …rentEpoch, runNextEpoch and runPreviousEpochs
    Manuel committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    4fcb524 View commit details
    Browse the repository at this point in the history
  3. 1. EPOCH variable in echo string, replaced with $1:

    echo "ERROR: Failed to fetch protocol parameters for epoch ${1}.";
    
    2. Requoted "$1" in condition [[ ${epoch} = "$1" ]] && matched=true && break
    Manuel committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    919b529 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0b89e4f View commit details
    Browse the repository at this point in the history
  5. updated processAllEpochs function to use epoch var from epoch list lo…

    …op, instead of input "$1"
    Manuel committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    2cdd481 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    4d78d02 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2024

  1. [Mithril] script updates (cardano-community#1785)

    ## Description
    1. Changes to the nginx mithril relay LB and squid mithril relay
       * Nginx mithril relay LB:
         * Uses stream protocol
         * proxy_connect_timeout of 10 seconds
         * max_fails set to 1
    * fail_timeout set to 10 seconds per relay (20 seconds for 2 relay
    configurations, 30 for 3, etc.)
       * Squid mithril relay:
    * Change to how ACL's and allow rulesare written for multiple block
    producers
    * Includes rules for non block producers (other relays, local relay IP,
    simplifies testing from other than block producer) etc.
    2. Enhances the verify_signer_registered function
    * Continues to check if mithril-signer has registered in the current
    epoch.
    * If the signer has registered for the current epoch it will also check
    if the signer registered two epochs earlier and reports if the signer is
    valid to sign certificates in the current epoch or not.
    3. Adds a new MITHRIL_HOME variable to the
    `scripts/cnode-helper-scripts/env` file.
       * Defaults to the current path of `${CNODE_HOME}/mithril`.
    * Provides the SPO a way to set a unique path for where mithril
    environment and mithril data-stores will be located.
    4. Changes to a status code for checks on mithril minimum versions.
    Sourced from past conversation w/ @Scitz0
    5. Adds functions **semantic_version_check** and
    **check_mithril_upgrade_safe**
    * GitHub Actions workflow now uses **check_mithril_upgrade_safe**
    against the repo `node-latest.txt` to determine if changing
    `mithril-latest.xt` is acceptable.
    * Not yet implemented in `guild-deploy.sh`, but the
    **check_mithril_upgrade_safe** should support both CI Actions and
    production node/signer environments. Open for implementation
    discussions.
    
    ## Where to start?
    I'd suggest review of each commit vs. reviewing the files changed. The
    commit for **Refactor mithril functions into mithril library.** will be
    quite large. There is a general section of functions used by multiple
    scripts, and then functions are sorted into blocks by the script that
    uses them, client, relay or signer. Each section is alphabetically
    sorted.
    
    ## Motivation and context
    1. Nginx relay/lb
     * Issues with current nginx config reported in Koios support channel.
     * Simplified testing of squid proxy when additional IPs are provided.
    * Offline relays were adding 60 second delays until nginx timeouts
    occured and LB moved onto the next relay.
     * Tuned the max fails to 1 and fail_timeout 10 seconds * # of relays.
    * Temporarily prevents re-using an offline relay if many requests get
    sent through the same LB
    2. Reduce confusion for new signer users who were not aware signing had
    a 2 epoch delay.
    3. Flexibility for SPO's to choose their own directories.
    4. Past discussions with @Scitz0 
    5. Past discussions with @rdlrt 
    
    ## How has this been tested?
    1. Nginx relay/squid
       * Direct curl requests on the block producer:
    `curl -4 --proxy http://127.0.0.1:3132
    https://aggregator.release-mainnet.api.mithril.network/aggregator`
    
    * Enabling tcpdump on port 3132 for all (squid) mithril relays,
    disabling the first relay in the mithril_relays upstream definition of
    the (nginx) mithril relay lb. Then timing the request while watching
    traffic from each mithril relay:
    `time curl -4 --proxy http://127.0.0.1:3132
    https://aggregator.release-mainnet.api.mithril.network/aggregator`.
    
    * The first request takes 10.0x seconds, to succeed, subsequent requests
    take sub 1 second and traffic shows round robin over the remaining
    online mithril relays. If round robin reaches the offline mithril relay
    before the fail_timout has expired it gets skipped and the traffic is
    again sent to the first online (second in the list) upstream mithril
    relay.
    
    2. Manually
    3. Test virtual machines altering path to `/opt/cardano/mithril`
    4. Testing locally and in forked repository github actions
    5. Testing locally and in forked repository github actions
    
    ![image](https://github.com/user-attachments/assets/3838f427-1f38-4808-8188-eb517def0153)
    
    ---------
    
    Co-authored-by: RdLrT <[email protected]>
    TrevorBenson and rdlrt authored Aug 10, 2024
    Configuration menu
    Copy the full SHA
    1b0b6b0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    48a3e02 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Configuration menu
    Copy the full SHA
    8fdf2a5 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Configuration menu
    Copy the full SHA
    731efe1 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Configuration menu
    Copy the full SHA
    d8dfec0 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Configuration menu
    Copy the full SHA
    89abf3a View commit details
    Browse the repository at this point in the history
  2. 1. fixed linting issues.

    2. replaced first six hours of new epoch condition for function: getLedgerData
    
    3. added print available epochs from blocklog table in blocklog db
     No slots found in blocklog table for epoch 222.
     choose from epochs in list:
      230 252 274 282 299 311 312 316 317 318......
    Manuel committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    2d214b7 View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. Configuration menu
    Copy the full SHA
    2963267 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. 1. Added check stability window on runNextEpoch

    2. changed: sed 's/"//g' >> "$tmpcsv"
       for tr -d '"' >> "$tmpcsv"
    on runCurrentEpoch, runNextEpoch, runPreviousEpochs
    Manuel committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    cf20483 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Configuration menu
    Copy the full SHA
    8016b09 View commit details
    Browse the repository at this point in the history