feat: onboard msl changes for new record event

[yashasvibajpai]

What are the changes introduced in this PR?

Resolves INT-651

We are adding support for a new event type record for Marketo Static List. This is a new alternative to the existing audiencelist event.

Please explain the objectives of your changes below:

Through record events, we aim to eliminate some observability inconsistencies that were present along with audiencelist events.
There are some dependent changes in rudder-server

Type of change

If the pull request is a bug-fix, enhancement or a refactor, please fill in the details on the changes made.

Current changes	New changes

If the pull request is a new feature,

Any technical or performance related pointers to consider with the change?

New record event spec:

{
      "type": "record",
      "action": "insert", // can be either insert or delete for Marketo Static List
      "context": {
        "sources": {
          "job_id": "",
          "version": "master",
          "job_run_id": "",
          "task_run_id": ""
        },
        "externalId": [
          {
            "type": "marketoStaticListId",
            "id": "5612"
          }
        ],
      },
      "recordId": "",
      "rudderId": "",
      "messageId": "",
      "fields": {
         "id": "[email protected]" // the lead ID to be inserted into the desired static list
      }
}

The other features of MSL, like batching maximum of 300 lead IDs into a single API call are intact. As of now, both record and audiencelist event types are supported, and the changes made here are backward-compatible.

Any new dependencies introduced with this change?

No generic utility has been modified in these changes, all logic concerning the new supported event type record is in a separate transformV2.js file.

If the PR has changes in more than 5 files, please mention why the changes were not split into multiple PRs.

There is a overall functionality change with a new transformation flow being introduced, hence all related changes needed to be in one PR for easier reference.

If there are multiple linear items associated with the PR changes, please elaborate on the reason:

INT-1007 : We had to move out of the earlier done modular approach on top of audiencelist event structure.

---

Developer checklist

• Please confirm if there are no breaking changes being introduced.
• Are all related docs linked with the PR?
• Are all changes tested?
• Does this change require any RudderStack documentation changes?
• Are relevant unit and component test-cases added?

Reviewer checklist

• Is the PR tagged correctly as per the changes?
• Verified if there are no security credentials or confidential information in the changes.

Notion

Summary by CodeRabbit

• New Features

  • Enhanced Marketo Static List integration with batch processing capabilities for record inputs.
  • Improved response handling with new responseBuilder and batchResponseBuilder functions.

• Improvements

  • Updated event processing logic to handle grouped inputs and separate processing for record and audience list events.
  • Refined metadata processing for more robust handling within the routing function.

• Bug Fixes

  • Corrected HTTP method and endpoint parameters in test data to align with expected API operations.

• Tests

  • Added comprehensive unit tests for the processRouterDest function to validate batch creation and API interaction.

• Chores

  • Disabled specific eslint rule to streamline code consistency.
  • Imported InstrumentationError for enhanced error tracking and handling.

[codecov]

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (571dbf5) 87.25% compared to head (e3c7950) 87.23%.
Report is 24 commits behind head on develop.

Files	Patch %	Lines
...c/v0/destinations/marketo_static_list/transform.js	88.37%	5 Missing ⚠️
...v0/destinations/marketo_static_list/transformV2.js	92.53%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2644      +/-   ##
===========================================
- Coverage    87.25%   87.23%   -0.03%     
===========================================
  Files          775      779       +4     
  Lines        28869    29042     +173     
  Branches      6780     6802      +22     
===========================================
+ Hits         25189    25334     +145     
- Misses        3340     3366      +26     
- Partials       340      342       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
7 Code Smells

35.7% Coverage
0.0% Duplication

[devops-github-rudderstack]

This PR is considered to be stale. It has been open for 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR.

[coderabbitai]

Important

Auto Review Skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository.

To trigger a single review, invoke the @coderabbitai review command.

Walkthrough

The updates involve significant enhancements to a Marketo Static List integration, focusing on the addition of new constants, the export of these constants, and the restructuring of objects and functions to support batch processing. The transform.js and transformV2.js files have seen the introduction of new functions and parameters to handle record inputs more efficiently, with improved error handling and response construction. Unit tests have been added to ensure the correct processing and routing of events.

Changes

File Path	Change Summary
.../marketo_static_list/testData/constants.js	Added and exported new constants; modified DEST_CONFIG and DEST_OBJECT.
.../marketo_static_list/transform.js	Imported processRecordInputs, renamed parameters, updated function signatures, and modified processRouterDest to handle grouped inputs.
.../marketo_static_list/transform.test.js	Introduced unit tests for processRouterDest function.
.../marketo_static_list/transformV2.js	Added new functions for batch processing and response construction; updated processRecordInputs for batched responses.
.../marketo_static_list/util.js	Disabled an eslint rule and imported InstrumentationError.
.../marketo_static_list/processor/data.ts	Updated method property and id parameters in the output object.

🐇✨
In the burrows of the code, where the data streams flow,
A rabbit hopped through, with changes in tow.
Batched events, constants new, tests to ensure,
The Marketo list, now robust and secure.
🌟📝

---

Tips

Chat with CodeRabbit Bot (@coderabbitai)

• If you reply to a review comment from CodeRabbit, the bot will automatically respond.
• To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
• Note: Review comments are made on code diffs or files, not on the PR overview.
• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• The JSON schema for the configuration file is available here.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

[coderabbitai]

Review Status

Actionable comments generated: 3

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 5c63d2c and 21ac726.

Files ignored due to filter (5)

• test/tests/data/marketo_static_list.json
• test/tests/data/marketo_static_list_router_input.json
• test/tests/data/marketo_static_list_router_metadata_input.json
• test/tests/data/marketo_static_list_router_metadata_output.json
• test/tests/data/marketo_static_list_router_output.json

Files selected for processing (7)

• src/v0/destinations/marketo_static_list/testData/constants.js (1 hunks)
• src/v0/destinations/marketo_static_list/transform.js (4 hunks)
• src/v0/destinations/marketo_static_list/transform.test.js (1 hunks)
• src/v0/destinations/marketo_static_list/transformV2.js (1 hunks)
• src/v0/destinations/marketo_static_list/util.js (1 hunks)
• test/integrations/destinations/marketo_static_list/processor/data.ts (2 hunks)
• test/integrations/destinations/marketo_static_list/router/data.ts (6 hunks)

Files not summarized due to errors (1)

• test/integrations/destinations/marketo_static_list/router/data.ts: Error: Message exceeds token limit

Files skipped from review due to trivial changes (1)

• src/v0/destinations/marketo_static_list/util.js

Additional comments: 14

src/v0/destinations/marketo_static_list/testData/constants.js (5)

• 1-45: The addition and export of new constants are correctly implemented.

• 8-8: Verify that the updated staticListId value in DEST_CONFIG is correct and intended.

• 25-25: Verify that the updated staticListId value in DEST_OBJECT.Config is correct and intended.

• 29-29: The addition of the IsProcessorEnabled property to DEST_OBJECT is correctly implemented.

• 31-35: The MESSAGE_SOURCES_CONTEXT constant is correctly implemented.

src/v0/destinations/marketo_static_list/transform.js (1)

• 95-103: The _processParams parameter is unused in the process function. If this parameter is intended for future use or to match a function signature interface, consider adding a comment to clarify its purpose.

src/v0/destinations/marketo_static_list/transform.test.js (2)

• 1-138: The test cases provided in the file cover a variety of scenarios and assert the expected behavior of the processRouterDest function with different input sizes and types. The use of a mock Axios adapter to simulate API responses is a good practice for unit testing external API interactions. The assertions check for the correct API endpoints, HTTP methods, and metadata handling, which aligns with the changes described in the pull request summary.

• 14-138: Verify that the hardcoded endpoints used in the test cases are not subject to frequent changes. If they are, consider making them configurable to avoid having to update the tests frequently.

src/v0/destinations/marketo_static_list/transformV2.js (3)

• 74-141: The logic for processing record inputs, including error handling and response building, appears to be well-structured and comprehensive.

• 57-66: The batchResponseBuilder function correctly chunks the lead IDs to avoid long URLs and creates appropriate batched responses.

• 74-141: Verify that the caller of processRecordInputs properly handles the promise returned by this asynchronous function.

test/integrations/destinations/marketo_static_list/processor/data.ts (2)

• 65-67: The change in the HTTP method from "POST" to "DELETE" and the update of the id parameters in the endpoint URL are consistent with the summary provided. Ensure that the backend service supports these changes and that the corresponding API endpoints are expecting these new id values.

• 83-85: The change in the HTTP method from "DELETE" to "POST" and the update of the id parameters in the endpoint URL are consistent with the summary provided. As with the previous hunk, verify that the backend service supports these changes and that the corresponding API endpoints are expecting these new id values.

test/integrations/destinations/marketo_static_list/router/data.ts (1)

• 1663-3643: > Note: This review was outside the patches, so it was mapped to the patch with the greatest overlap. Original lines [1-3643]

The test data for the new 'record' event type and the existing 'audiencelist' event type is structured correctly, with appropriate endpoint URLs, headers, and metadata. The batchedRequest objects are formed as expected, and the batched flag is set correctly according to the test case requirements. The placeholders used for tokens and IDs are appropriate for test data.

[coderabbitai]

Review Status

Actionable comments generated: 1

Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 21ac726 and 9e3694b.

Files selected for processing (2)

• src/v0/destinations/marketo_static_list/transform.js (4 hunks)
• src/v0/destinations/marketo_static_list/transformV2.js (1 hunks)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

90.5% Coverage
5.0% Duplication