feat: Add a new built-in test for taps to verify that fields and streams names are normalized

[edgarrmondragon]

Feature scope

Taps (catalog, state, stream maps, tests, etc.)

Description

Special characters and punctuation in stream and field names should be normalized to make everyone's lives easier:

• campaign:id -> campaign_id
• running+shoes -> running_shoes
• Growth (%) -> growth_pct

I propose a new built-in test that checks stream and field names againts a normalization spec. Perhaps:

import re

NORMAL = re.compile(r"[^a-zA-Z0-9_]")

SPECIAL_STRING = {
    "-1": "minus_one",
    "%": "pct",
    "(": "",
    ")": "",
}


def normalize_string(input_string):
    """Normalize and convert to lower snake case.

    Also replace special strings with their normalized version.
    """
    result = input_string.lower()
    for k, v in SPECIAL_STRING.items():
        result = result.replace(k, v)

    return NORMAL.sub("_", result)


ex1 = "Example 123"
assert normalize_string(ex1) == "example_123"

ex2 = "campaign.id"
assert normalize_string(ex2) == "campaign_id"

ex3 = "Growth (%)"
assert normalize_string(ex3) == "growth_pct"

ex4 = "Revenue (USD)"
assert normalize_string(ex4) == "revenue_usd"

---

If this is done, users would appreciate if we

• Make it easy to normalize names automatically and predictably
• Make it easy to turn the test off in case they don't care about name normalization