Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple Schema is nesting an object instead of honoring the mapping #582

Open
anthonyhi11 opened this issue Oct 15, 2024 · 0 comments
Open

Comments

@anthonyhi11
Copy link

PGSync version: 3.2.1

Postgres version: 16.4

Elasticsearch/OpenSearch version: 2.17.1

Redis version: 7

Python version: 3.11

Problem Description:
I have enjoyed working with pgsync so far, but I'm trying to do something very simple and I just can't get it to work how I'd expect it to. I'm hoping it's something very obvious... I've googled and looked at docs, but they all seem to assume this very simple example should just work.

Very simple example mapping:

{
    "mappings": {
        "properties": {
            "id": {
                "type": "text"
            },
            "child_id": {
                "type": "text"
            },
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            }
        }
    }
}

The schema.json:

    {
        "database": "example",
        "index": "example_index",
        "nodes": {
            "table": "parent_table",
            "columns": [
                "id"
            ],
            "children": [
                {
                    "table": "child_table",
                    "columns": [
                        "id", 
                        "first_name",
                        "last_name",
                        "email"
                    ],
                    "relationship": {
                        "variant": "object",
                        "type": "one_to_one"
                    },
                    "transform": {
                        "rename": {
                            "id": "child_id"
                        },
                        "mapping": {
                            "first_name": {
                                "type": "text"
                            },
                            "last_name": {
                                "type": "text"
                            },
                            "email": {
                                "type": "keyword"
                            }
                        }
                    
                    }
                }
            ]
        }
    }

The mapping gets overwritten and the child data comes in like this to form this type of mapping:

{
    "mappings": {
        "properties": {
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            },
            "email": {
                "type": "keyword"
            },
            "child_table": {
                "type": "object",
                "properties": {
                    "first_name": {
                        "type": "text"
                    },
                    "last_name": {
                        "type": "text"
                    },
                    "email": {
                        "type": "text"
                    }
                }
            }
        }
    }
}

I thought the mapping within the transform would make sure it knows to place it at the top level, but it isn't seeming to do that and is just creating an object with the fields nested within. All the data streaming from postgres is being saved to opensearch in the incorrect mapping. This wouldn't really be an issue since I could just use this new nested object, but I need to specify the mapping to be of type "keyword" and it doesn't handle that either. Any insight would be awesome!

Error Message (if any):



Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant