SQS Validation Sequence

---
title: SQS Validation Flow
---
sequenceDiagram
    filing-api->>s3: write csv
    s3->>cfpb-regtech-dev-s3-queue-test: sqs message
    sqs-csv-poller->>cfpb-regtech-dev-s3-queue-test: pull message
    sqs-csv-poller->>parquet-job: create k8s job
    parquet-job->>s3: convert csv to parquets
    parquet-job->>event bridge: push message
    event bridge->>cfpb-regtech-dev-pqs-validate: sqs message
    sqs-pqs-poller->>cfpb-regtech-dev-pqs-validate: pull message
    sqs-pqs-poller->>validator-job: create k8s job
    validator-job->>s3: scan parquets and validate
    validator-job->>s3: write result pqs
    validator-job->>event bridge: push message
    event bridge->>cfpb-regtech-dev-res-aggregate: sqs message
    sqs-res-poller->>cfpb-regtech-dev-res-aggregate: pull message
    sqs-res-poller->>aggregator-job: create k8s job
    aggregator-job->>s3: scan _res parquets
    aggregator-job->>aggregator-job: translate to report and json formats
    aggregator-job->>s3: write report csv
    aggregator-job->>database: store validation results in submission

Loading