LCFS - Fix Missing Organizations in NiFi ETL (TFRS to LCFS)

[AlexZorkin]

Description:
The NiFi-based ETL process, which relies on a Groovy script to handle schema migration, failed to include all organizations from TFRS to LCFS. Several organizations did not appear in LCFS after running the pipeline. This ticket addresses the debugging, resolution, and verification of the issue to ensure complete data transfer.

Purpose and benefit to user:
Ensuring that LCFS accurately reflects the organizations contained in TFRS maintains data consistency and reliability across the system.

Acceptance Criteria:

• Given the user is an ETL operator, when the pipeline runs, then all organizations from TFRS appear in LCFS.
• Given the user is an ETL operator, when the fix is applied, then logs and verification reports confirm that no organizations remain missing in LCFS.

Development Checklist:

• Identify the root cause of the missing organizations in the NiFi Groovy file.
• Apply a fix to the Groovy script to capture all organizations from TFRS.
• Test the updated pipeline with sample and production data to validate successful transfers.

Notes:
Validate any logs related to the NiFi process and confirm the presence of all expected organization records in LCFS.

[prv-proton]

When tested on local all organizations were loaded. May require re-implementation after clearing all data in the database. Based on it further work can be done.