Merging code for 'Utilizing Cloud Logging and Cloud Run to create Event Driven Data Pipelines' #283
Conversation
| @@ -0,0 +1,156 @@ | |||
| <?xml version="1.0" encoding="UTF-8"?> | |||
There was a problem hiding this comment.
Remove this entire file
tools/cloud_run/event_driven_bq_ingest/src/main/java/com/example/cloudrun/BQAccessor.java
Outdated
Show resolved
Hide resolved
tools/cloud_run/event_driven_bq_ingest/src/main/java/com/example/cloudrun/GCSAccessor.java
Outdated
Show resolved
Hide resolved
| @@ -0,0 +1,64 @@ | |||
| /* | |||
There was a problem hiding this comment.
My mistake; I see this is needed to map the message into a JSON object
There was a problem hiding this comment.
This file would be better named: BigQueryMetadataAuditLog
|
|
||
| @Value | ||
| @Builder | ||
| public class PubSubMessageProperties { |
There was a problem hiding this comment.
If you're going to have a class like this, you should have it also store bq job notifications which come in through pubsub.
| PubSubMessageParser.parsePubSubProperties(pubSubMessage); | ||
| if (pubSubMessageProperties == null) { | ||
| // parse pubsub message as bq job notification | ||
| PubSubMessageData pubSubMessageData = |
There was a problem hiding this comment.
Why parse bq job notifications differently than gcs notifications? Unify with one parser that handles both
| if (TRIGGER_FILE_NAME.equals(pubSubMessageProperties.getTriggerFile())) { | ||
| log.info("Found Trigger file, started BQ insert"); | ||
| BQAccessor.insertIntoBQ(pubSubMessageProperties, FILE_FORMAT); | ||
| // GCSAccessor.archiveFiles(pubSubMessageProperties); |
There was a problem hiding this comment.
Remove if this is not needed
|
|
||
| List<String> allSourceUris = new ArrayList<String>(); | ||
| // check for error | ||
| if (job.isDone()) { |
There was a problem hiding this comment.
This does not mean the job was successful, only that that job completed.
You must check the status of the job as follows:
if(job.isDone() && job.getStatus().getError() == null)
Co-authored-by: Daniel De Leo <[email protected]>
Co-authored-by: Daniel De Leo <[email protected]>
Co-authored-by: Daniel De Leo <[email protected]>
…le/cloudrun/BQAccessor.java Co-authored-by: Daniel De Leo <[email protected]>
…le/cloudrun/GCSAccessor.java Co-authored-by: Daniel De Leo <[email protected]>
Co-authored-by: Daniel De Leo <[email protected]>
|
|
||
| package com.example.cloudrun; | ||
|
|
||
| import com.google.cloud.bigquery.BigQueryError; |
Merging changes for parsing pubsub messages efficiently
Removed extra dependencies from pom and reformat code
@danieldeleo