Binary payloads #2560
Merged
Binary payloads #2560
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was
linked to
issues
Apr 17, 2023
preardon
approved these changes
Jul 10, 2023
preardon
pushed a commit
to preardon/Brighter
that referenced
this pull request
Nov 15, 2023
* First pass with Sqlite; errors * First pass at binary payload with Sqlite * Fix issues with SQL Parameter naming * Problem with Sqlite * Try to get Sqlite tests running * Erros with Sqlite * Fix Sqlite Inbox * Support binary payloads in Sqlite * Fix MySQL to support binary; Remove Sync suffix from Outbox name * Ensure bytes are written raw, not converted to a string, which is lossy * Run SQL Server tests on GA * Binary support for MSSQL; some naming clean up for Outboxes. Ensure that we encode * remove obsolete sql data types * Fix missing partition key header * Sqlite should support partition key * sqlite supports writing kafka partition key * Move Postgres to new format; add binary and partitionkey support * ensure consistency of headers * Fix unix timestamps * Fix issue with order of paged read * Fix tests broken now that headers don't auto-round times. * Fix up sql * Reduce precision of timestamp tests to improve success rate * Add an example for Kafka binary to make my life easier * Clean up errors in SQL tests * fix up sqlite * Fix mysql messages * missing files * Use the CreatedTime to ensure that we get the correct time * Fix SQL lite order by timestamp * Missing direction on sort * Make sure we can build * Kafka and Sqlite working with Serdes and Binary * Use RelationalDatebaseConfiguration not derived classes * Alter the interfaces to outboxes, to prevent need to cast * Fix base type * Move to new model for Dbconnection provision * Breadth of interface for async scenarios * Prevent sweeper running multiple threads * unit of work and connection provider * Ensure we register the new interfaces * entity framework is a type of transaction provider * Add commit and rollback to the provider * Binary not being set; relational db connection not transaction * use transactionconnectionprovider for lifetime * update non-binary Dapper sample * Flow move back to box transaction provider out * Move to a single abstraction for boxtransactionprovider * Modify for generic AddBrighter - need shifting of generic parameters to Outbox * Tests don't apply any more, default to an in memory outbox * Fix failing test * Fix failing mssql tests that open the connection * Make archiver non-generic * Make derived outboxes call useexternaloutbox * safety dance. * Builds; tests and samples need debugging * Ensure that we use the transaction on an external bus with an outbox * use reflection to remove need to pass generic type parameters * Remove DynamoDb extension for creation of outbox * Fix how we get the message mapper and transformer * Sample fixes * Was not marking dispatched as missing background thread * Add missing migrations to support multiple production databases * Dapper support for all dbs * safety dance * remove post overloads * safety dance * Fix issues with PostgresSql generated via Dapper.Extensions and schema created by FluentMigrator * Safety Dance prior to debug changes * Drop Dapper Extensions, simpler debugging * Move to Dapper; fix database access * Fixes to salutations for supporting all databases * Fix naming issues for the Salutations project * safewt dance * Fix DDL for Postgres * Fix exists SQL for inbox * Remove old DDL statements; improved exists checks * Improved outbox creation handling * Fix PostgreSQL Inbox issues * Fix inbox and outbox creation * Allow different transports * Ensure Kafka works on sample * Merged Kafka project, no longer needed * Remove redundant example elements, and improve README.md * Migrate ef core examples;fix issues with target framework type * Get SalutationAnalytics.csproj working with hostbuilder changes * Fix DynamoDb sample and tools * Remove out of date readmes; switch Dapper default to RMQ * ephemera that probably need removing from source control
preardon
pushed a commit
that referenced
this pull request
Nov 15, 2023
* First pass with Sqlite; errors * First pass at binary payload with Sqlite * Fix issues with SQL Parameter naming * Problem with Sqlite * Try to get Sqlite tests running * Erros with Sqlite * Fix Sqlite Inbox * Support binary payloads in Sqlite * Fix MySQL to support binary; Remove Sync suffix from Outbox name * Ensure bytes are written raw, not converted to a string, which is lossy * Run SQL Server tests on GA * Binary support for MSSQL; some naming clean up for Outboxes. Ensure that we encode * remove obsolete sql data types * Fix missing partition key header * Sqlite should support partition key * sqlite supports writing kafka partition key * Move Postgres to new format; add binary and partitionkey support * ensure consistency of headers * Fix unix timestamps * Fix issue with order of paged read * Fix tests broken now that headers don't auto-round times. * Fix up sql * Reduce precision of timestamp tests to improve success rate * Add an example for Kafka binary to make my life easier * Clean up errors in SQL tests * fix up sqlite * Fix mysql messages * missing files * Use the CreatedTime to ensure that we get the correct time * Fix SQL lite order by timestamp * Missing direction on sort * Make sure we can build * Kafka and Sqlite working with Serdes and Binary * Use RelationalDatebaseConfiguration not derived classes * Alter the interfaces to outboxes, to prevent need to cast * Fix base type * Move to new model for Dbconnection provision * Breadth of interface for async scenarios * Prevent sweeper running multiple threads * unit of work and connection provider * Ensure we register the new interfaces * entity framework is a type of transaction provider * Add commit and rollback to the provider * Binary not being set; relational db connection not transaction * use transactionconnectionprovider for lifetime * update non-binary Dapper sample * Flow move back to box transaction provider out * Move to a single abstraction for boxtransactionprovider * Modify for generic AddBrighter - need shifting of generic parameters to Outbox * Tests don't apply any more, default to an in memory outbox * Fix failing test * Fix failing mssql tests that open the connection * Make archiver non-generic * Make derived outboxes call useexternaloutbox * safety dance. * Builds; tests and samples need debugging * Ensure that we use the transaction on an external bus with an outbox * use reflection to remove need to pass generic type parameters * Remove DynamoDb extension for creation of outbox * Fix how we get the message mapper and transformer * Sample fixes * Was not marking dispatched as missing background thread * Add missing migrations to support multiple production databases * Dapper support for all dbs * safety dance * remove post overloads * safety dance * Fix issues with PostgresSql generated via Dapper.Extensions and schema created by FluentMigrator * Safety Dance prior to debug changes * Drop Dapper Extensions, simpler debugging * Move to Dapper; fix database access * Fixes to salutations for supporting all databases * Fix naming issues for the Salutations project * safewt dance * Fix DDL for Postgres * Fix exists SQL for inbox * Remove old DDL statements; improved exists checks * Improved outbox creation handling * Fix PostgreSQL Inbox issues * Fix inbox and outbox creation * Allow different transports * Ensure Kafka works on sample * Merged Kafka project, no longer needed * Remove redundant example elements, and improve README.md * Migrate ef core examples;fix issues with target framework type * Get SalutationAnalytics.csproj working with hostbuilder changes * Fix DynamoDb sample and tools * Remove out of date readmes; switch Dapper default to RMQ * ephemera that probably need removing from source control
iancooper
added a commit
that referenced
this pull request
Dec 12, 2023
* First pass with Sqlite; errors * First pass at binary payload with Sqlite * Fix issues with SQL Parameter naming * Problem with Sqlite * Try to get Sqlite tests running * Erros with Sqlite * Fix Sqlite Inbox * Support binary payloads in Sqlite * Fix MySQL to support binary; Remove Sync suffix from Outbox name * Ensure bytes are written raw, not converted to a string, which is lossy * Run SQL Server tests on GA * Binary support for MSSQL; some naming clean up for Outboxes. Ensure that we encode * remove obsolete sql data types * Fix missing partition key header * Sqlite should support partition key * sqlite supports writing kafka partition key * Move Postgres to new format; add binary and partitionkey support * ensure consistency of headers * Fix unix timestamps * Fix issue with order of paged read * Fix tests broken now that headers don't auto-round times. * Fix up sql * Reduce precision of timestamp tests to improve success rate * Add an example for Kafka binary to make my life easier * Clean up errors in SQL tests * fix up sqlite * Fix mysql messages * missing files * Use the CreatedTime to ensure that we get the correct time * Fix SQL lite order by timestamp * Missing direction on sort * Make sure we can build * Kafka and Sqlite working with Serdes and Binary * Use RelationalDatebaseConfiguration not derived classes * Alter the interfaces to outboxes, to prevent need to cast * Fix base type * Move to new model for Dbconnection provision * Breadth of interface for async scenarios * Prevent sweeper running multiple threads * unit of work and connection provider * Ensure we register the new interfaces * entity framework is a type of transaction provider * Add commit and rollback to the provider * Binary not being set; relational db connection not transaction * use transactionconnectionprovider for lifetime * update non-binary Dapper sample * Flow move back to box transaction provider out * Move to a single abstraction for boxtransactionprovider * Modify for generic AddBrighter - need shifting of generic parameters to Outbox * Tests don't apply any more, default to an in memory outbox * Fix failing test * Fix failing mssql tests that open the connection * Make archiver non-generic * Make derived outboxes call useexternaloutbox * safety dance. * Builds; tests and samples need debugging * Ensure that we use the transaction on an external bus with an outbox * use reflection to remove need to pass generic type parameters * Remove DynamoDb extension for creation of outbox * Fix how we get the message mapper and transformer * Sample fixes * Was not marking dispatched as missing background thread * Add missing migrations to support multiple production databases * Dapper support for all dbs * safety dance * remove post overloads * safety dance * Fix issues with PostgresSql generated via Dapper.Extensions and schema created by FluentMigrator * Safety Dance prior to debug changes * Drop Dapper Extensions, simpler debugging * Move to Dapper; fix database access * Fixes to salutations for supporting all databases * Fix naming issues for the Salutations project * safewt dance * Fix DDL for Postgres * Fix exists SQL for inbox * Remove old DDL statements; improved exists checks * Improved outbox creation handling * Fix PostgreSQL Inbox issues * Fix inbox and outbox creation * Allow different transports * Ensure Kafka works on sample * Merged Kafka project, no longer needed * Remove redundant example elements, and improve README.md * Migrate ef core examples;fix issues with target framework type * Get SalutationAnalytics.csproj working with hostbuilder changes * Fix DynamoDb sample and tools * Remove out of date readmes; switch Dapper default to RMQ * ephemera that probably need removing from source control
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to support binary payloads for messages. When an Outbox stores a binary payload as a UTF-8 string this can be lossy, so we would prefer to support native binary columns. However, as some preference exists for readable JSON for debugging, and for backwards compatibility, we are making support for binary payloads an option.
Because binary payloads may be large, we also take advantage of any Db support for handling large binary fields where appropriate.