Airbyte 1.2.0

Happy Halloween! We are excited to release the following set of improvements and changes.

🚀 Platform Changes

• File Transfers: Airbyte now supports transferring unstructured text data, non-text data and compressed files. This capability is in early access, and currently only available for moving data from the "SFTP Bulk" source to the "S3" destination. Support in the "S3" source is coming soon. File transfers allow you to copy raw files in Airbyte without parsing their contents. Bits are copied into the destination exactly as they appeared in the source. This is recommended for use with unstructured text data, non-text and compressed files. We currently support copying files up to 1GB in size. This is an entirely new way of moving data through Airbyte, as all pre-existing data movement methods involved parsing individual records.

• Custom Image Registry Support: Introduces support for specifying a custom Docker registry from which all platform images are pulled. This should simplify helm chart configurations for users who require images to be pulled from internally approved image repositories. Configuring the following will prefix all docker images with my-registry.foo.com. The value provided in the registry field must have a trailing slash:

global:
  image:
    registry: my-registry.foo.com/

• XML Support in the Connector Builder: Airbyte now supports custom API connectors built via the Airbyte Connector Builder which return responses as XML format.

🚀 Self-Managed Enterprise Changes

• Runner RBAC Role: Users provisioned with the 'Runner' role are able to start or stop syncs, and run backfills for individual connections (in the workspaces where they've been granted Runner permissions). Outside of these explicit tasks, users with the 'Runner' role have an entirely read-only experience (Self-Managed Enterprise only).

• Self-Managed Diagnostics: Diagnostics have been expanded to include deployment statistics. Diagnostics can be downloaded through the Organization Settings page. This exports a .zip file which can be easily shared with Airbyte Support (Self-Managed Enterprise only).
• Removing SSO Users: You may now remove SSO users from your Airbyte Organization. This removes them from the list of Organization Members.

Airbyte 1.1.0

🚀 Features

The 1.1.0 release includes the following enhancements and bug fixes.

Platform Releases

• Adds field hashing for Self-Managed Enterprise

• Adds the Connection Timeline, replacing the Job History for a connection. For assistance with the migration of existing jobs to the new timeline events, follow our guide.

• Redesigned the schema tab to improve the selection & deselection of streams and fields. Sync modes, cursor fields, and primary keys are also selected here.

• Custom Docker-based connectors can now be renamed and deleted

Support Experience

• Diagnostics can be downloaded through the Organization Settings page, which will export a JSON file to facilitate sharing with Support. This JSON file includes relevant infrastructure and connection information. (Self-Managed Enterprise only)

Billing & Licenses

• Self-Managed Enterprise users can now view directly in the UI when the license is expiring.

Changelog

See the Full Changelog

Airbyte 1.0

airbyte v1.0.0

Official launch page: https://airbyte.com/v1

This page includes new features and improvements to the Airbyte products we're excited to share as a part of the 1.0 release. Airbyte v1.0 contains many improvements and additions to enhance the reliability, scalability, and uses of Airbyte.

Moving forward, Airbyte will release official new platform versions on a monthly cadence, with the associated changelog.

Ease of deployment

• abctl is the easiest, quickest way to get started with Airbyte Self-managed. See our quickstart docs for more details.

• Native authentication by email and password is available so that any instance of Airbyte is secure by default.

Proven Reliability

• Automatic detection of dropped records ensures data is passed without fail through the Airbyte platform.

• Enhanced our notification suite to enable webhook integrations and added more contextual information about sync failures or schema changes.

• Connections now offer rate limited messaging, improved error handling, and live monitoring of ongoing syncs.

• Many sources, in particular S3, have become faster. With CDK 2.0, the protocol now uses Pydantic V2 and removes the serialization of each record. We also improved Databricks and S3 destinations to enable faster sync speeds, checkpointing, and modernization to our Destinations V2 framework.

Resiliency at Scale

• Refreshes bring an improved experience to resyncing all of your data again. This enables data to never be deleted from final tables during a historical resyncing of data.

• Resumable Full Refresh allows for large streams syncing in Full Refresh to sync without failures.

• Our database sources added resumability to reduce failures and stay resilient when incoming data is inconsistent. We gracefully handle record-level error in the destination field _airbyte_meta.errrors for large records or mistyped data. Read more in our typing & deduping documentation.

• Large initial CDC syncs are gracefully handled by consuming WAL and acknowledges logs periodically to ensure disk space is freed in a timely manner.

• Workloads provide a more scalable and reliable architecture to run sync jobs by separating scheduling and orchestration from data movement tasks. This improvement unlocks more automated management of workloads by managing job spikes and enables horizonal scaling (for Cloud and Enterprise users).

Tackling the long-tail of connectors

• We launched our connector Marketplace for our community-maintained connectors. Automated testing ensures our connectors stay high quality, and most connectors have been migrated to low/no-code.

• The Connector Builder seamlessly switches betweem UI and YAML modes, supports custom components, and contains extensive testing and error handling.

• AI Assist now builds connectors from scratch for you with just a link to the API docs.

• Anyone can now contribute new connectors directly from the builder to add a new connector directly to Airbyte's Marketplace. Additionally, connectors can be forked directly from the UI for faster edits, ensuring our connectors stay up-to-date.

• Airbyte supports writing to several vector store destinations, including pgVector, Pinecone, and Snowflake Cortex.

Self-Managed Enterprise

Self-Managed Enterprise extends on Airbyte 1.0 by introducing new classes of functionality: multitenancy and access management, enterprise source connectors , sensitive data masking and enterprise support – all while data never leaves your infrastructure.

• Support for Okta and OpenID Connect (OIDC) SSO
• Availability of Role-based Access Control and multitenancy
• Columns can now be hashed to protect sensitive data

Announcements

• To ensure adherence to security best practices, Airbyte is migrating all connectors to non-root versions. It is highly recommended that you upgrade your platform version to v0.63.9 or later before October 2024 to ensure your syncs continue to succeed.

• As we prepare to deprecate Docker Compose, we published a migration guide for those migrating from Docker Compose to abctl.

v0.64.7

55f8144 fix: restore secret persistence config (#14015)
da74ee2 chore: change wording on link to invoice (#14012)
4b2a524 chore: [SyncCatalog V2] remove debug code from catalog (#13994)
ab5be73 fix: [SyncCatalog V2] wrong "breaking changes message" position (#13993)
f9e7607 feat: add banners for manual and auto recharge to billing page (#14011)
1129cef chore: delete deprecated worker v1 helm and infra code. (#13984)
4070642 chore: mappers also rename source default cursor and source defined pk. (#14002)
559dc25 chore: delete remaining worker v1 config code (#13989)
342f72e refactor: remove connector rollout cli shortcut for actor IDs (#14001)
30eaac3 chore: deploy connector rollout worker on oss (#13887)
a1b5395 Bump helm chart version reference to 0.620.2
f5b500b chore: refactor workload acceptance tests to use dynamic feature flags (#13492)
0993ad0 Bump helm chart version reference to 0.620.0
43565b5 fix: parsing docker image when the full image name contains registry port (#13990)

v0.64.6

d1c36c4 feat: implement correct license statuses (#13894)
0665001 fix: properly generate the airbyte_type (#13998)
df7d84e chore: use separate message for empty exit code file. (#13997)
43f964a docs: add learn more link about recovering credentials in oss (#13951)
737e8c5 chore: delete all non-config worker v1 code. (#13988)
ca7ff83 chore(ai-assist): log assist service response codes (#13995)
4e6b440 feat: add organization balance API (#13964)

v0.64.5

Caution

This release contains the activation of a feature which mutates the type TIMESTAMP WITHOUT TIMEZONE in TIMESTAMP WITH TIMEZONE. This was unintentional and was fixed in the next release.
Existing Airbyte users should skip this version and directly upgrade to 0.64.6.

eb28d9b feat: introduce diagnostic tool api (#13845)
f264427 fix: uses SHA-256 not SHA-1 (#13970)
f9c23bd feat: virtualize all connections page (#13548)
9717c1c chore: add data-testid on hashing input for fullstory sessions (#13992)
dd882ff feat: connector progressive rollout CLI (#13888)
483f470 chore: rename the <PKCell /> and <CursorCell /> components to match the level in the catalog (#13991)
2a75339 chore: change cursor from checkbox to radio button (#13965)
74800c2 chore: add tooltip to source defined pk and cursor combobox (#13974)
e733b86 refactor: add useCurrentConnection hook (#13922)
2b4ba10 chore: add analytics tracking functions to new sync catalog (#13976)
512e493 feat: worker for connector progressive rollouts (#13886)
d74ce0d chore: ensure launcher fails on certain errors (#13987)
d2114e5 feat: update connector builder project contributionInfo after contributing (#13897)
d8af275 chore(ai-assist): add alpha badge (#13973)
b9cf0a9 feat: fork in builder buttons (#13787)
50b9b34 fix: remove typecasting and centering from running jobs in timeline ui (#13972)
26d3236 feat: add details field to known exception (#13930)
f812e04 feat: new & updated API endpoints and schema for connector rollouts (#13864)
5d8b230 refactor: rename commons logging (#13925)
53e3e4c chore: set the mapper flag to true (#13945)
3454fb5 refactor: move storage client to shared library (#13916)
bc2aefd feat: enable renaming of cursor and pk in mappers (#13959)
10af3de chore(frontend): update format command to match ci requirements (#13848)
d8a8a58 fix: identify streams with changed hashedFields as needing refresh (#13943)
181a2e9 fix: fix not applied height in case of empty table state (#13975)
bd254f0 chore: don't send known invalid events to datadog for timeline (#13952)
8190bb0 chore: make resource error as external error message (#13971)
54849f2 design: use standard empty state for dbt cloud transformations tab (#13944)
bc351cc fix: handle both null and empty namespaces for stream status completion (#13967)
8189ffd chore: default hashing config to set SHA-256 instead of SHA-1 (#13947)
311be67 feat: hashing UI feedback (#13931)
8476ebf fix: align schema refreshing message properly (#13963)
f249bee chore: update to micronaut 4.6.2 (#13940)
06196c9 fix: autoescape regex characters in sync catalog search (#13962)
5cf62e0 fix: don't display usage data if it doesn't match user's current timezone (#13961)
69b33f6 chore: upgrade base image to alpine3.20 (#13956)
9418f09 fix: add the connection string to all the application.yml files that need it (#13955)
4a5327f fix: catalog generation related to mappers (#13950)
5120384 chore: add tracking for contribute form focus (#13912)
dec1795 feat: deterministic workflow Id (#13948)
4174406 chore: hashing mapper populate meta.changes on errors (#13919)
06af5db chore: cleanup rolled out flags (#13906)
b739c1b Bump helm chart version reference to 0.594.0

v0.64.4

b32d1d2 feat: ensure keycloak master realm is properly configured on startup (#13821)
074558c feat: custom sync catalog search and expansion (#13924)
59e9bdd fix: a few wording adjustments (#13939)
db1a531 chore: add dd tracking for missing timeline event item (#13932)
8944018 fix: update "enable/disable all streams in namespace" implementation (#13937)
c0debfc feat: azure key vault persistence (#13791)
d7bc738 chore: change error message (#13914)
865e444 fix: add aws and vault env vars to init container app yml. (#13921)
ddde2b0 chore: remove all worker v1 / workload branching. (#13861)
6ad3f9d build: gradle 8.10.1 (#13926)
11be428 fix: apply mapper to the destination catalog in monopod (#13918)
a4d7371 fix: revert fix for modal scrollbars for now (#13927)
9c970c5 fix: show settings change events in timeline again (#13917)
b886328 feat(billing): add workspace usage graph and connection table (#13586)
12319bb feat: show base connector and contribution info in Builder UI (#13724)
b172a40 feat: allow forking from manifest-only connectors in Builder Fork page (#13671)
7b59c7b feat: return base actor definition version info and contribution info from connector builder project endpoints (#13829)
10eb8ca fix: conditionally create local-secrets-bean. (#13920)
fde00dd feat: store contribution info on project update (#13832)
597bb8b feat: show {enabled|disabled} count / {totalCount} in namespace row when tab filter is active (#13907)
fd94d28 chore: print container and pod info for monopod. (#13915)
cfdad4e chore: fix hashing mapper test (#13913)
671a36f chore: remove condition on setting the workload (#13911)
6c6b657 feat: schema propagation events in timeline UI (#13892)
9b701d5 chore: improve error message for timeout exceptions (#13883)
845eda4 design: reword notification settings for connection updates (#13901)
19f894f chore: add data and caching to AI assist (#13885)
0e63273 chore: hashing mapper returns hex instead of base64 (#13898)
832ead0 design: show tooltip for relative time toggle on sync status page (#13902)
b91926c chore: [SyncCatalog V2] always show expand/collapse control (#13884)
c9ff4d8 chore: remove connector name in use backend check (#13895)
6b35241 chore: fallback to true if getUseWorkloadApi null. (#13896)
0c98541 fix: handle loading state flicker in generate (#13857)
c4b1a9a fix: handle invalid mapper configs (#13825)
03cce50 refactor: remove dead instance_config options (#13891)
3cf3aa5 feat: return actor definition id from /contribute/generate (#13665)
568839c fix: remove old init sql (#13877)
823e7ed feat: endpoint to create a forked connector builder project (#13664)
6b5f842 feat: add endpoint to get connector builder project by definiton ID (#13784)
3171f4e fix: handle jobs with 0 attempts in the ui (#13875)
163f4d9 feat: add azure support for storage (#13608)
013c398 feat: hash mapper UI v0 (#13805)
bed7719 fix: prevent submitting sync catalog form from search input (#13881)
e47a59b feat: report new usage_category with Orb events (#13764)
ec83760 revert: chore: add data and caching to AI assist (#13880)
c87691a fix: add back failureOrigin enum values for normalization and dbt (#13830)
3893558 feat: defaults monopod to enabled for builds. (#13873)
b7f1ed0 revert: "revert: "feat: add internal api to support field hashing"" (#13865)
88de307 chore: add data and caching to AI assist (#13750)
9d5a39e feat: add new license info endpoint (#13711)
c769251 fix: support the unexpected json schema (#13871)
9a95174 feat(billing): new workspace usage endpoint (#13811)
1c62ddb chore: set monopod flags to true for accpetance tests. (#13856)
2565d7b fix: isTerminal(Pod) logic works with Monopod (#13854)
3824ae2 chore: clean up some permission logic (#13849)
5849764 fix: do not log duplicate connection disabled events (#13752)
98992ba fix(test): use local chart version for temporal-ui (#13866)
d6833d5 fix: don't allow invalid filter selections on timeline (#13855)
6cca28f revert: "feat: add internal api to support field hashing" (#13859)
bf799dd feat: add internal api to support field hashing (#13798)
aff5a15 Bump helm chart version reference to 0.551.0

v0.64.3

e6ade26 fix: include refreshes in early syncs query (#13852)
d378e75 fix(helm): remove pre-upgrade hook from service account (#13729)
a353a6a build: micronaut 4.6.1 (#13847)
d295dcd chore: enable node selector override (#13843)
84d599f feat(ai-assist): animated loading state (#13841)
c16e5be chore: [SyncCatalog V2] imrove collapse/expand row perfomace (#13840)
8a529d9 chore: removes unnecessary print statement that clutters logs. (#13844)
2531253 chore(helm): add logging and storage templates (#13807)
0c753eb chore: add basic mapper performance metrics to the sync summary (#13842)
ed7bf3e feat: progressive rollouts initial connector_rollout table & API (#13675)
6db66c0 feat: progressive rollouts table migration (#13819)
910f012 feat: support unknown type in the jsonSchema catalog (#13818)
efd0adf feat: split org members access management onto its own page (#13834)
4d99e61 fix: remove duplicate jobId on failures in "advanced view" of timeline (#13827)
cbfbfe8 chore: introduce AirbyteRecord and AirbyteJsonRecordAdapter (#13774)
d9e3dd5 fix: missing local string (#13831)

v0.64.2

0a8bc49 fix: update selected fields during schema propagation (#13727)
6ba68fd feat: add onboarding assist ui (#13705)
36b248d fix: wrap zendesk calls in safety nets (#13743)
cf0a3d4 chore: ensure both source/dest termination files are written on orchestrator exit (#13817)
a53c08a fix(magic variant): update missed code review comment (#13820)
fa846c1 fix: add entry for google-drive to publicApi definition mappers (#13788)
f621ae2 test: frontend e2e open and interact with logs from timeline (#13786)
7791151 feat: generate field from schema (#13808)
bab5ad7 chore(ai-assist): change highlight style to magic style (#13777)
8eb6884 feat: add temporal UI for local development (#13803)
7a85cec chore: update to Gradle 8.10 (#13813)
1cf3827 feat: auto grant instance admin to service accounts (#13806)
a9a7aca fix: use our UncaughtExceptionhandler over built-in (#13809)
9b99b39 feat: enable updates in development Temporal server (#13796)
8576a01 feat: apply mapping and generate destination catalog (#13739)
6361736 chore: pass correct env vars to read/write containers in megapod. Organize runtime env var code. (#13790)
805247b chore: jobs list -> micronaut data (#13422)
35dd8bc revert: "chore: matches req/limit for source/dest container memory." (#13795)
02583e3 fix: remove redundant uri from endpoint mapping (#13804)
b55bbb3 fix: move sleep to post heartbeat call + increase wait duration (#13775)
41e03fb feat: generate catalog for destination (#13688)
419b8bf feat: add UI warning for failed logins likely due to insecure cookies (#13801)
00270d7 chore: cleanup OrchestratorFetchesInputFromInit flag (#13734)
5eae8bd fix: add jobId back for sync fail events (#13709)
59dfc70 build: micronaut 4.6.0 (#13704)
8ccc517 fix: loosen frontend cron validation (#13792)
47fa764 chore: update frontend testing deps (#13778)
a38b774 Bump helm chart version reference to 0.524.0
8386ce6 chore: add keycloak client configs for workers on cloud (#13673)
4c567e4 feat: support keycloak client auth for cloud-server/airbyte-server requests (#13579)

v0.64.1

47f568b chore: add feature flag to disable auth header replacement (#13581)
6dfe3ca chore: matches req/limit for source/dest container memory. (#13782)
6fd9f15 fix: n+1 issue when doing the list_for_workspace calls on source and destination definitions (#13686)
27f3b41 feat: add execution type to exit value metric (#13780)
56e8660 feat: add caret button to ComboBox input (#13694)
41ef7b0 fix: fetch the exit value once (#13779)
47b59bb fix: make forms in modals have proper scrolling behavior (#13757)
a33b711 chore: emit dest exit value when non-0. (#13773)
d7bd50c refactor: move notification container to react portal (#13726)
2686543 feat: enable opening job logs anywhere (#13712)
fc57c6a feat: [SyncCatalog V2] group streams by namespace, add ability to enable/disable all streams and add the ability to edit destination namespace (#13578)
5843a62 chore: update CDK version following release (#13768)
157f77d feat: record metric for non-zero connector exit codes (#13770)
6c6c894 fix: use connection_disabled intl key for connection_disabled events (#13769)
5e2816e fix: retry links during validation w/backoff (#13766)
9244c1c feat: ensure memory footprint of monopod (#13761)
ff76c8b chore: ensure source/dest exit on orchestrator exit (#13751)
fb8eaa0 chore: cleanup forgotten code for FF (#13763)
4dd60c5 chore: update deps. (#13753)
99c4aa0 fix: properly position combobox options (#13690)
66f79b1 feat(ai-assist): add analytics events (#13742)
beac464 docs: remove docs references + startup script for docker compose (#13747)
d98a717 fix: [SyncCatalog V2] show deeply nested fields (#13746)
a4ee854 feat: debounce AI assist requests (#13652)
d542d52 chore: update ai button in modal to title (#13720)
8ee03b0 feat: add more explicit view all description (#13737)
f972c9b refactor: move k8 security context to singleton (#13679)
71ffcb7 fix: apm env vars need to be added to mutable list object (#13759)
37f7005 fix: remount global requests to avoid index issue (#13754)
6d0bd7c chore: make replication worker buffer size configurable (#13749)
b4d99c6 chore: fixing some comments (#13741)
37774d1 fix: bad variable name (#13736)
d0eb8a6 chore: update CDK version following release (#13748)
0a5715a fix: add "button" type to AI Button (#13721)
de7709a feat: frontend changes for publish modal when contributing to existing connector (behind feature flag) (#13670)
d376f5a chore: remove 'connectorBuilder.contributeToMarketplace' feature flag (#13717)
33d12df fix: actually pass connector apm env vars to connector (#13744)
986f00b fix: match casing & show number of synced streams in the status graph tooltip (#13745)
de2972c refactor: remove src/components/common directory (#13740)
b93873a feat: timeline events for enabling/disabling a connection, and connection settings updates (#13613)
f3023ca chore: use ellipsis character exclusively (#13693)
f03f774 feat: implement hashing mapper (#13607)
4ad8717 fix: fix StateWithIdTest to work when tests are run in random order (#13738)
e4b0285 chore: introduce mapper configurations (#13526)
17e3c6b chore: remove AutoBackfillOnNewColumns feature flag (#13735)
f5930e7 docs: add stories for LoadingSkeleton and BorderedTile (#13730)
4d5a9e0 fix: refresh creds for sync in init container (#13700)
e7482f3 fix: restore deleted script (#13731)
2639300 feat: organizational billing UI (#13649)
57c70eb fix: [SyncCatalog V2] add buttonPlaceholder for source-defined cursor catalog combobox (#13719)
698f43c chore: [SyncCatalog V2] add specific validation error for Pk and Cursor (#13650)
a30e048 Bump helm chart version reference to 0.493.1
2c8f465 Bump helm chart version reference to 0.493.0