Merge tag 'v14.11.0' into next-major

CHANGELOG.md

@@ -24,8 +24,36 @@
 # 14.10.4 Release notes
 
 ### Enhancements
+* On Windows devices Device Sync will additionally look up SSL certificates in the Windows Trusted Root Certification Authorities certificate store when establishing a connection. (PR [#7882](https://github.com/realm/realm-core/pull/7882))
+* Updated the return type of `LogCategory::get_category_names()` from `std::vector<const char*>` to `std::vector<std::string_view>`. ([PR #7879](https://github.com/realm/realm-core/pull/7879))
+* Added `realm_get_persisted_schema_version` for reading the version of the schema currently stored locally. (PR [#7873](https://github.com/realm/realm-core/pull/7873))
+* Added `realm_app_config_get_sync_client_config()` function to the C_API to get the sync_client_config value in `realm_app_config_t` if REALM_APP_SERVICES is enabled. If REALM_APP_SERVICES is not available, `realm_sync_client_config_new()` is available to create a new `sync_client_config_t` to use when initializing the sync manager. ([PR #7891](https://github.com/realm/realm-core/pull/7891))
+* Role and permissions changes no longer require a client reset to update the local realm. ([PR #7440](https://github.com/realm/realm-core/pull/7440))
+
+### Fixed
+* FLX download progress was only updated when bootstraps completed, making it always be 0 before the first completion and then forever 1. ([PR #7869](https://github.com/realm/realm-core/issues/7869), since v14.10.2)
+* Sync client can crash if a session is resumed while the session is being suspended. ([#7860](https://github.com/realm/realm-core/issues/7860), since v12.0.0)
+* If a sync session is interrupted by a disconnect or restart while downloading a bootstrap, stale data from the previous bootstrap may be included when the session reconnects and downloads the bootstrap. This can lead to objects stored in the database that do not match the actual state of the server and potentially leading to compensating writes. ([#7827](https://github.com/realm/realm-core/issues/7827), since v12.0.0)
+* Fixed unnecessary server roundtrips when there is no download to acknowledge ([#2129](https://jira.mongodb.org/browse/RCORE-2129), since v14.8.0).
+
+### Breaking changes
 * None.
 
+### Compatibility
+* Fileformat: Generates files with format v24. Reads and automatically upgrade from fileformat v10. If you want to upgrade from an earlier file format version you will have to use RealmCore v13.x.y or earlier.
+
+-----------
+
+### Internals
+* Protocol version has been updated to v14 to support server intiated bootstraps and role change updates without a client reset. ([PR #7440](https://github.com/realm/realm-core/pull/7440))
+* Add support for server initiated bootstraps. ([PR #7440](https://github.com/realm/realm-core/pull/7440))
+
+----------------------------------------------
+
+# 14.10.4 Release notes
+
+### Enhancements
+
 ### Fixed
 * When a public name is defined on a property, calling `realm::Results::sort()` or `realm::Results::distinct()` with the internal name could throw an error like `Cannot sort on key path 'NAME': property 'PersonObject.NAME' does not exist`. ([realm/realm-js#6779](https://github.com/realm/realm-js/issues/6779), since v12.12.0)
 
@@ -45,14 +73,14 @@
 # 14.10.3 Release notes
 
 ### Enhancements
-* "Next launch" metadata file actions are now performed in a multi-process safe manner ([#7576](https://github.com/realm/realm-core/pull/7576)).
+* "Next launch" metadata file actions are now performed in a multi-process safe manner. ([PR #7576](https://github.com/realm/realm-core/pull/7576))
 
 ### Fixed
 * Fixed a change of mode from Strong to All when removing links from an embedded object that links to a tombstone. This affects sync apps that use embedded objects which have a `Lst<Mixed>` that contains a link to another top level object which has been deleted by another sync client (creating a tombstone locally). In this particular case, the switch would cause any remaining link removals to recursively delete the destination object if there were no other links to it. ([#7828](https://github.com/realm/realm-core/issues/7828), since 14.0.0-beta.0)
 * Fixed removing backlinks from the wrong objects if the link came from a nested list, nested dictionary, top-level dictionary, or list of mixed, and the source table had more than 256 objects. This could manifest as `array_backlink.cpp:112: Assertion failed: int64_t(value >> 1) == key.value` when removing an object. ([#7594](https://github.com/realm/realm-core/issues/7594), since v11 for dictionaries)
 * Fixed the collapse/rejoin of clusters which contained nested collections with links. This could manifest as `array.cpp:319: Array::move() Assertion failed: begin <= end [2, 1]` when removing an object. ([#7839](https://github.com/realm/realm-core/issues/7839), since the introduction of nested collections in v14.0.0-beta.0)
 * wait_for_upload_completion() was inconsistent in how it handled commits which did not produce any changesets to upload. Previously it would sometimes complete immediately if all commits waiting to be uploaded were empty, and at other times it would wait for a server roundtrip. It will now always complete immediately. ([PR #7796](https://github.com/realm/realm-core/pull/7796)).
-* `realm_sync_session_handle_error_for_testing` parameter `is_fatal` was flipped changing the expected behavior. (#[7750](https://github.com/realm/realm-core/issues/7750)).
+* `realm_sync_session_handle_error_for_testing` parameter `is_fatal` was flipped changing the expected behavior. ([#7750](https://github.com/realm/realm-core/issues/7750))
 
 ### Breaking changes
 * None.

Package.swift

@@ -3,7 +3,7 @@
 import PackageDescription
 import Foundation
 
-let versionStr = "14.10.4"
+let versionStr = "14.11.0"
 let versionPieces = versionStr.split(separator: "-")
 let versionCompontents = versionPieces[0].split(separator: ".")
 let versionExtra = versionPieces.count > 1 ? versionPieces[1] : ""

dependencies.yml

@@ -1,7 +1,8 @@
 PACKAGE_NAME: realm-core
-VERSION: 14.10.4
+VERSION: 14.11.0
 OPENSSL_VERSION: 3.2.0
 ZLIB_VERSION: 1.2.13
 # https://github.com/10gen/baas/commits
 # 2f308db is 2024 July 10
 BAAS_VERSION: 2f308db6f65333728a101d1fecbb792f9659a5ce
+BAAS_VERSION_TYPE: githash

evergreen/config.yml

@@ -242,8 +242,9 @@ functions:
         shell: bash
         env:
           BAASAAS_API_KEY: "${baasaas_api_key}"
+          # BAAS_VERSION and VERSION_TYPE are set by realm-core/dependencies.yml
           BAASAAS_REF_SPEC: "${BAAS_VERSION}"
-          BAASAAS_START_MODE: "githash"
+          BAASAAS_START_MODE: "${BAAS_VERSION_TYPE|githash}"
         script: |-
           set -o errexit
           set -o verbose

src/realm.h

@@ -1241,6 +1241,11 @@ RLM_API realm_schema_t* realm_get_schema(const realm_t*);
  */
 RLM_API uint64_t realm_get_schema_version(const realm_t* realm);
 
+/**
+ * Get the schema version for this realm at the path.
+ */
+RLM_API uint64_t realm_get_persisted_schema_version(const realm_config_t* config);
+
 /**
  * Update the schema of an open realm.
  *
@@ -3018,8 +3023,6 @@ RLM_API void realm_app_config_set_metadata_mode(realm_app_config_t*,
 RLM_API void realm_app_config_set_metadata_encryption_key(realm_app_config_t*, const uint8_t[64]) RLM_API_NOEXCEPT;
 RLM_API void realm_app_config_set_security_access_group(realm_app_config_t*, const char*) RLM_API_NOEXCEPT;
 
-RLM_API realm_sync_client_config_t* realm_app_config_get_sync_client_config(realm_app_config_t*) RLM_API_NOEXCEPT;
-
 /**
  * Get an existing @a realm_app_credentials_t and return it's json representation
  * Note: the caller must delete the pointer to the string via realm_release
@@ -3741,8 +3744,21 @@ typedef void (*realm_async_open_task_completion_func_t)(realm_userdata_t userdat
 // callback runs.
 typedef void (*realm_async_open_task_init_subscription_func_t)(realm_thread_safe_reference_t* realm,
                                                                realm_userdata_t userdata);
-
+#if REALM_APP_SERVICES
+// If using App Services, the realm_sync_client_config_t instance is part of the
+// realm_app_config_t structure and this function returns a pointer to that
+// member property. The realm_sync_client_config_t reference returned by this
+// function should not be freed using realm_release.
+RLM_API realm_sync_client_config_t* realm_app_config_get_sync_client_config(realm_app_config_t*) RLM_API_NOEXCEPT;
+#else
+// If not using App Services, the realm_app_config_t structure is not defined, and
+// the real_sync_client_config_t structure returned by this function is meant to be
+// used with realm_sync_manager_create() to create a separate Sync Manager instance.
+// The realm_sync_client_config_t instance returned by this function will need to be
+// manually freed using realm_release.
 RLM_API realm_sync_client_config_t* realm_sync_client_config_new(void) RLM_API_NOEXCEPT;
+#endif // REALM_APP_SERVICES
+
 RLM_API void realm_sync_client_config_set_reconnect_mode(realm_sync_client_config_t*,
                                                          realm_sync_client_reconnect_mode_e) RLM_API_NOEXCEPT;
 RLM_API void realm_sync_client_config_set_multiplex_sessions(realm_sync_client_config_t*, bool) RLM_API_NOEXCEPT;

src/realm/object-store/c_api/app.cpp

@@ -279,6 +279,11 @@ RLM_API const char* realm_app_credentials_serialize_as_json(realm_app_credential
     });
 }
 
+RLM_API realm_sync_client_config_t* realm_app_config_get_sync_client_config(realm_app_config_t* app_config) noexcept
+{
+    return static_cast<realm_sync_client_config_t*>(&app_config->sync_client_config);
+}
+
 RLM_API realm_app_t* realm_app_create(const realm_app_config_t* app_config)
 {
     return wrap_err([&] {

src/realm/object-store/c_api/logging.cpp

@@ -91,7 +91,7 @@ RLM_API size_t realm_get_category_names(size_t num_values, const char** out_valu
         if (number_to_copy > num_values)
             number_to_copy = num_values;
         for (size_t n = 0; n < number_to_copy; n++) {
-            out_values[n] = vec[n];
+            out_values[n] = vec[n].data();
         }
     }
     return number_to_copy;

src/realm/object-store/c_api/schema.cpp

@@ -55,6 +55,23 @@ RLM_API uint64_t realm_get_schema_version(const realm_t* realm)
     return rlm->schema_version();
 }
 
+RLM_API uint64_t realm_get_persisted_schema_version(const realm_config_t* config)
+{
+    auto conf = RealmConfig();
+    conf.schema_version = ObjectStore::NotVersioned;
+    conf.path = config->path;
+    conf.encryption_key = config->encryption_key;
+
+    if (config->sync_config) {
+        conf.sync_config = nullptr;
+        conf.force_sync_history = true;
+    }
+
+    auto realm = Realm::get_shared_realm(conf);
+    uint64_t version = ObjectStore::get_schema_version(realm->read_group());
+    return version;
+}
+
 RLM_API bool realm_schema_validate(const realm_schema_t* schema, uint64_t validation_mode)
 {
     return wrap_err([&]() {

src/realm/object-store/c_api/sync.cpp

@@ -128,10 +128,12 @@ static Query add_ordering_to_realm_query(Query realm_query, const DescriptorOrde
     return realm_query;
 }
 
+#if !REALM_APP_SERVICES
 RLM_API realm_sync_client_config_t* realm_sync_client_config_new(void) noexcept
 {
     return new realm_sync_client_config_t;
 }
+#endif // !REALM_APP_SERVICES
 
 RLM_API void realm_sync_client_config_set_reconnect_mode(realm_sync_client_config_t* config,
                                                          realm_sync_client_reconnect_mode_e mode) noexcept

src/realm/object-store/c_api/types.hpp

@@ -617,9 +617,21 @@ struct realm_http_transport : realm::c_api::WrapC, std::shared_ptr<realm::app::G
     }
 };
 
+#if REALM_APP_SERVICES
+// This class doesn't support realm_release() since it is only meant to be used
+// as a CAPI-compatible reference to the SyncClientConfig member variable that
+// is part of AppConfig.
+// To avoid data misalignment or other conflicts with the original SyncClientConfig,
+// do not add any additional functions or member variables to this class.
+struct realm_sync_client_config final : realm::SyncClientConfig {
+    using SyncClientConfig::SyncClientConfig;
+};
+#else
+// This class must be freed using realm_release()
 struct realm_sync_client_config : realm::c_api::WrapC, realm::SyncClientConfig {
     using SyncClientConfig::SyncClientConfig;
 };
+#endif // REALM_APP_SERVICES
 
 struct realm_sync_config : realm::c_api::WrapC, realm::SyncConfig {
     using SyncConfig::SyncConfig;

src/realm/object-store/sync/async_open_task.cpp

@@ -65,6 +65,25 @@ void AsyncOpenTask::start(AsyncOpenCallback callback)
     session->revive_if_needed();
 }
 
+util::Future<ThreadSafeReference> AsyncOpenTask::start()
+{
+    auto pf = util::make_promise_future<ThreadSafeReference>();
+    start([promise = std::move(pf.promise)](ThreadSafeReference&& ref, std::exception_ptr e) mutable {
+        if (e) {
+            try {
+                std::rethrow_exception(e);
+            }
+            catch (...) {
+                promise.set_error(exception_to_status());
+            }
+        }
+        else {
+            promise.emplace_value(std::move(ref));
+        }
+    });
+    return std::move(pf.future);
+}
+
 void AsyncOpenTask::cancel()
 {
     std::shared_ptr<SyncSession> session;

src/realm/object-store/sync/async_open_task.hpp

@@ -21,6 +21,7 @@
 
 #include <realm/util/checked_mutex.hpp>
 #include <realm/util/functional.hpp>
+#include <realm/util/future.hpp>
 
 #include <memory>
 #include <vector>
@@ -47,13 +48,21 @@ class AsyncOpenTask : public std::enable_shared_from_this<AsyncOpenTask> {
                            std::shared_ptr<realm::SyncSession> session, bool db_open_for_the_first_time);
     AsyncOpenTask(const AsyncOpenTask&) = delete;
     AsyncOpenTask& operator=(const AsyncOpenTask&) = delete;
+
     // Starts downloading the Realm. The callback will be triggered either when the download completes
     // or an error is encountered.
     //
     // If multiple AsyncOpenTasks all attempt to download the same Realm and one of them is canceled,
     // the other tasks will receive a "Cancelled" exception.
     void start(AsyncOpenCallback callback) REQUIRES(!m_mutex);
 
+    // Starts downloading the Realm. The future will be fulfilled either when the download completes
+    // or an error is encountered.
+    //
+    // If multiple AsyncOpenTasks all attempt to download the same Realm and one of them is canceled,
+    // the other tasks will receive a cancelled Status
+    util::Future<ThreadSafeReference> start() REQUIRES(!m_mutex);
+
     // Cancels the download and stops the session. No further functions should be called on this class.
     void cancel() REQUIRES(!m_mutex);
 

src/realm/object-store/sync/sync_session.hpp

@@ -30,9 +30,9 @@
 #include <realm/util/future.hpp>
 #include <realm/version_id.hpp>
 
+#include <map>
 #include <mutex>
 #include <unordered_map>
-#include <map>
 
 namespace realm {
 class DB;
@@ -345,6 +345,8 @@ class SyncSession : public std::enable_shared_from_this<SyncSession> {
             return session.get_appservices_connection_id();
         }
 
+        // Supported commands can be found in `handleTestCommandMessage()`
+        // in baas/devicesync/server/qbs_client_handler_functions.go
         static util::Future<std::string> send_test_command(SyncSession& session, std::string request)
         {
             return session.send_test_command(std::move(request));

src/realm/sync/CMakeLists.txt

@@ -108,13 +108,11 @@ elseif(REALM_HAVE_OPENSSL)
     target_link_libraries(Sync PUBLIC OpenSSL::SSL)
 endif()
 
-if(WIN32 AND NOT WINDOWS_STORE)
-    target_link_libraries(Sync INTERFACE Version.lib)
-    if(CMAKE_VERSION VERSION_LESS "3.21")
-        # This is needed for OpenSSL, but CMake's FindOpenSSL didn't declare it
-        # on the OpenSSL::Crypto target until CMake 3.21.0. 
-        target_link_libraries(Sync INTERFACE Crypt32.lib)
+if(WIN32)
+    if(NOT WINDOWS_STORE)
+        target_link_libraries(Sync INTERFACE Version.lib)
     endif()
+    target_link_libraries(Sync INTERFACE Crypt32.lib)
 endif()
 
 install(TARGETS Sync EXPORT realm