Merge pull request #1521 from CesiumGS/shared-assets

Cache images across glTFs to avoid duplication
CesiumGS · Nov 1, 2024 · 203c697 · 203c697
2 parents 681dc06 + 28333a1
commit 203c697
Show file tree

Hide file tree

Showing 57 changed files with 2,936 additions and 2,073 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -5,6 +5,7 @@
 ##### Additions :tada:
 
 - Added `CircumscribedGroundHeight` property to `CesiumSunSky`. It defaults to 0, which is consistent with the previous behavior. It can be set to a larger value (like 40) to avoid dark splotchy artifacts when zoomed out far from the globe in certain tilesets where geometry extends very far beyond the ellipsoid in the low-detail representation, such as Google Photorealistic 3D Tiles.
+- Added a shared assets system that ensures external images referenced by different glTFs will only be loaded once per image. Previously, these images would be loaded again for each glTF that referenced them. This significantly reduces memory usage for tilesets that reuse the same textures.
 
 ##### Fixes :wrench:
 
@@ -165,16 +166,16 @@ In addition to the above, this release updates [cesium-native](https://github.co
 
 ##### Additions :tada:
 
- - Added support for multiple Cesium ion servers by creating `CesiumIonServer` data assets.
+- Added support for multiple Cesium ion servers by creating `CesiumIonServer` data assets.
 
 In addition to the above, this release updates [cesium-native](https://github.com/CesiumGS/cesium-native) from v0.30.0 to v0.31.0. See the [changelog](https://github.com/CesiumGS/cesium-native/blob/main/CHANGES.md) for a complete list of changes in cesium-native.
 
 ### v2.1.0 - 2023-12-01
 
 ##### Additions :tada:
 
- - Added support for styling with property textures in `EXT_structural_metadata`.
- - Significantly improved tile download performance by adding `HttpThreadActiveFrameTimeInSeconds=0.001` to `Engine.ini`. This results in a major performance improvement for all tilesets, particularly Google Photorealistic 3D Tiles.
+- Added support for styling with property textures in `EXT_structural_metadata`.
+- Significantly improved tile download performance by adding `HttpThreadActiveFrameTimeInSeconds=0.001` to `Engine.ini`. This results in a major performance improvement for all tilesets, particularly Google Photorealistic 3D Tiles.
 - Added `HttpMaxConnectionsPerServer=40` to `Engine.ini`. By default, only 16 connections are allowed, which limits the performance when downloading tiles.
 
 ##### Fixes :wrench:
@@ -194,7 +195,7 @@ This release no longer supports Unreal Engine v5.0. Unreal Engine v5.1, v5.2, or
 ##### Breaking Changes :mega:
 
 - Removed `FCesiumIntegerColor`, `FCesiumFloatColor`, `UCesiumFeatureTexturePropertyBlueprintLibrary::GetIntegerColorFromTextureCoordinates` and `UCesiumFeatureTexturePropertyBlueprintLibrary::GetFloatColorFromTextureCoordinates`. Check out the [upgrade guide](Documentation/upgrade-to-2.0-guide.md) for how retrieve metadata from property textures with the new API.
-- Renamed `GetTextureCoordinateIndex` to `GetUnrealUVChannel` in both `UCesiumFeatureIdTextureBlueprintLibrary` and `UCesiumPropertyTexturePropertyBlueprintLibrary`. Contrary to what the documentation claimed, this function retrieved the index of the texture coordinate set in the *Unreal static mesh*, which is not necessarily equal to the texture coordinate set index in the *glTF primitive*. For the latter value, use `GetGltfTextureCoordinateSetIndex` instead.
+- Renamed `GetTextureCoordinateIndex` to `GetUnrealUVChannel` in both `UCesiumFeatureIdTextureBlueprintLibrary` and `UCesiumPropertyTexturePropertyBlueprintLibrary`. Contrary to what the documentation claimed, this function retrieved the index of the texture coordinate set in the _Unreal static mesh_, which is not necessarily equal to the texture coordinate set index in the _glTF primitive_. For the latter value, use `GetGltfTextureCoordinateSetIndex` instead.
 - Removed the old "exclusion zones" feature, which has been deprecated since v1.11.0. Use `CesiumCartographicPolygon` or `CesiumTileExcluder` instead.
 
 ##### Additions :tada:
@@ -343,7 +344,7 @@ In addition to the above, this release updates [cesium-native](https://github.co
 ##### Fixes :wrench:
 
 - Added a workaround for an apparent bug in Unreal Engine 5.1 that prevented collisions from working with Cesium3DTilesets.
-- Fixed a bug that could cause the `AGlobeAwareDefaultPawn` / `DynamicPawn`  to suddenly move to a very high height for one render frame just as it arrives at its destination during a flight.
+- Fixed a bug that could cause the `AGlobeAwareDefaultPawn` / `DynamicPawn` to suddenly move to a very high height for one render frame just as it arrives at its destination during a flight.
 
 In addition to the above, this release updates [cesium-native](https://github.com/CesiumGS/cesium-native) from v0.25.0 to v0.25.1. See the [changelog](https://github.com/CesiumGS/cesium-native/blob/main/CHANGES.md) for a complete list of changes in cesium-native.
 
@@ -406,7 +407,7 @@ This will be the _last_ release that supports Unreal Engine v4.27. Future versio
 
 ##### Additions :tada:
 
-- The `FlyToAltitudeProfileCurve`, `FlyToProgressCurve`, `FlyToMaximumAltitudeCurve`, `FlyToDuration`, and `FlyToGranularityDegrees` properties of `GlobeAwareDefaultPawn`  / `DynamicPawn` may now be read and written from Blueprints.
+- The `FlyToAltitudeProfileCurve`, `FlyToProgressCurve`, `FlyToMaximumAltitudeCurve`, `FlyToDuration`, and `FlyToGranularityDegrees` properties of `GlobeAwareDefaultPawn` / `DynamicPawn` may now be read and written from Blueprints.
 - Added an option on `Cesium3DTileset` to ignore the `KHR_materials_unlit` extension entirely and use normal lighting and shadows.
 - Added `CreateNavCollision` property to `Cesium3DTileset`. When enabled, `CreateNavCollision` is called on the static meshes created for tiles.
 

diff --git a/Source/CesiumRuntime/Private/Cesium3DTileset.cpp b/Source/CesiumRuntime/Private/Cesium3DTileset.cpp
@@ -8,16 +8,18 @@
 #include "Cesium3DTilesSelection/Tile.h"
 #include "Cesium3DTilesSelection/TilesetLoadFailureDetails.h"
 #include "Cesium3DTilesSelection/TilesetOptions.h"
+#include "Cesium3DTilesSelection/TilesetSharedAssetSystem.h"
 #include "Cesium3DTilesetLoadFailureDetails.h"
 #include "Cesium3DTilesetRoot.h"
 #include "CesiumActors.h"
+#include "CesiumAsync/SharedAssetDepot.h"
 #include "CesiumBoundingVolumeComponent.h"
 #include "CesiumCamera.h"
 #include "CesiumCameraManager.h"
 #include "CesiumCommon.h"
 #include "CesiumCustomVersion.h"
 #include "CesiumGeospatial/GlobeTransforms.h"
-#include "CesiumGltf/ImageCesium.h"
+#include "CesiumGltf/ImageAsset.h"
 #include "CesiumGltf/Ktx2TranscodeTargets.h"
 #include "CesiumGltfComponent.h"
 #include "CesiumGltfPointsSceneProxyUpdater.h"
@@ -40,6 +42,7 @@
 #include "Engine/TextureRenderTarget2D.h"
 #include "Engine/World.h"
 #include "EngineUtils.h"
+#include "ExtensionImageAssetUnreal.h"
 #include "GameFramework/PlayerController.h"
 #include "Kismet/GameplayStatics.h"
 #include "LevelSequenceActor.h"
@@ -781,16 +784,14 @@ class UnrealResourcePreparer
       Cesium3DTilesSelection::TileLoadResult&& tileLoadResult,
       const glm::dmat4& transform,
       const std::any& rendererOptions) override {
-    CesiumGltf::Model* pModel =
-        std::get_if<CesiumGltf::Model>(&tileLoadResult.contentKind);
-    if (!pModel)
+    CreateGltfOptions::CreateModelOptions options(std::move(tileLoadResult));
+    if (!options.pModel) {
       return asyncSystem.createResolvedFuture(
           Cesium3DTilesSelection::TileLoadResultAndRenderResources{
-              std::move(tileLoadResult),
+              std::move(options.tileLoadResult),
               nullptr});
+    }
 
-    CreateGltfOptions::CreateModelOptions options;
-    options.pModel = pModel;
     options.alwaysIncludeTangents = this->_pActor->GetAlwaysIncludeTangents();
     options.createPhysicsMeshes = this->_pActor->GetCreatePhysicsMeshes();
 
@@ -807,16 +808,21 @@ class UnrealResourcePreparer
 
     const CesiumGeospatial::Ellipsoid& ellipsoid = tileLoadResult.ellipsoid;
 
-    TUniquePtr<UCesiumGltfComponent::HalfConstructed> pHalf =
-        UCesiumGltfComponent::CreateOffGameThread(
+    CesiumAsync::Future<UCesiumGltfComponent::CreateOffGameThreadResult>
+        pHalfFuture = UCesiumGltfComponent::CreateOffGameThread(
+            asyncSystem,
             transform,
-            options,
+            std::move(options),
             ellipsoid);
 
-    return asyncSystem.createResolvedFuture(
-        Cesium3DTilesSelection::TileLoadResultAndRenderResources{
-            std::move(tileLoadResult),
-            pHalf.Release()});
+    return MoveTemp(pHalfFuture)
+        .thenImmediately(
+            [](UCesiumGltfComponent::CreateOffGameThreadResult&& result)
+                -> Cesium3DTilesSelection::TileLoadResultAndRenderResources {
+              return Cesium3DTilesSelection::TileLoadResultAndRenderResources{
+                  std::move(result.TileLoadResult),
+                  result.HalfConstructed.Release()};
+            });
   }
 
   virtual void* prepareInMainThread(
@@ -862,7 +868,7 @@ class UnrealResourcePreparer
   }
 
   virtual void* prepareRasterInLoadThread(
-      CesiumGltf::ImageCesium& image,
+      CesiumGltf::ImageAsset& image,
       const std::any& rendererOptions) override {
     auto ppOptions =
         std::any_cast<FRasterOverlayRendererOptions*>(&rendererOptions);
@@ -875,7 +881,7 @@ class UnrealResourcePreparer
 
     if (pOptions->useMipmaps) {
       std::optional<std::string> errorMessage =
-          CesiumGltfReader::GltfReader::generateMipMaps(image);
+          CesiumGltfReader::ImageDecoder::generateMipMaps(image);
       if (errorMessage) {
         UE_LOG(
             LogCesium,
@@ -885,17 +891,31 @@ class UnrealResourcePreparer
       }
     }
 
+    // TODO: sRGB should probably be configurable on the raster overlay.
+    bool sRGB = true;
+
+    const ExtensionImageAssetUnreal& extension =
+        ExtensionImageAssetUnreal::getOrCreate(
+            CesiumAsync::AsyncSystem(nullptr), // TODO
+            image,
+            sRGB,
+            pOptions->useMipmaps,
+            std::nullopt);
+
+    // Because raster overlay images are never shared (at least currently!), the
+    // future should already be resolved by the time we get here.
+    check(extension.getFuture().isReady());
+
     auto texture = CesiumTextureUtility::loadTextureAnyThreadPart(
         image,
         TextureAddress::TA_Clamp,
         TextureAddress::TA_Clamp,
         pOptions->filter,
         pOptions->useMipmaps,
         pOptions->group,
-        // TODO: sRGB should probably be configurable on the raster overlay.
-        true,
-        std::nullopt,
-        nullptr);
+        sRGB,
+        std::nullopt);
+
     return texture.Release();
   }
 
@@ -954,7 +974,7 @@ class UnrealResourcePreparer
     const Cesium3DTilesSelection::TileContent& content = tile.getContent();
     const Cesium3DTilesSelection::TileRenderContent* pRenderContent =
         content.getRenderContent();
-    if (pRenderContent) {
+    if (pMainThreadRendererResources != nullptr && pRenderContent != nullptr) {
       UCesiumGltfComponent* pGltfContent =
           reinterpret_cast<UCesiumGltfComponent*>(
               pRenderContent->getRenderResources());
@@ -984,7 +1004,7 @@ class UnrealResourcePreparer
       UCesiumGltfComponent* pGltfContent =
           reinterpret_cast<UCesiumGltfComponent*>(
               pRenderContent->getRenderResources());
-      if (pGltfContent) {
+      if (pMainThreadRendererResources != nullptr && pGltfContent != nullptr) {
         pGltfContent->DetachRasterTile(
             tile,
             rasterTile,
@@ -1941,7 +1961,38 @@ void ACesium3DTileset::updateLastViewUpdateResultState(
     const Cesium3DTilesSelection::ViewUpdateResult& result) {
   TRACE_CPUPROFILER_EVENT_SCOPE(Cesium::updateLastViewUpdateResultState)
 
-  if (!this->LogSelectionStats) {
+  if (this->DrawTileInfo) {
+    const UWorld* World = GetWorld();
+    check(World);
+
+    const TSoftObjectPtr<ACesiumGeoreference> Georeference =
+        ResolveGeoreference();
+    check(Georeference);
+
+    for (Cesium3DTilesSelection::Tile* tile : result.tilesToRenderThisFrame) {
+
+      CesiumGeometry::OrientedBoundingBox obb =
+          Cesium3DTilesSelection::getOrientedBoundingBoxFromBoundingVolume(
+              tile->getBoundingVolume(),
+              Georeference->GetEllipsoid()->GetNativeEllipsoid());
+
+      FVector unrealCenter =
+          Georeference->TransformEarthCenteredEarthFixedPositionToUnreal(
+              VecMath::createVector(obb.getCenter()));
+
+      FString text = FString::Printf(
+          TEXT("ID %s (%p)"),
+          UTF8_TO_TCHAR(
+              Cesium3DTilesSelection::TileIdUtilities::createTileIdString(
+                  tile->getTileID())
+                  .c_str()),
+          tile);
+
+      DrawDebugString(World, unrealCenter, text, nullptr, FColor::Red, 0, true);
+    }
+  }
+
+  if (!this->LogSelectionStats && !this->LogSharedAssetStats) {
     return;
   }
 
@@ -1972,24 +2023,40 @@ void ACesium3DTileset::updateLastViewUpdateResultState(
         result.tilesWaitingForOcclusionResults;
     this->_lastMaxDepthVisited = result.maxDepthVisited;
 
-    UE_LOG(
-        LogCesium,
-        Display,
-        TEXT(
-            "%s: %d ms, Visited %d, Culled Visited %d, Rendered %d, Culled %d, Occluded %d, Waiting For Occlusion Results %d, Max Depth Visited: %d, Loading-Worker %d, Loading-Main %d, Loaded tiles %g%%"),
-        *this->GetName(),
-        (std::chrono::high_resolution_clock::now() - this->_startTime).count() /
-            1000000,
-        result.tilesVisited,
-        result.culledTilesVisited,
-        result.tilesToRenderThisFrame.size(),
-        result.tilesCulled,
-        result.tilesOccluded,
-        result.tilesWaitingForOcclusionResults,
-        result.maxDepthVisited,
-        result.workerThreadTileLoadQueueLength,
-        result.mainThreadTileLoadQueueLength,
-        this->LoadProgress);
+    if (this->LogSelectionStats) {
+      UE_LOG(
+          LogCesium,
+          Display,
+          TEXT(
+              "%s: %d ms, Visited %d, Culled Visited %d, Rendered %d, Culled %d, Occluded %d, Waiting For Occlusion Results %d, Max Depth Visited: %d, Loading-Worker %d, Loading-Main %d, Loaded tiles %g%%"),
+          *this->GetName(),
+          (std::chrono::high_resolution_clock::now() - this->_startTime)
+                  .count() /
+              1000000,
+          result.tilesVisited,
+          result.culledTilesVisited,
+          result.tilesToRenderThisFrame.size(),
+          result.tilesCulled,
+          result.tilesOccluded,
+          result.tilesWaitingForOcclusionResults,
+          result.maxDepthVisited,
+          result.workerThreadTileLoadQueueLength,
+          result.mainThreadTileLoadQueueLength,
+          this->LoadProgress);
+    }
+
+    if (this->LogSharedAssetStats && this->_pTileset) {
+      const Cesium3DTilesSelection::TilesetSharedAssetSystem::ImageDepot&
+          imageDepot = *this->_pTileset->getSharedAssetSystem().pImage;
+      UE_LOG(
+          LogCesium,
+          Display,
+          TEXT(
+              "Images shared asset depot: %d distinct assets, %d inactive assets pending deletion (%d bytes)"),
+          imageDepot.getAssetCount(),
+          imageDepot.getInactiveAssetCount(),
+          imageDepot.getInactiveAssetTotalSizeBytes());
+    }
   }
 }