Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Splat shader #12364

Draft
wants to merge 62 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
0103673
initial vertex calcs for splats
keyboardspecialist Jun 7, 2024
ea53962
removed early returns. we seem to hit them all the time
keyboardspecialist Jun 8, 2024
60022f4
covariance fixes
keyboardspecialist Jun 12, 2024
ba77e31
hack to switch primitive type points to triangle_strips
keyboardspecialist Jun 13, 2024
8d67853
refactor, use inverse matrices
keyboardspecialist Jun 17, 2024
9aaf292
final point primitive experiment. moving to quads
keyboardspecialist Jun 18, 2024
8c3aea4
quad drawing.
keyboardspecialist Jun 19, 2024
7a3f472
new splat pipeline files
keyboardspecialist Jun 21, 2024
4f9c886
shader org
keyboardspecialist Jun 21, 2024
b235c09
switched rendering to not need vertex buffer attr
keyboardspecialist Jun 21, 2024
df28196
tweaks
keyboardspecialist Jun 21, 2024
bdf4eb5
detect KHR_gaussian_splatting extension and set props and prime for p…
keyboardspecialist Jun 26, 2024
7a4c864
Buffer rework. inject attribute buffer for screen space quad at gltf …
keyboardspecialist Jun 28, 2024
c6b8d25
splats but bad projection
keyboardspecialist Jul 1, 2024
a03abdd
shader cleanup, simplification
keyboardspecialist Jul 1, 2024
7d5b09a
minor tweaks
keyboardspecialist Jul 1, 2024
0620dca
tweaks
keyboardspecialist Jul 1, 2024
fe340a4
Fixed position and color buffers
keyboardspecialist Jul 1, 2024
e352ca2
scaling?
keyboardspecialist Jul 1, 2024
3e74c74
added new GAUSSIAN_SPLAT pass
keyboardspecialist Jul 2, 2024
d5f4069
pipeline execution for splats
keyboardspecialist Jul 2, 2024
13dfa78
splats sorted via countSort
keyboardspecialist Jul 2, 2024
31e8d13
invalidates splat commands when camera moves
keyboardspecialist Jul 3, 2024
947d9e0
gaurded splat vertex setup in builddrawcommand
keyboardspecialist Jul 3, 2024
9862eeb
command rebuild under control
keyboardspecialist Jul 3, 2024
0069114
guard gaussian pipeline stage
keyboardspecialist Jul 3, 2024
427d490
sorting fixes
keyboardspecialist Jul 4, 2024
fcfc3df
cleanup
keyboardspecialist Jul 8, 2024
f4657c6
opacity fix
keyboardspecialist Jul 9, 2024
2270dae
small tweak to re-enable instanceDivisor check on attr 0
keyboardspecialist Jul 12, 2024
fb180f6
splat scale now adjustable with 3DTile style
keyboardspecialist Jul 12, 2024
43c08cf
toggle between points and splats at runtime
keyboardspecialist Jul 12, 2024
2b1e72a
Merge branch 'main' into splat-shader
keyboardspecialist Oct 9, 2024
52b3d61
fixes from latest merge
keyboardspecialist Oct 22, 2024
f207786
shader updates improves performance and visual quality
keyboardspecialist Oct 25, 2024
18d75d2
staging for demo
keyboardspecialist Nov 4, 2024
ddceea8
Add unsigned int pixel datatype
keyboardspecialist Nov 13, 2024
d2441cf
guassian texture pipeline
keyboardspecialist Nov 14, 2024
3c7bae6
splat index attr
keyboardspecialist Nov 15, 2024
6061b17
splatting texture pipeline updates and improvements
keyboardspecialist Nov 20, 2024
b2f5aa2
Lots of pipeline changes.
keyboardspecialist Nov 22, 2024
361312b
temporary wasm package integration for CI and sandcastle until NPM
keyboardspecialist Nov 22, 2024
1991389
missing files
keyboardspecialist Nov 22, 2024
baf322d
Rendering fixes and tweaks
keyboardspecialist Dec 6, 2024
bf4ab9d
lets try and clean up this branch with the new prettier changes
keyboardspecialist Dec 10, 2024
de96ac6
sorting fixes in texture mode
keyboardspecialist Dec 11, 2024
276a410
support scaling in texture mode
keyboardspecialist Dec 11, 2024
fb0559b
add myself
keyboardspecialist Dec 11, 2024
6874e72
more sensible dequant positions
keyboardspecialist Dec 11, 2024
2ed9de4
force a command rebuild if the sorter isnt ready
keyboardspecialist Dec 11, 2024
62116b5
wrap await
keyboardspecialist Dec 11, 2024
f0237a7
back up bad change
keyboardspecialist Dec 11, 2024
c858f03
Merge branch 'main' into splatting-prettier-fixup
keyboardspecialist Jan 6, 2025
7142df3
remove unneeded deps
keyboardspecialist Jan 7, 2025
eb1db94
Added a sandcastle example for 3D Tiles Gaussian Splatting
weegeekps Jan 8, 2025
811338a
Merge branch 'main' into splatting-prettier-fixup
weegeekps Jan 10, 2025
ce0976f
Renamed the splatting wasm module
weegeekps Jan 10, 2025
635659e
Committing the wasm package temporarily
weegeekps Jan 10, 2025
edca8a8
Add orientedBoundingBox to gsplat commands
keyboardspecialist Jan 15, 2025
a327c88
typo on final sort calc
keyboardspecialist Jan 16, 2025
b0bde10
Merge branch 'main' into splat-shader
ggetz Jan 16, 2025
ef2ec34
add weighted distance from center to sort
keyboardspecialist Jan 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .markdownlintignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/node_modules
/ThirdParty
/Tools/**
temp_wasm/*

LICENSE.md
108 changes: 108 additions & 0 deletions Apps/Sandcastle/gallery/3D Tiles Gaussian Splatting.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta
name="viewport"
content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no"
/>
<meta
name="description"
content="Use Viewer to start building new applications or easily embed Cesium into existing applications."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This description should be updated to a summary of the example. Any key words here will also be helpful, as they will help this example show up when a user searches.

Something along the lines of "Load a gaussian splat dataset of a cell tower as a 3D tileset." would be great.

/>
<meta name="cesium-sandcastle-labels" content="Showcases, 3D Tiles" />
<title>Cesium Demo</title>
<script type="text/javascript" src="../Sandcastle-header.js"></script>
<script type="module" src="../load-cesium-es6.js"></script>
</head>
<body class="sandcastle-loading" data-sandcastle-bucket="bucket-requirejs.html">
<style>
@import url(../templates/bucket.css);
</style>
<div id="cesiumContainer" class="fullSize"></div>
<div id="loadingOverlay"><h1>Loading...</h1></div>
<div id="toolbar"></div>
<script id="cesium_sandcastle_script">
window.startup = async function (Cesium) {
"use strict";
//Sandcastle_Begin
Cesium.Ion.defaultAccessToken =
"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI2MWEyNWI4NS02ZWVmLTQyZTEtOTRjZi02NjljOWNhOWViNGEiLCJpZCI6MTQwNTg1LCJpYXQiOjE3MzYzNjg4OTR9.rIAEC387JLBYOPt3w1fliKTWZgTGCQYJlG8o5HJsTwY";
const viewer = new Cesium.Viewer("cesiumContainer");
viewer.scene.debugShowFramesPerSecond = true;

const ORBIT_RANGE_FACTOR = 3;
const ORBIT_PITCH_DIVIDEND = 8;

let cellTowerTileset;
Cesium.Cesium3DTileset.fromIonAssetId(2917325, {
maximumScreenSpaceError: 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is maximumScreenSpaceError: 1 needed? Does that indicate a need for us to change something in the source code to minimize any configuration for the end user?

}).then((tileset) => {
cellTowerTileset = tileset;
viewer.scene.primitives.add(cellTowerTileset);
setupCamera();
});
Comment on lines +39 to +45
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going forward, we prefer async/await syntax over promise chains:

Suggested change
Cesium.Cesium3DTileset.fromIonAssetId(2917325, {
maximumScreenSpaceError: 1,
}).then((tileset) => {
cellTowerTileset = tileset;
viewer.scene.primitives.add(cellTowerTileset);
setupCamera();
});
const cellTowerTileset = await Cesium.Cesium3DTileset.fromIonAssetId(2917325, {
maximumScreenSpaceError: 1,
});
viewer.scene.primitives.add(cellTowerTileset);
setupCamera();

To make eslint happy, you may also need to move the call to setupCamera to after it is defined.

let startTime;
let isRunning = false;

function setupCamera() {
const boundingSphere = cellTowerTileset.boundingSphere;
const heading = 0;
const range = boundingSphere.radius * ORBIT_RANGE_FACTOR;
const pitch = -Math.PI / ORBIT_PITCH_DIVIDEND;

viewer.camera.lookAt(
boundingSphere.center,
new Cesium.HeadingPitchRange(heading, pitch, range),
);

return true;
}

function startCameraOrbit() {
if (!setupCamera()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When would this return false?

return;
}
startTime = performance.now();
isRunning = true;
requestAnimationFrame(cameraOrbitTickCallback);
}

function cameraOrbitTickCallback(timestamp) {
if (!isRunning) {
return;
}

const boundingSphere = cellTowerTileset.boundingSphere;
const elapsedSeconds = (timestamp - startTime) / 1000;
const heading = (elapsedSeconds / 15) * Math.PI * 2;

const range = boundingSphere.radius * ORBIT_RANGE_FACTOR;
const pitch = -Math.PI / ORBIT_PITCH_DIVIDEND;

viewer.camera.lookAt(
boundingSphere.center,
new Cesium.HeadingPitchRange(heading, pitch, range),
);

requestAnimationFrame(cameraOrbitTickCallback);
}

Sandcastle.addToolbarButton("Start Orbit", startCameraOrbit);
Sandcastle.addToolbarButton("Stop Orbit", () => {
isRunning = false;
});
//Sandcastle_End
Sandcastle.finishedLoading();
};
if (typeof Cesium !== "undefined") {
window.startupCalled = true;
window.startup(Cesium).catch((error) => {
"use strict";
console.error(error);
});
}
</script>
</body>
</html>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute to Cesiu
- [Adam Morris](https://github.com/weegeekps)
- [Luke McKinstry](https://github.com/lukemckinstry)
- [Ryan Veenstra](https://github.com/r-veenstra)
- [Jason Sobotka](https://github.com/keyboardspecialist)
- [Northrop Grumman](http://www.northropgrumman.com)
- [Joseph Stein](https://github.com/nahgrin)
- [EOX IT Services GmbH](https://eox.at)
Expand Down
1 change: 1 addition & 0 deletions eslint.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ export default [
"packages/engine/Source/Shaders/**/*",
"Specs/jasmine/*",
"**/*/SpecList.js",
"temp_wasm/*",
],
},
{
Expand Down
5 changes: 5 additions & 0 deletions gulpfile.js
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,11 @@ export async function prepare() {
"Tools/jsdoc/cesium_template/static/styles/prism.css",
);

copyFileSync(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is nit-picky, but can we move this line up with the other source code dependencies up around line 351? It will help it not get lost among the other dev processes like building documentation.

"node_modules/@cesium/wasm-splats/wasm_splats_bg.wasm",
"packages/engine/Source/ThirdParty/wasm_splats_bg.wasm",
);

// Copy jasmine runner files into Specs
const files = await globby([
"node_modules/jasmine-core/lib/jasmine-core",
Expand Down
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@
],
"dependencies": {
"@cesium/engine": "^13.1.0",
"@cesium/widgets": "^10.1.0"
"@cesium/widgets": "^10.1.0",
"@cesium/wasm-splats": "file:./temp_wasm/pkg"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root dependencies here should only include the packages in this repo.

The @cesium/wasm-splats dependency should be moved to @cesium/engine/package.json, since that's the code which depends on the wasm module.

},
"devDependencies": {
"@playwright/test": "^1.41.1",
Expand Down Expand Up @@ -159,4 +160,4 @@
"packages/engine",
"packages/widgets"
]
}
}
43 changes: 43 additions & 0 deletions packages/engine/Source/Core/PixelFormat.js
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,15 @@ const PixelFormat = {
*/
RGBA: WebGLConstants.RGBA,

/**
* Integral versions of above types. Required if using integral textures.
*/

RED_INTEGER: WebGLConstants.RED_INTEGER,
RG_INTEGER: WebGLConstants.RG_INTEGER,
RGB_INTEGER: WebGLConstants.RGB_INTEGER,
RGBA_INTEGER: WebGLConstants.RGBA_INTEGER,

/**
* A pixel format containing a luminance (intensity) channel.
*
Expand Down Expand Up @@ -190,14 +199,18 @@ const PixelFormat = {
PixelFormat.componentsLength = function (pixelFormat) {
switch (pixelFormat) {
case PixelFormat.RGB:
case PixelFormat.RGB_INTEGER:
return 3;
case PixelFormat.RGBA:
case PixelFormat.RGBA_INTEGER:
return 4;
case PixelFormat.LUMINANCE_ALPHA:
case PixelFormat.RG:
case PixelFormat.RG_INTEGER:
return 2;
case PixelFormat.ALPHA:
case PixelFormat.RED:
case PixelFormat.RED_INTEGER:
case PixelFormat.LUMINANCE:
return 1;
default:
Expand All @@ -217,6 +230,10 @@ PixelFormat.validate = function (pixelFormat) {
pixelFormat === PixelFormat.RG ||
pixelFormat === PixelFormat.RGB ||
pixelFormat === PixelFormat.RGBA ||
pixelFormat === PixelFormat.RED_INTEGER ||
pixelFormat === PixelFormat.RG_INTEGER ||
pixelFormat === PixelFormat.RGB_INTEGER ||
pixelFormat === PixelFormat.RGBA_INTEGER ||
pixelFormat === PixelFormat.LUMINANCE ||
pixelFormat === PixelFormat.LUMINANCE_ALPHA ||
pixelFormat === PixelFormat.RGB_DXT1 ||
Expand Down Expand Up @@ -499,6 +516,32 @@ PixelFormat.toInternalFormat = function (pixelFormat, pixelDatatype, context) {
}
}

if (pixelDatatype === PixelDatatype.INT) {
switch (pixelFormat) {
case PixelFormat.RGBA_INTEGER:
return WebGLConstants.RGBA32I;
case PixelFormat.RGB_INTEGER:
return WebGLConstants.RGB32I;
case PixelFormat.RG_INTEGER:
return WebGLConstants.RG32I;
case PixelFormat.RED_INTEGER:
return WebGLConstants.R32I;
}
}

if (pixelDatatype === PixelDatatype.UNSIGNED_INT) {
switch (pixelFormat) {
case PixelFormat.RGBA_INTEGER:
return WebGLConstants.RGBA32UI;
case PixelFormat.RGB_INTEGER:
return WebGLConstants.RGB32UI;
case PixelFormat.RG_INTEGER:
return WebGLConstants.RG32UI;
case PixelFormat.RED_INTEGER:
return WebGLConstants.R32UI;
}
}

return pixelFormat;
};

Expand Down
4 changes: 3 additions & 1 deletion packages/engine/Source/Renderer/Context.js
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ function Context(canvas, options) {
} = defaultValue(options, {});

// Override select WebGL defaults
webglOptions.alpha = defaultValue(webglOptions.alpha, false); // WebGL default is true
webglOptions.alpha = defaultValue(webglOptions.alpha, true); // WebGL default is true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain a bit why this was changed?

It look like a fundamental change which could cause unintended effects on the rest of the codebase.

webglOptions.stencil = defaultValue(webglOptions.stencil, true); // WebGL default is false
webglOptions.powerPreference = defaultValue(
webglOptions.powerPreference,
Expand Down Expand Up @@ -1308,6 +1308,7 @@ function beginDraw(
bindFramebuffer(context, framebuffer);
applyRenderState(context, renderState, passState, false);
shaderProgram._bind();

context._maxFrameTextureUnitIndex = Math.max(
context._maxFrameTextureUnitIndex,
shaderProgram.maximumTextureUnitIndex,
Expand Down Expand Up @@ -1382,6 +1383,7 @@ function continueDraw(context, drawCommand, shaderProgram, uniformMap) {
} else {
count = va.numberOfVertices;
}

if (instanceCount === 0) {
context._gl.drawArrays(primitiveType, offset, count);
} else {
Expand Down
5 changes: 3 additions & 2 deletions packages/engine/Source/Renderer/Pass.js
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ const Pass = {
OPAQUE: 7,
TRANSLUCENT: 8,
VOXELS: 9,
OVERLAY: 10,
NUMBER_OF_PASSES: 11,
GAUSSIAN_SPLATS: 10,
OVERLAY: 11,
NUMBER_OF_PASSES: 12,
};
export default Object.freeze(Pass);
14 changes: 14 additions & 0 deletions packages/engine/Source/Renderer/Texture.js
Original file line number Diff line number Diff line change
Expand Up @@ -340,9 +340,17 @@ function loadBufferSource(texture, source) {
pixelDatatype,
width,
);

let glErr;
gl.pixelStorei(gl.UNPACK_ALIGNMENT, unpackAlignment);
gl.pixelStorei(gl.UNPACK_PREMULTIPLY_ALPHA_WEBGL, false);
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, false);
glErr = gl.getError();
if (glErr !== 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and below: Opt to throw a RuntimeError over logging to the console. This will allow downstream code to handle errors accordingly.

console.log(
`pixelStorei err ${glErr}, format ${pixelFormat} internalFormat ${internalFormat}`,
);
}

let { arrayBufferView } = source;
if (flipY) {
Expand All @@ -365,6 +373,12 @@ function loadBufferSource(texture, source) {
PixelDatatype.toWebGLConstant(pixelDatatype, context),
arrayBufferView,
);
glErr = gl.getError();
if (glErr !== 0) {
console.log(
`texImage2D err ${glErr}, format ${pixelFormat} internalFormat ${internalFormat}`,
);
}

if (defined(source.mipLevels)) {
let mipWidth = width;
Expand Down
50 changes: 42 additions & 8 deletions packages/engine/Source/Renderer/VertexArray.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import RuntimeError from "../Core/RuntimeError.js";
import Buffer from "./Buffer.js";
import BufferUsage from "./BufferUsage.js";
import ContextLimits from "./ContextLimits.js";
import AttributeType from "../Scene/AttributeType.js";

function addAttribute(attributes, attribute, index, context) {
const hasVertexBuffer = defined(attribute.vertexBuffer);
Expand Down Expand Up @@ -648,6 +649,7 @@ VertexArray.fromGeometry = function (options) {
componentDatatype = ComponentDatatype.FLOAT;
}

let attrProps = {};
vertexBuffer = undefined;
if (defined(attribute.values)) {
vertexBuffer = Buffer.createVertexBuffer({
Expand All @@ -658,16 +660,48 @@ VertexArray.fromGeometry = function (options) {
),
usage: bufferUsage,
});

attrProps = {
index: attributeLocations[name],
vertexBuffer: vertexBuffer,
value: attribute.value,
componentDatatype: componentDatatype,
componentsPerAttribute: attribute.componentsPerAttribute,
normalize: attribute.normalize,
};
}

//if we already have a typedArray lets use it
if (defined(attribute.typedArray)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change does seem to make sense, but we should confirm this doesn't affect any downstream processes with glTF loading and the like. I'm not sure off the top of my head if there are special cases where we tweak the contents of the typed array based on certain conditions.

vertexBuffer = Buffer.createVertexBuffer({
context: context,
typedArray: attribute.typedArray,
usage: bufferUsage,
});

attrProps = {
index: attributeLocations[name],
vertexBuffer: vertexBuffer,
value: undefined,
componentDatatype: componentDatatype,
componentsPerAttribute: AttributeType.getNumberOfComponents(
attribute.type,
),
normalize: attribute.normalized,
instanceDivisor: attribute.instanceDivisor,
};
}

vaAttributes.push({
index: attributeLocations[name],
vertexBuffer: vertexBuffer,
value: attribute.value,
componentDatatype: componentDatatype,
componentsPerAttribute: attribute.componentsPerAttribute,
normalize: attribute.normalize,
});
vaAttributes.push(attrProps);

// vaAttributes.push({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this if it's no longer needed.

// index: attributeLocations[name],
// vertexBuffer: vertexBuffer,
// value: attribute.value,
// componentDatatype: componentDatatype,
// componentsPerAttribute: attribute.componentsPerAttribute,
// normalize: attribute.normalize,
// });
}
}
}
Expand Down
2 changes: 2 additions & 0 deletions packages/engine/Source/Renderer/createUniform.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ function createUniform(gl, activeUniform, uniformName, location) {
case gl.SAMPLER_2D:
case gl.SAMPLER_CUBE:
return new UniformSampler(gl, activeUniform, uniformName, location);
case gl.UNSIGNED_INT_SAMPLER_2D:
return new UniformSampler(gl, activeUniform, uniformName, location);
case gl.INT:
case gl.BOOL:
return new UniformInt(gl, activeUniform, uniformName, location);
Expand Down
Loading
Loading