From 59015ba194f8026168dcfd7c9ca6478c43e563dd Mon Sep 17 00:00:00 2001 From: Matt Kuruc Date: Fri, 9 Feb 2024 15:13:01 -0500 Subject: [PATCH 1/2] Add initial draft of visualization schema proposal --- proposals/vprim/README.md | 121 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 proposals/vprim/README.md diff --git a/proposals/vprim/README.md b/proposals/vprim/README.md new file mode 100644 index 0000000..93bfe16 --- /dev/null +++ b/proposals/vprim/README.md @@ -0,0 +1,121 @@ +# Visualizing the Unseeable + +## Overview +OpenUSD models both fantastical and real world scenes with physically modeled geometry and materials. There's an increasing need to render nondiegetic visualizations that do not participate in physically based rendering and simulation. These may be user defined labels or curves describing a path through a scene. They may be curves representing vector fields (or even captured light paths!). The traditional method of tagging `Gprim`s with `guide` purpose has not been sufficient in part because users are looking to visualize in non-physical ways (like screen space widths). These nondiegetic visualizations require different descriptions than those the current `UsdGeom` domain currently specifies. + +### Nondiegetic visualizations and `guide`s +We use "nondiegetic" as a handle for describing objects intended to be a part of any primary visualizations of the scene but aren't physically present. If `Gprims` are diegetic like dialogue on audio track, we're concerned with how to represent the visualization equivalent of the orchestral score. Nondigetic effects cannot be removed without fundamentally changing the viewing experience but hold a different set of constraints and idealized representations than their diegetic cousins. + +We view nondiegetic visualizations as orthogonal to the `guide` `purpose` attribute. In our audio track analogy, `guide` serves the role of a DVD commentary, an overlay adding additional context and visual information that aren't a part of the primary viewing experience and may be optionally experienced. + +## Background +OpenUSD was developed as path tracing and physically based rendering were maturing in the visual effects production pipeline. As such, scenes are modeled as physical 3-D geometry. `Points` are spheres and `BasisCurves` are tubes (or user-oriented ribbons) that can be ray traced from any direction. They were decidedly not the screen oriented disks and ribbons previously used in rasterizers. + +Users are currently seeking a way to visualize data through schemas added to OpenUSD. A user may want describe a "red line" to draw attention to a particular object or a "green curve" to represent available paths towards exits during building design. + +### Schema Slicing +One instance of these new schemas is `CurveStyleAPI`, proposed as an API schema for the existing `Curves` schemas. The API schema makes three transformative changes. It transforms the meaning of the `widths` attribute to be a screen space size (`constant` or `uniform` only), changes the appearence of joints and end caps, and resticts the materials allowed to be bound to a restricted set of "styling" materials. + +We believe this creates a situation we're calling "schema slicing" (to adapt the term "object slicing" from C++). In that language, object slicing occurs when the properties and behavior of a derived type are "sliced" off when interfacing with an object via a higher level type. Using `CurveStyleAPI` to transform the curves into screen space defined primitives similarly creates the potential for slicing. A user interacting with the prim using the `BasisCurves` schema and not the full prim definition will have effectively "sliced" off the reinterpretation of widths and other modifications to the underlying definition. Utlimately, "slicing" complicates interchange and standardization. + +### Disaggregating `hermite` out of `BasisCurves` +As a historical analogue, the `hermite` basis was previously a part of `BasisCurves` but was left unimplemented in Storm and many renderers. We learned the primary usage was to describe paths and not as a renderable representation, ultimately seperating it out into its own schema. This change made `BasisCurves` simpler for renderers to fully support and let `HermiteCurves` stand on its own with a decoupled `points` and `tangents` representation and without the need to formulate support for the different `wrap` modes of `BasisCurves`. + +We similarly wonder if all variations of the current set of `Curves` are needed to describe screen space visualizations and when adding support, how renderers and other consumers know which to prioritize to establish compatability. A new axis of variation that transform the existing geometric schemas complicates interchange and compatability of the `UsdGeom` domain. We suggest new schema domain with a new set of schemas may be the best place to drive development of nondiegetic primitives separate from their diegetic `UsdGeom` cousins. + +## Proposal +Introduce a class of visualization primitives that are defined in a 3-D scene, but whose imaging definition is allowed to include nondiegetic screen effects and with a relaxed definition of boundability. Let's call them `Vprims` (visualization primitives). Boundability as a schema partioning mechanism is already employed in `UsdLux` which classifies light schemas as either boundable (ie. `RectLight` and `SphereLight`) or non-boundable (`DomeLight`). + +Let's use points as a motivating example. Consider an OpenUSD file that contains a mesh reconstruction of data from a point cloud scan. A user might benefit from visualizing the points on top of the mesh to aid in debugging some errors in the reconstruction. The user might choose the `Points` schema to encode positions. And they might be able to hide the source point cloud from rendering contexts using `purpose` tagging and/or variant sets. But they will be unable to make the points all the same size in camera space in a specification compliant renderer. Preview optimizations that might give a user their desired effect like those found in Storm exist primarily for performance reasons and may not persist across renderers, sessions, and future releases. + +A visualization focused specification of points could look something like this-- + +``` +def VizPoints "PointCloud" { + point3d[] points = [...] + float[] pixels = [...] + color3f[] primvars:displayColor = [(1, 0, 0)] +} +``` + +`VizPoints` could be rendered in camera but may also be composited against a depth buffer from a physically based renderer. + +### Styling +`Vprims` still support primvars and would be primarily styled using `displayColor` and `displayOpacity`. + +We'd like to leave open the possibility of a `viz` material purpose as a holder for describing nondiegetic styles best described through material network, but this proposal doesn't have a concrete definition of this yet. + +### API Schema and Grouping Adapters +This proposal frames screen space points (and paths) visualization primitives as fundamentally different from their `Gprim` counterparts, warranting their own first class `Vprim` typed schemas. Does this proposal set the precedent that other `Gprim` types like `Cube` or `Cylinder` require visualization equivalents if they wanted to be adapted into the "visualization" domain? We'd suggest no and offer two solutions that have prior art in OpenUSD. + +If a primitive retains its underlying definition, `UsdLux` already provides a solution to the problem of adapting `Gprim`s into another domain. Rather than a `MeshLight`, the `Mesh` `Gprim` is adapted via an API schema. `Gprim`s could be similarly "adapted" into the visualization domain. To avoid "schema slicing" an adaptation must not change the underlying definition or require their own distinct `purpose` or `visibility` tagging. Consideration must be also given to if or how these schemas affect interactive picking. A non-slicing visualization effect might include sillhoutte outlines, glows, or overlays applied to a `Gprim` to style or highlight its importance. + +For `Gprim`s that need to be aligned, anchored, or otherwise nondiegtically constrained for visualization, we suggest that a custom `Imageable` be introduced-- (`VizAnchor` or `VizGroup`). Just as `UsdGeomPointInstancer` is allowed to introduce a break in a transformation hierarchy to introduce vectorized instancing, these grouping adapters would be responsible for introducing nondiegetic transformations of the existing `Gprim` domain (including screen relative placement). We'd propose a Macbeth color chart as an example of geometry that users might want nondiegetically constrained to a screen position in an asset's OpenUSD. +``` +( + defaultPrim = "Asset" +) + +def Xform "Asset" { + def Scope "Geometry" { + def Sphere "sphere" {} + } +} + +def VizScope "Visualization" { + # Collection of properties that anchor descendants to the lower right + # corner of screen spaces + def Plane "MacbethChart" {} +} +``` +### Boundability +Because `Vprim`s are allowed to be screen dependent, they may have a relaxed notion of boundability and may complicate extent computation of `Boundable`s like `PointInstancer`s and `SkelRoot`s (and `PointInstancer`s of `SkelRoot`s). It may desirable for users to optionally specify a "reference" or "approximate" extent. + +### Purpose +The traditional purpose rules still apply. Visualization primitives may but are not required to be tagged as `guide`s. There may be complicated visualizations that warrant `render` and `proxy` tagging as well. + +### Sample Set of Visualization Primitives +A proposed initial set of visualization primitives is listed below. Additional iterations of this document should ensure that the the visualization primitive concept can accomodate existing proposals. Further evaluation should be done to see if existing `Gprim`s might be more appropriate as `Vprim`s (like `HermiteCurves` which is explicitly called out as "not for final rendering"). +* `VizPoints`: Points with 3-D positions but screen space widths +* `VizVolumeSamples`: List of 3-D points at which to sample fields in a `UsdVol`. The field type can drive the output style (`vector` renders as arrows, `density` or `color` render as screen space points) or could be otherwise configured. +* `VizPolylines`: Align with existing curve styling proposals +* `VizPaths`: Cubically interpolated curves that align with existing curve styling proposals +* `VizText`: Align with existing text schema proposals + +### Note on Schema Slicing +It's worth noting that this proposal does not believe that all applications or usages of API schemas are inherently "slicing". Most API schemas are used to augment, annotate, or adapt prims for usage by other domains, but generally leave the original prim definition untouched. Using API schemas to describe "optional" features of a concrete typed schema (that might otherwise be described with an enabling `bool` flag) would generally not be considered slicing as well. + +## Alternative Approaches +### Annotation Specific Schemas +Some suggest that the needs would be better served with discrete purposed annotation primitives (ie. `AnnotationLabel`, `AnnotationPaths`, `AnnotationPoints`). The door should be cleanly shut to using visualzation primitives for imposter billboard cards or generalized non-photorealistic rendering. This choice was not the preferred approach because there is a need for visualization primitives outside of the annotation space, and that annotations specific properties could extend the set of visualization primitives through additional schemas. + +### Introduce New Typed Schemas Without the `Vprim` Categorization +Readers may agree that some of these nondiegetic primitives are distinct enough to warrant new schemas, but may reject the premise that screen independence is necessarily a fundemental property of `Gprim`s. The USD Glossary says `Gprim`s are just drawable, transformable, and boundable. The problems of how to bound, transform (and otherwise collide / intersect) screen space objects should be clarified and reconciled within the existing `Gprim` definition, rather than introduce a new type. This is not the preferred approach of this proposal which sees value having a high level classification for diegetic and nondiegetic primitives that clients can use to organize scenes and imaging pipelines around. + +### Disaggregate Boundability out of `Gprim` +The proposal suggests that `Vprim`s required a relaxed definition of boundability. One could imagine that just as `visibility` was disaggregated out of `Gprim` into its own schema, boundability could disaggregated as well so that not every `Gprim` requires a traditional `extent`. + +We think there might be other core properties (`doublesided`) and behaviors that warrant removal from the "visualization" domain so this isn't our preferred approach. + +## Summary +To preserve compatability, supportability, and interchange of existing `UsdGeom` schemas while extending USD's ability to describe nondiegetic visualizations, a new domain should be introduced. The `Vprim` domain would allow visualizations to be described in relation to both screen and other primitives and would avoid introducing schemas that might "slice" the current set of `Gprim`s. + +## Questions +* As formulated `Vprims` cannot be `PointBased` which are strictly `Gprims`. Are there any cases where dispatching behavior off of the more general `PointBased` without knowing the concrete type is important? +* When defining screen space sizes, what is the best choice of units? Is it pixels or something else? +* How should scaling be treated for screen space objects? + +## Future Work + +### Hiding +As currently described, visualization primitives should be strictly hidden via depth. However, one could imagine customizing hiding for these prims. Some visualizations might want to hide based on a particular reference point for the entire string of text. Some visualizations might want to express priority with respect to other objects in the scene. Hiding is another axis where the specification may eventually end up diverging from the `Gprim` space. + +### Breaking the Fourth Wall +Some care needs to be taken to establish best practices and rules for visibility of screen-constrained primitives. While we generally would expect nondiegetic primitives to not participate in physically based lighting, consider screen constrained "context spheres" as reference objects. They "break the fourth wall" and capture light and reflections. It should be possible to specify this behavior even when using nondiegetic transformations. + +### Styling Schemas +Universal material model description has required a plugin system build around `Sdr` to allow for disparate renderer and site specific materials models to be integrated into OpenUSD. Even the simplest material models in OpenUSD (ie. `UsdPreviewSurface`) requires the notion of a network and connections. + +Visualization style, on the other hand, is a more compact problem. Users often want to specify a single color, image, or pattern to style an object. There's potentially some inspiration to be drawn from 2-D tooling where often the same user interface is used to style text, lines, and other shapes. + +We think the simplicity afforded by configuring stylization with simple declarative schemas that don't require material networks is worth pursuing in the future, but is beyond the scope of this proposal. From 3f419c9bf5da2b9048d369704d515fe5764f2679 Mon Sep 17 00:00:00 2001 From: Matt Kuruc Date: Fri, 5 Apr 2024 14:49:30 -0400 Subject: [PATCH 2/2] Revise proposal to emphasize applied schemas and anchoring transforms --- proposals/vprim/README.md | 305 ++++++++++++++++++++++++++++++-------- 1 file changed, 242 insertions(+), 63 deletions(-) diff --git a/proposals/vprim/README.md b/proposals/vprim/README.md index 93bfe16..d1d11fb 100644 --- a/proposals/vprim/README.md +++ b/proposals/vprim/README.md @@ -1,34 +1,84 @@ # Visualizing the Unseeable - +Copyright © 2024, NVIDIA Corporation, version 1.1 ## Overview -OpenUSD models both fantastical and real world scenes with physically modeled geometry and materials. There's an increasing need to render nondiegetic visualizations that do not participate in physically based rendering and simulation. These may be user defined labels or curves describing a path through a scene. They may be curves representing vector fields (or even captured light paths!). The traditional method of tagging `Gprim`s with `guide` purpose has not been sufficient in part because users are looking to visualize in non-physical ways (like screen space widths). These nondiegetic visualizations require different descriptions than those the current `UsdGeom` domain currently specifies. +OpenUSD models both fantastical and real world scenes with physically +modeled geometry and materials. There's an increasing need to render +nondiegetic visualizations that do not participate in physically based +rendering and simulation. These may be screen anchored labels or paths +through a scene. They may be vector fields (or even captured light +paths!). The traditional method of tagging `Gprim`s with `guide` +purpose has not been sufficient in part because users are looking to +style these visualizations with nondiegetic effects (screen dependent +positioning and sizing, outlines and overlays, etc.). These +nondiegetic visualizations often require different descriptions than +those the current `UsdGeom` domain currently specifies. + +Nondiegetic visualizations fall into three categories. +* Nondiegetic drawables used to describe screen and other +extrinsically constrainted effects (ie. markup curves or labels) +* Nondiegetic transformations that may introduce screen and other +extrinsic contraints on top of existing diegetic geometry +* Applied nondiegetic styling schemas that can be add screen and +extrinsically constrained effects to existing `Gprim`s (ie. like a +sillhoutte outline or overlays). + +This proposal focuses mostly on examples of nondiegetic drawables. But +this does not aim to suggest that every diegetic `Gprim` requires a +nondiegetic reflection. Nondiegetic transformations and styles should +be appliable to the existing set of `Gprim`s as well. ### Nondiegetic visualizations and `guide`s -We use "nondiegetic" as a handle for describing objects intended to be a part of any primary visualizations of the scene but aren't physically present. If `Gprims` are diegetic like dialogue on audio track, we're concerned with how to represent the visualization equivalent of the orchestral score. Nondigetic effects cannot be removed without fundamentally changing the viewing experience but hold a different set of constraints and idealized representations than their diegetic cousins. - -We view nondiegetic visualizations as orthogonal to the `guide` `purpose` attribute. In our audio track analogy, `guide` serves the role of a DVD commentary, an overlay adding additional context and visual information that aren't a part of the primary viewing experience and may be optionally experienced. - -## Background -OpenUSD was developed as path tracing and physically based rendering were maturing in the visual effects production pipeline. As such, scenes are modeled as physical 3-D geometry. `Points` are spheres and `BasisCurves` are tubes (or user-oriented ribbons) that can be ray traced from any direction. They were decidedly not the screen oriented disks and ribbons previously used in rasterizers. - -Users are currently seeking a way to visualize data through schemas added to OpenUSD. A user may want describe a "red line" to draw attention to a particular object or a "green curve" to represent available paths towards exits during building design. - -### Schema Slicing -One instance of these new schemas is `CurveStyleAPI`, proposed as an API schema for the existing `Curves` schemas. The API schema makes three transformative changes. It transforms the meaning of the `widths` attribute to be a screen space size (`constant` or `uniform` only), changes the appearence of joints and end caps, and resticts the materials allowed to be bound to a restricted set of "styling" materials. - -We believe this creates a situation we're calling "schema slicing" (to adapt the term "object slicing" from C++). In that language, object slicing occurs when the properties and behavior of a derived type are "sliced" off when interfacing with an object via a higher level type. Using `CurveStyleAPI` to transform the curves into screen space defined primitives similarly creates the potential for slicing. A user interacting with the prim using the `BasisCurves` schema and not the full prim definition will have effectively "sliced" off the reinterpretation of widths and other modifications to the underlying definition. Utlimately, "slicing" complicates interchange and standardization. - -### Disaggregating `hermite` out of `BasisCurves` -As a historical analogue, the `hermite` basis was previously a part of `BasisCurves` but was left unimplemented in Storm and many renderers. We learned the primary usage was to describe paths and not as a renderable representation, ultimately seperating it out into its own schema. This change made `BasisCurves` simpler for renderers to fully support and let `HermiteCurves` stand on its own with a decoupled `points` and `tangents` representation and without the need to formulate support for the different `wrap` modes of `BasisCurves`. - -We similarly wonder if all variations of the current set of `Curves` are needed to describe screen space visualizations and when adding support, how renderers and other consumers know which to prioritize to establish compatability. A new axis of variation that transform the existing geometric schemas complicates interchange and compatability of the `UsdGeom` domain. We suggest new schema domain with a new set of schemas may be the best place to drive development of nondiegetic primitives separate from their diegetic `UsdGeom` cousins. +We use "nondiegetic" as a handle for describing objects intended to be +a part of any primary visualizations of the scene but aren't +physically present. If `Gprims` are diegetic like dialogue on audio +track, we're concerned with how to represent the visualization +equivalent of the orchestral score. Nondiegetic effects cannot be +removed without fundamentally changing the viewing experience but hold +a different set of constraints and idealized representations than +their diegetic cousins. + +We view nondiegetic visualizations as orthogonal to the `guide` +`purpose` attribute. In our audio track analogy, `guide` serves the +role of a DVD commentary, an overlay adding additional context and +visual information that aren't a part of the primary viewing +experience and may be optionally experienced. ## Proposal -Introduce a class of visualization primitives that are defined in a 3-D scene, but whose imaging definition is allowed to include nondiegetic screen effects and with a relaxed definition of boundability. Let's call them `Vprims` (visualization primitives). Boundability as a schema partioning mechanism is already employed in `UsdLux` which classifies light schemas as either boundable (ie. `RectLight` and `SphereLight`) or non-boundable (`DomeLight`). - -Let's use points as a motivating example. Consider an OpenUSD file that contains a mesh reconstruction of data from a point cloud scan. A user might benefit from visualizing the points on top of the mesh to aid in debugging some errors in the reconstruction. The user might choose the `Points` schema to encode positions. And they might be able to hide the source point cloud from rendering contexts using `purpose` tagging and/or variant sets. But they will be unable to make the points all the same size in camera space in a specification compliant renderer. Preview optimizations that might give a user their desired effect like those found in Storm exist primarily for performance reasons and may not persist across renderers, sessions, and future releases. - -A visualization focused specification of points could look something like this-- +To address the need for standardized and interopable nondiegetic +drawables, transformations, and styling, introduce a new domain +`UsdViz` for nondiegetic visualizations primitives. + +This domain would have three primary bases: +* `Vprims`: a set of typed nondiegetic drawables that remove some of +the constriants of `Gprim` (ie. intrinsic boundability) +* `VizAnchor`: a typed nondiegetic transformations that removes some +of the contraints of `Xform` (ie. intrinsic transformability) +* `VizStyleAPI`: a set of applied API schemas to add nondiegetic +effects to existing `Gprim`s. + +The idea of `Vprims` and `VizStyleAPI` was partially inspired by work +done in `UsdLux`. Boundability as a schema partioning mechanism was +introduced to classify light schemas as either boundable (ie. +`RectLight` and `SphereLight`) or non-boundable (`DomeLight`). Applied +schema adapters were introduced to make any `Gprim` a light without +having to redefine in `UsdLux`. + +### Example: `VizPoints` +Let's use points as a motivating example of a `Vprim`. Consider an +OpenUSD file that contains a mesh reconstruction of data from a point +cloud scan. A user might benefit from visualizing the points on top of +the mesh to aid in debugging some errors in the reconstruction. The +user might choose the `Points` schema to encode positions. And they +might be able to hide the source point cloud from rendering contexts +using `purpose` tagging and/or variant sets. But they will be unable +to make the points all the same size in camera space in a +specification compliant renderer. Preview optimizations that might +give a user their desired effect like those found in Storm exist +primarily for performance reasons and may not persist across +renderers, sessions, and future releases. + +A visualization focused specification of points could look something +like this-- ``` def VizPoints "PointCloud" { @@ -38,19 +88,55 @@ def VizPoints "PointCloud" { } ``` -`VizPoints` could be rendered in camera but may also be composited against a depth buffer from a physically based renderer. - -### Styling -`Vprims` still support primvars and would be primarily styled using `displayColor` and `displayOpacity`. +`VizPoints` could be rendered in camera but may also be composited +against a depth buffer from a physically based renderer. -We'd like to leave open the possibility of a `viz` material purpose as a holder for describing nondiegetic styles best described through material network, but this proposal doesn't have a concrete definition of this yet. +`Vprims` would still support primvars and would be primarily styled +using `displayColor` and `displayOpacity`. We'd like to leave open the +possibility of a `viz` material purpose as a holder for describing +nondiegetic styles best described through material network, but this +proposal doesn't have a concrete definition of this yet. -### API Schema and Grouping Adapters -This proposal frames screen space points (and paths) visualization primitives as fundamentally different from their `Gprim` counterparts, warranting their own first class `Vprim` typed schemas. Does this proposal set the precedent that other `Gprim` types like `Cube` or `Cylinder` require visualization equivalents if they wanted to be adapted into the "visualization" domain? We'd suggest no and offer two solutions that have prior art in OpenUSD. +### Example: `VizOverlayAPI` +This proposal frames screen space points (and paths) visualization +primitives as fundamentally different from their `Gprim` counterparts, +warranting their own first class `Vprim` typed schemas. It's important +that other `Gprim` types like `Cube` or `Cylinder` that retain their +underlying definition don't require nondiegetic reflections. -If a primitive retains its underlying definition, `UsdLux` already provides a solution to the problem of adapting `Gprim`s into another domain. Rather than a `MeshLight`, the `Mesh` `Gprim` is adapted via an API schema. `Gprim`s could be similarly "adapted" into the visualization domain. To avoid "schema slicing" an adaptation must not change the underlying definition or require their own distinct `purpose` or `visibility` tagging. Consideration must be also given to if or how these schemas affect interactive picking. A non-slicing visualization effect might include sillhoutte outlines, glows, or overlays applied to a `Gprim` to style or highlight its importance. +Consider an important primitive that you want to highlight in the +context of visualization. -For `Gprim`s that need to be aligned, anchored, or otherwise nondiegtically constrained for visualization, we suggest that a custom `Imageable` be introduced-- (`VizAnchor` or `VizGroup`). Just as `UsdGeomPointInstancer` is allowed to introduce a break in a transformation hierarchy to introduce vectorized instancing, these grouping adapters would be responsible for introducing nondiegetic transformations of the existing `Gprim` domain (including screen relative placement). We'd propose a Macbeth color chart as an example of geometry that users might want nondiegetically constrained to a screen position in an asset's OpenUSD. +``` +def Sphere "grey_sphere" (apiSchemas=["VizOverlayAPI"]) +{ + double radius = 4 + color3f primvars:displayColor (0.3, 0.3, 0.3) + color3f viz:overlay:color = (0.6, 0.0, 0.0) + float viz:overlay:opacity = 0.5 + token viz:overlay:visability = "inherited" +} +``` +This adds a nondiegetic overlay and outline for visualizations to a +grey sphere. The viewer will see the red transparently overlayed, but +the sphere is still a grey sphere from the perspective of light +transport (reflected and refracted light). + +Consideration must be also given to if or how these schemas affect +interactive picking. A non-slicing visualization effect might include +sillhoutte outlines or overlays applied to a `Gprim` to style or +highlight its importance. + +### Example: `VizScreenAnchor` +For `Gprim`s that need to be aligned, anchored, or otherwise +nondiegtically constrained for visualization, we suggest that a custom +`Imageable` be introduced. Just as `UsdGeomPointInstancer` is allowed +to introduce a break in a transformation hierarchy to introduce +vectorized instancing, these grouping adapters would be responsible +for introducing nondiegetic transformations of the existing `Gprim` +domain (including screen relative placement). We'd propose a Macbeth +color chart as an example of geometry that users might want +nondiegetically constrained to a screen position in an asset's OpenUSD. ``` ( defaultPrim = "Asset" @@ -62,60 +148,153 @@ def Xform "Asset" { } } -def VizScope "Visualization" { - # Collection of properties that anchor descendants to the lower right - # corner of screen spaces +def VizAnchor "Visualization" { + # Collection of properties that anchor descendants to + # the lower right corner of screen def Plane "MacbethChart" {} } ``` -### Boundability -Because `Vprim`s are allowed to be screen dependent, they may have a relaxed notion of boundability and may complicate extent computation of `Boundable`s like `PointInstancer`s and `SkelRoot`s (and `PointInstancer`s of `SkelRoot`s). It may desirable for users to optionally specify a "reference" or "approximate" extent. -### Purpose -The traditional purpose rules still apply. Visualization primitives may but are not required to be tagged as `guide`s. There may be complicated visualizations that warrant `render` and `proxy` tagging as well. +A nondiegetic transformation could be used to orient an object to + camera or provide a floating region for a label around an object. ### Sample Set of Visualization Primitives -A proposed initial set of visualization primitives is listed below. Additional iterations of this document should ensure that the the visualization primitive concept can accomodate existing proposals. Further evaluation should be done to see if existing `Gprim`s might be more appropriate as `Vprim`s (like `HermiteCurves` which is explicitly called out as "not for final rendering"). +A proposed initial set of visualization primitives is listed below. +Additional iterations of this document should ensure that the the +visualization primitive concept can accomodate existing proposals. +Further evaluation should be done to see if existing `Gprim`s might be +more appropriate as `Vprim`s (like `HermiteCurves` which is explicitly +called out as "not for final rendering"). * `VizPoints`: Points with 3-D positions but screen space widths -* `VizVolumeSamples`: List of 3-D points at which to sample fields in a `UsdVol`. The field type can drive the output style (`vector` renders as arrows, `density` or `color` render as screen space points) or could be otherwise configured. +* `VizVolumeSamples`: List of 3-D points at which to sample fields in +a `UsdVol` volume. The field type can drive the output style (`vector` +renders as arrows, `density` or `color` render as screen space points) +or could be otherwise configured. * `VizPolylines`: Align with existing curve styling proposals -* `VizPaths`: Cubically interpolated curves that align with existing curve styling proposals -* `VizText`: Align with existing text schema proposals +* `VizPaths`: Cubically interpolated curves that align with existing +curve styling proposals +* `VizLabel`: Align with existing text schema proposals +* `VizOverlayAPI`: Add a nondiegetic color overlay on top of an +existing `Gprim` +* `VizSilhoutteAPI`: Add a nondiegetic edge on top of an existing +`Gprim` +* `VizScreenAnchor`: Nondiegetic screen anchoring transformation -### Note on Schema Slicing -It's worth noting that this proposal does not believe that all applications or usages of API schemas are inherently "slicing". Most API schemas are used to augment, annotate, or adapt prims for usage by other domains, but generally leave the original prim definition untouched. Using API schemas to describe "optional" features of a concrete typed schema (that might otherwise be described with an enabling `bool` flag) would generally not be considered slicing as well. +### Schema Slicing +One important consideration when introducing schemas in this domain is +to whether or not they are new drawables (`Vprim`s) or styles on top +of existing drawables (`Gprim` + `VizStyleAPI`). + +An iteration of the `LineStyle` [proposal](https://github.com/PixarAnimationStudios/OpenUSD-proposals/tree/main/proposals/LineStyle) +suggests the existing `Curves` schemas can be adapted into screen +space via API schema. The API schema makes three transformative +changes. It transforms the meaning of the `widths` attribute to be a +screen space size (`constant` or `uniform` only), changes the +appearence of joints and end caps, and resticts the materials allowed +to be bound to a restricted set of "styling" materials. + +The current code for bounding `UsdGeomCurves` takes the maximum value +of the `widths` attribute to pad the extent of its `points`. Consider +a scene with `metersPerUnit = 1.0`. If a prim uses the `CurveStyleAPI` +to redefine a 1 pixel constant width, the +`UsdGeomCurves::ComputeExtent` code is unawair of the redefinition and +interpret the `widths` as object space meters, padding the curve +`extent` by a meter. + +We term this ambiguity "schema slicing" (to adapt the term "object +slicing" from C++). In that language, object slicing occurs when the +properties and behavior of a derived type are "sliced" off when +interfacing with an object via a higher level type. Using an API +schema to transform the curves into screen space defined primitives +similarly slices off the space redefinition for contexts like extent +computation. Utlimately, "slicing" complicates interchange and +standardization. + +This proposal does not believe that all applications or usages of API +schemas are inherently "slicing". Most API schemas are used to +augment, annotate, or adapt prims for usage by other domains, but +generally leave the original prim's meaning untouched. + +#### Disaggregating `hermite` out of `BasisCurves` +As a historical analogue, the `hermite` basis was previously a part of +`BasisCurves` but was left unimplemented in Storm and many renderers. +We learned the primary usage was to describe paths and not as a +renderable representation, ultimately seperating it out into its own +schema. This change made `BasisCurves` simpler for renderers to fully +support and let `HermiteCurves` stand on its own with a decoupled +`points` and `tangents` representation and without the need to +formulate support for the different `wrap` modes of `BasisCurves`. +Disaggregating schemas based on usage and client needs can lead to +better support and interopability for all. ## Alternative Approaches ### Annotation Specific Schemas -Some suggest that the needs would be better served with discrete purposed annotation primitives (ie. `AnnotationLabel`, `AnnotationPaths`, `AnnotationPoints`). The door should be cleanly shut to using visualzation primitives for imposter billboard cards or generalized non-photorealistic rendering. This choice was not the preferred approach because there is a need for visualization primitives outside of the annotation space, and that annotations specific properties could extend the set of visualization primitives through additional schemas. +Some suggest that the needs would be better served with discrete +purposed annotation primitives (ie. `AnnotationLabel`, +`AnnotationPaths`, `AnnotationPoints`). The door should be cleanly +shut to using visualzation primitives for imposter billboard cards or +generalized non-photorealistic rendering. This choice was not the +preferred approach because there is a need for visualization +primitives outside of the annotation space, and that annotations +specific properties could extend the set of visualization primitives +through additional schemas. ### Introduce New Typed Schemas Without the `Vprim` Categorization -Readers may agree that some of these nondiegetic primitives are distinct enough to warrant new schemas, but may reject the premise that screen independence is necessarily a fundemental property of `Gprim`s. The USD Glossary says `Gprim`s are just drawable, transformable, and boundable. The problems of how to bound, transform (and otherwise collide / intersect) screen space objects should be clarified and reconciled within the existing `Gprim` definition, rather than introduce a new type. This is not the preferred approach of this proposal which sees value having a high level classification for diegetic and nondiegetic primitives that clients can use to organize scenes and imaging pipelines around. +Readers may agree that some of these nondiegetic primitives are +distinct enough to warrant new schemas, but may reject the premise +that screen independence is necessarily a fundemental property of +`Gprim`s. The USD Glossary says `Gprim`s are just drawable, +transformable, and boundable. The problems of how to bound, transform +(and otherwise collide / intersect) screen space objects should be +clarified and reconciled within the existing `Gprim` definition, +rather than introduce a new type. This is not the preferred approach +of this proposal which sees value having the high level classification +for diegetic and nondiegetic primitives that clients can use to +organize scene processing and imaging pipelines around. ### Disaggregate Boundability out of `Gprim` -The proposal suggests that `Vprim`s required a relaxed definition of boundability. One could imagine that just as `visibility` was disaggregated out of `Gprim` into its own schema, boundability could disaggregated as well so that not every `Gprim` requires a traditional `extent`. +The proposal suggests that `Vprim`s required a relaxed definition of +boundability. One could imagine that just boundability could +disaggregated as well so that not every `Gprim` requires a traditional +`extent`. -We think there might be other core properties (`doublesided`) and behaviors that warrant removal from the "visualization" domain so this isn't our preferred approach. +This would be a large change requiring schema upgrades for almost all +current `Gprim` usage. We also think there might be other core +properties (`doublesided`) and behaviors that warrant removal from the +"visualization" domain so this isn't our preferred approach. ## Summary -To preserve compatability, supportability, and interchange of existing `UsdGeom` schemas while extending USD's ability to describe nondiegetic visualizations, a new domain should be introduced. The `Vprim` domain would allow visualizations to be described in relation to both screen and other primitives and would avoid introducing schemas that might "slice" the current set of `Gprim`s. +To preserve compatability, supportability, and interchange of existing +`UsdGeom` schemas while extending USD's ability to describe +nondiegetic visualizations, a new domain of applied and typed schemas +should be introduced. This domain would allow visualizations to be +described in relation to both screen and other primitive. ## Questions -* As formulated `Vprims` cannot be `PointBased` which are strictly `Gprims`. Are there any cases where dispatching behavior off of the more general `PointBased` without knowing the concrete type is important? -* When defining screen space sizes, what is the best choice of units? Is it pixels or something else? +* As formulated `Vprims` cannot be `PointBased` which are strictly +`Gprims`. Are there any cases where dispatching behavior off of the +more general `PointBased` without knowing the concrete type is +important? +* When defining screen space sizes, what is the best choice of units? +Is it pixels or something else? * How should scaling be treated for screen space objects? ## Future Work ### Hiding -As currently described, visualization primitives should be strictly hidden via depth. However, one could imagine customizing hiding for these prims. Some visualizations might want to hide based on a particular reference point for the entire string of text. Some visualizations might want to express priority with respect to other objects in the scene. Hiding is another axis where the specification may eventually end up diverging from the `Gprim` space. +As currently described, visualization primitives should be strictly +hidden via depth. However, one could imagine customizing hiding for +these prims. Some visualizations might want to hide based on a +particular reference point for the entire string of text. Some +visualizations might want to express priority with respect to other +objects in the scene. Hiding is another axis where the specification +may eventually end up diverging from the `Gprim` space. ### Breaking the Fourth Wall -Some care needs to be taken to establish best practices and rules for visibility of screen-constrained primitives. While we generally would expect nondiegetic primitives to not participate in physically based lighting, consider screen constrained "context spheres" as reference objects. They "break the fourth wall" and capture light and reflections. It should be possible to specify this behavior even when using nondiegetic transformations. - -### Styling Schemas -Universal material model description has required a plugin system build around `Sdr` to allow for disparate renderer and site specific materials models to be integrated into OpenUSD. Even the simplest material models in OpenUSD (ie. `UsdPreviewSurface`) requires the notion of a network and connections. - -Visualization style, on the other hand, is a more compact problem. Users often want to specify a single color, image, or pattern to style an object. There's potentially some inspiration to be drawn from 2-D tooling where often the same user interface is used to style text, lines, and other shapes. - -We think the simplicity afforded by configuring stylization with simple declarative schemas that don't require material networks is worth pursuing in the future, but is beyond the scope of this proposal. +Some care needs to be taken to establish best practices and rules for +visibility of screen-constrained primitives. While we generally would +expect nondiegetic primitives to not participate in physically based +lighting, consider screen constrained "context spheres" as reference +objects. They "break the fourth wall" and capture light and +reflections. It should be possible to opt into this behavior when +using nondiegetic transformations. \ No newline at end of file