Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Add SKSLEffect. #17981

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions build/SkiaSharp.props
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
<PackageReference Condition="'$(IncludeWasmSkia)' == 'true'" Include="SkiaSharp.NativeAssets.WebAssembly" Version="2.88.9" />
</ItemGroup>
<ItemGroup Condition="'$(AvsIncludeSkiaSharp3)' == 'true'">
<PackageReference Include="SkiaSharp" Version="3.118.0-preview.1.2" />
<PackageReference Condition="'$(IncludeLinuxSkia)' == 'true'" Include="SkiaSharp.NativeAssets.Linux" Version="3.118.0-preview.1.2" />
<PackageReference Condition="'$(IncludeWasmSkia)' == 'true'" Include="SkiaSharp.NativeAssets.WebAssembly" Version="3.118.0-preview.1.2" />
<PackageReference Include="SkiaSharp" Version="3.118.0" />
<PackageReference Condition="'$(IncludeLinuxSkia)' == 'true'" Include="SkiaSharp.NativeAssets.Linux" Version="3.118.0" />
<PackageReference Condition="'$(IncludeWasmSkia)' == 'true'" Include="SkiaSharp.NativeAssets.WebAssembly" Version="3.118.0" />
</ItemGroup>
</Project>
Binary file added samples/ControlCatalog/Assets/noise.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions samples/ControlCatalog/ControlCatalog.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<Nullable>enable</Nullable>
<IncludeAvaloniaGenerators>true</IncludeAvaloniaGenerators>
<AvsIncludeSkiaSharp3>true</AvsIncludeSkiaSharp3>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avalonia 11.x will stay compatible with SkiaSharp 2.88, so such change cant' be merged. We need to have code paths for both 2.88 and 3.x.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose some of user-facing version-specific APIs can live in separate Avalonia.Skia.VersionSpecific.2.88 and Avalonia.Skia.VersionSpecific.3.0 packages.
@maxkatz6

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How feasible would it be to avoid SkiaSharp types in the new public API? This way it would be easier for us to maintain and abstract different versions internally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@maxkatz6 This pr is add the sksl effect, which means that it depends on sksl. Or do you want to design a new shader language, alsl (avalonia shader language) ?

Copy link
Member

@maxkatz6 maxkatz6 Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. I mean SKIA shaders, but not SkiaSharp API (not public API at least, internally we can bridge different versions/implementations of SkiaSharp).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How feasible would it be to avoid SkiaSharp types in the new public API? This way it would be easier for us to maintain and abstract different versions internally.

Only SkiaSharp with my pr can works now.
This requires RuntimeShader, and SkiaSharp hasn't exposed this api yet.
And i think it's hard to avoid SkiaSharp types. The SKImageFilter can be many types of things, like images, shaders which is not exists in avalonia.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically we don't need SKRuntimeFilter and can work with SKShader. The way it would work is by manually creating an intermediate texture and SKCanvas for subsequent drawing calls and using it as an input for SKShader.

That way it can be compatible with 2.88.
So the API could accept SKSL source + Avalonia.Media.Bitmap for inputs.

</PropertyGroup>
<ItemGroup>
<Compile Update="**\*.xaml.cs">
Expand Down
3 changes: 3 additions & 0 deletions samples/ControlCatalog/MainView.xaml
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,9 @@
<TabItem Header="Screens">
<pages:ScreenPage />
</TabItem>
<TabItem Header="SKSLEffect">
<pages:SKSLEffectPage />
</TabItem>
<FlyoutBase.AttachedFlyout>
<Flyout>
<StackPanel Width="152" Spacing="8">
Expand Down
12 changes: 12 additions & 0 deletions samples/ControlCatalog/Pages/SKSLEffectPage.xaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<UserControl xmlns="https://github.com/avaloniaui"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"
x:Class="ControlCatalog.Pages.SKSLEffectPage">
<UniformGrid Columns="3" Height="200">
<Rectangle x:Name="Rectangle0" Fill="Red" Margin="20"/>
<Rectangle x:Name="Rectangle1" Fill="Red" Margin="20"/>
<Rectangle x:Name="Rectangle2" Fill="Red" Margin="20"/>
</UniformGrid>
</UserControl>
185 changes: 185 additions & 0 deletions samples/ControlCatalog/Pages/SKSLEffectPage.xaml.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using Avalonia;
using Avalonia.Animation;
using Avalonia.Controls;
using Avalonia.Controls.Shapes;
using Avalonia.Data;
using Avalonia.Markup.Xaml;
using Avalonia.Media;
using Avalonia.Platform;
using Avalonia.Skia.Effects;
using Avalonia.Styling;
using SkiaSharp;

namespace ControlCatalog.Pages
{
public partial class SKSLEffectPage : UserControl
{
public SKSLEffectPage()
{
InitializeComponent();
InitEffect();
}

private void InitializeComponent()
{
AvaloniaXamlLoader.Load(this);
}

private void InitEffect()
{
var rectangle1 = this.FindControl<Rectangle>("Rectangle1")!;
var shaderBuilder = CreateSimpleShaderBuilder();
if (shaderBuilder != null)
{
rectangle1.Effect = new SKSLEffect(shaderBuilder)
{
ChildShaderNames = ["src"],
Inputs = [null],
};
}

var rectangle2 = this.FindControl<Rectangle>("Rectangle2")!;
try
{
var effect = new DissolveSKSLEffect();
effect.Progress = 0.5f;
effect[!DissolveSKSLEffect.ResolutionProperty] = new Binding
{
Source = rectangle2,
Path = "Bounds.Size",
};
rectangle2.Effect = effect;

var animation = new Animation
{
Duration = TimeSpan.FromSeconds(2),
IterationCount = IterationCount.Infinite,
PlaybackDirection = PlaybackDirection.Alternate,
Children = {
new KeyFrame
{
Setters =
{
new Setter(DissolveSKSLEffect.ProgressProperty, 0f),
},
KeyTime = TimeSpan.FromSeconds(0),
},
new KeyFrame
{
Setters =
{
new Setter(DissolveSKSLEffect.ProgressProperty, 1f),
},
KeyTime = TimeSpan.FromSeconds(2),
}
}
};

_ = animation.RunAsync(effect);
}
catch
{
// Do not crash.
}
}

private SKRuntimeShaderBuilder? CreateSimpleShaderBuilder()
{
var sksl = @"
uniform shader src;

float4 main(float2 coord) {
return src.eval(coord).bgra;
}
";
var effect = SKRuntimeEffect.CreateShader(sksl, out var str);
if (effect != null)
{
return new SKRuntimeShaderBuilder(effect);
}
else
{
return null;
}
}
}

public class DissolveSKSLEffect : SKSLEffect
{
public static readonly StyledProperty<float> ProgressProperty = AvaloniaProperty.Register<DissolveSKSLEffect, float>(nameof(Progress), default);

public float Progress
{
get => GetValue(ProgressProperty);
set => SetValue(ProgressProperty, value);
}

public static readonly StyledProperty<Size> ResolutionProperty = AvaloniaProperty.Register<DissolveSKSLEffect, Size>(nameof(Resolution), default);

public Size Resolution
{
get => GetValue(ResolutionProperty);
set => SetValue(ResolutionProperty, value);
}

public DissolveSKSLEffect() : base(CreateShaderBuilder())
{
ChildShaderNames = ["src"];
Inputs = [null];
AffectsRender<SKSLEffect>(ProgressProperty);

RegisterUniform("progress", ProgressProperty);
RegisterUniform("resolution", ResolutionProperty);
}

private static SKRuntimeShaderBuilder? s_shaderBuilder;
private static SKRuntimeShaderBuilder CreateShaderBuilder()
{
if (s_shaderBuilder != null)
{
return s_shaderBuilder;
}

var sksl = @"
uniform float2 resolution;
uniform shader src;
uniform shader noise;
uniform float2 noiseResolution;
uniform float progress;

float4 main(float2 coord) {
float val = noise.eval(fract(coord / resolution) * noiseResolution).x;

if(val < progress)
{
return src.eval(coord);
}
else
{
return float4(0,0,0,0);
}
}
";
var effect = SKRuntimeEffect.CreateShader(sksl, out var str);
if (effect != null)
{
var noise = AssetLoader.Open(new Uri("avares://ControlCatalog/Assets/noise.png"));
var noiseImage = SKImage.FromEncodedData(noise);
var noiseImageShader = SKShader.CreateImage(noiseImage);
var builder = new SKRuntimeShaderBuilder(effect);
builder.Uniforms["noiseResolution"] = new SKSize(noiseImage.Width, noiseImage.Height);
builder.Children["noise"] = noiseImageShader;
s_shaderBuilder = builder;
return s_shaderBuilder;
}
else
{
throw new NotSupportedException();
}
}
}
}
7 changes: 6 additions & 1 deletion src/Avalonia.Base/Media/Effects/EffectExtesions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ internal static Thickness GetEffectOutputPadding(this IEffect? effect)
return new Thickness(Math.Max(0, 0 - rc.X),
Math.Max(0, 0 - rc.Y), Math.Max(0, rc.Right), Math.Max(0, rc.Bottom));
}
if (effect is IShaderEffect)
{
// Shader effect should not have padding.
return default;
}

throw new ArgumentException("Unknown effect type: " + effect.GetType());
}
Expand Down Expand Up @@ -53,4 +58,4 @@ internal static bool EffectEquals(this IImmutableEffect? immutable, IEffect? rig
return immutable.Equals(right);
return false;
}
}
}
12 changes: 12 additions & 0 deletions src/Avalonia.Base/Media/Effects/IShaderEffect.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Avalonia.Media
{
public interface IShaderEffect : IEffect
Copy link
Member

@kekekeks kekekeks Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Custom effects need to report their expected padding, i. e. the area outside of the affected drawing bounds they will alter (e. g. drop-shadow(5px 5px) would add (0,0,5,5) padding for renderer invalidation).

Note that this can depend on uniform values.

Copy link
Contributor Author

@kkwpsv kkwpsv Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think shader effect shouldn't has padding?
The coord is sksl input, and the color is output. Coord cannot be changed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SKImageFilter can have padding, that's how our blur and drop shadow effects work. They report padding to the compositor and it extends the intermediate texture bounds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, right now the intermediate texture size always matches the render target size. That's something we've planned to change but never actually implemented.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SKImageFilter can have padding, that's how our blur and drop shadow effects work. They report padding to the compositor and it extends the intermediate texture bounds.

But runtimeshader seems to be not?
I'll read skia source code to confirm it.

Copy link
Member

@kekekeks kekekeks Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "padding" is just shader sampling the source at a different coordinate than it's currently producing output for. E. g. a simple implementation of drop-shadow(5,5) shader would just sample src at (-5,-5) from coord and use float4(1-alpha, 1-alpha, 1-alpha, alpha) for output.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "padding" is just shader sampling the source at a different coordinate than it's currently producing output for. E. g. a simple implementation of drop-shadow(5,5) shader would just sample src at (-5,-5) from coord and use float4(1-alpha, 1-alpha, 1-alpha, alpha) for output.

I have read skia source and usage of GetEffectOutputPadding.

GetEffectOutputPadding affect dirty rect and bounds in Avalonia compositor.
The drop shadow implementation is here.
It has blur and MatrixTransform. And they can affect rendering bounds. So we need padding for compositor.
The implementation seems to be not same as you said?

And i think runtimeshader cannot affect. It's just a fragment shader and a subset of SKimageFilter.
Or can you give me a runtimeshader example where padding is needed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In WPF it's a normal thing for ShaderEffect to extend the affected Visual's render bounds. We should do the same. If a custom SKImageFilter somehow doesn't support that, we need to manually manage the intermediate texture and use SKShader instead of SKImageFilter.

{
}
}
1 change: 1 addition & 0 deletions src/Skia/Avalonia.Skia/Avalonia.Skia.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
<IncludeLinuxSkia>true</IncludeLinuxSkia>
<IncludeWasmSkia>true</IncludeWasmSkia>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<AvsIncludeSkiaSharp3>true</AvsIncludeSkiaSharp3>
</PropertyGroup>
<ItemGroup>
<EmbeddedResource Include="Assets\NoiseAsset_256X256_PNG.png" />
Expand Down
7 changes: 7 additions & 0 deletions src/Skia/Avalonia.Skia/DrawingContextImpl.Effects.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using System;
using Avalonia.Media;
using Avalonia.Skia.Effects;
using SkiaSharp;

namespace Avalonia.Skia;
Expand Down Expand Up @@ -44,6 +45,12 @@ public void PopEffect()
return SKImageFilter.CreateDropShadow((float)drop.OffsetX, (float)drop.OffsetY, sigma, sigma, color);
}

if (effect is ISKSLEffect skslEffect)
{
var builder = skslEffect.ShaderBuilder;
return SKImageFilter.CreateRuntimeShader(builder, skslEffect.MaxSampleRadius, skslEffect.ChildShaderNames, skslEffect.Inputs);
}

return null;
}

Expand Down
94 changes: 94 additions & 0 deletions src/Skia/Avalonia.Skia/Effects/SKSLEffect.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
using System.Collections.Generic;
using Avalonia.Logging;
using Avalonia.Media;
using SkiaSharp;

namespace Avalonia.Skia.Effects
{
public class SKSLEffect : Effect, ISKSLEffect, IMutableEffect
{
public SKRuntimeShaderBuilder ShaderBuilder { get; set; }

public float MaxSampleRadius { get; set; } = 0f;

public string[] ChildShaderNames { get; set; } = [];

public SKImageFilter?[] Inputs { get; set; } = [];

private readonly Dictionary<string, AvaloniaProperty> _uniformProperties = new Dictionary<string, AvaloniaProperty>();

public SKSLEffect(SKRuntimeShaderBuilder builder)
{
ShaderBuilder = builder;
}

public void RegisterUniform(string name, AvaloniaProperty<int> property) => _uniformProperties.Add(name, property);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How uniform updates are supposed to be propagated to the composition thread?

I think we need to make effects to use the same infrastructure as brushes do (ICompositionRenderResource/ICompositorSerializable)

Copy link
Contributor Author

@kkwpsv kkwpsv Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be stored in the copided SkRuntimeShaderBuilder in ToImuatable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would invalidation work after the initial layout/render pass?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess AffectsRender<T> is still public in Effect.cs. It's something we are planning to remove though

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm using AffectsRender currently.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When calling RegisterUniform, we can call AffectsRender internally?

Copy link
Member

@kekekeks kekekeks Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would still prefer switching to the compositor-aware infra, so we could make SKSLShader disposable and implement lifetime management for inputs/builder as well.


public void RegisterUniform(string name, AvaloniaProperty<float> property) => _uniformProperties.Add(name, property);

public void RegisterUniform(string name, AvaloniaProperty<Size> property) => _uniformProperties.Add(name, property);

public IImmutableEffect ToImmutable()
{
SKRuntimeShaderBuilder builder = new SKRuntimeShaderBuilder(ShaderBuilder);
foreach (var property in _uniformProperties)
{
var value = GetValue(property.Value);
if (value is int intVal)
{
builder.Uniforms[property.Key] = intVal;
}
else if (value is float floatVal)
{
builder.Uniforms[property.Key] = floatVal;
}
else if (value is Size sizeVal)
{
float[] val = [(float)sizeVal.Width, (float)sizeVal.Height];
builder.Uniforms[property.Key] = val;
}
else
{
Logger.TryGet(LogEventLevel.Error, "Effect")?.Log(this, $"Unsupported uniform type: {value?.GetType() ?? null}");
}
}

return new ImmutableSKSLEffect(builder, MaxSampleRadius, ChildShaderNames, Inputs);
}
}

public class ImmutableSKSLEffect : ISKSLEffect, IImmutableEffect
{
public SKRuntimeShaderBuilder ShaderBuilder { get; }

public float MaxSampleRadius { get; }

public string[] ChildShaderNames { get; }

public SKImageFilter?[] Inputs { get; }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with having ShaderBuilder and Inputs as such properties is lifetime management. There is no clear ownership of ImmutableSKSLEffect once it's passed to the compositor, so we are not allowed to manually call Dispose on ShaderBuilder and SKImageFilters, yet they might hold a huge chunk of unmanaged memory by referencing bitmaps. So the API is prone to "soft" unmanaged memory leaks when GC isn't aware that it's supposed to collect collect gen2 to free a few GB of native bitmaps.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kekekeks Do you means using GC.AddMemoryPressure to add the memory pressure to GC? Or the GC untimely collect the object to release the skia resource?

Copy link
Contributor

@Gillibald Gillibald Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ownership of allocated objects needs to be transferred to the compositor and only the compositor knows when to free those objects. By implementing ICompositionRenderResource etc. you make sure these objects are properly ref counted and will be cleaned up at the right time.

ICompositorSerializable allows you to sync mutable states with the compositor, which will then hold an immutable copy of that resource. Using that mechanism makes AffectsRender redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which will then hold an immutable copy of that resource.

Creating a copy of the SKImageFilter seems to be not possible. It's immutable.
Users may want to reuse SKImageFilter. So the ownership shouldn't be transferred to the compositor?
But maybe we can use ICompositionRenderResource to let users know when the compositor is no longer using the object.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as objects are properly ref counted this is fine

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users may want to reuse SKImageFilter

Compositor shouldn't be calling Dispose directly either, it should be calling a user-specified callback as a means to report that it's not using the resource anymore.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e. g. void AddInput(SKImageFilter filter, Action? release = null);


public ImmutableSKSLEffect(SKRuntimeShaderBuilder builder, float maxSampleRadius, string[] childShaderNames, SKImageFilter?[] inputs)
{
ShaderBuilder = builder;
MaxSampleRadius = maxSampleRadius;
ChildShaderNames = childShaderNames;
Inputs = inputs;
}

public bool Equals(IEffect? other)
{
return false;
}
}

public interface ISKSLEffect : IShaderEffect
{
SKRuntimeShaderBuilder ShaderBuilder { get; }

float MaxSampleRadius { get; }

string[] ChildShaderNames { get; }

SKImageFilter?[] Inputs { get; }
}
}
Loading