Layer Mask Modes

[mrtripie]

I've been trying to come up with a simple yet powerful solution to layer masks, here's my idea:

• No dedicated MaskLayer type

• Any layer can be added as a child of any other layer (and any layer can have multiple children)

• When a layer is a child of a NON-Group-Layer (ie: PixelLayer, ShapeLayer, TilemapLayer, FillLayer):

  • It becomes a mask layer. It will a dropdown for mask modes (shared with blend modes) with a few different mask modes to choose from:
    • Transparency masks: Possibly ALPHA_MULTIPLY, INVERTED_ALPHA_MULTIPLY, ALPHA_SUBTRACT, and INVERTED_ALPHA_SUBTRACT
      • Multiply mask modes will multiply the alpha channel of this mask layer with the alpha of the parent before blending the parent
      • Subtract mask modes will subtract the alpha channel of this mask from the alpha of the parent before blending the parent
      • Inverted mask modes will first invert the mask's alpha channel (1 - alpha) and then multiply or subtract
      • I think that with solid black/white alphas, ALPHA_MULTIPLY will produce an effect similar to a Intersect Boolean operation in a vector/3d program, and INVERTED_ALPHA_MULTIPLY will produce a Difference Boolean. (The opposite with ALPHA_SUBTRACT/INVERTED_ALPHA_SUBTRACT)
      • 
    • A NO_BLEND mode: There's interest in FX Layers #619 / Layer FX having some of their parameters controlled by a layer's texture. This mode will not effect blending at all (similar to hiding the layer, except you will still be able to use tools to modify it), which will be good for using as Layer FX masks
      • 

• When a layer is a child of a GroupLayer it will behave like this:

  • The child can use any blend mode as usual (it won't be a mask), such as NORMAL, or OVERLAY
  • The child layers will be blended first, then the group will be blended together with the rest of the layers that have the same parent as the group
  • To have masks on a Group Layer, the child can have a blend mode very similar to the the mask modes:
    • First the group will blend each layer below them, then when it gets to the mask the alpha multiply or subtract operation will be done.
    • If the masks are at the top of the group layer it should behave exactly the same to the user
    • If there is a non-mask-blend-mode layer above the mask-blend-mode layer it will not be masked by them, which is additional behavior in this case (I think it should be consistent because all direct children of a NON-Group-Layer will be using mask modes)
    • 
  • If a GroupLayer is a child of a NON-Group-Layer, it will become a mask, but the GroupLayer's children will be blended normally
    • 

• Should these use the same variable on BaseLayer as blend modes or a seperate mask_mode variable?

  • Separating them has the advantage that you can drag and drop a layer onto another layer, which will turn it into a mask, and then drag it out of that layer, returning to the previous blend mode
  • When saving they could be saved as a single "mode" value in the JSON data.

• Pros (vs a dedicated MaskLayer type):

  • Don't need to write/maintain MaskLayer/MaskCel classes
  • Any layer type can become a mask (PixelLayer, ShapeLayer, TilemapLayer, FillLayer, etc.)
  • The layer moving code is already quite complicated and difficult to modify, and adding additional rules for MaskLayers of which layer types can go where will make it more so, especially when adding more support for moving multiple layers at once.
    • As any layer be a child of any other layer with no special rules, it simplifies layer moving code (can remove the Layer's accepts_child method and conditions based on it)

• Cons (vs a dedicated MaskLayer type):

  • PixelLayer masks will have full RGBA data even though it only uses one channel. Potentially we can add an option to add a PixelLayer with L8 as its color space. I've tested this by replacing all FORMAT_RGBA8 in Pixelorama with FORMAT_L8, and all tools basically just work
  • More steps to setup for the user (Add grayscale layer, add as child, rename to "Mask"), though there could be a button or menu option somewhere that could automatically set it up
  • As a mask can be a child of a mask this might cause the shader generation code to be a little more complicated? Not sure.