Editing raster layer masks in PSD file via API

Overview

To automate PSD format editing and change PSD file without Adobe® Photoshop® you can use the Aspose.PSD API provided below. There are C# and .NET code snippets that can help you to modify PSD files.

Using PSD Layer and Vector Masks we are able to hide and show layer pixels without permanently deleting them. Raster masks are also called a layer mask or user mask. Access to both raster and vector masks in Aspose.PSD is provided through the LayerMaskData layer property which can be an instance of ‘LayerMaskDataShort’ and ‘LayerMaskDataFull’ classes that are child classes of abstract ‘LayerMaskData’ class. If a layer has both raster and vector masks then LayerMaskDataFull instance is provided. If a layer has only a raster or a vector mask then LayerMaskDataShort instance is provided. If the LayerMaskData property is null then the layer has no masks or only a disabled vector mask.

	A raster mask and a disabled vector mask LayerMaskDataShort A disabled raster mask  LayerMaskDataShort A raster mask and a vector mask  LayerMaskDataFull A raster mask  LayerMaskDataShort A vector mask  LayerMaskDataShort A disabled vector mask null (But vector resource is present)

How to get a layer raster mask in the PSD file?

At first, we should find out if a layer has both vector and layer masks:

Below provided sample code demonstrates how to get a layer raster mask

LayerMaskDataFull fullMask = layer.LayerMaskData as LayerMaskDataFull;
if (fullMask != null)
{
    var left = fullMask.EnclosingLeft;
    var right = fullMask.EnclosingRight;
    var top = fullMask.EnclosingTop;
    var bottom = fullMask.EnclosingBottom;
    var maskRectangle = fullMask.UserMaskRectangle;
    var imageData = fullMask.UserMaskData;
    var defaultColor = fullMask.BackgroundColor;
    var flags = fullMask.RealFlags;
    bool isDisabled = (flags & LayerMaskFlags.Disabled) != 0;
}

Otherwise, type of the LayerMaskData layer property is LayerMaskDataShort. In this case, let’s check if the layer has only a raster mask by checking the Flags property. It should not contain LayerMaskFlags.UserMaskFromRenderingOtherData flag, otherwise the mask is a vector mask cache**.**

Getting mask code snippet:

LayerMaskDataShort mask = (LayerMaskDataShort)layer.LayerMaskData;
if ((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) == 0)
{
    var left = fullMask.Left;
    var right = fullMask.Right;
    var top = fullMask.Top;
    var bottom = fullMask.Bottom;
    var maskRectangle = mask.MaskRectangle;
    var imageData = mask.ImageData;
    var defaultColor = newMask.DefaultColor;
    var flags = newMask.Flags;
    bool isDisabled = (flags & LayerMaskFlags.Disabled) != 0;
}

If you need to extract a raster mask as LayerMaskDataShort (for further manipulations) even when both masks present, the LayerMaskDataFull should be extracted and converted to LayerMaskDataShort. The following code can be used for both cases:

Extracting a raster mask from PSD

LayerMaskDataShort GetRasterMask(LayerMaskData mask)
    {
        LayerMaskDataFull fullMask = mask as LayerMaskDataFull;
        if (mask == null ||
((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0 && fullMask == null))
        {
            return null;
        }

        if (fullMask != null)
        {
            return new LayerMaskDataShort()
            {
                Left = fullMask.EnclosingLeft,
                Right = fullMask.EnclosingRight,
                Top = fullMask.EnclosingTop,
                Bottom = fullMask.EnclosingBottom,
                ImageData = fullMask.UserMaskData,
                DefaultColor = fullMask.DefaultColor,
                Flags = fullMask.RealFlags
            };
        }

How to check if a layer in the PSD file has a raster mask?

The following C# code may help you to check if a layer has a raster mask:

How to know if raster mask applied to PSD Layer

bool HasLayerRasterMask(Layer layer)
{
        var mask = layer.LayerMaskData;
        var fullMask = mask as LayerMaskDataFull;
        if (mask == null || 
   ((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0 && fullMask == null))
        {
            return false;
        }

        return true;
}

How to remove / add / update a layer raster mask in the PSD file?

Just removing / adding / updating the LayerMaskData is not enough for correct saving because channels are not updated; though it may provide correct rendering. This does not change the mask channels:

layer.LayerMaskData = newMaskData;

We should use the layer AddLayerMask method for removing / adding / updating.

This adds/updates both the mask and channels:

layer.AddLayerMask(newMaskData);

This removes both the mask and channels:

layer.AddLayerMask(null);

Removing a layer raster mask in the PSD image

Firstly, we check if the mask is in the short format and if it not vector one we can just call the AddLayerMask method with null to delete the raster mask. But if it is in the full format we have to convert it to a short format thus leaving the vector mask only. For removing a layer mask this following C# .NET code snippet can be used:

Code snippet how to remove Layer Mask from PSD File.

void RemoveRasterMask(Layer layer)
    {
        LayerMaskData mask = layer.LayerMaskData;
        LayerMaskDataFull fullMask = mask as LayerMaskDataFull;
        if (mask == null ||
((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0 && fullMask == null))
        {
            throw new Exception("This layer has no raster mask.");
        }

        if (fullMask == null)
        {
            layer.AddLayerMask(null);
            return;
        }

        var vectorMask = new LayerMaskDataShort();
        vectorMask.Flags = fullMask.Flags;
        vectorMask.MaskRectangle = fullMask.MaskRectangle;
        vectorMask.DefaultColor = fullMask.DefaultColor;
        vectorMask.ImageData = fullMask.ImageData;
        layer.AddLayerMask(vectorMask);
    }

Updating a layer raster mask in the PSD image

This is straightforward:  if the mask is in the short format we have to change ImageData and MaskRectangle if need, otherwise UserMaskData and UserMaskRectangle should be changed. The following C# .NET code snippet can be used for updating a layer mask:

Update PSD Layer Mask with C#

void RemoveRasterMask(Layer layer)
    {
        LayerMaskData mask = layer.LayerMaskData;
        LayerMaskDataFull fullMask = mask as LayerMaskDataFull;
        if (mask == null ||
((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0 && fullMask == null))
        {
            throw new Exception("This layer has no raster mask.");
        }

        if (fullMask == null)
        {
            layer.AddLayerMask(null);
            return;
        }

        var vectorMask = new LayerMaskDataShort();
        vectorMask.Flags = fullMask.Flags;
        vectorMask.MaskRectangle = fullMask.MaskRectangle;
        vectorMask.DefaultColor = fullMask.DefaultColor;
        vectorMask.ImageData = fullMask.ImageData;
        layer.AddLayerMask(vectorMask);
    }

Here is an example of possible actions that change a raster mask. This one inverts a layer user mask:

Update PSD Layer Mask with C#

        void InvertMask(LayerMaskData mask)
        {
            byte[] maskBytes;
            LayerMaskDataFull fullMask = mask as LayerMaskDataFull;
            if (fullMask == null)
            {
                maskBytes = mask.ImageData;
            }
            else
            {
                maskBytes = fullMask.UserMaskData;
            }

            for (int i = 0; i < maskBytes.Length; i++)
            {
                maskBytes[i] = (byte)~maskBytes[i];
            }
        }

Updating a vector mask in the PSD file when a layer raster mask is present

It is supposed that a user has already changed a vector path resource. Then it can update the vector mask simply by calling the AddLayerMask layer method:

Update PSD Layer Vector Mask with C#

void UpdateVectorMask(Layer layer)
    {
        var vectorMask = layer.LayerMaskData;
        if (vectorMask == null || (vectorMask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) == 0)
        {
            throw new Exception("This layer has no vector mask.");
        }

        layer.AddLayerMask(vectorMask);
    }

Adding a layer raster mask in the PSD file

If a layer has no mask we can add the given raster mask simply by calling the AddLayerMask layer method.

If the mask has no UserMaskFromRenderingOtherData** flag then it already has a raster mask and we have to update it as described above. Otherwise, if this mask is in a short format we convert it to the full format. If not we use it as is. Then update UserMaskData, UserMaskRectangle, and other properties with the given mask properties. The following C# .NET code snippet can be used for adding (updating) a layer mask:

Add new Layer Mask to PSD

void AddRasterMask(LayerMaskData mask, LayerMaskDataShort newShortMask)
    {
        LayerMaskData mask = layer.LayerMaskData;
        LayerMaskDataFull fullMask = mask as LayerMaskDataFull;
        var hasRasterMask = fullMask != null ||
        (mask != null && (mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) == 0);

        if (hasRasterMask)
        {
            Console.WriteLine("This layer has a raster mask already, updating.");
        }

        if (mask != null)
        {
            if (fullMask != null)
            {
                // let's update user raster mask in a full one.
                fullMask.RealFlags = newMask.Flags;
                fullMask.DefaultColor = newMask.DefaultColor;
                fullMask.UserMaskRectangle = newMask.MaskRectangle;
                fullMask.UserMaskData = newMask.ImageData;
                newMask = fullMask;
            }
            else if ((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0)
            {
                // let's convert the short raster mask to a full one.
                fullMask = new LayerMaskDataFull();
                fullMask.Flags = mask.Flags;
                fullMask.MaskRectangle = mask.MaskRectangle;
                fullMask.ImageData = mask.ImageData;
                fullMask.RealFlags = newMask.Flags;
                fullMask.DefaultColor = newMask.DefaultColor;
                fullMask.UserMaskRectangle = newMask.MaskRectangle;
                fullMask.UserMaskData = newMask.ImageData;
                newMask = fullMask;
            }
        }

        // Adds or updates a mask
        layer.AddLayerMask(newMask);
    }

How to check if a layer mask is enabled?

To find out layer raster mask enabled state we can check LayerMaskFlags.Disabled flag state in the Flags property for LayerMaskDataShort or in the RealFlags for LayerMaskDataFull. The following C# .NET code snippet can be used for getting a layer mask enabled state:

Check if a mask is enabled:

bool GetRasterMaskState(Layer layer)
            {
                var mask = layer.LayerMaskData;
                var fullMask = mask as LayerMaskDataFull;
                if (mask == null || ((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0 &&
                                     fullMask == null))
                {
                    throw new Exception("This layer has no raster mask.");
                }

                if (fullMask != null)
                {
                    return (fullMask.RealFlags & LayerMaskFlags.Disabled) == 0;
                }
                else
                {
                    return (mask.Flags & LayerMaskFlags.Disabled) == 0;
                }
            }

How to enable or disable a raster layer mask?

To enable or disable a layer raster mask we can change the LayerMaskFlags.Disabled flag state in the Flags property for LayerMaskDataShort or in the RealFlags for LayerMaskDataFull. The following C# .NET code snippet can be used for changing a layer mask enabled state:

Enable or disable Raster Layer Mask:

void SetRasterMaskState(Layer layer, bool isEnabled)
            {
                var mask = layer.LayerMaskData;
                var fullMask = mask as LayerMaskDataFull;
                if (mask == null || ((mask.Flags & LayerMaskFlags.UserMaskFromRenderingOtherData) != 0 &&
                                     fullMask == null))
                {
                    throw new Exception("This layer has no raster mask."));
                }

                if (fullMask != null)
                {
                    if (isEnabled)
                    {
                        fullMask.RealFlags = fullMask.RealFlags & ~LayerMaskFlags.Disabled;
                    }
                    else
                    {
                        fullMask.RealFlags = fullMask.RealFlags | LayerMaskFlags.Disabled;
                    }
                }
                else
                {
                    {
                        if (isEnabled)
                        {
                            mask.Flags = mask.Flags & ~LayerMaskFlags.Disabled;
                        }
                        else
                        {
                            mask.Flags = mask.Flags | LayerMaskFlags.Disabled;
                        }
                    }
                }
            }