Create a privacy-preserving image style transfer application using Concrete ML

[zaccherinij]

Overview

Concrete ML makes it easier for data scientists to use Fully Homomorphic Encryption (FHE) by automating the conversion of machine learning models into their homomorphic versions.

Style Transfer, a technique in computer vision and graphics, creates a new image by merging the content of one image with the style of another image or set of images. The objective of style transfer is to produce an image that retains the original content while adopting the visual style of the other image(s).

Several neural network architectures can be used for style transfer, including:

• Convolutional Neural Networks (CNNs), like VGG-16 and VGG-19.
• Encoder-Decoder Networks.
• Perceptual Loss Networks, which use a loss function based on feature maps from a pre-trained network.
• Transformers, especially Vision Transformers (ViTs).
• Neural Style Transfer (NST), which optimizes an image to match content and style features from a pre-trained CNN.

Style transfer often involves processing personal images, such as photos of individuals, which may contain sensitive information. Protecting the privacy of these images is crucial to prevent unauthorized access or misuse. Unauthorized access to images can lead to various security risks, such as identity theft, phishing attacks, or other malicious activities.

Your challenge is to implement a style transfer application using Concrete ML, focusing on maintaining data privacy throughout the process. Your task is to develop a pipeline that takes an image and changes its style. The style can be defined either in the model or by using a second image. In the second case, one image is used for content and one for style. The system should generate a new image that combines the content of the first with the new style, all while keeping the data secure and encrypted.

What we expect

We expect you to demonstrate the effectiveness of your solution by providing examples and discussing tradeoffs you made to make your model compatible with FHE computation.

To make the task manageable on common PCs, the input images can be small (for example, 32x32 or 48x48 which works well for portrait pictures). You may perform some pre-processing steps on clear data before running the FHE-style transfer circuit. We suggest that approaches that only perform a forward-pass of a neural network to do style transfer should be the most amenable to work with the quantization necessary to use FHE.

We expect that your submission contains:

• A report on the method and technical choices you made
• A notebook showing off your model on some images

Reward

🥇Best submission: up to €5,000.

To be considered the best submission, a solution must be efficient, and effective and demonstrate a deep understanding of the core problem. Alongside the technical correctness, it should also be submitted with a clean code, clear explanations, and a complete documentation.

🥈Second-best submission: up to €3,000.

For a solution to be considered the second-best submission, it should be both efficient and effective. The code should be neat and readable, while its documentation might not be as exhaustive as the best submission, it should cover the key aspects of the solution.

🥉Third-best submission: up to €2,000.

The third-best submission is one that presents a solution that effectively tackles the challenge at hand, even if it may have certain areas of improvement in terms of efficiency or depth of understanding. Documentation should be present, covering the essential components of the solution.

Reward amounts are decided based on code quality, model accuracy scores, and speed performance on a m6i.metal AWS server. When multiple solutions of comparable scope are submitted they are compared based on the accuracy metrics and computation times.

Related links and references

• Concrete ML documentation
• Concrete ML repository

👉 Register

Step 1: Registration

Click here to register for the Bounty that you want to participate. Fill out the registration form with your information. Once you fill out the form, you will receive a confirmation email with a link to the submission portal for when you are ready to submit your code.

Note

Check your spam folder in case you don't receive the confirmation email. If you haven't received it within 24 hour, please contact us by email at [email protected].

Step 2: Work on the Challenge

Read through the Bounty details and requirements carefully. Use the provided resources and create your own GitHub repository to store your code.
If you have any questions during your work, feel free to comment directly in the Bounty issue and our team will be happy to assist you.

Step 3: Submission

Once you have completed your work, upload your completed work to the submission portal using the link provided in the confirmation email.

Note

The deadline for submission is September, 8th 2024 (23:59, Anywhere On Earth). Late submissions will not be considered.

We wish you the best of luck with the challenge!

✅ Support

• Comment on this issue with any questions regarding this bounty.
• Email for private questions: [email protected].
• Join the Zama community channels here.

[c-gamble]

Ran into this issue

[bcm-at-zama]

@c-gamble , I moved your issue / question here, zama-ai/concrete-ml#759, to keep clarity in the bounty issue. Your question is more related to Concrete ML than to the bounty, you could have had the same issue working on another subject.

[bcm-at-zama]

@c-gamble : would you mind removing #127 (comment) comment please, and replace it by a link to zama-ai/concrete-ml#759 please?

[c-gamble]

Done @bcm-at-zama!

[pandasamanvaya]

Hey @bcm-at-zama. Will a notebook using a fork of Concrete ML(consisting few additions) considered a valid submission?

[andrei-stoian-zama]

Hi, yes, we can consider a submission with a fork of CML, if you can make a PR on those diffs to the CML repo (e.g. it passes tests). If your submission is considered as a winning one, you will need to go through the process of merging those diffs in the public CML.

[bcm-at-zama]

Thanks @andrei-stoian-zama

@pandasamanvaya : How many differences do you have in your fork? Can I ask on what subject(s) they are? Why did you need this differences? To understand what's missing / wrong in current Concrete ML. Thx

[pandasamanvaya]

Hey @bcm-at-zama. Some of the ONNX operators like sqrt etc. are missing. So, I was thinking of adding them in a fork which I could use in my notebook

[bcm-at-zama]

OK! If you need support or want to apply your merges directly in Concrete ML, make PRs in https://github.com/zama-ai/concrete-ml. The developers will guide you, eg making the right choices or tests.

[pandasamanvaya]

Thanks @bcm-at-zama for the suggestion. I would be happy to make any contribution to Concrete ML.