The Backdoor in a Fine-tuned Model —Is Your Data Safe?

Imagine downloading a ready-made model to fine-tune it for your specific needs. You then share the model, but what if that model was secretly designed to steal your data? Recent findings reveal a shocking new threat: hidden “privacy backdoors” in pretrained models. These sneaky backdoors can allow attackers to reconstruct your sensitive information—even if you’ve taken steps to protect it.

A recent paper titled ‘Privacy Backdoors: Stealing Data with Corrupted Pretrained Models’ is authored by Shanglun Feng and Florian Tramer. It has been showing these threats.

Mary, an expert machine learning engineer with extensive experience in deep learning and transformers, plays the role of the ‘attacker’ in our story. With a calculated plan, she decides to implant a privacy backdoor into a model she is preparing to upload to Hugging Face. She names it ‘The Best PII Classifier,’ a transformer-based classifier designed to detect whether any given text contains Personally Identifiable Information (PII). Using her advanced skills, Mary adds a hidden trapdoor into the model, enabling it to retain the original data from any fine-tuning process as part of the model itself. Later, when the model becomes accessible to her, she will be able to retrieve this sensitive information. Once everything is in place, Mary uploads the compromised model to Hugging Face, ready to exploit her creation.

Unaware of Mary’s intentions, John downloads her ‘Best PII Classifier’ model to his laptop. He decides to fine-tune the model using his own PII data to enhance its performance before putting it to use. When he runs the model, he observes the expected improvement and, satisfied with the results, decides to share his work with other developers. He uploads the fine-tuned model back to Hugging Face. However, unbeknownst to him, this action has compromised his PII data.

How it’s done?

How is it done? Here’s the idea in a nutshell. Let’s demonstrate this using one of the simplest neural networks, the Multilayer Perceptron (MLP). See the video below.

References:

Feng, S., & Tramèr, F. (2024). Privacy Backdoors: Stealing Data with Corrupted Pretrained Models. arXiv preprint arXiv:2404.00473.‏

Paper explained by Yannic Kilcher

All images presented in this content were generated using DALL-E, an AI image creation tool developed by OpenAI. These images are not photographs and may not accurately represent real-world objects or scenes.

Tags: Finetuned Models, Privacy Backdoors, Corrupted Pretrained Models,AI