As the tech industry races to develop new AI models, Google alleges that “private sector” entities have been trying to reverse-engineer its Gemini chatbot by bombarding it with prompts intended to leak its secrets.
As PCMag reports, Google floated the accusation in a report issued last week that examined malicious activity targeting the chatbot, which has received positive reviews and a flood of new traffic after it was updated with the Gemini 3 model.
Google’s report, which covers Q4, doesn’t name the offenders. But the company says it “observed and mitigated frequent model extraction attacks from private sector entities all over the world and researchers seeking to clone proprietary logic.”
Model extraction isn’t your typical hacker-led “break-in.” Instead of exploiting a software glitch or infiltrating a corporate network, these attacks leverage legitimate access via Gemini’s API, which Google sells to software developers who want to build their apps around the chatbot.
Typically, Gemini jumps to the final answer after processing the user’s prompt, bypassing the “show your work” thinking behind the output. However, Google claims private-sector entities have been trying to “coerce the model into outputting full reasoning processes” by submitting carefully crafted prompts designed to reveal the chatbot’s internal logic.
For example, “One identified attack instructed Gemini that the ‘…language used in the thinking content must be strictly consistent with the main language of the user input,’” the company noted. Over 100,000 prompts were tied to the model extraction attacks, the report added. “This activity effectively represents a form of intellectual property (IP) theft,” Google alleges.
We’ll be curious to see whether the company takes legal action and, if so, what the basis for a suit would be. But for now, Google says the activities are a Terms of Service violation, enabling it to pull the plug on offending users.
In addition, the company is warning other AI developers to watch out for model extraction attacks. Although the threat doesn’t affect consumers, the company fears that a private entity could learn Gemini’s trade secrets, apply them to its own AI models, and then offer them however it’d like, including for shady or malicious purposes.
“For example, a custom model tuned for financial data analysis could be targeted by a commercial competitor seeking to create a derivative product, or a coding model could be targeted by an adversary wishing to replicate capabilities in an environment without guardrails,” the company adds.
—
Photo Credit: rafapress / Shutterstock.com