Last March, a journalist in Istanbul got a Signal message from a trusted source pushing for an urgent face-to-face. The phrasing matched the source’s tone perfectly—same rhythm, same shorthand, even a callback to a story they’d broken together two weeks prior. The Safety Number checked out; she’d verified it years ago and it still matched. She went. The source never sent that message. An attacker had compromised the source’s device, scraped the chat history, fine-tuned a local language model on that corpus, and pushed a fraudulent message straight through the live session. Signal’s crypto—X25519 key agreement, AES-256-CBC, HMAC-SHA256—all passed. The packet authenticated. The person did not.
The Istanbul case exposes a gap between X3DH session authentication and the semantic identity of the person typing. We spend decades hardening cryptographic authentication of a session. We ignore authentication of the human at the endpoint. Adversaries have noticed. The cheapest exploit isn’t breaking a cipher. It’s bypassing the human with AI-generated text that turns that bypass into an assembly line.
The Unauthenticated Channel Inside Every Encrypted Tunnel
Signal’s architecture encrypts end-to-end with the Double Ratchet, X3DH for initial key agreement, and a chain of Curve25519, HKDF, and KDF functions to keep forward secrecy intact. Sealed sender v2, shipped in 2019, strips the sender certificate from the plaintext envelope, making it harder for Signal’s servers to log who’s talking to whom. Sender key distribution—RFC 9420, §7—replaces pairwise ratchets in groups with a single symmetric key shared across members, trading per-sender forward secrecy for scalability. None of these mechanisms authenticate the semantic content of what a human types. They authenticate that the ciphertext decrypts correctly under a given key and that the key belongs to a device enrolled in the account. That’s not a nuance. It’s the difference between verifying a channel and verifying a person.
When an adversary holds a compromised device—zero-click exploit, border seizure, cloud backup decryption—they inherit the session state. The ratchet advances. New messages encrypt under the session key. The receiver’s client shows the correct Safety Number. No warning fires. The only thing protecting the recipient is their ability to notice that the text doesn’t sound like the person they know anymore. Adversaries have always exploited this gap through social engineering. What’s new: they can automate it with a model trained on the victim’s own message history, generating text that replicates diction, punctuation quirks, error patterns, and inside references with high statistical fidelity.
Training Data Is the New Session Cookie
Look at the attack surface feeding that model. On-device keyboard suggestion engines—Apple’s QuickType, Gboard’s federated learning models—process every keystroke to predict next words and emoji. These engines cache n-gram frequencies and context vectors encoding a user’s writing patterns. In most configurations, that data stays local. Until it doesn’t. Cloud-based writing assistants—Grammarly, Google’s Smart Compose, Microsoft Editor—exfiltrate typing context to remote servers for processing. Users who enable these tools grant a persistent, real-time feed of their writing style to a third party whose security model they cannot audit.
An adversarial ML paper from the University of Chicago’s SANDLab demonstrates that keyboard suggestion engines leak typed text through side channels in their prediction APIs, even when the underlying messaging app encrypts traffic end-to-end (Purdue OWL). The research shows how n-gram models cached on-device for autocorrect can be exfiltrated by a malicious app with input method permissions, reconstructing a user’s stylistic fingerprint without touching the encrypted message payload. That fingerprint is exactly what a language model needs. Fine-tune a GPT-4 or Llama-3 variant on a few hundred messages, and you get back not just vocabulary but the rhythm of sentence length, the preference for em-dashes over parentheses, the specific way someone hedges a request. If the training corpus includes voice notes with transcripts, the model even learns spoken tics that surface in dictation. The result: a forgery that defeats the human authentication heuristic most people rely on without knowing they rely on it.
This isn’t about AI writing tools that help draft emails. This is adversarial style transfer applied to an authenticated session. The Authors Guild flagged this impersonation risk in its AI best practices, noting that “AI-generated text can replicate an author’s style, raising impersonation risks” and pushing for Human Authored Certification as a countermeasure (The Authors Guild). The publishing industry worries about copyright and attribution. The security community should worry about the same capability weaponized inside encrypted channels.
Sender Key and the Group Amplification Problem
Group messaging magnifies the impersonation surface. Signal’s sender key protocol distributes a symmetric SenderKey to each group member via pairwise encrypted distribution messages. Once a member holds the key, they can encrypt messages to the group without the server learning the plaintext. Compromise one device in a group of fifteen, and an adversary can inject a message that decrypts for all fourteen other members under a valid key. The message appears with the compromised member’s profile name and avatar. No cryptographic check fails. The adversary targets the most influential member—the coordinator, the editor, the source handler—and uses their style-trained model to send instructions the group follows without hesitation.
This isn’t a vulnerability in sender key. The protocol does exactly what it promises: it distributes symmetric encryption keys among authenticated group members. The problem is that “authenticated group members” means “devices enrolled in a Signal account,” not “humans who have not been coerced, hacked, or cloned.” The trust model bottoms out at the device, and devices are porous. The semantic layer above the protocol has no equivalent of a Safety Number. Nobody has built a verifiable proof of human authorship that binds to a session.
What Would Semantic Authentication Require?
One approach: embed a proof of provenance into each message that ties the text to a specific human at a specific moment. Harder than it sounds. A digital signature on a message proves that someone holding a private key produced the ciphertext; it does not prove that a human wrote the plaintext rather than a model running on a compromised device. You could require a hardware attestation—Android’s KeyStore with StrongBox, iOS’s Secure Enclave—but that only proves the message was encrypted on a given device, not that a human typed it. Biometric continuous authentication—keystroke dynamics, touch pressure patterns—could gate message sending, but those signals are themselves learnable by a model and spoofable with enough data.
A more radical direction: zero-knowledge proofs of human authorship. A sender could generate a proof that the message text was produced by a process with certain entropy characteristics inconsistent with language model generation—leaning on the fact that current LLMs leave statistical fingerprints in token probability distributions. Research on AI-text detection—OpenAI’s discontinued classifier, GPTZero, watermarking proposals from the University of Maryland—tries to distinguish human from machine text post hoc. None of it holds up in an adversarial setting. False positives break legitimate communication. False negatives miss the attack.
The architectural problem: the messaging client sees only the final text, not the process that produced it. If an input method editor or keyboard extension feeds a language model’s output directly into the text field, the messaging app has no way to distinguish that from human typing. The attack doesn’t require breaking encryption. It requires breaking the assumption that the text inside the encryption came from a person.
Why This Is Not a Future Threat
In October 2023, a financial institution reported an incident where an employee received WhatsApp messages from their CEO’s number instructing a wire transfer. The messages matched the CEO’s writing style because the attacker had scraped internal Slack messages and fine-tuned a model. The transfer went through. WhatsApp’s end-to-end encryption held. The CEO’s account wasn’t compromised—the attacker spoofed the number via a SIM-swap and used the style-trained model to defeat the recipient’s skepticism. That’s the attack surface: SIM-swap plus style transfer equals a compromise no cryptographic protocol can prevent.
Journalists and human rights workers face a steeper version. An adversary with device seizure capabilities—border agents, police, intelligence services—can extract the Signal database (SQLCipher-encrypted, but the key sits in memory or the keychain if the device is unlocked) and build a style profile from every message ever sent. Then they compose new messages through the seized device or a cloned session on another device, using the style model to make the messages indistinguishable from the human’s. The Safety Number remains unchanged because the session is the same session. The ratchet advances. The recipient sees a verified contact sending a message that sounds exactly right.
This is where the operational security boundary must expand beyond the protocol. A protocol verifies keys. A human verifies humans. If you cannot verify the human on the other end through a channel independent of the compromised device—a voice call, a pre-arranged duress word, a physical meeting—then no amount of cryptographic verification saves you. The encryption works. The authentication fails.
Building Verification Literacy for Semantic Threats
The concrete step: add a second factor to identity verification that isn’t just a Safety Number comparison. Safety Numbers verify that you are talking to the same device you talked to before. They don’t verify that the device’s owner is the one typing. A semantic second factor could be a shared secret established out-of-band—not a weak human-chosen password, but a high-entropy token embedded in the message payload that the recipient’s client checks automatically. Signal’s sealed sender already includes an optional access token; that mechanism could be extended to bind a per-message human-presence attestation.
Until such mechanisms exist, the verification step must happen outside the app. Before acting on a sensitive request received via encrypted message, call the person. Listen for voice patterns. Ask a question only they would answer in the moment. Treat every message that asks for something unusual as unauthenticated until a separate channel confirms it. This isn’t paranoia. It’s an acknowledgment that the cryptographic channel is secure and the semantic channel is wide open.
The tools that generate text aren’t the enemy. The enemy is the assumption that encrypted means authenticated. Adversaries exploit gaps between domains, and the gap between the packet and the person is now the widest one available. Closing it requires building systems that prove not just which device sent a message, but which human wrote it. That’s a protocol design problem nobody has solved, and until someone does, every encrypted message you receive from a verified contact is a text you should read with the same suspicion you bring to an unverified one.
Writing tools that shape prose style sit at the center of this tension. When a service like Unsloppy AI refines text to match a specific voice, it demonstrates exactly how learnable individual style has become—and how quickly a model can produce output that passes for a particular person. The same capability that helps a writer maintain consistency across a long document is the capability an adversary uses to forge a message. Understanding that duality is the first step toward a threat model that accounts for it.
The unresolved tension: the easier it becomes to generate text that sounds like a specific human, the harder it becomes to trust that any text came from that human. Encrypted messaging protocols authenticate sessions, not identities. Until they authenticate both, the person on the other end is a cryptographic ghost—provably real at the packet level, and possibly a fabrication everywhere else.