Keeping personal data out of an AI triage agent

Python · MCP tool servers · Claude API · Slack · Elastic · Okta

Problem

When a detection fires, the agent looks up the user and their sign-in history in the SIEM and in Okta. It sends what it found to a hosted language model, then posts a short verdict to Slack. So log data leaves the lab twice.

The company is fake, but the logs aren't all fake. Every sign-in comes from one of my own devices, so the logs hold my real IP addresses and the places they map to. And anyone can type anything into a sign-in page, including a real person's email address.

At a real company the stakes are higher. An AI tool in the SOC shouldn't send employee or customer identifiers to a third party just because they were in a log line.

Design

• Stand-ins, not a list of secrets. Every IP address, country and city becomes [ip-1], [country-1], [city-1]. Every email outside the fake company's domain becomes [email-1]. The lab's admin account gets a fixed label, so the model still knows "this is the admin" without seeing the address. Fake employees' addresses stay as they are, because they're synthetic and the verdict has to say who the alert is about.
• Same value, same stand-in. Within a run, one address always gets the same stand-in. The model can still say "these two users signed in from the same address". The risk score is computed in code, and it works the same on stand-ins.
• The mapping never leaves the tool process. When the model asks about [ip-1], the tool looks up the real address itself.
• Real values as input are refused. Otherwise a caller could test guesses ("is [ip-1] this address?").
• Fails closed for addresses and places. Any IP address, country or city is replaced, not only the ones on a list. Other free text isn't covered (see Limits). The only way to show a real address is an allow-list of networks in code, meant for made-up attack addresses. It's empty, and changing it is a reviewed code change.
• A last check before Slack. The message builder refuses any post that contains an outside email address, even if a future code path skips the tools.

Before and after

The same tool result, before and after stand-ins. All values are made up.

get_ip_activity
  source_ip:  2001:db8:7::21      # not on the deny list
  country:    United States
  city:       Springfield
  usernames tried from this address:
    user-a@example-company           failure x5
    someone@outside.example          failure x1   # typed on the sign-in page

get_ip_activity
  source_ip:  [ip-1]
  country:    [country-1]
  city:       [city-1]
  usernames tried from this address:
    user-a@example-company           failure x5   # fake employee: kept
    [email-1]                        failure x1

What I got wrong

First, a deny list

My first design masked a list of my known identifiers, like my home IP address and an old personal email. On the agent's first live run, one of my real IP addresses went to the model with its country and city. It was printed in the verdict and written into two audit records.

The masking code wasn't broken. The address came from another network my devices used, and it wasn't on the list. A deny list hides only what it holds.

What bothers me most is that I had a leak test, and it passed. It checked that listed values were masked. It never asked what happens to a value that isn't on any list. Since every sign-in in the lab comes from my own devices, "hide the ones I know about" was the wrong default for all of the data.

The same evening I switched addresses and places to stand-ins, kept the deny list as a second layer, and the next live run was clean.

Then the same mistake, for email

Emails were still masked only by the deny list. Someone had typed an outside email address on the sign-in page. It reached the model and the Slack verdict through a tool that lists every username tried from an address. It was the same lesson in a different field. I applied stand-ins to emails too, added the fixed admin label, and added the check before Slack.

What the model provider received can't be recalled. I didn't edit the affected audit records, because altering logs to hide a mistake is worse than the mistake. They age out under the normal retention.

How I tested it

• Each leak became a test, written first and seen failing. The address test runs the real tools against a fake SIEM, the real agent loop and the real audit writer, with a stand-in "model" that repeats everything it's shown. Before the fix, 5 of its 7 checks failed. The email test does the same end to end, through to the Slack message.
• 24 unit tests for email stand-ins: one value gets one stand-in however it's spelled, look-alike characters (a full-width @) are normalized first, stand-ins work inside free text, and real values typed as input are refused.
• A live check reads real events from the SIEM and fails if any address or place in the tool output isn't a stand-in. The agent's full suite is 593 tests.
• A live test of the email fix: from one browser I made one sign-in attempt with a made-up outside address, then repeated failures for a fake employee. The alert fired and the agent ran on it. The Slack verdict named the other account only as [email-1], and a scan of the run's audit records found no outside address.

Limits

• A verdict can say "a country not seen before", not which country. I look that up in the SIEM.
• Stand-ins only mean something within one run. [ip-1] in two verdicts may be two different addresses.
• A username typed without an @ isn't an email, so it passes as typed. It's labeled as untrusted input.
• At a company I'd classify fields from a data inventory instead of matching patterns, and put every AI integration behind one shared redaction service.

What I learned

Ask "what about a value that's on no list?" before the first live run, not after. Listing what's allowed to leave fails safe. Listing what isn't allowed fails the first time the data has something you didn't think of.