Okta in Terraform, and a detection I tested both ways

Okta · Terraform · Elastic · Sigma

Problem

Most attacks on a company go through its identity provider. I wanted my fake company's Okta setup to be reviewable. When someone adds a user or changes a group by hand, nobody reviews it and there's no record of why.

I also wanted my first detection to be tested. A rule that has never run against real events can miss the attack. It can also fire so often that people stop reading it.

What I built

The directory has 30 fake employees in 8 departments and 13 groups. A CSV file plays the HR system. To add or remove someone, I edit the CSV, run terraform plan, read the plan and then apply it. If anyone changes Okta by hand, the next plan shows the difference.

The Terraform API token only works from my home network. Admin sign-in uses Okta push with number matching, so an attacker can't send a stream of push prompts and hope I tap yes.

Okta's audit log goes into Elastic through Elastic Agent. The agent signs in as its own OAuth app with one scope, okta.logs.read. I didn't reuse the Terraform token for this, because that token can create and change users.

The rule looks for 5 or more failed sign-ins for one user within 5 minutes, which suggests someone is guessing a password (MITRE ATT&CK T1110.001). I wrote the base query in Sigma and converted it for Elastic. The count lives in the Elastic rule. I tested two Sigma backends, and neither could produce a Kibana threshold rule.

$ sigma convert -t lucene -p detections/pipelines/okta-ecs.yml \
    detections/okta/okta_repeated_failed_logins.yml
event.dataset:okta.system AND (event.action:user.session.start AND event.outcome:failure)

# Elastic threshold rule on top of that query:
#   group by   user.name
#   threshold  >= 5
#   runs every 5 minutes, looks back 6 minutes

A script deploys the rule with an API key that can only manage detection rules. The key expires after 90 days.

How I tested it

Test cases next to the rule

I keep test cases in a file next to the rule. Each case says whether the rule should fire, and why.

fire     five failures for one user in four minutes
         5 matching events, same user.name

no_fire  four failures, then a success (forgot the password)
         only 4 failures; the success doesn't match

no_fire  five failures spread over five users (spraying)
         1 per user: a known gap, spraying needs
         its own rule, grouped by source.ip

events in each case look like:
{"event": {"dataset": "okta.system", "action": "user.session.start",
           "outcome": "failure"},
 "user": {"name": "user-a@example-company"},
 "okta": {"outcome": {"reason": "VERIFICATION_ERROR"}},
 "source": {"ip": "<documentation range>"}}

The third case is a gap I know about. Password spraying tries many users once each, so a per-user count never reaches 5. Catching it needs a separate rule that groups by source IP.

Live test with real Okta events

Then I ran it against real sign-ins with two test users. User A failed 6 times in 31 seconds and got one alert with a count of 6. User B failed 4 times and got no alert. A few minutes later, user A failed 7 more times and got a second alert with a count of 7. The rule counted the two bursts separately.

Both test users were staged, so Okta saw them as users who can't sign in. These weren't real wrong passwords, and no lockout could happen. Testing a real wrong password and a lockout is my next step.

Revoking the collector's access

I also tested what happens when the collector loses access. I deactivated its Okta app, and Okta refused it 2 seconds later. When I turned the app back on, access didn't come back. Deactivating had removed its admin role and its scope grant. The admin console still showed the scope as granted, but Okta's own log showed it wasn't.

After I restored both, the collector picked up from its bookmark. It caught up on 31 events with no gaps and no duplicates. I wrote a recovery checklist from what I found.

What I learned

• The cases where the rule stays quiet matter as much as the one where it fires. Without the forgot-my-password case, I'd have no evidence the threshold is right.
• I need to say what a test didn't cover. My failed sign-ins came from staged users, so I still haven't seen a real wrong password.
• When the admin console and the audit log disagree, I go with the log.
• Rules should match on the user's ID. I renamed my own admin account during setup, and the older events still show the old name.
• It pays to check tool limits on day one. Learning early that Sigma couldn't express the threshold decided where each part of the rule lives.