Missing logs that weren't missing
What this demonstrates: Root-cause analysis on security telemetry. I measured instead of trusting an alarming log line, built an independent check that matches every event by ID, and dropped my own conclusion when the evidence contradicted it.
Homelab project: a fake company, real systems, my own test traffic.
Problem
After I rebooted the lab, the agent that ships Okta's audit log to my SIEM logged Exporting failed. Dropping data. It had tried to send events while Elasticsearch was still starting.
Losing security logs with no warning would be serious. Every detection after that point would be missing data, and nothing would tell me. I read the line as proof of loss and wrote up a root cause: during an outage, events can be lost for good. I drafted six possible fixes.
I was wrong, and I only found out by measuring.
What I built
Before changing anything, I wrote a gap check to measure loss directly. It takes a time window and lists every event ID in Okta's audit log. Then it looks up each ID in Elastic.
A missing event is LOST if it's older than the agent's bookmark, because the agent won't fetch it again. It's PENDING if it's newer, because the next poll should pick it up.
The check only reads. It uses an Okta app that can only read logs, and a read-only Elastic account. Exit code 0 means nothing is missing, 1 means events are missing, and 2 means the check couldn't run.
It started with 25 unit tests that use fakes. The first live run failed because my test data had guessed the agent's file format wrong. I fixed the code and copied the real format into the tests.
How I tested it
17:01:09 elasticsearch exporter "Exporting failed. Dropping data."
dropped_items: 3
17:01:18 Elasticsearch reachableOne boot from the record. Every boot showed the same pattern: one "Dropping data" line 21 to 25 seconds after the first poll, then Elasticsearch reachable 9 to 12 seconds later.
| Test | Okta | Elastic | Lost |
|---|---|---|---|
| History check across all 5 boots since setup | 109 | 109 | 0 |
| A 19-minute outage while the agent kept running | 23 | 23 | 0 |
| A planned outage with an agent restart in the middle | 2 | 2 | 0 |
I matched every event by its ID.
$ okta-gap-check.py --since 2026-09-28T18:00-07:00 --until 2026-10-01T10:00-07:00
Window: 2026-09-28 18:00:00.000 PDT -> 2026-10-01 10:00:00.000 PDT
Okta: 109 events
Elastic: 109 of them
Missing: 0 lost, 0 pending
exit 0The labels follow the check's real output format, and the window and counts come from the investigation record. I left out the Bookmark line, and the command line is illustrative.
One boot settled it. The dropped batch held 3 events: the bookmark event and two new ones. Both new events were stored 12 seconds after the "Dropping data" line and 3 seconds after Elasticsearch came up. That was before the next poll. If my theory were right, only the newest event could have come back, and only on that next poll. Both came back early, so the agent had retried the batch.
event event time (PT) type stored in Elastic
A 09-28 7:37:54 PM app.oauth2.token.grant.access_token 09-28 7:39:24 PM (bookmark event)
B 09-29 12:15:45 PM app.oauth2.token.grant 09-29 5:01:21 PM
C 09-29 12:15:45 PM app.oauth2.token.grant 09-29 5:01:21 PM
5:01:09 PM "Dropping data", dropped_items: 3
5:01:18 PM Elasticsearch reachable
5:01:21 PM B and C stored <- before the next poll
5:01:44 PM next pollA re-fetch only brings back the newest event, and only on the next poll. B and C were both stored 23 seconds before that poll, so the batch was retried, not lost.
In the restart test, one event was still waiting in memory when I restarted the agent. Its bookmark hadn't moved, so it fetched the event again after the restart and stored it. The bookmark moved only after that.
-
5:00:44 PM · step 1 of 7
First poll after boot
The agent asks Okta for events since its bookmark. Okta has the bookmark event plus two newer ones, B and C. Elasticsearch is still starting.
← Back -
5:01:09 PM · step 2 of 7
The scary line
The exporter can't reach Elasticsearch. It logs that it's dropping data.
Exporting failed. Dropping data. dropped_items: 3
-
my theory · step 3 of 7
What I predicted
If the batch were really dropped and the bookmark had moved on, only the newest event would come back, on the next re-fetch. One of B and C would be lost for good.
-
5:01:18 PM · step 4 of 7
Elasticsearch comes up
Nine seconds later, Elasticsearch accepts connections.
-
5:01:21 PM · step 5 of 7
B and C are both stored
Three seconds after that, both new events are in Elastic. That's before any re-fetch could happen.
B app.oauth2.token.grant stored 5:01:21 PM C app.oauth2.token.grant stored 5:01:21 PM
-
5:01:44 PM · step 6 of 7
The next poll
The poll my theory depended on comes 23 seconds too late to explain it. The batch was retried and delivered.
-
conclusion · step 7 of 7
Retracted
"Dropping data" counts a failed send attempt, not a discarded event. I struck through the root cause, kept it in the record, and closed the issue as not a bug.
The retraction
I closed the issue as not a bug and applied no fix. I kept my wrong analysis in the decision record, struck through, with a note on where it went wrong.
- I treated a log message as proof. The line counts failed send attempts. It doesn't count lost events.
- I based the theory on a batch of 1 event. With one event, a retry and a re-fetch look the same.
- I never saw the bookmark move before delivery. I only looked at it after things had recovered.
Still untested: an outage longer than the agent's memory queue can hold, and a multi-event batch across an agent restart. The gap check covers those cases. It now runs every hour and alerts me on any LOST event or if it can't run. The next write-up covers that alerting.
The investigation did find one real issue. If the agent restarts while Elasticsearch is down, it can't start until Elasticsearch is back. Collection pauses, but nothing is lost, because it resumes from its bookmark.
What I learned
- A log line is something to check. It isn't a finding.
- The gap check took less time to build than any of the six fixes, and it showed I didn't need them.
- My small samples couldn't tell the two explanations apart. I needed the one case where they predicted different results.
- I keep wrong analysis in the record. Whoever reads it next, including me, can see what I got wrong and why.