Synthetic security logs for an Elastic SIEM lab — Check Point, Nginx, IIS, Windows / Active Directory, CrowdStrike Falcon and SentinelOne — plus the detection rules, ML jobs and Kibana assets to detect the attack they describe.
Seven products, one incident: the perimeter shows the break-in, the web servers show what got through, the endpoints show what ran, and Active Directory shows where it went.
Every log is emitted in the product's own native format and parsed by the official Elastic integration pipeline. There is no hand-written ingest pipeline anywhere in this repository, which is what keeps the field names and values honest.
Most synthetic log generators invent their field names. The result looks plausible in Discover and then quietly fails: detection rules never fire because they search for values the real product never produces.
This project takes the opposite approach. Each generator writes the raw artifact a real product would write, hands it to the vendor's own Elastic parser, and lets that parser decide what the ECS fields become.
A few things that were wrong when this was built by assumption, and what the real products actually emit:
| Field | Assumed | Actual |
|---|---|---|
checkpoint.severity |
"High", "Critical" |
a number, "1"–"5" |
checkpoint.confidence_level |
"High" |
a number, "1"–"5" |
checkpoint.protection_type |
"signature" / "none" |
"IPS" / "anomaly" |
checkpoint.protection_name |
"SQLi", "XSS", "RCE" |
"Command Injection Over HTTP Payload", "Web Servers Directory Traversal", … |
http_status (Check Point) |
present | does not exist — status codes belong to the web server |
event.action (Windows) |
"logon-success", "process-creation" |
logged-in, created-process |
event.dataset (Windows) |
"ad.security_csv" |
system.security |
SeverityName (CrowdStrike) |
a number | a word, and the word is what drives event.severity |
ruleInfo.severity (SentinelOne) |
a number | a word - low/medium/high/critical |
The Check Point severity case is the instructive one. The official pipeline
runs convert checkpoint.severity -> event.severity type: long. Feed it the
word "Critical" and the conversion fails silently — event.severity is
simply never populated, and every rule that filters on severity goes dark.
The endpoint products invert that convention, which is precisely why guessing
does not scale across vendors. CrowdStrike and SentinelOne both key
event.severity off the word (Low → 21, Medium → 47, High → 73,
Critical → 99) and send a number along as a secondary field. Same idea,
opposite convention, and three different numeric scales between the three
products.
One more CrowdStrike trap worth knowing: its pipeline strips any
crowdstrike.event field whose value is 0, along with null, "", "-",
"N/A" and "NA". A rule searching for LateralMovement:0 matches nothing,
because by the time the rule runs the field no longer exists.
| Source | Format | Data stream |
|---|---|---|
| Check Point | Log Exporter syslog (RFC 5424) | logs-checkpoint.firewall-* |
| Nginx | combined access log + error log | logs-nginx.access-*, logs-nginx.error-* |
| IIS | W3C Extended (IIS 7+ default fields) | logs-iis.access-* |
| Windows Security | Winlogbeat-shaped documents | logs-system.security-* |
| Windows System / Application | Winlogbeat-shaped documents | logs-system.system-*, logs-system.application-* |
| CrowdStrike Falcon | SIEM Connector JSON envelope | logs-crowdstrike.falcon-* |
| SentinelOne | threat and alert API JSON | logs-sentinel_one.threat-*, logs-sentinel_one.alert-* |
Check Point blades: Firewall (VPN-1 & FireWall-1), IPS (SmartDefense),
URL Filtering, Anti-Bot, Identity Awareness, SmartConsole audit.
Windows Event IDs: 4624, 4625, 4634, 4647, 4672, 4688, 4698, 4720, 4728, 4740, 4768, 4769, 4771, 4776, 4798, 1102 (Security); 7045, 7036, 7040, 6005 (System).
CrowdStrike event types: EppDetectionSummaryEvent, IncidentSummaryEvent, FirewallMatchEvent, UserActivityAuditEvent, AuthActivityAuditEvent.
All sources tell one story, so Attack Discovery has something to correlate:
1. Recon Check Point IPS: port scan + scanner signatures
Nginx / IIS: 404 sweep for .env, .git/config, wp-login.php
2. Initial Access Check Point IPS: SQLi / XSS / command injection / traversal
Nginx: 403 for what the gateway blocked, 500 for what got through
3. Foothold IIS: web shell reached at /app/upload/report.aspx (200)
Windows 4688: cmd.exe, net.exe, powershell.exe under w3wp.exe
4. Credential Windows 4771 / 4625 / 4776: password spray against the DC
Access Windows 4740: a legitimate account locked out as collateral
Windows 4769: RC4 service ticket — Kerberoasting
5. Escalation Windows 4624 + 4672: successful logon with elevated token
6. Persistence Windows 4720 + 4728: new account added to Domain Admins
Windows 7045: service installed from C:\Windows\Temp
Windows 4698: scheduled task with an encoded PowerShell payload
7. Evasion Windows 1102: Security log cleared
8. C2 Check Point Anti-Bot: command-and-control traffic
CrowdStrike FirewallMatchEvent: beacon leaving the network
Check Point audit: failed SmartConsole logins from the pivot
The two EDR agents describe the same incident from the endpoint:
CrowdStrike cmd.exe / nltest.exe spawned under w3wp.exe (High)
credential theft tool executed and NOT blocked (Critical)
service binary masquerading in C:\Windows\Temp (High)
wevtutil clearing the Security log (Defense Evasion)
console brute force, then a detection edited from the pivot
CrowdScore incident with LateralMovement set
SentinelOne credential dumping tool, malicious and NOT mitigated
STAR rule: encoded PowerShell under the IIS worker (High)
STAR rule: remote service creation on the DC (Critical)
Stage 2 is worth a note. In Check Point, Prevent versus Detect is a setting on the protection inside the Threat Prevention profile, not something that varies per request. Protections left in Staging/Detect forward the payload to the origin — which is exactly why the same request appears in both the Check Point log and the Nginx log, and why cross-source correlation has anything to work with.
- Python 3.9+ (no third-party packages for the generators)
requests, for the deploy script only:pip install requests- An Elastic Stack with Fleet, and permission to install integration packages
# 1. Install the integration packages (REQUIRED FIRST — see below)
export KIBANA_URL="https://kibana.lab.example:5601"
export ES_URL="https://elasticsearch.lab.example:9200"
export ES_API_KEY="<base64_api_key>"
python3 deploy_elastic_assets.py --step packages
# 2. Generate and ship the logs
python3 checkpoint_log_generator.py --push \
--es-host "$ES_URL" --es-api-key "$ES_API_KEY" \
--namespace seclab --no-verify-tls --days-back 5
python3 ad_log_generator.py --push \
--es-host "$ES_URL" --es-api-key "$ES_API_KEY" \
--namespace seclab --no-verify-tls --days-back 5
python3 edr_log_generator.py --push \
--es-host "$ES_URL" --es-api-key "$ES_API_KEY" \
--namespace seclab --no-verify-tls --days-back 5
# 3. Create the ML jobs, now that the datafeeds have history to learn from
python3 deploy_elastic_assets.py --step ml-jobs
# 4. Create the detection rules
python3 deploy_elastic_assets.py --step rulesStep 1 is not optional. Without the integration packages, the index templates
and their default_pipeline do not exist, so documents land with dynamic
mappings and are never parsed.
Inspect before sending anything:
python3 checkpoint_log_generator.py --dry-run
python3 ad_log_generator.py --dry-run
python3 edr_log_generator.py --dry-run
python3 deploy_elastic_assets.py --dry-run
python3 deploy_elastic_assets.py --list-rulesA — syslog to an Elastic Agent (the real production path for Check Point):
python3 checkpoint_log_generator.py --send-syslog \
--target-host 10.10.5.20 --target-port 9001 --protocol udpB — Bulk API straight into the official data stream (for labs with no Agent):
python3 ad_log_generator.py --push --es-host ... --namespace seclabEither way the integration's own pipeline does the parsing. Nginx and IIS logs can also be written to disk as real files for an Agent file input to tail:
python3 ad_log_generator.py --iis-file u_ex.log
python3 checkpoint_log_generator.py \
--nginx-access-file access.log --nginx-error-file error.log42 rules across four types. --list-rules prints the full table.
| Type | Count | Used for |
|---|---|---|
query |
30 | single-event signatures |
threshold |
6 | brute force, scanning, error bursts |
machine_learning |
4 | volume anomalies against the learned baseline |
eql |
2 | cross-stage and cross-product correlation |
The EQL rule is the one that matters most for Attack Discovery — it collapses three separate alerts into one incident:
sequence by host.name with maxspan=2h
[ any where event.code in ("4625", "4771", "4776") and event.outcome == "failure" ]
[ any where event.code == "4624" ]
[ any where event.code in ("4728", "4732", "4756") ]
Brute force is a threshold rule, not a query rule, and that is deliberate.
The normal baseline genuinely contains the occasional 4625 — users mistype their
passwords — so a plain event.code:"4625" rule false-positives forever.
Three rules use field names that only exist after the Check Point pipeline runs, because that pipeline renames them:
| Check Point sends | The rule must use |
|---|---|
checkpoint.malware_action |
event.category:"malware" |
checkpoint.matched_category |
rule.category |
checkpoint.client_ip |
source.ip |
These survive under checkpoint.*: attack, attack_info, protection_name,
protection_type, confidence_level, operation, audit_status,
administrator.
| Job | Detector | Datafeed filter |
|---|---|---|
seclab_checkpoint_ips_anomaly |
high_count |
checkpoint.protection_name exists |
seclab_windows_auth_failure_anomaly |
high_count |
codes 4625/4771/4776, outcome failure |
seclab_web_http_error_anomaly |
high_count |
http.response.status_code >= 400 |
seclab_edr_detection_anomaly |
high_count |
event.kind: alert across both EDR products |
None of them partitions by user or source IP, which is intentional. The
usernames a password spray tries (administrator, svc-sql, backupadmin)
never appear in the baseline, so a per-user partition leaves the model
cold-started and it misses the burst entirely. Counting total failures per
bucket across all users is what actually detects it.
Nothing here was inferred from memory. The sources:
- Check Point format and field names — the grok and
kvprocessors of the official Elasticcheckpointintegration package, cross-checked against real Check Point logs in Elastic's test corpus - Check Point protection names — Check Point Advisories (CPAI-2018-0567, CPAI-2023-0799, CPAI-2025-0043)
- Check Point severity / confidence scales — sk116254
- Windows event structure — real captured events in Elastic's Winlogbeat test corpus
- Windows EventData fields — Microsoft Learn Security Auditing, e.g. Event 4624
- Event ID → ECS mapping — not written by hand; produced by the official
system.securitypipeline - Nginx and IIS formats — the official grok patterns of the Elastic
nginxandiispackages, plus Configure Logging in IIS - CrowdStrike and SentinelOne — the real fixtures and ingest pipelines of
the Elastic
crowdstrikeandsentinel_onepackages. Every enumerated value (tactic, technique, objective, threat classification, mitigation status, severity scale) is one that actually appears in those fixtures. Where no authoritative public list exists — SentinelOne's numericactivityTypecodes — that data stream is simply not generated rather than filled in from memory
Each generator's module docstring cites the specific source for its own fields.
The output was checked against the official patterns rather than eyeballed:
269/269 Check Point lines parsed by the official grok + kv
200/200 Nginx access lines parsed by the official grok
48/48 Nginx error lines parsed by the official grok
183/183 IIS access lines parsed by the official grok
76/76 Check Point fields present in the official fields.yml
327/327 CrowdStrike records every field path exists in the official corpus
111/111 SentinelOne records every field path exists in the official corpus
42/42 detection rules match the generated data (across 5 random seeds)
The rule check simulates the pipeline transformations on the generated data and evaluates every rule's KQL against the result. It caught three rules that would have silently matched nothing — the renamed Check Point fields listed above.
All four files are clean under pyright --strict.
Every address and domain in this repository is drawn from ranges reserved for documentation, so nothing here labels real infrastructure as malicious:
- RFC 5737 —
192.0.2.0/24(legitimate visitors),198.51.100.0/24(opportunistic scanners),203.0.113.0/24(the attacker) - RFC 2606 — the
.exampleTLD, both for the C2 and phishing domains and for the Active Directory domain itself (corp.example) - RFC 1918 —
10.0.0.0/8for the internal lab network
One consequence: geoip enrichment will not resolve for these addresses, so
source.geo.* stays empty. Substitute a routable range in ATTACKER_IPS /
SCANNER_IPS if a demo needs the map populated.
Everything else is a neutral placeholder as well — hostnames (DC01,
WEB-PROD-01, cp-gw01), accounts (j.smith, m.jones, svc-backup,
cpadmin) and the CORP NetBIOS name. Change them in the constants at the top
of each generator to match your own lab.
| File | Purpose |
|---|---|
checkpoint_log_generator.py |
Check Point Log Exporter syslog + Nginx access/error |
ad_log_generator.py |
Windows Security / System / Application + IIS access |
edr_log_generator.py |
CrowdStrike Falcon + SentinelOne threat and alert |
deploy_elastic_assets.py |
packages, space, data views, ML jobs, 42 detection rules |
ad_log_generator_csv.py |
superseded — the original CSV approach, kept to document what changed and why |
ad_log_generator_csv.py depends on a hand-written ad-csv-pipeline that the
deploy script no longer creates, so its --push will not parse. It is kept as a
worked example of the failure mode this project exists to avoid.
Every document is tagged, so lab data is easy to separate:
tags: "SECLAB" every run
tags: "ATTACKSIM-<id>" one specific run (printed when the generator finishes)
On the syslog path the marker goes into Check Point's comment field — a field
that genuinely exists and holds free text — rather than an invented one, so
nothing fabricated ends up in the checkpoint.* namespace.
python3 deploy_elastic_assets.py --step cleanupRemoves the rules, ML jobs and data streams. Leaves the space, the data views and the integration packages alone, since those are safe to reuse. A 404 on any line is normal — it just means that asset was never created.
MIT