Skip to content

About

Synthetic security logs for an Elastic SIEM lab

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Elastic Attack Simulation Log Generator

Synthetic security logs for an Elastic SIEM lab — Check Point, Nginx, IIS, Windows / Active Directory, CrowdStrike Falcon and SentinelOne — plus the detection rules, ML jobs and Kibana assets to detect the attack they describe.

Seven products, one incident: the perimeter shows the break-in, the web servers show what got through, the endpoints show what ran, and Active Directory shows where it went.

Every log is emitted in the product's own native format and parsed by the official Elastic integration pipeline. There is no hand-written ingest pipeline anywhere in this repository, which is what keeps the field names and values honest.


Why this exists

Most synthetic log generators invent their field names. The result looks plausible in Discover and then quietly fails: detection rules never fire because they search for values the real product never produces.

This project takes the opposite approach. Each generator writes the raw artifact a real product would write, hands it to the vendor's own Elastic parser, and lets that parser decide what the ECS fields become.

A few things that were wrong when this was built by assumption, and what the real products actually emit:

Field Assumed Actual
checkpoint.severity "High", "Critical" a number, "1"–"5"
checkpoint.confidence_level "High" a number, "1"–"5"
checkpoint.protection_type "signature" / "none" "IPS" / "anomaly"
checkpoint.protection_name "SQLi", "XSS", "RCE" "Command Injection Over HTTP Payload", "Web Servers Directory Traversal", …
http_status (Check Point) present does not exist — status codes belong to the web server
event.action (Windows) "logon-success", "process-creation" logged-in, created-process
event.dataset (Windows) "ad.security_csv" system.security
SeverityName (CrowdStrike) a number a word, and the word is what drives event.severity
ruleInfo.severity (SentinelOne) a number a word - low/medium/high/critical

The Check Point severity case is the instructive one. The official pipeline runs convert checkpoint.severity -> event.severity type: long. Feed it the word "Critical" and the conversion fails silently — event.severity is simply never populated, and every rule that filters on severity goes dark.

The endpoint products invert that convention, which is precisely why guessing does not scale across vendors. CrowdStrike and SentinelOne both key event.severity off the word (Low → 21, Medium → 47, High → 73, Critical → 99) and send a number along as a secondary field. Same idea, opposite convention, and three different numeric scales between the three products.

One more CrowdStrike trap worth knowing: its pipeline strips any crowdstrike.event field whose value is 0, along with null, "", "-", "N/A" and "NA". A rule searching for LateralMovement:0 matches nothing, because by the time the rule runs the field no longer exists.


What it generates

Source Format Data stream
Check Point Log Exporter syslog (RFC 5424) logs-checkpoint.firewall-*
Nginx combined access log + error log logs-nginx.access-*, logs-nginx.error-*
IIS W3C Extended (IIS 7+ default fields) logs-iis.access-*
Windows Security Winlogbeat-shaped documents logs-system.security-*
Windows System / Application Winlogbeat-shaped documents logs-system.system-*, logs-system.application-*
CrowdStrike Falcon SIEM Connector JSON envelope logs-crowdstrike.falcon-*
SentinelOne threat and alert API JSON logs-sentinel_one.threat-*, logs-sentinel_one.alert-*

Check Point blades: Firewall (VPN-1 & FireWall-1), IPS (SmartDefense), URL Filtering, Anti-Bot, Identity Awareness, SmartConsole audit.

Windows Event IDs: 4624, 4625, 4634, 4647, 4672, 4688, 4698, 4720, 4728, 4740, 4768, 4769, 4771, 4776, 4798, 1102 (Security); 7045, 7036, 7040, 6005 (System).

CrowdStrike event types: EppDetectionSummaryEvent, IncidentSummaryEvent, FirewallMatchEvent, UserActivityAuditEvent, AuthActivityAuditEvent.


The attack chain

All sources tell one story, so Attack Discovery has something to correlate:

1. Recon           Check Point IPS: port scan + scanner signatures
                   Nginx / IIS: 404 sweep for .env, .git/config, wp-login.php

2. Initial Access  Check Point IPS: SQLi / XSS / command injection / traversal
                   Nginx: 403 for what the gateway blocked, 500 for what got through

3. Foothold        IIS: web shell reached at /app/upload/report.aspx (200)
                   Windows 4688: cmd.exe, net.exe, powershell.exe under w3wp.exe

4. Credential      Windows 4771 / 4625 / 4776: password spray against the DC
   Access          Windows 4740: a legitimate account locked out as collateral
                   Windows 4769: RC4 service ticket — Kerberoasting

5. Escalation      Windows 4624 + 4672: successful logon with elevated token

6. Persistence     Windows 4720 + 4728: new account added to Domain Admins
                   Windows 7045: service installed from C:\Windows\Temp
                   Windows 4698: scheduled task with an encoded PowerShell payload

7. Evasion         Windows 1102: Security log cleared

8. C2              Check Point Anti-Bot: command-and-control traffic
                   CrowdStrike FirewallMatchEvent: beacon leaving the network
                   Check Point audit: failed SmartConsole logins from the pivot

The two EDR agents describe the same incident from the endpoint:

CrowdStrike        cmd.exe / nltest.exe spawned under w3wp.exe (High)
                   credential theft tool executed and NOT blocked (Critical)
                   service binary masquerading in C:\Windows\Temp (High)
                   wevtutil clearing the Security log (Defense Evasion)
                   console brute force, then a detection edited from the pivot
                   CrowdScore incident with LateralMovement set

SentinelOne        credential dumping tool, malicious and NOT mitigated
                   STAR rule: encoded PowerShell under the IIS worker (High)
                   STAR rule: remote service creation on the DC (Critical)

Stage 2 is worth a note. In Check Point, Prevent versus Detect is a setting on the protection inside the Threat Prevention profile, not something that varies per request. Protections left in Staging/Detect forward the payload to the origin — which is exactly why the same request appears in both the Check Point log and the Nginx log, and why cross-source correlation has anything to work with.


Requirements

  • Python 3.9+ (no third-party packages for the generators)
  • requests, for the deploy script only: pip install requests
  • An Elastic Stack with Fleet, and permission to install integration packages

Quick start

# 1. Install the integration packages (REQUIRED FIRST — see below)
export KIBANA_URL="https://kibana.lab.example:5601"
export ES_URL="https://elasticsearch.lab.example:9200"
export ES_API_KEY="<base64_api_key>"
python3 deploy_elastic_assets.py --step packages

# 2. Generate and ship the logs
python3 checkpoint_log_generator.py --push \
    --es-host "$ES_URL" --es-api-key "$ES_API_KEY" \
    --namespace seclab --no-verify-tls --days-back 5

python3 ad_log_generator.py --push \
    --es-host "$ES_URL" --es-api-key "$ES_API_KEY" \
    --namespace seclab --no-verify-tls --days-back 5

python3 edr_log_generator.py --push \
    --es-host "$ES_URL" --es-api-key "$ES_API_KEY" \
    --namespace seclab --no-verify-tls --days-back 5

# 3. Create the ML jobs, now that the datafeeds have history to learn from
python3 deploy_elastic_assets.py --step ml-jobs

# 4. Create the detection rules
python3 deploy_elastic_assets.py --step rules

Step 1 is not optional. Without the integration packages, the index templates and their default_pipeline do not exist, so documents land with dynamic mappings and are never parsed.

Inspect before sending anything:

python3 checkpoint_log_generator.py --dry-run
python3 ad_log_generator.py --dry-run
python3 edr_log_generator.py --dry-run
python3 deploy_elastic_assets.py --dry-run
python3 deploy_elastic_assets.py --list-rules

Two ways in

A — syslog to an Elastic Agent (the real production path for Check Point):

python3 checkpoint_log_generator.py --send-syslog \
    --target-host 10.10.5.20 --target-port 9001 --protocol udp

B — Bulk API straight into the official data stream (for labs with no Agent):

python3 ad_log_generator.py --push --es-host ... --namespace seclab

Either way the integration's own pipeline does the parsing. Nginx and IIS logs can also be written to disk as real files for an Agent file input to tail:

python3 ad_log_generator.py --iis-file u_ex.log
python3 checkpoint_log_generator.py \
    --nginx-access-file access.log --nginx-error-file error.log

Detection rules

42 rules across four types. --list-rules prints the full table.

Type Count Used for
query 30 single-event signatures
threshold 6 brute force, scanning, error bursts
machine_learning 4 volume anomalies against the learned baseline
eql 2 cross-stage and cross-product correlation

The EQL rule is the one that matters most for Attack Discovery — it collapses three separate alerts into one incident:

sequence by host.name with maxspan=2h
  [ any where event.code in ("4625", "4771", "4776") and event.outcome == "failure" ]
  [ any where event.code == "4624" ]
  [ any where event.code in ("4728", "4732", "4756") ]

Brute force is a threshold rule, not a query rule, and that is deliberate. The normal baseline genuinely contains the occasional 4625 — users mistype their passwords — so a plain event.code:"4625" rule false-positives forever.

Three rules use field names that only exist after the Check Point pipeline runs, because that pipeline renames them:

Check Point sends The rule must use
checkpoint.malware_action event.category:"malware"
checkpoint.matched_category rule.category
checkpoint.client_ip source.ip

These survive under checkpoint.*: attack, attack_info, protection_name, protection_type, confidence_level, operation, audit_status, administrator.


ML jobs

Job Detector Datafeed filter
seclab_checkpoint_ips_anomaly high_count checkpoint.protection_name exists
seclab_windows_auth_failure_anomaly high_count codes 4625/4771/4776, outcome failure
seclab_web_http_error_anomaly high_count http.response.status_code >= 400
seclab_edr_detection_anomaly high_count event.kind: alert across both EDR products

None of them partitions by user or source IP, which is intentional. The usernames a password spray tries (administrator, svc-sql, backupadmin) never appear in the baseline, so a per-user partition leaves the model cold-started and it misses the burst entirely. Counting total failures per bucket across all users is what actually detects it.


Where the field definitions come from

Nothing here was inferred from memory. The sources:

  • Check Point format and field names — the grok and kv processors of the official Elastic checkpoint integration package, cross-checked against real Check Point logs in Elastic's test corpus
  • Check Point protection names — Check Point Advisories (CPAI-2018-0567, CPAI-2023-0799, CPAI-2025-0043)
  • Check Point severity / confidence scales — sk116254
  • Windows event structure — real captured events in Elastic's Winlogbeat test corpus
  • Windows EventData fields — Microsoft Learn Security Auditing, e.g. Event 4624
  • Event ID → ECS mapping — not written by hand; produced by the official system.security pipeline
  • Nginx and IIS formats — the official grok patterns of the Elastic nginx and iis packages, plus Configure Logging in IIS
  • CrowdStrike and SentinelOne — the real fixtures and ingest pipelines of the Elastic crowdstrike and sentinel_one packages. Every enumerated value (tactic, technique, objective, threat classification, mitigation status, severity scale) is one that actually appears in those fixtures. Where no authoritative public list exists — SentinelOne's numeric activityType codes — that data stream is simply not generated rather than filled in from memory

Each generator's module docstring cites the specific source for its own fields.


Verification

The output was checked against the official patterns rather than eyeballed:

269/269  Check Point lines   parsed by the official grok + kv
200/200  Nginx access lines  parsed by the official grok
 48/48   Nginx error lines   parsed by the official grok
183/183  IIS access lines    parsed by the official grok
 76/76   Check Point fields   present in the official fields.yml
327/327  CrowdStrike records  every field path exists in the official corpus
111/111  SentinelOne records  every field path exists in the official corpus
 42/42   detection rules      match the generated data (across 5 random seeds)

The rule check simulates the pipeline transformations on the generated data and evaluates every rule's KQL against the result. It caught three rules that would have silently matched nothing — the renamed Check Point fields listed above.

All four files are clean under pyright --strict.


Simulated identities

Every address and domain in this repository is drawn from ranges reserved for documentation, so nothing here labels real infrastructure as malicious:

  • RFC 5737 — 192.0.2.0/24 (legitimate visitors), 198.51.100.0/24 (opportunistic scanners), 203.0.113.0/24 (the attacker)
  • RFC 2606 — the .example TLD, both for the C2 and phishing domains and for the Active Directory domain itself (corp.example)
  • RFC 1918 — 10.0.0.0/8 for the internal lab network

One consequence: geoip enrichment will not resolve for these addresses, so source.geo.* stays empty. Substitute a routable range in ATTACKER_IPS / SCANNER_IPS if a demo needs the map populated.

Everything else is a neutral placeholder as well — hostnames (DC01, WEB-PROD-01, cp-gw01), accounts (j.smith, m.jones, svc-backup, cpadmin) and the CORP NetBIOS name. Change them in the constants at the top of each generator to match your own lab.


Files

File Purpose
checkpoint_log_generator.py Check Point Log Exporter syslog + Nginx access/error
ad_log_generator.py Windows Security / System / Application + IIS access
edr_log_generator.py CrowdStrike Falcon + SentinelOne threat and alert
deploy_elastic_assets.py packages, space, data views, ML jobs, 42 detection rules
ad_log_generator_csv.py superseded — the original CSV approach, kept to document what changed and why

ad_log_generator_csv.py depends on a hand-written ad-csv-pipeline that the deploy script no longer creates, so its --push will not parse. It is kept as a worked example of the failure mode this project exists to avoid.


Filtering the simulated data

Every document is tagged, so lab data is easy to separate:

tags: "SECLAB"              every run
tags: "ATTACKSIM-<id>"      one specific run (printed when the generator finishes)

On the syslog path the marker goes into Check Point's comment field — a field that genuinely exists and holds free text — rather than an invented one, so nothing fabricated ends up in the checkpoint.* namespace.


Cleanup

python3 deploy_elastic_assets.py --step cleanup

Removes the rules, ML jobs and data streams. Leaves the space, the data views and the integration packages alone, since those are safe to reuse. A 404 on any line is normal — it just means that asset was never created.


License

MIT

About

Synthetic security logs for an Elastic SIEM lab

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages