Event-Driven Configuration Drift Remediation

Production-style demo: a Linux administrator changes a protected configuration file, Linux auditing detects the change, Filebeat ships the event to Kafka, Event-Driven Ansible evaluates the signal, and AAP restores the approved configuration. For a click-to-launch storefront that shows Ansible owning a Jinja template (no Kafka or EDA), start with Simple Config Drift.

configuration change
  → Linux audit event (auditd)
  → Filebeat
  → Kafka
  → Event-Driven Ansible
  → AAP remediation

Presenter guide

Two parts: one-time setup, then show the demo.

Part 1 — One-time setup

1. Sync the Ansible Product Demos project from your fork branch.

2. Run **APD

   Single demo setup** with category infrastructure.

3. Launch **Infrastructure

   Config Drift - Full Setup** and complete the survey (region, owner, environment).

That single workflow:

• Checks whether aws_rhel8 and aws_rhel9 are in inventory — if not, stops with a message to run Deploy Cloud Stack in AWS first
• Provisions the Kafka queue (aws_kafka)
• Syncs AWS inventory
• Deploys auditd + Filebeat on aws_rhel* with Kafka output
• Activates the EDA rulebook (config_drift_kafka)

When it finishes, the demo is ready to go.

Part 2 — Show the demo

Option A — drift from AAP (recommended for presenters):

1. Open http://PUBLIC_IP/ on the Kafka broker (aws_kafka) — a live dashboard shows friendly lines like /etc/ssh/sshd_config modified — event published when drift happens.

2. Run **LINUX

   Config Drift - Introduce SSHD Drift** (default limit aws_rhel* hits both RHEL workers).

3. Watch Automation Decisions — activation config_drift_kafka consumes the event.

4. Watch Automation Execution — **LINUX

   SSHD Configuration Remediation** launches (once per host, throttled to every 20 seconds).

5. Confirm on the workers: sudo grep -i Root /etc/ssh/sshd_config → PermitRootLogin no.

Option B — drift manually on a host:

sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
sudo grep -i Root /etc/ssh/sshd_config

Same EDA → remediation flow follows within ~20 seconds.

What to say

• Detection is broad — any write to /etc/ssh/sshd_config triggers the pipeline (audit key sshd_config_change).
• Remediation is narrow (on purpose) — the playbook only restores one line (PermitRootLogin no). That keeps the first demo easy to explain.
• Graduate to the whole file — swap the remediation playbook for a Jinja template that owns the full sshd_config (same idea as Simple Config Drift), widen the audit watch if needed, and keep the same Kafka → EDA path.
• No polling — the OS emits an event; EDA reacts in near real time.

Prerequisites

• Deploy Cloud Stack in AWS already run (aws_rhel8, aws_rhel9 in inventory)

• **APD

  Single demo setup** — category infrastructure

• RHSM Registration credential — org ID and activation key (see **LINUX

  Register RHEL with RHSM**)

• Automation Decisions (EDA) on AAP
• SSH access via APD Machine Credential

One-shot setup (recommended)

1. Verifies aws_rhel8 and aws_rhel9 are in inventory (fails with guidance if missing)
2. Provisions Kafka (aws_kafka)
3. Syncs AWS inventory
4. Deploys auditd + Filebeat with Kafka output on aws_rhel*
5. Activates the EDA rulebook

flowchart LR
  check[Check Cloud Stack]
  feedback[Cloud stack required]
  kafka[Provision Kafka]
  sync[Sync Inventory]
  filebeat[Deploy Filebeat]
  eda[Setup EDA]

  check -->|success| kafka
  check -->|failure| feedback
  kafka --> sync --> filebeat --> eda

In the AAP workflow visualizer, the main path runs left to right. When workers are missing, Check Cloud Stack fails to a feedback node that tells you to run Deploy Cloud Stack in AWS first.

Survey prompts: AWS region, owner tag, and environment (same as cloud stack deploy).

Manual setup (step by step)

If you prefer individual jobs:

1. Deploy Cloud Stack in AWS — aws_rhel8, aws_rhel9 in inventory

2. **LINUX

   Register RHEL with RHSM** — default _hosts: aws_rhel*

3. **Infrastructure

   AWS - Provision Kafka Queue**

4. Sync AWS Inventory

5. **LINUX

   Config Drift - Deploy Audit and Filebeat** — Filebeat output: kafka

6. **Infrastructure

   Setup Rulebook for Kafka Queue - Config Drift & Remediation**

Host targeting

aws_rhel* does not match reports. If your job run includes reports, you used a broader limit than aws_rhel* (for example a group or *).

Unreachable hosts are skipped (ignore_unreachable) so one bad SSH target does not block the rest of the fleet.

Job templates

Build status

Stage 1 — auditd

Setup

1. Run **APD

   Single demo setup** with infrastructure.

2. Run **Infrastructure

   Config Drift - Full Setup**, or deploy auditd manually with **LINUX

   Config Drift - Deploy Audit and Filebeat** — default _hosts: aws_rhel*.

What was deployed

Persistent rule in /etc/audit/rules.d/99-sshd-config.rules:

-w /etc/ssh/sshd_config -p wa -k sshd_config_change

Verify rule is loaded

sudo auditctl -l | grep sshd_config

Trigger a test change

sudo cp -a /etc/ssh/sshd_config /tmp/sshd_config.bak
sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config
grep PermitRootLogin /etc/ssh/sshd_config

Inspect audit events

sudo ausearch -k sshd_config_change -ts recent

Understanding the audit records

One file edit typically produces multiple correlated records:

Restore after testing

sudo cp -a /tmp/sshd_config.bak /etc/ssh/sshd_config

Stage 2 — Filebeat

Filebeat is installed and configured automatically by the same job template.

• Elastic 8.x filebeat package with the auditd module
• Reads /var/log/audit/audit.log*
• Console output (pretty JSON) for validation before Kafka is wired in Stage 3

Verify Filebeat is running

sudo systemctl status filebeat
sudo journalctl -u filebeat -f

Trigger a test change and watch events

sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config

In another session on the host:

sudo journalctl -u filebeat -f

You should see parsed JSON with audit fields, hostname, and @timestamp.

Foreground debug:

sudo systemctl stop filebeat
sudo filebeat -e -c /etc/filebeat/filebeat.yml

Stage 3 — Kafka

Full step-by-step walkthrough: Config Drift — Kafka Queue (provision broker, wire Filebeat, consume and validate sshd_config_change events).

Quick reference

1. Run **Infrastructure

   AWS - Provision Kafka Queue** (same region/owner as cloud stack).

2. Sync AWS Inventory so aws_kafka appears.

3. Re-run **LINUX

   Config Drift - Deploy Audit and Filebeat** with Filebeat output: kafka.

4. On aws_kafka, consume topic linux-audit-events and filter for sshd_config_change (see kafka walkthrough for 9094 admin listener).

Presenter note: Kafka and the cloud stack share aws-test-sg. Deploy Cloud Stack in AWS (Create VPC) now includes TCP 9095 for EDA. If EDA stops firing after an idempotent cloud redeploy on an older environment, see Kafka troubleshooting and EDA recovery.

Stage 4 — Event-Driven Ansible

Full walkthrough: Config Drift — EDA.

1. Sync the Ansible Product Demos EDA project.

2. Run **Infrastructure

   Setup Rulebook for Kafka Queue - Config Drift & Remediation**.

3. Confirm activation config_drift_kafka is running.

4. Run **LINUX

   Config Drift - Introduce SSHD Drift** or edit sshd_config on a worker — **LINUX

   SSHD Configuration Remediation** should launch automatically.

Stage 5 — AAP remediation

Why it matters

• Demonstrates a credible production pattern: audit at the OS, ship via a log agent, route through Kafka, decide in EDA, remediate in AAP
• Ansible is not polling for drift — infrastructure emits a signal when configuration changes
• Maps to enterprise deployments where Kafka may be replaced by a SIEM or cloud event bus

See other Infrastructure demos

Browse all Infrastructure demos →