Hardware Health

Hardware Health is early warning, not prediction. It reads the signals that hardware gives out before it fails (reallocated sectors, corrected memory errors, a fan slowing down, a PCIe link retraining) and turns them into a status per host and per component, with an incident and a notification when something gets worse.

A small agent runs on each server and sends two things to your LogTide project:

• Metrics named hw.*, sent as OTLP gauges to POST /v1/otlp/metrics.
• Events from the kernel log and the BMC event log, sent as logs with service hardware.

LogTide registers each host the first time it sees hw.* metrics from it, evaluates it every two minutes and shows the result on Monitor → Hardware. Nothing else is needed: the agent uses the same project API key as your logs.

• Bare metal, homelab and on-premises servers are the target. Disks, board sensors, EDAC and a BMC are all visible there.
• Virtual machines and most cloud instances expose no hardware sensors. The agent detects virtualization and reports it. Such hosts get a VM badge and the status No sensors instead of a health status, and the host page explains that sensors are not available. Run the agent on the hypervisor host, not inside the guests.
• A host that goes completely dark shows as stale. If no hw.* data point arrives for more than 10 minutes, the host is marked stale. A stale host is shown but does not open an incident. To be told when a machine dies or loses power, pair it with a heartbeat monitor from Uptime that the server pings on a schedule.
• Not covered in this release: RAID controller tools (MegaCLI, storcli), UPS (NUT), Redfish, per-host custom thresholds, and Windows or macOS hosts.

• LogTide 1.7.0 or later (self-hosted or Cloud) and a project API key.
• A Linux host. The agent is a single Python 3 script with no dependencies.
• smartmontools for disk data. Included in the Docker image.
• ipmitool and a BMC (iDRAC, iLO, IPMI) for PSU, voltage and sensor data from the board. Optional, included in the Docker image.
• The EDAC kernel modules loaded for ECC memory counters (/sys/devices/system/edac/mc must exist). Needs ECC RAM and a supporting platform.
• Root or a privileged container, because SMART, IPMI and /dev/kmsg need it. See Permissions.

Each collector is independent. A host without a BMC still reports disks, memory and temperatures; the missing collector is listed as unavailable on the host page.

Create a project API key first (Project settings → API keys). Use a dedicated key for hardware agents so you can rotate it separately. The examples use the LogTide Cloud API URL; for a self-hosted instance set LOGTIDE_URL to your own API base URL.

The agent lives in its own repository, logtide-dev/logtide-hw-agent (MIT license), and is versioned separately from LogTide. Agent 1.x works with LogTide 1.7.0 and later. The Docker image is logtide/hw-agent.

# docker/.env
HW_AGENT_API_KEY=lp_your_ingestion_key_here

docker compose --profile hardware up -d

docker run -d --name logtide-hw-agent --restart unless-stopped --privileged \
  -e LOGTIDE_URL=https://api.logtide.dev \
  -e LOGTIDE_API_KEY=lp_your_key \
  -e HOST_ROOT=/host \
  -v /sys:/host/sys:ro \
  -v /proc:/host/proc:ro \
  -v /etc/hostname:/host/etc/hostname:ro \
  -v logtide-hw-agent-state:/var/lib/logtide-hw-agent \
  logtide/hw-agent:latest

sudo apt install -y python3 smartmontools ipmitool
sudo install -d /opt/logtide-hw-agent /etc/logtide
sudo curl -fsSL https://raw.githubusercontent.com/logtide-dev/logtide-hw-agent/main/logtide_hw_agent.py \
  -o /opt/logtide-hw-agent/logtide_hw_agent.py
sudo curl -fsSL https://raw.githubusercontent.com/logtide-dev/logtide-hw-agent/main/logtide-hw-agent.service \
  -o /etc/systemd/system/logtide-hw-agent.service
sudo tee /etc/logtide/hw-agent.env >/dev/null <<'ENV'
LOGTIDE_URL=https://api.logtide.dev
LOGTIDE_API_KEY=lp_your_key
ENV
sudo chmod 600 /etc/logtide/hw-agent.env
sudo systemctl daemon-reload
sudo systemctl enable --now logtide-hw-agent

# Docker
docker run --rm --privileged -e HOST_ROOT=/host -e LOGTIDE_HW_DRY_RUN=1 \
  -v /sys:/host/sys:ro -v /proc:/host/proc:ro -v /etc/hostname:/host/etc/hostname:ro \
  logtide/hw-agent:latest --once

# systemd
sudo sh -c 'set -a; . /etc/logtide/hw-agent.env; LOGTIDE_HW_DRY_RUN=1 python3 /opt/logtide-hw-agent/logtide_hw_agent.py --once'

The agent is configured with environment variables.

The command line flag --once runs one collection, sends it and exits. In the compose stack the agent reads HW_AGENT_API_KEY and HW_AGENT_IMAGE from .env.

The agent runs as root because SMART queries, /dev/kmsg and /dev/ipmi0 need raw device access. It only reads: it does not write to disks, change fan settings or modify the BMC. In the container, the setup is:

• /sys and /proc are mounted read-only under /host and read through HOST_ROOT. The container's own /sys/class/net only shows the container's network namespace, so NIC counters would be wrong without the host mount.
• --privileged gives the container the host's /dev (SATA and NVMe disks, /dev/kmsg, /dev/ipmi0) and the capabilities smartctl needs.
• The /etc/hostname mount makes the host name match the real host instead of the container id. You can set LOGTIDE_HOSTNAME instead.
• The state volume keeps the BMC event log position across restarts, so old events are not sent again.

Without --privileged you need --cap-add SYS_RAWIO --cap-add SYS_ADMIN --cap-add SYSLOG plus one --device per disk (/dev/sda, /dev/nvme0), --device /dev/kmsg, and --device /dev/ipmi0 if present. This is more fragile, because seccomp and AppArmor profiles vary between distributions, and privileged is the tested path.

Every data point carries the attributes host.name, hw.type and hw.id (the device, sensor or controller it belongs to). All metrics are gauges. For disks, hw.id is the drive's serial number (or its /dev/disk/by-id name when there is none), not sda, so a drive keeps its history when the kernel renames it after a reboot; the current kernel name is in the device attribute. Counters such as error totals are sent as the raw cumulative value, and LogTide works on the growth over a window, so a reboot that resets a counter does not look like an improvement or a spike.

The metrics are ordinary OTLP metrics, so you can also chart them in the Metrics Explorer or with a metric_chart panel on a custom dashboard.

Each host has six components: Disks, Memory, CPU, Cooling, Power and Network. A component is ok, warning, critical or unknown (no data). The host status is the worst of its components, or stale when the host stopped reporting. Every finding on the host page names the component, the device and the value that crossed the threshold.

Two more checks raise a component to warning so that a silent failure is not shown as healthy:

• Missing device. A disk, power supply, fan or network interface that reported in the last 24 hours and is gone from the latest data (disk.missing, power.psu_missing, cooling.fan_missing, network.interface_missing). A dead drive often disappears instead of reporting an error.
• Collector down. A collector now reports 0 although its component had data in the last 24 hours (agent.collector_down), for example because smartctl started failing. The host page names the collector.

• Only physical network interfaces are evaluated. Bridges, VLANs, VPNs and container interfaces are ignored.
• Rules that look at growth need history. Right after you install the agent, growth-based findings can take up to 24 hours (or 7 days for the 7-day rules) to become meaningful.
• The thresholds are fixed in this release and shared by the backend and the UI. Per-host custom thresholds are not available yet.

Besides metrics, the agent forwards hardware events as logs:

• Kernel log. The agent tails /dev/kmsg from the moment it starts (no replay of old messages) and keeps only lines that look like hardware problems: disk I/O errors, ATA link resets, EDAC and machine check messages, PCIe AER errors, thermal throttling, NIC link changes.
• BMC System Event Log. New entries from ipmitool sel elist are forwarded once. The agent remembers the last entry it sent in its state directory.

Each event is a log with service hardware, the host name as hostname, and metadata hw.event.source (kmsg or sel) and hw.event.type (disk, memory, cpu, pcie, power, thermal, network or other). Kernel lines are at least warnings: priority 0 to 2 is critical, 3 is error, anything else is warning. BMC entries are warnings, and critical when the event mentions an uncorrectable error, a failure or a predictive failure. When the BMC logs that a condition cleared (a Deasserted entry), the event is sent at info level, with hw.sel.direction set to deasserted, and no detection rule fires on it. BMC events also carry hw.sel.id and hw.sel.timestamp. The host page lists the host's events from the last 7 days, newest first, and you can search them in the log explorer by service hardware.

The Hardware Health Pack is a bundle of 21 Sigma rules, in the new hardware pack category, that match hardware problems in kernel and BMC messages. Open Alerts, choose Detection packs and enable it like any other pack. Its detections show up in the recent detections on the Alerts page.

The rules match on the message text, case-insensitively and whatever the service name, so they fire on agent events and on plain kernel lines that reach LogTide some other way. If you already ship kernel logs through syslog, journald or Fluent Bit, the pack works without the agent; you just do not get the metrics and the host page. The rules carry no MITRE ATT&CK tags, because hardware faults are not attacks.

BMC recoveries are logged as ... | Deasserted. Every rule skips those lines, except Power supply removed, which exists to catch a supply that is no longer present. For general Sigma usage see the security monitoring guide.

• When a host gets worse (ok to warning or critical, or warning to critical), LogTide opens or updates an incident with source Hardware in the incidents inbox and sends a notification. A warning maps to medium severity, a critical to critical. There is at most one active hardware incident per host.
• If the host improves from critical to warning, the incident drops to medium severity and no new notification is sent.
• When the host returns to ok, the incident is resolved and a recovery notification is sent.
• The notification uses the Hardware health event type. Choose its destinations (email, Slack, Discord, Teams, PagerDuty, webhook) under Settings → Notifications → Routing, as described in Notification Routing. Without defaults, an email goes to your organization owners and admins when SMTP is configured. Owners and admins also get an in-app notification when a host degrades.
• Webhook destinations receive hardware.health.degraded and hardware.health.recovered events, see Webhooks. PagerDuty gets one PagerDuty incident per hardware incident, closed on recovery.
• Mute a host on its detail page while you work on it (organization owners and admins can). A muted host is still evaluated and shown, but opens no incidents and sends no notifications. If it recovers while muted, its open incident is resolved silently.
• Forget host resolves the host's open incident and removes the host from the list. It comes back by itself if the agent keeps sending data.
• A stale host does not open an incident. Use a heartbeat monitor for that, see above.