Service Monitoring, Log Pipelines & Custom Dashboards

Added

Service health monitoring and status pages (#152)

Proactive uptime monitoring with auto-generated public status pages.

• 3 monitor types: HTTP/HTTPS (configurable method, expected status, headers, body assertion), TCP ping, and heartbeat (alert when no ping received within grace window)
• HTTP config: per-monitor httpConfig with method, expectedStatus, custom headers, and body assertion (contains or regex) - stored as JSONB, validated via Zod
• Per-monitor severity: incident severity is configurable per monitor (critical, high, medium, low, informational) instead of hardcoded high
• BullMQ-style polling: worker checks all due monitors every 30s, batched in groups of 20 concurrent checks via Promise.allSettled
• TimescaleDB storage: monitor_results hypertable with 7-day compression, 30-day retention, and monitor_uptime_daily continuous aggregate refreshed hourly
• State machine: consecutive failure tracking with configurable threshold, atomic incident dedup guard (WHERE incident_id IS NULL), auto-resolve on recovery
• Auto-incident creation: when failure threshold is crossed, a SIEM incident is created with source: 'monitor' and linked via monitor_id; notifications sent via existing email/webhook channels
• Public status page (/status/:projectSlug): Uptime Kuma-inspired design with 45-day heartbeat bar grid, per-monitor uptime badge, overall status banner, custom CSS tooltips, light/dark mode toggle
• Status page access control: configurable visibility per project - disabled (default), public, password-protected, or org-members-only
• Scheduled maintenances: create maintenance windows with start/end times; active maintenances suppress monitor incident creation and display a banner on the status page
• Manual status incidents: create public incident communications (investigating → identified → monitoring → resolved) with update timeline, independent from SIEM incidents
• Heartbeat endpoint: POST /api/v1/monitors/:id/heartbeat accepts both API key and session auth, rate-limited to 600/min
• Project slugs: auto-generated from project name on creation, unique per org, backfilled for existing projects via migration
• Dashboard UI (/dashboard/monitoring): monitor list with project selector, create/edit/delete forms with client-side validation, detail page with refresh button, uptime chart, recent checks, copy heartbeat URL
• Monitoring navigation: added to sidebar under “Detect” section alongside Alerts and Security

Log parsing and enrichment pipelines

Define multi-step processing rules that automatically parse and enrich incoming log messages before they are stored.

• 5 built-in parsers: nginx (combined log format), apache (identical to nginx), syslog (RFC 3164 and RFC 5424), logfmt, and JSON message body
• Custom grok patterns: %{PATTERN:field} and %{PATTERN:field:type} syntax with 22 built-in patterns (IPV4, WORD, NOTSPACE, NUMBER, POSINT, DATA, GREEDYDATA, QUOTEDSTRING, METHOD, URIPATH, HTTPDATE, etc.) and optional type coercion (:int, :float)
• GeoIP enrichment: extract country, city, coordinates, timezone, and ISP data from any IP field using the embedded MaxMind GeoLite2 database
• Async processing via BullMQ: pipelines run as background jobs after ingestion - zero impact on ingestion latency
• Project-scoped vs org-wide: pipelines can target a specific project or apply to all projects in the organization; project-specific pipelines take priority over org-wide ones
• Pipeline preview: test any combination of steps against a sample log message and inspect per-step extracted fields and the final merged result before saving
• YAML import/export: import pipeline definitions from YAML with name, description, enabled, and steps fields; upserts (replace existing pipeline for the same scope)
• In-memory cache: getForProject caches the resolved pipeline per project for 5 minutes, automatically invalidated on create/update/delete
• Settings UI (/dashboard/settings/pipelines): list, enable/disable toggle, create, edit, and delete pipelines with live org-switch reactivity ($effect instead of onMount)
• Step builder: interactive UI for adding, reordering, and configuring parser, grok, and geoip steps with per-type configuration forms
• Pipeline edit page redirects to the list when the active organization is switched, preventing stale-ID errors

Custom dashboards with configurable panels (#151)

User-built dashboards replace the previous fixed /dashboard page, with team-specific views across all observability domains.

• 9 panel types covering every data source: time series, single stat, top-N table, live log stream, alert status (logs/alerts), metric chart and metric stat (OTLP metrics with avg/sum/min/max/count/last/p50/p95/p99 aggregations), trace latency (p50/p95/p99 from spans), detection events (SIEM by severity), monitor status (uptime + response time)
• Panel registry architecture: adding a new panel type touches only 6 files (shared types, backend Zod schema, backend fetcher, frontend panel component, frontend config form, frontend registry entry); the renderer, container, store, and routes never need to change
• Drag-and-drop reorder via svelte-dnd-action with optimistic local state and a single PUT save
• Drag-to-resize with bottom-right pointer-event handle, snapping to grid units; constrained by per-type min width/height from the registry
• Responsive 12/6/1 column grid that collapses panels to 6 columns on tablet (640-1024px) and 1 column on mobile (<640px); stored widths are always in the canonical 12-col reference and scale proportionally
• Auto-created Default dashboard per organization, idempotent via Postgres unique-violation guard, replicating the previous fixed layout (4 stat cards + log volume + top services + top error messages) so existing users see no visual change
• Inline edit mode with toggle, no separate edit page; pending changes are kept in a snapshot and discarded on Cancel
• Per-panel configuration dialogs with type-specific forms (level toggles, intervals, aggregation pickers, percentile selectors)
• Dashboard switcher dropdown in the page header with personal/shared distinction, create, delete (default protected), import, export
• YAML import/export: dashboards round-trip through YAML for version-controlling alongside infrastructure code; import regenerates panel IDs and uses JSON_SCHEMA to block JS-tag prototype pollution
• Versioned JSON schema (schema_version: 1) with a migration framework in @logtide/shared: each version writes a MigrationFn indexed by target version, migrateDashboard walks the chain on every read; clamps out-of-range versions defensively
• Cross-org isolation guard: every panel data fetch verifies that config.projectId belongs to the requesting org, preventing data leaks via crafted YAML imports or stale references
• Batch panel data endpoint (POST /:id/panels/data): single round-trip fetches all panel data via Promise.allSettled, individual panel errors do not fail the dashboard
• Organization scoping: dashboards are org-scoped with optional is_personal flag (only visible to creator) and created_by tracking; partial unique indexes prevent multiple defaults per (org, project) scope
• Migration 039_custom_dashboards.sql: JSONB panels column with GIN index for future panel-type filtering, partial unique indexes for default scope guarantees

Fixed

• Status page slug collision: getPublicStatus now filters by status_page_public flag instead of returning the first project matching the slug, preventing cross-org data leaks
• createMonitor not transactional: monitor and monitor_status inserts are now wrapped in db.transaction() to prevent orphaned monitors
• mapMonitor typed: replaced any parameter with proper MonitorWithStatusRow interface for compile-time safety
• Org membership check optimized: monitoring routes now use a single SELECT WHERE user_id AND organization_id query instead of fetching all user orgs and scanning in JS
• Redundant DB read eliminated: processCheckResult now receives status data from the already-fetched monitor object instead of issuing a second SELECT
• Target validation on update: PUT endpoint now validates target format against monitor type (HTTP must start with http:///https://, TCP must contain :)
• $derived.by fix: monitor detail page uptime calculation now uses $derived.by() instead of $derived(() => ...) for correct Svelte 5 reactivity
• @const placement: replaced invalid {@const} inside <div> elements with {#if}/{:else} blocks for Svelte 5 compatibility
• uptimePct type coercion: Postgres ROUND() returns numeric as string - status page now coerces to number before calling .toFixed()
• Default failureThreshold aligned: frontend form default changed from 3 to 2 to match backend default
• Test setup cleanup: added monitor_results, monitor_status, monitors, incident_alerts to global beforeEach cleanup