LogTide
v1.0.0-beta
Released
Feature

1.0 Beta: Tenant Isolation, Metering & Capabilities, Hooks & Webhook Delivery

First beta of the 1.0 line. Tenant data isolation audit, per-org metering with a capability/quota system, typed lifecycle hooks, a hardened outbound webhook dispatcher, a structured audit log primitive, and fail-closed PII masking.

  • Tenant data isolation audit across every backend data-access path, with a CI tripwire and isolation test suite (#228, #219)
  • Metering + capability system: per-org usage measurement, feature gates, and enforceable limits/quotas (#212, #214)
  • Typed lifecycle hooks for ingestion, query, alert evaluation and webhook dispatch (#216)
  • Outbound webhook delivery: HMAC signing, retry with backoff, dead-letter queue and SSRF protection (#218)
  • Audit log primitive: typed actions, actor types, outcomes and per-org retention (#217)
  • PII masking is now fail-closed at ingestion — records that fail masking are rejected, never stored unmasked

First beta of the 1.0 line. The headline work since 0.9.7 is the tenant data isolation audit that hardens every backend data-access path before the stable cut, and the metering + capability system pair that gives every organization usage measurement, feature gates and enforceable limits/quotas (ingestion, spans, storage) — the foundation for plan tiers without changing OSS behavior. A typed lifecycle hooks surface lands alongside it, letting operators observe, mutate or reject ingestion, query, alert-evaluation and webhook-dispatch without forking. A reusable outbound webhook delivery system also lands — HMAC signing, retry with backoff, a dead-letter queue and centralized SSRF protection — onto which every existing webhook sender is migrated, closing three unguarded fetch paths in the process.

Security

  • PII masking is now fail-closed at ingestion: previously, if the masking step threw, the batch was stored unmasked with only a console.warn. maskLogBatch now reports per-record failures and the ingestion service rejects exactly those records before the reservoir write (the whole batch if rule compilation itself fails). Rejected records are reported back to the client in a new optional rejected: [{ index, reason: 'pii_masking_failed' }] field on the ingest response (omitted when empty, so existing clients are unaffected) and as partialSuccess.rejectedLogRecords on the OTLP logs endpoint. No unmasked data can reach any storage engine through ingestLogs
  • Cross-tenant log reads via unvalidated ?projectId on the logs query API (#228, audit #219): all 10 endpoints in the query module accepted a ?projectId= parameter and queried with it directly, without verifying it belonged to the authenticated API key’s bound project. Fixed with a shared resolveQueryProjectId guard that returns 403 when the requested projectId does not match the key’s project, locked by a dedicated isolation test
  • Cross-tenant trace/span reads via the same pattern on the traces query API (#228): the identical flaw existed on 8 trace/span read endpoints, fixed with the same guard and locked by an isolation test. The metrics query API was already safe
  • Application-layer scoping gaps on tenant-table queries (defense in depth, #228): a sweep of every tenant-table access path closed several queries missing an organization_id / project_id filter (PII-masking, SIEM, exceptions, correlation). Session-auth routes now Zod-validate organizationId as a uuid instead of a raw cast
  • Dependency security updates ahead of the 1.0 cut: vitest / @vitest/coverage-v8 to 3.2.6, and transitive esbuild (≥ 0.28.1) and shell-quote (≥ 1.8.4) pins bumped via the root pnpm overrides. No vulnerable version remains in the lockfile

Added

  • Capability system: per-organization feature gates, static limits and usage quotas (#214): a typed capability registry is the single source of truth for 12 initial capabilities across three kinds — boolean gates (auth.sso, detection.advanced, audit.enabled, isolation.dedicated), static numeric limits (alerts.max_rules, notifications.max_channels, apikeys.max, audit.retention_days) and metered usage quotas (ingestion.max_bytes_monthly, ingestion.max_events_monthly, storage.max_bytes, tracing.max_spans_monthly). All defaults are OSS-permissive (booleans enabled, limits/quotas unlimited), so self-hosted behavior is unchanged until an operator sets a cap. Per-org overrides live in a new organization_entitlements table; a cache-backed resolver merges rows over registry defaults, fails open on DB errors, and is swappable so a hosted distribution can source entitlements from subscription state without patching core. Enforcement: assertCapability (403), assertWithinLimit (403) and assertWithinUsageQuota (429, hard-blocks log and OTLP span ingestion). Read API GET /api/v1/capabilities returns the merged set for UI gating; admin endpoints manage per-org overrides, surfaced in an “Entitlements” card on the admin organization page
  • Resource usage metering per organization and project (#212): a storage-agnostic way to record and aggregate consumption, surfaced as a “Usage” section in the dashboard. A new metering_events TimescaleDB hypertable is fed by a non-blocking, loss-tolerant in-process recorder that buffers metering.record(...) calls and batch-inserts on a size threshold, flush interval, and graceful shutdown — dropping under back-pressure rather than blocking the request path. Log ingestion records logs.ingested.bytes and logs.ingested.events fire-and-forget after the reservoir write (OTLP ingestion is metered too). Read APIs GET /api/v1/usage and GET /api/v1/usage/breakdown answer “what is being ingested” by type, project, service and log level. Tunable via METERING_ENABLED, METERING_FLUSH_INTERVAL_MS and METERING_FLUSH_MAX_BUFFER
  • Span and storage metering recording sites (#212 follow-up): span ingestion now records spans.ingested, activating the tracing.max_spans_monthly quota end-to-end. A daily StorageSnapshotJob records a per-(organization, project) storage.snapshot estimating stored bytes as the logical bytes ingested within the org’s retention window — engine-agnostic by design — which activates storage.max_bytes. Usage pages gain a “Current storage (estimated)” stat and a daily storage trend
  • Capability usage vs plan limits on the usage page: a “Plan limits” section shows current consumption against each configured capability limit as a color-coded progress bar (green < 80%, amber 80–99%, red ≥ 100%), backed by a new GET /api/v1/usage/capabilities endpoint that joins live usage to the configured cap for every measurable capability. A null limit renders as “Unlimited”, so OSS defaults read correctly
  • Audit log primitive: typed actions, actor types, outcomes and per-org retention (#217): the audit log grows from a free-string event sink into a structured security primitive. A canonical action registry (70 actions across 13 families like org.*, apikey.*, rule.*, auth.*, data.*) is the single source of truth — actions are a TypeScript string-literal union, so typos fail to compile. A new auditLogService.record() API reads organization, actor (user/apiKey/system), IP and user agent from the request context, so callsites only state the action and target; it supports atomic recording inside a transaction, awaited fire-and-safe inserts that never fail the request, and a flush buffer for high-volume access logging. All 57 legacy callsites were converted; new coverage includes failed local logins (auth.login_failed, previously not recorded at all). Per-org audit_retention_days (1–3650, NULL = keep forever) is editable from the admin org page and enforced by the daily retention job. The audit page gains actor-type, outcome and time-range filters
  • Lifecycle hooks at ingestion, query, alert evaluation and webhook dispatch (#216): a small, typed set of named extension points (beforeIngest, beforeQuery, beforeAlertEvaluation, beforeWebhookDispatch), no-op in OSS (nothing registered by default, a hasHandlers guard keeps the hot paths at zero overhead). Handlers run sequentially, receive a typed per-phase context with documented mutable fields, and may abort by throwing HookRejectionError(code, message, statusCode); unexpected hook errors fail closed. Operators register handlers without forking via HOOKS_MODULES. Contract documented in the new Lifecycle Hooks guide
  • after- lifecycle hook phases* (#216 follow-up): afterIngest (batch counts and rejection reasons), afterAlertTriggered (post-persist trigger facts) and afterWebhookDispatch (per delivery-attempt result). Fire-and-forget: handler errors are logged and never block or mutate the operation; contexts are frozen read-only snapshots
  • Generic outbound webhook delivery infrastructure (#218): a reusable dispatcher that centralizes every outbound HTTP delivery behind one well-tested module. Capabilities: optional HMAC-SHA256 signing (X-Logtide-Signature: t=<unix>,v1=<hex>, timing-safe to verify); retry with exponential backoff (1s, 5s, 25s, 2m, 10m, transient-only); a dead-letter queue modeled as webhook_deliveries.status='dead', listable and replayable from the dashboard; SSRF protection by reusing the safeFetch guard; per-organization concurrency limiting; and a bounded delivery log exposed via an org-scoped API and a “Webhook Deliveries” page under Settings. All five existing webhook senders were migrated onto it — the error/monitor/incident paths previously used a bare fetch with no SSRF guard, closing three sibling gaps. Documented in the new Webhooks guide
  • Request context propagation across HTTP, jobs and the DB layer (#222, closes #213): a new AsyncLocalStorage-backed RequestContext primitive in @logtide/shared/context. A Fastify plugin establishes the context right after auth resolves; BullMQ producers piggy-back _ctx on payloads and consumers wrap processors in context.run / runAsSystem; cron callbacks run under runAsSystem. The pg.Pool is patched to prepend a /* req=... */ SQL comment so requests are traceable in slow-query logs, and each reservoir engine injects the same correlation in its native form (Timescale SQL comment, ClickHouse query_id + log_comment, MongoDB $comment)
  • Tenant isolation audit, test suite and CI tripwires (#228, #219): a living audit document, a dedicated isolation test suite (the createIsolatedTenants fixture plus per-area tests for query, traces, crud, api-key auth, audit-log and metering) that runs in CI, a static tripwire (check:tenant-scoping) that flags Kysely queries on tenant tables lacking org/project scoping and fails the CI typecheck on new unscoped sites, an opt-in runtime Kysely guard (TENANT_GUARD=1), and a PR template carrying a tenant-safety checklist. The codified model: organization is a hard boundary for everyone, API keys are project-scoped, session users are org-wide by current policy
  • Capability enforcement completed across resource creation (#214 follow-up): the previously defined notifications.max_channels and apikeys.max limits are now enforced at channel/key creation (API keys counted org-wide), and two new limit keys land with enforcement: sigma.max_active_rules and dashboards.max_custom. All defaults stay unlimited for self-hosted OSS
  • Ingestion health visibility for admins: a new GET /api/v1/admin/stats/ingestion-health endpoint aggregates the last 24h of new ingestion health counters (ingestion.pii_rejected, ingestion.detection_enqueue_failed, and more) plus SIEM enrichment availability, surfaced as an “Ingestion health (24h)” card that highlights in red when failures occurred. Sigma-detection and exception-parsing job enqueues now retry once and record a counter on failure instead of dropping silently
  • Scheduled email digest reports — groundwork, disabled (#209, part of #154): the foundation for periodic email summaries of log activity. Intentionally disabled in this beta — the worker leaves the scheduler unregistered and no digests are sent. The service, schema and tests are merged so the remaining scope can be finished. What landed: digest_configs / digest_recipients tables, an ICronRegistry abstraction with native-cron implementations for both BullMQ and Graphile, and a generator that computes log volume from the hourly-stats continuous aggregate
  • Test debt paid down: six backend coverage exclusions are gone, the 12 previously untested @logtide/shared Zod schemas gain dedicated tests, CI now runs the shared and frontend unit suites alongside the backend, and frontend component testing is enabled (Svelte 5 + testing-library). A dedicated Playwright journey covers the full traces flow (list, filters, waterfall, span panel and trace/log correlation)
  • Migrations 044–048: digest_email_reports (044), metering_events (045), organization_entitlements (046), webhook_deliveries + webhook_delivery_attempts (047), and additive actor_type/actor_id/outcome columns on the audit_log hypertable (048)

Changed

  • BREAKING: unified webhook event envelope (#218 follow-up): every outbound webhook delivery now serializes to one envelope { id: "evt_<uuid>", type, version: 1, occurredAt, organizationId, projectId, data } instead of four bespoke payload shapes. Event types are alert.triggered, incident.created, error.detected, monitor.status_changed and channel.test; the per-type data payloads keep their previous snake_case fields minus event_type/timestamp. Envelope and per-type Zod schemas ship in @logtide/shared (webhookEnvelopeSchema, parseWebhookEvent); deliveries carry a new X-Logtide-Event-Version: 1 header. See the Webhooks guide for the full envelope reference
  • OTLP log metadata shape: resource attributes now land under metadata.resource instead of being spread flat (log-record attributes keep the flat namespace and win collisions there; nothing is lost). Structured log bodies (kvlist/array) keep their decoded structure under metadata['otel.body'] alongside the stringified message. Logs ingested before this change keep their old shape
  • Reservoir log query params now require projectId (#228): projectId was optional on the reservoir query/count/aggregate params, which is what allowed an unscoped log read to compile. It is now required, with an explicit GLOBAL_SCOPE sentinel for the handful of intentional platform-wide reads

Fixed

  • Sigma search by MITRE technique/tactic/tag always 500ed: the queries compared TEXT[] columns with a ::jsonb cast and failed on every request since they shipped; they now use proper text-array containment and are covered by regression tests
  • Migration prefix collision that could break production migrate (#229): two migration pairs shared a numeric prefix, which Kysely’s position-by-position validation could reject. Fixed by renumbering the files plus an idempotent pre-migration repair that renames the matching kysely_migration rows on already-applied databases; a new CI guard rejects future duplicate prefixes. Verified on both a fresh schema and a database seeded into the pre-fix state
  • DELETE /api/v1/sigma/rules/:id and custom-dashboard update/delete returned 500 instead of 404 for a missing or cross-org id (#229): these now map to a proper 404. None leaked or mutated another tenant’s data — the scoping already prevented the operation; this only corrects the HTTP status
  • Silent failures made visible across admin and monitoring UI: the admin version check no longer claims “up to date” when GitHub is unreachable; the monitoring page no longer swallows incidents/maintenance load errors; GeoLite2/IPsum unavailability warns once per outage instead of spamming every lookup
  • API-key last_used write amplification on hot keys (#222): every authenticated request updated api_keys.last_used, producing ~100 updates/sec on a single row under load. The update is now debounced to at most once per 60s per key