Traces Live Tail, Dashboard Panels & Async Ingestion Buffer

Added

• Traces live tail (SSE): new GET /api/v1/traces/stream Server-Sent Events endpoint polls the traces table once a second and emits new trace_ids as they appear, filtered by projectId, service (CSV) and error. On the frontend the traces page now has a “Live tail” switch + rows-limit selector (50/100/200/500/1000, persisted in localStorage) in the filter bar; incoming traces are prepended and the list is capped at the chosen limit. The tracesEventSource is torn down on onDestroy and on every filter-key change
• Traces row click-to-expand with inline span list: clicking a trace row now toggles an inline panel below it that fetches GET /api/v1/traces/:traceId/spans once (cached per trace_id for the session) and shows a compact table with Service / Operation / Kind / Duration / Status for each span. The “View” action still opens the full trace detail page
• Traces keyboard shortcuts: / focuses the Trace ID input, r refreshes, j/k move the selection down/up with scroll-into-view, enter expands/collapses the selected trace. Registered via shortcutsStore under scope traces and unregistered on destroy
• Traces export (JSON / CSV): new Export popover in the filter bar produces a client-side download of the current list (disabled during live tail or when the list is empty). CSV covers start_time / service / operation / duration_ms / span_count / error / trace_id
• Monitoring entry in the command palette: cmd+k now lists Monitoring under Navigation (icon + g o shortcut hint), matching the sidebar’s Detect group so the page is reachable without leaving the keyboard
• trace_volume dashboard panel: new panel type that plots span count over time (optional errors line), configurable time range (1h/6h/24h/7d) and optional service filter. Mirrors the shape of the existing logs “Log Volume” panel but for OTLP traces. Reads the spans_hourly_stats / spans_daily_stats continuous aggregate first, falling back to the raw spans hypertable with date_trunc only when the cagg is empty for the window (covers the end_offset=1h refresh lag on freshly-ingested data). TimescaleDB-only
• activity_overview dashboard panel: unified multi-series timeline combining logs, log errors, spans, span errors, detection events and alert triggers on a common bucket grid (hourly for ≤24h, daily for 7d/30d). Each series is individually toggleable in the config form. Per-source queries run in parallel and prefer the logs_*_stats, spans_*_stats, detection_events_*_stats continuous aggregates for cost; if a cagg returns zero rows for the window the fetcher falls back to the raw hypertable (logs, spans, detection_events) for that source only, so the panel stays correct on freshly-ingested data without running a raw scan in the hot path. Alerts always come from alert_history + alert_rules (no cagg exists for it)

Changed

• Monitoring page reorganized into tabs: what was a single 1000-line scroll with three unrelated sections (Monitors, Incidents, Maintenance) plus a status-page config block wedged at the top is now a Tabs.Root with four tabs — Monitors, Incidents, Maintenance, Status page — each with its own header, its own primary CTA, and its own empty state. The Monitors tab gains a summary card grid (Total / Up / Down / Paused, each card clickable to filter the list by that status), a search input that matches name / target / type, and a “Clear filters” shortcut. Every row now has a one-click Pause/Resume button (previously buried in the edit form). Status-page visibility, password, slug and embed-badge config moved into the Status page tab. No changes to underlying API calls or permissions
• Admin dashboard sidebar collapses on mobile: the 256px admin sidebar was previously always visible, taking up a huge chunk of the viewport on phones. It now hides below lg: and opens as a fixed drawer triggered by a hamburger in a new mobile-only sticky header that also shows the current section name (Dashboard / Users / Organizations / etc). The drawer has a backdrop, is dismissed on nav-click or backdrop-click, and locks body scroll while open. Desktop behaviour unchanged
• Dashboard footer wraps properly on mobile: was a single justify-between row that squeezed “LogTide / Alpha v0.9.2” against ”© 2026 LogTide · Documentation · GitHub” on narrow screens. Now stacks the brand block over the links block below sm: (each block keeps its own horizontal flow with flex-wrap gap-x-4 gap-y-1), and “Documentation” shortens to “Docs” under sm: to fit on a single line. Container padding also drops from px-6 py-4 to px-4 py-3 on mobile
• User settings promoted from a dialog to a full page at /dashboard/account. The cramped scrollable UserSettingsDialog modal (profile + password + tutorial restart + danger zone stacked inside a 32rem-wide overlay) is now a proper route using a max-w-3xl container with each concern split into its own rounded-lg border bg-card section (Profile / Change password / Onboarding / Danger zone). The dropdown menu item still says “User Settings” but now navigates instead of opening an overlay. UserSettingsDialog.svelte is removed. The delete-account confirmation remains a nested AlertDialog since that one is genuinely destructive and modal-appropriate
• Public status page polished for mobile and clarity: outer container shrinks from px-4 py-10 to px-3 py-6 on phones (only sm: and up gets the roomy desktop padding). The 45-day uptime bars drop their min-w-[6px] floor to 4px on mobile so the whole row fits comfortably on 320px viewports. The footer (“Last updated …” and “Powered by LogTide”) grows from text-[10px] to text-xs with a taller top margin, and “LogTide” is now styled as text-primary font-medium hover:underline so the link affordance is obvious (previously only a hover recolor). Incident update timestamps/status labels also bumped from text-[10px] to text-xs. The monitor-type badge (HTTP/TCP/etc.) hides below sm to free up the row for the name + uptime%. The password input switches from a fixed w-64 to w-full max-w-xs so it never overflows narrow phones
• Monitoring forms moved into modal dialogs: the previous inline create/edit monitor form, the “New incident” form, the “Post incident update” form and the “Schedule maintenance” form were expanding/collapsing sections that shifted the whole page layout when opened. They now open as centered Dialog modals with max-h-[90vh] overflow-y-auto so tall forms (monitor create) scroll independently of the page. The per-row “Post update” Dialog is a single instance driven by a showUpdateForm id + an updatingIncident derived lookup, replacing the one-per-row inline expand. Form state, validations and submit handlers are untouched
• Traces filter UI reworked into the same two-row pill bar as the search page: row 1 keeps Time Range (TimeRangePicker in a popover), the Live tail switch, the List/Map view toggle and the Export menu. Row 2 exposes every filter as its own always-visible pill — Project (single-select), Services (multi-select), Status (tri-state: all / errors / ok), Duration range (min/max ms inputs), Trace ID (direct-navigate to the trace detail page). Pills switch to the secondary variant when a non-default value is set; “Clear all” appears when any filter is active. Mobile popovers use w-[Xpx] max-w-[90vw] so they never overflow the viewport. handleTimeRangeChange syncs picker state back into timeRangeType / customFromTime / customToTime so the trigger label survives popover close/remount
• Traces backend accepts multi-value projectId and service (CSV on the query string) and two new minDurationMs / maxDurationMs bounds on GET /api/v1/traces. The cross-project access check loops over every project in the list. tracesService.listTraces widens its input types to string | string[] and forwards to reservoir.queryTraces, which already supported these filters
• Search page paginator now uses the same ellipsis/windowed control as the traces page: shows up to four numbered buttons plus sentinel … on each side when totalPages > 7. Falls back to the previous “Page N” label when the backend doesn’t return a total (storage engines without total in the query response). “Previous” / “Next” labels hide on <sm screens so only the chevron shows on phones
• Empty states wait for the first load to complete before rendering on the search, traces, errors and security-incidents pages. Each page now tracks a hasLoadedOnce flag flipped in the finally branch of its loader; until the first fetch resolves (success or failure) the page shows its skeleton instead of the “No Logs Yet / Start sending logs from your applications” onboarding card. Prevents the flash where an org that already has data showed the onboarding CTA during the initial render between mount and the first query. When a fetch legitimately returns zero rows, the page also distinguishes “nothing matches these filters” (with a Clear-filters shortcut) from “project has no data at all” (the full onboarding)
• Search page filter UI reworked into a two-row pill bar: row 1 keeps Search + mode, a Time Range popover trigger, Live Tail and Export inline. Row 2 exposes every filter as its own always-visible pill (Projects, Services, Hostnames, Levels, Trace ID, Session ID, Metadata) — each pill displays the current value (e.g. “Service: api-gateway”, “Levels: 2”) and opens its own popover for editing. Pills switch to the secondary variant when a non-default value is set so active filters are visible at a glance. A “Clear all” link appears at the end of the row when any filter is active. Popover widths use w-[min(Xpx,calc(100vw-1rem))] so they never overflow the viewport on phones. No semantic change to the underlying filter behavior (state, query params, live-tail flow, metadata apply/clear) - purely a layout/affordance pass replacing the previous six-column “Filters” Card
• Auto-created Default dashboard now uses the new activity_overview panel instead of the logs-only time_series “Log Volume” chart. Existing organizations keep their current dashboard untouched; the swap only affects orgs that don’t yet have a default (i.e. new signups or orgs where the default was deleted and re-seeded via ensureDefaultExists)
• Opt-in async buffer for reservoir writes: new ReservoirBuffered decorator sits between the ingestion API and the storage engine. When enabled via RESERVOIR_BUFFER_ENABLED=true, POST /api/v1/ingest returns as soon as records are accepted into a shard-partitioned queue; a flush consumer pool drains to storage asynchronously. Two transports ship in the box: an optimized in-memory queue with signal-based wakeup (single-instance, not crash-safe) and a Redis Streams transport with consumer groups, XAUTOCLAIM-based stale reclaim, atomic MULTI/EXEC nack, and a DLQ side stream (multi-instance, durable). Circuit breaker bypasses to sync ingestion when the buffer fills beyond a configurable pending threshold or after repeated flush failures in a rolling window. Off by default; see docs/async-buffer/ for when to enable and the per-engine benchmark table
• IReservoir shared interface: public type that Reservoir and ReservoirBuffered both implement, so downstream code (backend monitoring, etc.) can type against IReservoir without caring which implementation is live
• Prometheus-style buffer metrics: reservoir_buffer_enqueued_total, reservoir_buffer_bypass_total, reservoir_buffer_flush_success_total, reservoir_buffer_flush_failure_total, reservoir_buffer_flush_duration_ms (histogram), reservoir_buffer_dlq_total, reservoir_buffer_breaker_state (0 closed / 1 open / 2 half-open). All labelled by record kind and shard where applicable
• Warning at startup when buffer is enabled on non-Timescale engines: benchmarks show the buffer regresses p95 under saturation on ClickHouse and MongoDB (the bottleneck is the flush side, which the buffer cannot hide). The backend now logs a clear warning with a link to the docs when RESERVOIR_BUFFER_ENABLED=true is set together with STORAGE_ENGINE=clickhouse or STORAGE_ENGINE=mongodb
• k6 load-test script load-tests/buffered-vs-sync.js (pnpm --filter @logtide/backend load:buffered-vs-sync): constant-arrival-rate 100 req/s for 3 min, batch size 10, reports p50/p95/p99 + error rate. The file lives in the gitignored load-tests/ directory; only the npm script is checked in
• Graceful shutdown now drains the reservoir buffer in parallel with Fastify app.close(): previously serialized, so a slow app.close() (BullMQ workers, websocket connections) could exhaust the container stop timeout before shutdownReservoir() ran. The new order calls them via Promise.all, giving the buffer its full RESERVOIR_BUFFER_GRACEFUL_SHUTDOWN_MS window to drain regardless of how long app.close() takes
• Redis client in the backend’s buffer path is now explicitly quit()-ed on shutdown: RedisStreamTransport documents that it does not own the client, so the backend owns the lifecycle. Without this, the Redis connection was torn down by the process exit instead of closing cleanly
• Buffer start-up failure is fail-fast: when RESERVOIR_BUFFER_ENABLED=true and the consumer pool cannot start (e.g. Redis down at boot), the backend logs a CRITICAL message and process.exit(1) instead of silently continuing with a non-flushing buffer
• RedisStreamTransport hardening: XPENDING now filters by consumer name in claimStale, so delivery-count mapping is accurate even on busy streams with claims from other consumers; getStats.oldestPendingAgeMs now tracks the oldest not-yet-delivered entry (via XRANGE with exclusive start past the PEL tail) to match the documented contract; nack uses MULTI/EXEC so the DLQ write and the original ACK either both land or neither does; enqueueMany inspects pipeline exec() results and throws on the first error instead of swallowing them; default consumer name changed from consumer-${pid} to ${hostname}-${pid}-${randomHex} to avoid collisions between containers with the same PID
• FlushConsumer isolates partial failures by kind: the three ingest calls (ingest, ingestSpans, ingestMetrics) now run via Promise.allSettled; the breaker records success only when all three succeed; metrics reflect per-kind reality; DLQ only counts the failed kinds on retry exhaustion. An optional FlushLogger can be injected; default falls back to console
• FlushConsumerPool relaxes drain check to pendingRecords == 0: previously also required inflightRecords == 0, which never converged when another backend instance held claims on the same Redis consumer group. Consumer-task crashes are now surfaced via logger.error instead of being swallowed by Promise.allSettled. Drain poll bumped from 50 ms to 200 ms to reduce chatter
• InMemoryTransport uses signal-based wakeup: dequeue no longer polls every 50 ms; enqueues wake waiters directly via a per-shard resolver list, and stop() unblocks waiters promptly. enqueueMany now batches records into a single Array.push(...entries) per shard plus one wake, eliminating the N-await bottleneck that made the in-memory buffer slower than sync on fast TimescaleDB workloads (p95 dropped from 4224 ms to 24 ms in the 3-min 100 req/s test)
• Reservoir exposes getEngine() on the concrete Reservoir class (not on IReservoir): replaces the (inner as unknown as { engine }).engine cast inside ReservoirBuffered’s constructor. Kept out of the public interface so backend consumers typed against IReservoir cannot reach the engine and bypass the decorator
• ReservoirBuffered.ingest/ingestSpans/ingestMetrics return the correct IngestResult shape: { ingested, failed, durationMs } instead of the previous { inserted } masked by as unknown as casts. durationMs measures the enqueue time, so the value reflects the real cost imposed on the caller