Architecture
Understanding LogTide's system architecture and design decisions.
System Overview
LogTide follows a modern microservices architecture with clear separation of concerns:
User → Organizations (1:N) → Projects (1:N) → API Keys → Logs
- Organizations - Top-level isolation for companies/teams. Each user can belong to multiple organizations.
- Projects - Logical grouping within organizations (e.g., "production", "staging"). Complete data isolation.
- API Keys - Project-scoped keys for secure log
ingestion and query. Prefixed with
lp_. - Logs - Time-series data stored via the Reservoir storage abstraction, supporting both TimescaleDB and ClickHouse with automatic compression and retention policies.
Technology Stack
Runtime: Node.js 20+
Framework: Fastify
Language: TypeScript 5
ORM: Kysely (type-safe SQL)
Queue: BullMQ (Redis) or graphile-worker (PostgreSQL)
Validation: Zod schemas
Framework: SvelteKit 5 (Runes)
Language: TypeScript 5
Styling: TailwindCSS
Components: shadcn-svelte
Charts: ECharts
State: Svelte stores
Abstraction: Reservoir (pluggable engine)
Engine A: TimescaleDB (PostgreSQL extension)
Engine B: ClickHouse (columnar OLAP)
Compression: Automatic (both engines)
Retention: Configurable policies
Cache: Redis 7 (optional)
Proxy: Nginx
Container: Docker
Orchestration: Docker Compose
Monorepo: pnpm workspaces
Core Components
Backend Server (Fastify)
High-performance API server handling log ingestion, query, and management endpoints. Modular architecture with feature-based modules:
auth/- Authentication and user management-
ingestion/- Log ingestion with batch support query/- Log search and filteringalerts/- Alert rule managementdashboard/- Statistics and aggregations
Worker Process (BullMQ)
Background job processor for alert evaluation, notifications, and data retention. Runs independently from the main API server.
Frontend Dashboard (SvelteKit)
Modern, reactive UI with real-time log streaming, search, alerts management, and organization administration. Server-side rendering for optimal performance.
Storage Engines (Reservoir)
Log and span storage is managed by the Reservoir abstraction layer, which provides a unified API over two pluggable engines:
- TimescaleDB (default) — PostgreSQL extension with hypertables, continuous aggregates, and native compression. Best for deployments that already run PostgreSQL.
- ClickHouse — Columnar OLAP database optimized for high-volume analytical queries. Best for very high ingestion rates or large-scale deployments.
Data Flow
Log Ingestion Flow
-
Client sends logs via
POST /api/v1/ingestwith API key - Backend validates API key and extracts project ID
- Logs are validated against Zod schema
- Batch insert via Reservoir (TimescaleDB or ClickHouse)
- Alert evaluator job is triggered (BullMQ)
- Logs are broadcast to active SSE streams
Alert Processing Flow
- Worker evaluates all enabled alert rules (every minute)
- For each rule, query logs matching conditions
- If threshold exceeded, create alert instance
- Send notifications (email/webhook) via configured channels
- Update alert status and last triggered timestamp
Log Retention
LogTide supports customizable log retention policies per organization, allowing administrators to control how long logs are stored before automatic deletion.
Range: 1 to 365 days
Default: 90 days
Scope: Organization-level (applies to all projects within the organization)
Cleanup: Daily at 2:00 AM (server time)
Admin Configuration
Only system administrators can modify retention settings. This is done through the Admin Panel under Organization Details:
- Navigate to
Admin Panel → Organizations - Click on the organization you want to configure
- Find the "Log Retention Policy" card
- Enter the desired retention period (1-365 days)
- Click "Save" to apply the changes
User Visibility
Regular users can view their organization's retention policy in read-only mode:
- Navigate to
Organization Settings - View the "Log Retention Policy" card showing the current retention period
- Contact your administrator if you need to change the retention policy
Cleanup Process
The retention cleanup runs as a background worker job:
- Schedule: Daily at 2:00 AM server time
- Startup: Also runs 2 minutes after worker starts
- Process: Deletes logs older than the retention period for each organization
- Logging: All cleanup operations are logged internally for audit purposes
Important Notes
- Log deletion is permanent and cannot be undone
- Only the
logstable is affected by retention policies - Other data (spans, alert history, etc.) follows separate retention rules
- TimescaleDB's global 90-day policy may still apply as a safety net