LogTide
Login Get Started

Architecture

Understanding LogTide's system architecture and design decisions.

System Overview

LogTide follows a modern microservices architecture with clear separation of concerns:

Data Hierarchy 
User → Organizations (1:N) → Projects (1:N) → API Keys → Logs
  • Organizations - Top-level isolation for companies/teams. Each user can belong to multiple organizations.
  • Projects - Logical grouping within organizations (e.g., "production", "staging"). Complete data isolation.
  • API Keys - Project-scoped keys for secure log ingestion and query. Prefixed with lp_.
  • Logs - Time-series data stored in TimescaleDB with automatic compression and retention policies.

Technology Stack

Backend

Runtime: Node.js 20+

Framework: Fastify

Language: TypeScript 5

ORM: Kysely (type-safe SQL)

Queue: BullMQ + Redis

Validation: Zod schemas

Frontend

Framework: SvelteKit 5 (Runes)

Language: TypeScript 5

Styling: TailwindCSS

Components: shadcn-svelte

Charts: ECharts

State: Svelte stores

Database

RDBMS: PostgreSQL 16

Extension: TimescaleDB

Time-series: Hypertables

Compression: Automatic

Retention: Configurable policies

Infrastructure

Cache: Redis 7

Proxy: Nginx

Container: Docker

Orchestration: Docker Compose

Monorepo: pnpm workspaces

Core Components

Backend Server (Fastify)

High-performance API server handling log ingestion, query, and management endpoints. Modular architecture with feature-based modules:

  • auth/ - Authentication and user management
  • ingestion/ - Log ingestion with batch support
  • query/ - Log search and filtering
  • alerts/ - Alert rule management
  • dashboard/ - Statistics and aggregations

Worker Process (BullMQ)

Background job processor for alert evaluation, notifications, and data retention. Runs independently from the main API server.

Frontend Dashboard (SvelteKit)

Modern, reactive UI with real-time log streaming, search, alerts management, and organization administration. Server-side rendering for optimal performance.

TimescaleDB

PostgreSQL extension optimized for time-series data. Automatic partitioning, compression, and retention policies for efficient long-term log storage.

Data Flow

Log Ingestion Flow

  1. Client sends logs via POST /api/v1/ingest with API key
  2. Backend validates API key and extracts project ID
  3. Logs are validated against Zod schema
  4. Batch insert into TimescaleDB hypertable
  5. Alert evaluator job is triggered (BullMQ)
  6. Logs are broadcast to active SSE streams

Alert Processing Flow

  1. Worker evaluates all enabled alert rules (every minute)
  2. For each rule, query logs matching conditions
  3. If threshold exceeded, create alert instance
  4. Send notifications (email/webhook) via configured channels
  5. Update alert status and last triggered timestamp

Log Retention

LogTide supports customizable log retention policies per organization, allowing administrators to control how long logs are stored before automatic deletion.

Retention Configuration

Range: 1 to 365 days

Default: 90 days

Scope: Organization-level (applies to all projects within the organization)

Cleanup: Daily at 2:00 AM (server time)

Admin Configuration

Only system administrators can modify retention settings. This is done through the Admin Panel under Organization Details:

  1. Navigate to Admin Panel → Organizations
  2. Click on the organization you want to configure
  3. Find the "Log Retention Policy" card
  4. Enter the desired retention period (1-365 days)
  5. Click "Save" to apply the changes

User Visibility

Regular users can view their organization's retention policy in read-only mode:

  1. Navigate to Organization Settings
  2. View the "Log Retention Policy" card showing the current retention period
  3. Contact your administrator if you need to change the retention policy

Cleanup Process

The retention cleanup runs as a background worker job:

  • Schedule: Daily at 2:00 AM server time
  • Startup: Also runs 2 minutes after worker starts
  • Process: Deletes logs older than the retention period for each organization
  • Logging: All cleanup operations are logged internally for audit purposes

Important Notes

  • Log deletion is permanent and cannot be undone
  • Only the logs table is affected by retention policies
  • Other data (spans, alert history, etc.) follows separate retention rules
  • TimescaleDB's global 90-day policy may still apply as a safety net

Key Design Decisions

Why TimescaleDB?
Native time-series optimizations, automatic compression, built-in retention policies, and PostgreSQL compatibility make it ideal for log storage with high ingestion rates.
Why Fastify?
Excellent performance, native TypeScript support, schema validation, plugin ecosystem, and lower overhead compared to Express make it perfect for high-throughput log ingestion.
Why SvelteKit 5?
Modern reactivity with Runes, excellent performance, built-in SSR, file-based routing, and minimal bundle size provide the best developer and user experience.
Edit this page on GitHub