# LogTide — Full Content

> The Official LogTide Platform. A privacy-first, open-source alternative to Datadog and Splunk. GDPR-compliant log management with native Sigma rules and threat detection. Self-host or use our EU-based cloud.

Source: https://logtide.dev/


# Integrations

---

# Angular Application Logging Integration

> Add structured logging to Angular applications with ErrorHandler, HttpInterceptor, and standalone or NgModule providers.

*Source: https://logtide.dev/integrations/angular/*

---

LogTide's Angular SDK provides a custom ErrorHandler for automatic error capture, an HttpInterceptor for distributed tracing, and providers for both standalone and NgModule-based applications.

## Why use LogTide with Angular?

- **Automatic error capture**: Custom ErrorHandler catches all uncaught errors
- **HTTP tracing**: HttpInterceptor adds traceparent headers to outgoing requests
- **Component context**: Errors include component name and lifecycle hook
- **Single setup**: `provideLogtide()` registers everything in one call
- **Both styles**: Supports standalone and NgModule-based apps

## Prerequisites

- Angular 17+ (standalone) or Angular 14+ (NgModule)
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/angular
```

## Standalone Setup (Angular 17+)

```typescript
// main.ts
import { bootstrapApplication } from '@angular/platform-browser';
import { provideLogtide } from '@logtide/angular';
import { AppComponent } from './app/app.component';

bootstrapApplication(AppComponent, {
  providers: [
    provideLogtide({
      dsn: 'https://lp_abc123@api.logtide.dev',
      service: 'angular-app',
      environment: 'production',
      release: '1.0.0',
    }),
  ],
});
```

## NgModule Setup

```typescript
// app.module.ts
import { NgModule } from '@angular/core';
import { getLogtideProviders } from '@logtide/angular';

@NgModule({
  providers: [
    ...getLogtideProviders({
      dsn: 'https://lp_abc123@api.logtide.dev',
      service: 'angular-app',
      environment: 'production',
    }),
  ],
  bootstrap: [AppComponent],
})
export class AppModule {}
```

## Automatic Error Capture

Uncaught errors in components, services, and templates are captured automatically with context:

```typescript
@Component({ selector: 'app-dashboard', template: '...' })
export class DashboardComponent implements OnInit {
  ngOnInit() {
    // If this throws, LogTide captures it with:
    // - component: 'DashboardComponent'
    // - hook: 'ngOnInit'
    // - route information
    this.loadData();
  }
}
```

## HTTP Interceptor

Outgoing HttpClient requests automatically include tracing:

```typescript
@Injectable()
export class UserService {
  constructor(private http: HttpClient) {}

  getUser(id: string) {
    // traceparent header added automatically
    // Failed requests captured as errors
    return this.http.get<User>(`/api/users/${id}`);
  }
}
```

## Manual Logging in Components

```typescript
import { hub } from '@logtide/core';

@Component({ selector: 'app-checkout', template: '...' })
export class CheckoutComponent {
  purchase() {
    hub.addBreadcrumb({
      type: 'ui',
      category: 'ui',
      message: 'Purchase button clicked',
      timestamp: Date.now(),
    });

    try {
      // process purchase...
      hub.captureLog('info', 'Purchase completed', { orderId: '123' });
    } catch (error) {
      hub.captureError(error, {
        tags: { module: 'checkout' },
      });
    }
  }
}
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Angular SDK Reference](/docs/sdks/angular/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# Apache HTTP Server Log Integration

> Send Apache httpd access logs, error logs, and mod_security audit events to LogTide with structured JSON parsing.

*Source: https://logtide.dev/integrations/apache-httpd/*

---

Apache HTTP Server (httpd) is one of the most widely deployed web servers on the internet. This guide shows you how to send Apache access logs, error logs, and mod_security audit events to LogTide with full structured parsing for powerful queries, alerting, and compliance reporting.

## Why use LogTide with Apache?

- **Unified visibility**: Combine access logs, error logs, and security audit trails in one place
- **Structured analysis**: Parse Apache logs into queryable fields for fast investigation
- **Security monitoring**: Detect web attacks, brute-force attempts, and suspicious activity in real time
- **Compliance audit trail**: Meet GDPR, PCI-DSS, and SOC 2 requirements with centralized, tamper-evident logging
- **Virtual host tracking**: Monitor traffic and errors across multiple virtual hosts from a single dashboard
- **Performance insights**: Track response times, request sizes, and status code distribution

## Prerequisites

- Apache HTTP Server 2.4+ (any platform)
- LogTide instance with API key
- Fluent Bit or Vector for log forwarding
- `mod_log_json` (optional, for native JSON logging)

## Quick Start (5 minutes)

### Step 1: Configure Apache JSON Log Format

The simplest approach uses Apache's built-in `CustomLog` directive with a JSON format string.

Edit your Apache configuration (`/etc/apache2/apache2.conf` or `/etc/httpd/conf/httpd.conf`):

```apache
# JSON access log format
LogFormat "{\"time\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"remote_addr\":\"%a\",\"remote_user\":\"%u\",\"request_method\":\"%m\",\"request_uri\":\"%U%q\",\"protocol\":\"%H\",\"status\":%>s,\"body_bytes_sent\":%B,\"request_time_us\":%D,\"http_referer\":\"%{Referer}i\",\"http_user_agent\":\"%{User-Agent}i\",\"http_x_forwarded_for\":\"%{X-Forwarded-For}i\",\"vhost\":\"%v\",\"server_port\":%{local}p,\"request_id\":\"%{unique_id}e\"}" json_combined

# Use the JSON format for access logs
CustomLog /var/log/apache2/access.json json_combined

# Error log (use default format; parsed by Fluent Bit)
ErrorLog /var/log/apache2/error.log
LogLevel warn
```

Enable `mod_unique_id` for request correlation:

```bash
# Debian/Ubuntu
sudo a2enmod unique_id

# RHEL/CentOS
# Add to httpd.conf:
# LoadModule unique_id_module modules/mod_unique_id.so
```

Reload Apache:

```bash
sudo apachectl configtest && sudo systemctl reload apache2
```

### Step 2: Set Up Fluent Bit

Create `/etc/fluent-bit/fluent-bit.conf`:

```ini
[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Path          /var/log/apache2/access.json
    Tag           apache.access
    Refresh_Interval 5

[INPUT]
    Name          tail
    Path          /var/log/apache2/error.log
    Parser        apache_error
    Tag           apache.error
    Refresh_Interval 5

[FILTER]
    Name          parser
    Match         apache.access
    Key_Name      log
    Parser        json
    Reserve_Data  On

[FILTER]
    Name          modify
    Match         apache.*
    Add           service apache

[FILTER]
    Name          modify
    Match         apache.access
    Add           log_type access

[FILTER]
    Name          modify
    Match         apache.error
    Add           log_type error
    Add           level error

[FILTER]
    Name          lua
    Match         apache.access
    script        /etc/fluent-bit/apache_message.lua
    call          create_message

[OUTPUT]
    Name          http
    Match         apache.*
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key YOUR_API_KEY
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

Create `/etc/fluent-bit/apache_message.lua`:

```lua
function create_message(tag, timestamp, record)
    local method = record["request_method"] or "GET"
    local uri = record["request_uri"] or "/"
    local status = record["status"] or 0
    record["message"] = string.format("%s %s %d", method, uri, status)

    -- Convert microseconds to milliseconds
    local us = tonumber(record["request_time_us"]) or 0
    record["request_time_ms"] = us / 1000

    -- Set level based on status code
    local s = tonumber(status) or 0
    if s >= 500 then
        record["level"] = "error"
    elseif s >= 400 then
        record["level"] = "warn"
    else
        record["level"] = "info"
    end

    return 1, timestamp, record
end
```

Create `/etc/fluent-bit/parsers.conf`:

```ini
[PARSER]
    Name        json
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S%z

[PARSER]
    Name        apache_error
    Format      regex
    Regex       ^\[(?<time>[^\]]+)\] \[(?<module>[^\]]*)\] \[pid (?<pid>\d+)\] (?:\[client (?<client>[^\]]+)\] )?(?<message>.*)$
    Time_Key    time
    Time_Format %a %b %d %H:%M:%S.%L %Y
```

Start Fluent Bit:

```bash
sudo systemctl enable fluent-bit
sudo systemctl start fluent-bit
```

### Step 3: Verify

```bash
# Generate traffic
curl -I http://localhost/

# Check in LogTide with filter: service:apache
```

## Virtual Host Logging

### Per-VirtualHost Log Configuration

Configure separate logging for each virtual host:

```apache
<VirtualHost *:80>
    ServerName app.example.com
    DocumentRoot /var/www/app

    # Per-vhost JSON access log
    CustomLog /var/log/apache2/vhosts/app.example.com-access.json json_combined

    # Per-vhost error log
    ErrorLog /var/log/apache2/vhosts/app.example.com-error.log
    LogLevel warn
</VirtualHost>

<VirtualHost *:80>
    ServerName api.example.com
    DocumentRoot /var/www/api

    CustomLog /var/log/apache2/vhosts/api.example.com-access.json json_combined
    ErrorLog /var/log/apache2/vhosts/api.example.com-error.log
    LogLevel warn
</VirtualHost>
```

Collect all virtual host logs with a wildcard:

```ini
[INPUT]
    Name          tail
    Path          /var/log/apache2/vhosts/*-access.json
    Tag           apache.access.vhost
    Refresh_Interval 5
```

### Conditional Logging

Log only specific requests (e.g., exclude health checks):

```apache
# Don't log health check requests
SetEnvIf Request_URI "^/health$" no_log
SetEnvIf Request_URI "^/readyz$" no_log

CustomLog /var/log/apache2/access.json json_combined env=!no_log
```

## mod_log_json (Native JSON)

For cleaner JSON output, use `mod_log_json` if available:

```bash
# Install on Debian/Ubuntu
sudo apt-get install libapache2-mod-log-json
sudo a2enmod log_json
```

Configuration:

```apache
LogFormat json_combined
JSONLog On
JSONLogFields time remote_addr request_method request_uri status body_bytes_sent request_time http_referer http_user_agent

CustomLog /var/log/apache2/access.json json_combined
```

## mod_security Audit Logging

### Enable mod_security Audit Logs

mod_security provides a web application firewall (WAF) with detailed audit logging:

```bash
# Install mod_security
sudo apt-get install libapache2-mod-security2
sudo a2enmod security2
```

Configure audit logging:

```apache
# /etc/modsecurity/modsecurity.conf
SecRuleEngine On

# Audit logging
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABCDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log

# JSON audit log (mod_security 3.x)
SecAuditLogFormat JSON
```

### Forward mod_security Logs to LogTide

Add a Fluent Bit input for mod_security:

```ini
[INPUT]
    Name          tail
    Path          /var/log/apache2/modsec_audit.log
    Tag           apache.modsecurity
    Parser        json
    Refresh_Interval 5

[FILTER]
    Name          record_modifier
    Match         apache.modsecurity
    Record        service apache
    Record        log_type modsecurity
    Record        level warning
```

### OWASP Core Rule Set

Install the OWASP CRS for comprehensive protection:

```bash
sudo apt-get install modsecurity-crs

# Enable in Apache
sudo cp /etc/modsecurity/crs/crs-setup.conf.example /etc/modsecurity/crs/crs-setup.conf
```

Events from OWASP rules appear in LogTide with details about the matched rule and attack type.

## Production Configuration

### Enhanced Log Format with Security Fields

```apache
# Production JSON format with security-relevant fields
LogFormat "{\"time\":\"%{%Y-%m-%dT%H:%M:%S%z}t\",\"remote_addr\":\"%a\",\"remote_user\":\"%u\",\"request_method\":\"%m\",\"request_uri\":\"%U%q\",\"protocol\":\"%H\",\"status\":%>s,\"body_bytes_sent\":%B,\"request_time_us\":%D,\"http_referer\":\"%{Referer}i\",\"http_user_agent\":\"%{User-Agent}i\",\"http_x_forwarded_for\":\"%{X-Forwarded-For}i\",\"http_x_real_ip\":\"%{X-Real-Ip}i\",\"vhost\":\"%v\",\"server_port\":%{local}p,\"request_id\":\"%{unique_id}e\",\"ssl_protocol\":\"%{SSL_PROTOCOL}x\",\"ssl_cipher\":\"%{SSL_CIPHER}x\",\"content_type\":\"%{Content-Type}o\",\"request_bytes\":%I,\"response_bytes\":%O,\"connection_status\":\"%X\"}" json_production

CustomLog /var/log/apache2/access.json json_production
```

Connection status values:

| Value | Meaning |
|-------|---------|
| `X` | Connection aborted before response completed |
| `+` | Connection may be kept alive |
| `-` | Connection closed after response |

### Log Rotation

Configure logrotate for Apache logs:

```bash
# /etc/logrotate.d/apache2-json
/var/log/apache2/access.json {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if invoke-rc.d apache2 status > /dev/null 2>&1; then
            invoke-rc.d apache2 reload > /dev/null
        fi
    endscript
}
```

## Docker Setup

### docker-compose.yml

```yaml
services:
  apache:
    image: httpd:2.4-alpine
    container_name: apache
    volumes:
      - ./httpd.conf:/usr/local/apache2/conf/httpd.conf:ro
      - ./vhosts:/usr/local/apache2/conf/extra:ro
      - apache_logs:/var/log/apache2
      - ./htdocs:/usr/local/apache2/htdocs:ro
    ports:
      - "80:80"
      - "443:443"

  fluent-bit:
    image: fluent/fluent-bit:latest
    container_name: fluent-bit
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - ./fluent-bit/apache_message.lua:/etc/fluent-bit/apache_message.lua:ro
      - apache_logs:/var/log/apache2:ro
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      - apache
    restart: unless-stopped

volumes:
  apache_logs:
```

### Docker stdout/stderr Alternative

Configure Apache to log to stdout for Docker-native log collection:

```apache
# In httpd.conf
ErrorLog /proc/self/fd/2
CustomLog /proc/self/fd/1 json_combined
```

Then use Docker's Fluentd log driver as shown in the [Docker integration guide](/integrations/docker/).

## Vector Configuration (Alternative)

If you prefer Vector over Fluent Bit:

```toml
# /etc/vector/vector.toml

[sources.apache_access]
type = "file"
include = ["/var/log/apache2/access.json"]
read_from = "end"

[transforms.parse_access]
type = "remap"
inputs = ["apache_access"]
source = '''
. = parse_json!(.message)
.service = "apache"
.log_type = "access"
.request_time_ms = to_float(.request_time_us) / 1000.0 ?? 0.0
.level = if .status >= 500 { "error" } else if .status >= 400 { "warn" } else { "info" }
.message = join!([.request_method, .request_uri, to_string!(.status)], separator: " ")
'''

[sources.apache_error]
type = "file"
include = ["/var/log/apache2/error.log"]
read_from = "end"

[transforms.parse_error]
type = "remap"
inputs = ["apache_error"]
source = '''
. = parse_apache_log!(.message, format: "error")
.service = "apache"
.log_type = "error"
.level = "error"
'''

[sinks.logtide]
type = "http"
inputs = ["parse_access", "parse_error"]
uri = "https://YOUR_LOGTIDE_HOST/api/v1/ingest"
method = "post"
encoding.codec = "json"

[sinks.logtide.request.headers]
X-API-Key = "${LOGTIDE_API_KEY}"
Content-Type = "application/json"
```

## Detection Rules

Create alerts for common Apache issues:

### High Error Rate

```
service:apache AND status:>=500
```

Threshold: >10 errors in 5 minutes.

### Slow Requests

```
service:apache AND request_time_ms:>3000
```

Requests taking longer than 3 seconds.

### mod_security Alerts

```
service:apache AND log_type:modsecurity
```

Any occurrence triggers an alert for WAF events.

### Brute Force Detection

```
service:apache AND status:401
```

Threshold: >20 failures in 1 minute from the same `remote_addr`.

### Directory Traversal Attempts

```
service:apache AND request_uri:*../*
```

Any occurrence -- this is always suspicious.

### Large Request Bodies

```
service:apache AND request_bytes:>1048576
```

Requests over 1MB may indicate upload abuse.

## Verification

After setup, verify in LogTide that structured fields are available:

| Field | Example Value |
|-------|---------------|
| service | apache |
| status | 200 |
| request_method | GET |
| request_uri | /api/users?page=1 |
| request_time_ms | 45.2 |
| remote_addr | 192.168.1.100 |
| vhost | app.example.com |
| request_id | ZX1234abc |

## Performance Metrics

| Metric | Value | Notes |
|--------|-------|-------|
| Log overhead | <1ms per request | JSON format with `mod_unique_id` |
| Disk I/O | ~600 bytes/request | Full production format |
| Memory (Fluent Bit) | ~30MB | Tailing access + error logs |
| Memory (Vector) | ~50MB | With JSON parsing transforms |

## Privacy Considerations

For GDPR and PCI-DSS compliance:

### IP Anonymization

Use `mod_removeip` or a custom log format:

```apache
# Option 1: Use mod_remoteip to anonymize
# Option 2: Custom format that truncates IP
LogFormat "{\"remote_addr\":\"%{REMOTE_ADDR_ANON}e\",...}" json_anon

# Set anonymized IP via RewriteRule
RewriteEngine On
RewriteRule .* - [E=REMOTE_ADDR_ANON:%{REMOTE_ADDR}]
```

### Sensitive Header Filtering

Never log `Authorization` or `Cookie` headers:

```apache
LogFormat "{\"http_authorization\":\"[REDACTED]\",...}" json_safe
```

### Retention Policy

Configure LogTide retention to match your compliance requirements. Typical values:

- **GDPR**: 30-90 days for web server logs
- **PCI-DSS**: 1 year minimum for security-relevant logs
- **SOC 2**: 1 year minimum

## Troubleshooting

### JSON logs not valid

1. Check for special characters in request URIs:
   ```bash
   tail -1 /var/log/apache2/access.json | python3 -m json.tool
   ```

2. Ensure proper escaping in the `LogFormat` directive. Apache does not escape JSON special characters in `%U` and `%q` by default. Consider using `mod_log_json` for proper escaping.

### Error log not being parsed

1. Verify the error log format matches the parser regex:
   ```bash
   tail -5 /var/log/apache2/error.log
   ```

2. Apache 2.4 uses a different error log format than 2.2. Ensure your parser matches the version you're running.

### Logs not appearing in LogTide

1. Check Apache is writing to the expected path:
   ```bash
   ls -la /var/log/apache2/access.json
   tail -f /var/log/apache2/access.json
   ```

2. Verify Fluent Bit can read the files:
   ```bash
   sudo systemctl status fluent-bit
   sudo journalctl -u fluent-bit -f
   ```

3. Test LogTide connectivity:
   ```bash
   curl -X POST https://YOUR_LOGTIDE_HOST/api/v1/ingest \
     -H "X-API-Key: YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"logs":[{"service":"apache","level":"info","message":"test"}]}'
   ```

### mod_security audit log too large

Reduce audit log verbosity:

```apache
# Only log relevant status codes
SecAuditLogRelevantStatus "^5"

# Reduce logged parts
SecAuditLogParts ABZ
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log collection
- [systemd Integration](/integrations/systemd/) - Collect Apache logs via journald
- [PHP Integration](/integrations/php/) - Application-level logging from PHP
- [Security Monitoring](/use-cases/security-monitoring/) - Build alerting rules on Apache events
- [Compliance Audit Trail](/use-cases/compliance-audit-trail/) - Meet compliance requirements


---

# Apache Kafka Log Pipeline Integration

> Use Apache Kafka as a log transport layer to buffer and route high-volume log streams into LogTide reliably.

*Source: https://logtide.dev/integrations/kafka/*

---

When you're generating millions of log events per second, shipping directly to LogTide may not be enough. Apache Kafka acts as a durable buffer between your applications and LogTide, providing backpressure handling, replay capability, and multi-consumer fanout. This guide shows you how to build a Kafka-backed log pipeline.

## Why Kafka for log transport?

- **Backpressure handling**: Applications never block on log shipping — Kafka absorbs bursts
- **Durability**: Logs are persisted to disk before acknowledgment — no data loss on restarts
- **Replay**: Re-process historical logs by resetting consumer offsets
- **Fanout**: Send the same log stream to LogTide, S3 archival, and real-time alerting
- **Decoupling**: Applications don't need to know about LogTide — they publish to Kafka
- **Throughput**: Kafka handles millions of messages per second per partition

## Architecture

```d2
direction: right
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

app1: App 1 { class: src }
app2: App 2 { class: src }
app3: App 3 { class: src }
nginx: nginx { class: src }
kafka: Kafka Cluster { shape: queue; class: hub }
lt: LogTide { class: hub }
s3: S3 Archive { shape: cylinder; class: src }
alerting: Alerting { class: src }

app1 -> kafka { class: flow }
app2 -> kafka { class: flow }
app3 -> kafka { class: flow }
nginx -> kafka { class: flow }
kafka -> lt { class: flow }
kafka -> s3 { class: flow }
kafka -> alerting { class: flow }
```

Applications produce logs to Kafka topics. A Kafka Connect sink or Fluent Bit consumer ships logs to LogTide. Other consumers can archive to S3 or trigger real-time alerts.

## Prerequisites

- Apache Kafka 3.0+ (or Redpanda as a Kafka-compatible alternative)
- Fluent Bit or a custom consumer
- LogTide instance with API key
- Docker or Kubernetes for deployment

## Kafka Topic Setup

### Create Log Topics

```bash
# Create a topic for application logs
kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic logs.application \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=86400000 \        # 24 hour retention
  --config retention.bytes=10737418240 \  # 10 GB max per partition
  --config compression.type=lz4

# Create a topic for infrastructure logs
kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic logs.infrastructure \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=86400000 \
  --config compression.type=lz4

# Create a topic for security events (higher retention)
kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --topic logs.security \
  --partitions 6 \
  --replication-factor 3 \
  --config retention.ms=604800000 \       # 7 day retention
  --config compression.type=lz4
```

### Topic Naming Convention

```
logs.{category}

Examples:
  logs.application       # App-level logs
  logs.infrastructure    # nginx, systemd, etc.
  logs.security          # Auth events, SIEM
  logs.{service-name}    # Per-service topics (optional)
```

## Producing Logs to Kafka

### Node.js Producer

```typescript
// lib/kafka-logger.ts
import { Kafka, Partitioners } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'my-service',
  brokers: process.env.KAFKA_BROKERS!.split(','),
});

const producer = kafka.producer({
  createPartitioner: Partitioners.DefaultPartitioner,
  allowAutoTopicCreation: false,
});

await producer.connect();

interface LogEvent {
  level: string;
  message: string;
  service: string;
  metadata?: Record<string, unknown>;
}

export async function sendLog(event: LogEvent) {
  await producer.send({
    topic: 'logs.application',
    messages: [{
      // Partition by service name for ordering
      key: event.service,
      value: JSON.stringify({
        ...event,
        timestamp: new Date().toISOString(),
        hostname: process.env.HOSTNAME,
      }),
      headers: {
        'content-type': 'application/json',
      },
    }],
  });
}

// Graceful shutdown
process.on('SIGTERM', async () => {
  await producer.disconnect();
});
```

### Python Producer

```python
# lib/kafka_logger.py
import json
import os
import socket
from datetime import datetime, timezone
from confluent_kafka import Producer

conf = {
    'bootstrap.servers': os.environ['KAFKA_BROKERS'],
    'client.id': 'my-service',
    'compression.type': 'lz4',
    'linger.ms': 50,         # Batch for 50ms
    'batch.num.messages': 1000,
}

producer = Producer(conf)

def send_log(level: str, message: str, service: str, **metadata):
    event = {
        'level': level,
        'message': message,
        'service': service,
        'metadata': metadata,
        'timestamp': datetime.now(timezone.utc).isoformat(),
        'hostname': socket.gethostname(),
    }

    producer.produce(
        topic='logs.application',
        key=service.encode('utf-8'),
        value=json.dumps(event).encode('utf-8'),
    )

    # Trigger delivery callbacks
    producer.poll(0)

def flush():
    producer.flush(timeout=10)
```

## Consuming Logs into LogTide

### Fluent Bit Kafka Consumer

```ini
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Log_Level     info

# Consume from Kafka
[INPUT]
    Name          kafka
    Tag           kafka.logs
    Brokers       kafka-1:9092,kafka-2:9092,kafka-3:9092
    Topics        logs.application,logs.infrastructure,logs.security
    Group_Id      logtide-consumer
    Format        json
    Poll_Ms       100

# Output to LogTide
[OUTPUT]
    Name          http
    Match         *
    Host          api.logtide.dev
    Port          443
    URI           /api/v1/ingest/single
    Format        json_lines
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
    Retry_Limit   5
```

### Custom Consumer (Node.js)

For more control, use a custom consumer:

```typescript
// consumer.ts
import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'logtide-consumer',
  brokers: process.env.KAFKA_BROKERS!.split(','),
});

const consumer = kafka.consumer({
  groupId: 'logtide-ingest',
  maxWaitTimeInMs: 100,
  maxBytesPerPartition: 1048576, // 1MB per partition
});

const LOGTIDE_API_URL = process.env.LOGTIDE_API_URL!;
const LOGTIDE_API_KEY = process.env.LOGTIDE_API_KEY!;

const buffer: any[] = [];
const BATCH_SIZE = 100;
const FLUSH_INTERVAL = 5000;

async function flushToLogTide() {
  if (buffer.length === 0) return;

  const batch = buffer.splice(0, BATCH_SIZE);

  const response = await fetch(`${LOGTIDE_API_URL}/api/v1/ingest`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Key': LOGTIDE_API_KEY,
    },
    body: JSON.stringify({ logs: batch }),
  });

  if (!response.ok) {
    console.error(`LogTide ingest failed: ${response.status}`);
    // Put back in buffer for retry
    buffer.unshift(...batch);
  }
}

setInterval(flushToLogTide, FLUSH_INTERVAL);

await consumer.connect();
await consumer.subscribe({
  topics: ['logs.application', 'logs.infrastructure', 'logs.security'],
});

await consumer.run({
  eachMessage: async ({ topic, message }) => {
    try {
      const event = JSON.parse(message.value!.toString());
      event._topic = topic;
      buffer.push(event);

      if (buffer.length >= BATCH_SIZE) {
        await flushToLogTide();
      }
    } catch (error) {
      console.error('Failed to parse message:', error);
    }
  },
});
```

## Docker Setup

```yaml
# docker-compose.yml
services:
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_LOG_RETENTION_HOURS: 24
      KAFKA_LOG_SEGMENT_BYTES: 1073741824
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: false
      CLUSTER_ID: kafka-logtide-cluster-001
    volumes:
      - kafka_data:/var/lib/kafka/data

  kafka-consumer:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit-kafka.conf:/fluent-bit/etc/fluent-bit.conf:ro
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      - kafka

volumes:
  kafka_data:
```

## Kubernetes Deployment

```yaml
# kafka-consumer-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logtide-kafka-consumer
spec:
  replicas: 3  # One per Kafka partition group
  selector:
    matchLabels:
      app: logtide-kafka-consumer
  template:
    metadata:
      labels:
        app: logtide-kafka-consumer
    spec:
      containers:
        - name: consumer
          image: fluent/fluent-bit:latest
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc/
          env:
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: api-key
            - name: KAFKA_BROKERS
              value: "kafka-0.kafka:9092,kafka-1.kafka:9092,kafka-2.kafka:9092"
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "512Mi"
              cpu: "500m"
      volumes:
        - name: config
          configMap:
            name: kafka-consumer-config
```

## Monitoring the Pipeline

### Consumer Lag

Track consumer lag to ensure LogTide keeps up:

```bash
kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --group logtide-consumer \
  --describe
```

### Pipeline Health Metrics

| Metric | Healthy | Warning | Critical |
|--------|---------|---------|----------|
| Consumer lag | <1000 | 1000-10000 | >10000 |
| Throughput | >1000 msgs/s | - | <100 msgs/s |
| Partition balance | Even | 2x skew | 5x skew |
| Error rate | <0.01% | 0.01-1% | >1% |

## Performance Tuning

### Producer Tuning

```
linger.ms=50          # Batch window (50ms)
batch.size=65536      # Max batch size (64KB)
compression.type=lz4  # Compress batches
acks=1                # Leader acknowledgment (balance speed/durability)
```

### Consumer Tuning

```
fetch.min.bytes=1048576    # Wait for 1MB before fetching
fetch.max.wait.ms=500      # Max wait time
max.poll.records=500       # Process 500 records per poll
```

### Partition Strategy

- **12 partitions** for high-throughput topics (allows 12 parallel consumers)
- **6 partitions** for moderate topics
- **3 partitions** for low-volume security events

## Troubleshooting

### Consumer lag increasing

1. Check consumer health: `kafka-consumer-groups.sh --describe`
2. Add more consumer instances (up to partition count)
3. Check if LogTide is accepting logs (connection issues?)
4. Increase batch size for more efficient shipping

### Messages not being consumed

1. Verify topic exists: `kafka-topics.sh --list`
2. Check consumer group: `kafka-consumer-groups.sh --describe`
3. Verify Fluent Bit Kafka plugin configuration

### Data loss during restarts

Ensure `acks=all` for critical topics:

```
acks=all
min.insync.replicas=2
```

This guarantees logs survive broker failures.

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log routing to Kafka
- [Kubernetes Integration](/integrations/kubernetes/) - Kafka on K8s
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Alert on Kafka streams
- [Node.js Integration](/integrations/nodejs/) - Application log producers


---

# ASP.NET Core Logging Integration

> Add structured logging to ASP.NET Core applications with ILogger integration, Entity Framework tracing, and Docker deployment.

*Source: https://logtide.dev/integrations/dotnet/*

---

LogTide's C# SDK integrates with ASP.NET Core's built-in `ILogger` infrastructure, providing structured logging with automatic request tracing, Entity Framework query logging, and dependency injection support. This guide covers middleware setup, the ILogger provider, background services, and production deployment.

## Why use LogTide with ASP.NET Core?

- **ILogger integration**: Drop-in provider that works with the standard `ILogger<T>` interface
- **Request middleware**: Automatic HTTP request/response logging with timing and trace IDs
- **Dependency injection**: Register once, inject everywhere via ASP.NET Core DI
- **Entity Framework**: Log slow queries and database operations with execution time
- **Background services**: Trace `IHostedService` and worker processes
- **Structured logging**: First-class support for scopes, message templates, and semantic values
- **Non-blocking**: Async batching keeps request latency unaffected

## Prerequisites

- .NET 8.0+ (.NET 9.0 supported)
- ASP.NET Core
- LogTide instance with API key

## Installation

Install the `LogTide.SDK` NuGet package:

```bash
dotnet add package LogTide.SDK
```

Or via the Package Manager Console:

```powershell
Install-Package LogTide.SDK
```

Or add directly to your `.csproj`:

```xml
<PackageReference Include="LogTide.SDK" Version="0.9.4" />
```

## Quick Start

### 1. Configure Services

```csharp
// Program.cs
using LogTide.SDK.Core;

var builder = WebApplication.CreateBuilder(args);

// Register LogTide
builder.Services.AddLogTide(options =>
{
    options.ApiUrl = builder.Configuration["LogTide:ApiUrl"]!;
    options.ApiKey = builder.Configuration["LogTide:ApiKey"]!;
    options.DefaultService = "dotnet-app";
    options.BatchSize = 50;
    options.FlushIntervalMs = 5000;
    options.GlobalMetadata = new Dictionary<string, object>
    {
        ["environment"] = builder.Environment.EnvironmentName,
        ["version"] = typeof(Program).Assembly.GetName().Version?.ToString() ?? "unknown"
    };
});

// Add the LogTide ILogger provider
builder.Logging.AddLogTide();

builder.Services.AddControllers();
var app = builder.Build();

app.UseLogTideMiddleware();
app.MapControllers();
app.Run();
```

### 2. Configure App Settings

```json
// appsettings.json
{
  "LogTide": {
    "ApiUrl": "https://your-logtide-instance.example.com",
    "ApiKey": "lp_your_api_key_here"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.EntityFrameworkCore": "Warning"
    }
  }
}
```

For production, use environment variables or user secrets:

```bash
export LogTide__ApiUrl="https://your-logtide-instance.example.com"
export LogTide__ApiKey="lp_your_api_key_here"
```

### 3. Use in Controllers

```csharp
// Controllers/UsersController.cs
using Microsoft.AspNetCore.Mvc;

[ApiController]
[Route("api/[controller]")]
public class UsersController : ControllerBase
{
    private readonly ILogger<UsersController> _logger;
    private readonly UserService _userService;

    public UsersController(ILogger<UsersController> logger, UserService userService)
    {
        _logger = logger;
        _userService = userService;
    }

    [HttpGet("{id}")]
    public async Task<IActionResult> GetUser(int id)
    {
        _logger.LogInformation("Fetching user profile for {UserId}", id);

        var user = await _userService.GetByIdAsync(id);
        if (user is null)
        {
            _logger.LogWarning("User {UserId} not found", id);
            return NotFound();
        }

        return Ok(user);
    }

    [HttpPost]
    public async Task<IActionResult> CreateUser([FromBody] CreateUserRequest request)
    {
        _logger.LogInformation("Creating user with email {Email}", request.Email);

        try
        {
            var user = await _userService.CreateAsync(request);
            _logger.LogInformation("User {UserId} created successfully", user.Id);
            return CreatedAtAction(nameof(GetUser), new { id = user.Id }, user);
        }
        catch (DuplicateEmailException)
        {
            _logger.LogWarning("Duplicate email address: {Email}", request.Email);
            return Conflict(new { error = "Email already exists" });
        }
    }
}
```

## ASP.NET Core Middleware

### Request Logging Middleware

Create middleware that logs every request with timing, trace IDs, and user context:

```csharp
// Middleware/LogTideRequestMiddleware.cs
using System.Diagnostics;
using LogTide.SDK.Core;

public class LogTideRequestMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogTideClient _client;
    private static readonly HashSet<string> SkipPaths = new()
    {
        "/health", "/ready", "/metrics", "/swagger"
    };

    public LogTideRequestMiddleware(RequestDelegate next, ILogTideClient client)
    {
        _next = next;
        _client = client;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // Generate or extract trace ID
        var traceId = context.Request.Headers["X-Trace-ID"].FirstOrDefault()
            ?? Activity.Current?.TraceId.ToString()
            ?? Guid.NewGuid().ToString();

        context.Items["TraceId"] = traceId;
        context.Response.Headers["X-Trace-Id"] = traceId;

        var stopwatch = Stopwatch.StartNew();

        try
        {
            await _next(context);
        }
        catch (Exception ex)
        {
            stopwatch.Stop();
            LogRequest(context, stopwatch.ElapsedMilliseconds, traceId, ex);
            throw;
        }

        stopwatch.Stop();

        if (!ShouldSkip(context.Request.Path))
        {
            LogRequest(context, stopwatch.ElapsedMilliseconds, traceId);
        }
    }

    private void LogRequest(HttpContext context, long durationMs, string traceId, Exception? exception = null)
    {
        var statusCode = exception != null ? 500 : context.Response.StatusCode;
        var level = statusCode >= 500 ? "error" : statusCode >= 400 ? "warning" : "info";

        var metadata = new Dictionary<string, object>
        {
            ["method"] = context.Request.Method,
            ["path"] = context.Request.Path.ToString(),
            ["statusCode"] = statusCode,
            ["durationMs"] = durationMs,
            ["traceId"] = traceId,
            ["ip"] = GetClientIp(context),
            ["userAgent"] = context.Request.Headers.UserAgent.ToString(),
        };

        if (context.User.Identity?.IsAuthenticated == true)
        {
            metadata["userId"] = context.User.FindFirst("sub")?.Value ?? "unknown";
        }

        if (exception != null)
        {
            metadata["error"] = exception.Message;
            metadata["errorType"] = exception.GetType().Name;
            metadata["stackTrace"] = exception.StackTrace?.Substring(0, Math.Min(exception.StackTrace.Length, 1000)) ?? "";
        }

        _client.Log(level, $"{context.Request.Method} {context.Request.Path} {statusCode}", metadata);
    }

    private static string GetClientIp(HttpContext context)
    {
        var forwarded = context.Request.Headers["X-Forwarded-For"].FirstOrDefault();
        if (!string.IsNullOrEmpty(forwarded))
            return forwarded.Split(',')[0].Trim();

        return context.Connection.RemoteIpAddress?.ToString() ?? "unknown";
    }

    private static bool ShouldSkip(PathString path)
    {
        return SkipPaths.Any(p => path.StartsWithSegments(p));
    }
}

// Extension method for clean registration
public static class LogTideMiddlewareExtensions
{
    public static IApplicationBuilder UseLogTideMiddleware(this IApplicationBuilder builder)
    {
        return builder.UseMiddleware<LogTideRequestMiddleware>();
    }
}
```

### Request Logging Output

Each request generates a structured log:

```json
{
  "level": "info",
  "message": "GET /api/users/42 200",
  "service": "dotnet-app",
  "metadata": {
    "method": "GET",
    "path": "/api/users/42",
    "statusCode": 200,
    "durationMs": 15,
    "traceId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "ip": "192.168.1.1",
    "userId": "42",
    "environment": "Production"
  }
}
```

## ILogger Provider Integration

The LogTide ILogger provider routes all `ILogger<T>` calls to LogTide while preserving scopes and message templates:

```csharp
// Services/OrderService.cs
public class OrderService
{
    private readonly ILogger<OrderService> _logger;
    private readonly AppDbContext _db;

    public OrderService(ILogger<OrderService> logger, AppDbContext db)
    {
        _logger = logger;
        _db = db;
    }

    public async Task<Order> ProcessOrderAsync(int orderId)
    {
        using (_logger.BeginScope(new Dictionary<string, object>
        {
            ["OrderId"] = orderId,
            ["Operation"] = "ProcessOrder"
        }))
        {
            _logger.LogInformation("Starting order processing");

            var order = await _db.Orders
                .Include(o => o.Items)
                .FirstOrDefaultAsync(o => o.Id == orderId);

            if (order is null)
            {
                _logger.LogWarning("Order {OrderId} not found", orderId);
                throw new OrderNotFoundException(orderId);
            }

            try
            {
                await ChargePaymentAsync(order);
                order.Status = OrderStatus.Completed;
                await _db.SaveChangesAsync();

                _logger.LogInformation(
                    "Order {OrderId} processed. Total: {Total:C}, Items: {ItemCount}",
                    orderId, order.Total, order.Items.Count);

                return order;
            }
            catch (PaymentFailedException ex)
            {
                _logger.LogError(ex,
                    "Payment failed for order {OrderId}. Amount: {Amount:C}",
                    orderId, order.Total);
                throw;
            }
        }
    }
}
```

## Dependency Injection

Register LogTide services and inject them throughout the application:

```csharp
// Program.cs
builder.Services.AddLogTide(options =>
{
    options.ApiUrl = builder.Configuration["LogTide:ApiUrl"]!;
    options.ApiKey = builder.Configuration["LogTide:ApiKey"]!;
    options.DefaultService = "dotnet-app";
});

// Register application services
builder.Services.AddScoped<OrderService>();
builder.Services.AddScoped<UserService>();
builder.Services.AddScoped<AuditService>();
```

### Direct Client Injection

For cases where you need the LogTide client directly (outside ILogger):

```csharp
// Services/AuditService.cs
public class AuditService
{
    private readonly ILogTideClient _logtide;

    public AuditService(ILogTideClient logtide)
    {
        _logtide = logtide;
    }

    public void LogAdminAction(string action, string userId, string resourceType, string resourceId)
    {
        _logtide.Log("info", $"Admin action: {action}", new Dictionary<string, object>
        {
            ["action"] = action,
            ["userId"] = userId,
            ["resourceType"] = resourceType,
            ["resourceId"] = resourceId,
            ["timestamp"] = DateTimeOffset.UtcNow.ToString("o"),
            ["source"] = "admin-panel"
        });
    }
}
```

## Entity Framework Logging

### Log Slow Queries

Intercept Entity Framework queries to log slow or problematic operations:

```csharp
// Data/LogTideDbInterceptor.cs
using Microsoft.EntityFrameworkCore.Diagnostics;
using System.Data.Common;
using System.Diagnostics;
using LogTide.SDK.Core;

public class LogTideDbInterceptor : DbCommandInterceptor
{
    private readonly ILogTideClient _client;
    private readonly int _slowQueryThresholdMs;

    public LogTideDbInterceptor(ILogTideClient client, int slowQueryThresholdMs = 100)
    {
        _client = client;
        _slowQueryThresholdMs = slowQueryThresholdMs;
    }

    public override DbDataReader ReaderExecuted(
        DbCommand command, CommandExecutedEventData eventData, DbDataReader result)
    {
        LogQuery(command, eventData.Duration);
        return base.ReaderExecuted(command, eventData, result);
    }

    public override ValueTask<DbDataReader> ReaderExecutedAsync(
        DbCommand command, CommandExecutedEventData eventData,
        DbDataReader result, CancellationToken cancellationToken = default)
    {
        LogQuery(command, eventData.Duration);
        return base.ReaderExecutedAsync(command, eventData, result, cancellationToken);
    }

    public override void CommandFailed(DbCommand command, CommandErrorEventData eventData)
    {
        _client.Log("error", "Database query failed", new Dictionary<string, object>
        {
            ["query"] = command.CommandText[..Math.Min(command.CommandText.Length, 500)],
            ["durationMs"] = eventData.Duration.TotalMilliseconds,
            ["error"] = eventData.Exception.Message,
            ["errorType"] = eventData.Exception.GetType().Name
        });

        base.CommandFailed(command, eventData);
    }

    private void LogQuery(DbCommand command, TimeSpan duration)
    {
        var durationMs = duration.TotalMilliseconds;

        if (durationMs >= _slowQueryThresholdMs)
        {
            _client.Log("warning", "Slow SQL query detected", new Dictionary<string, object>
            {
                ["query"] = command.CommandText[..Math.Min(command.CommandText.Length, 500)],
                ["durationMs"] = Math.Round(durationMs, 2),
                ["thresholdMs"] = _slowQueryThresholdMs
            });
        }
    }
}
```

### Register the Interceptor

```csharp
// Program.cs
builder.Services.AddDbContext<AppDbContext>((serviceProvider, options) =>
{
    options.UseNpgsql(builder.Configuration.GetConnectionString("DefaultConnection"));

    var logtideClient = serviceProvider.GetRequiredService<ILogTideClient>();
    options.AddInterceptors(new LogTideDbInterceptor(logtideClient, slowQueryThresholdMs: 100));
});
```

## Background Service Logging

Log from `IHostedService` and background workers:

```csharp
// Services/OrderProcessingService.cs
public class OrderProcessingService : BackgroundService
{
    private readonly ILogger<OrderProcessingService> _logger;
    private readonly IServiceScopeFactory _scopeFactory;

    public OrderProcessingService(
        ILogger<OrderProcessingService> logger,
        IServiceScopeFactory scopeFactory)
    {
        _logger = logger;
        _scopeFactory = scopeFactory;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Order processing service started");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                using var scope = _scopeFactory.CreateScope();
                var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();

                var pendingOrders = await db.Orders
                    .Where(o => o.Status == OrderStatus.Pending)
                    .Take(10)
                    .ToListAsync(stoppingToken);

                if (pendingOrders.Count > 0)
                {
                    _logger.LogInformation(
                        "Processing {Count} pending orders", pendingOrders.Count);

                    foreach (var order in pendingOrders)
                    {
                        using (_logger.BeginScope(new Dictionary<string, object>
                        {
                            ["OrderId"] = order.Id,
                            ["Task"] = "BackgroundProcessing"
                        }))
                        {
                            try
                            {
                                await ProcessSingleOrder(order, db);
                                _logger.LogInformation("Order {OrderId} processed", order.Id);
                            }
                            catch (Exception ex)
                            {
                                _logger.LogError(ex,
                                    "Failed to process order {OrderId}", order.Id);
                                order.Status = OrderStatus.Failed;
                            }
                        }
                    }

                    await db.SaveChangesAsync(stoppingToken);
                }

                await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                _logger.LogInformation("Order processing service stopping");
                break;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Order processing service encountered an error");
                await Task.Delay(TimeSpan.FromSeconds(60), stoppingToken);
            }
        }

        _logger.LogInformation("Order processing service stopped");
    }

    private async Task ProcessSingleOrder(Order order, AppDbContext db)
    {
        order.Status = OrderStatus.Processing;
        await db.SaveChangesAsync();
        // ... payment, notifications, etc.
        order.Status = OrderStatus.Completed;
    }
}
```

### Register the Background Service

```csharp
// Program.cs
builder.Services.AddHostedService<OrderProcessingService>();
```

## Health Check Integration

Add a LogTide connectivity health check:

```csharp
// HealthChecks/LogTideHealthCheck.cs
using Microsoft.Extensions.Diagnostics.HealthChecks;
using LogTide.SDK.Core;

public class LogTideHealthCheck : IHealthCheck
{
    private readonly ILogTideClient _client;

    public LogTideHealthCheck(ILogTideClient client)
    {
        _client = client;
    }

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        var isHealthy = _client.IsConnected;

        return Task.FromResult(isHealthy
            ? HealthCheckResult.Healthy("LogTide connection is active")
            : HealthCheckResult.Degraded("LogTide connection is unavailable"));
    }
}
```

```csharp
// Program.cs
builder.Services.AddHealthChecks()
    .AddCheck<LogTideHealthCheck>("logtide");
```

## Docker Deployment

### Dockerfile

```dockerfile
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src

COPY *.csproj .
RUN dotnet restore

COPY . .
RUN dotnet publish -c Release -o /app/publish --no-restore

FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine AS runtime
WORKDIR /app

# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

COPY --from=build /app/publish .

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
    CMD wget -q --spider http://localhost:8080/health || exit 1

ENTRYPOINT ["dotnet", "MyApp.dll"]
```

### Docker Compose

```yaml
# docker-compose.yml
services:
  api:
    build: .
    ports:
      - "8080:8080"
    environment:
      - ASPNETCORE_ENVIRONMENT=Production
      - ASPNETCORE_URLS=http://+:8080
      - LogTide__ApiUrl=${LOGTIDE_API_URL}
      - LogTide__ApiKey=${LOGTIDE_API_KEY}
      - ConnectionStrings__DefaultConnection=Host=db;Database=myapp;Username=user;Password=pass
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  pgdata:
```

## Graceful Shutdown

ASP.NET Core handles shutdown automatically when LogTide is registered via `AddLogTide()`. The SDK's `IDisposable` implementation flushes pending logs on host shutdown. For manual control:

```csharp
// Program.cs
var app = builder.Build();

app.Lifetime.ApplicationStopping.Register(() =>
{
    var client = app.Services.GetRequiredService<ILogTideClient>();
    client.Flush();
});
```

## Performance

| Metric | Value |
|--------|-------|
| Middleware overhead | <1ms per request |
| Memory overhead | ~12MB |
| Network calls | 1 per batch (50 logs) |
| EF interceptor overhead | <0.5ms per query |
| Background service overhead | Negligible |
| ILogger provider overhead | <0.2ms per log entry |

## Troubleshooting

### Logs not appearing

1. Verify configuration values are loaded correctly:
   ```csharp
   var config = builder.Configuration.GetSection("LogTide");
   Console.WriteLine($"ApiUrl: {config["ApiUrl"]}");
   ```
2. Ensure `AddLogTide()` is called on both `Services` and `Logging`
3. Check that the middleware is registered before `MapControllers()`

### ILogger scopes not appearing in LogTide

Ensure you are using `BeginScope` with a `Dictionary<string, object>`:

```csharp
using (_logger.BeginScope(new Dictionary<string, object>
{
    ["OrderId"] = orderId
}))
{
    _logger.LogInformation("Processing order");
}
```

### Entity Framework logs too verbose

Raise the EF Core log level in `appsettings.json`:

```json
{
  "Logging": {
    "LogLevel": {
      "Microsoft.EntityFrameworkCore": "Warning",
      "Microsoft.EntityFrameworkCore.Database.Command": "Warning"
    }
  }
}
```

### High memory usage in background services

When processing large batches, create a new DI scope per iteration and dispose it properly to release the LogTide batch buffer:

```csharp
using var scope = _scopeFactory.CreateScope();
// ... process batch
// Scope disposes here, flushing logs
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container deployment patterns
- [Kubernetes Integration](/integrations/kubernetes/) - Production K8s deployment
- [PostgreSQL Integration](/integrations/postgresql/) - Database logging correlation
- [Compliance Audit Trail](/use-cases/compliance-audit-trail/) - Regulatory logging
- [Security Monitoring](/use-cases/security-monitoring/) - Threat detection and alerting


---

# Astro Error Tracking & Logging Integration

> Professional Astro logging and error tracking with automatic capture, server/client monitoring, and Web Vitals.

*Source: https://logtide.dev/integrations/astro-frontend/*

---

LogTide's Astro integration provides professional-grade observability for both server-side and client-side code. It includes automatic unhandled error capture, middleware-based request logging, and Core Web Vitals collection.

## Why use LogTide with Astro?

- **Hybrid monitoring**: Track errors in server-side `Astro.js` components and client-side interactive islands.
- **Middleware logging**: Capture every request's status, duration, and potential errors automatically.
- **Core Web Vitals**: Monitor your app's performance (LCP, FID, CLS) as perceived by real users.
- **Breadcrumbs**: Automatically track clicks and network requests in the browser.
- **Distributed Tracing**: Connect frontend requests to backend logs for end-to-end visibility.

## Prerequisites

- Astro 3.0+ or 4.0+
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/core @logtide/browser
```

## Client-Side Setup

Initialize LogTide as early as possible in your layout or on specific pages using a `<script>` tag:

```astro
---
// src/layouts/Layout.astro
---
<script>
  import { initLogtide } from '@logtide/browser';

  initLogtide({
    dsn: import.meta.env.PUBLIC_LOGTIDE_DSN,
    service: 'astro-frontend',
    environment: import.meta.env.MODE,
    release: '1.0.0',
  });
</script>

<slot />
```

## Server-Side Setup (Middleware)

To capture server-side errors and request logs, use Astro middleware:

```typescript
// src/middleware.ts
import { defineMiddleware } from 'astro:middleware';
import { hub } from '@logtide/core';

// Initialize core SDK for server-side
hub.init({
  dsn: import.meta.env.LOGTIDE_DSN,
  service: 'astro-server',
});

export const onRequest = defineMiddleware(async (context, next) => {
  const startTime = Date.now();
  
  try {
    const response = await next();
    
    // Log successful requests
    hub.captureLog('info', `Request: ${context.request.method} ${context.url.pathname}`, {
      status: response.status,
      duration: Date.now() - startTime,
    });
    
    return response;
  } catch (error) {
    // Capture server-side errors
    hub.captureError(error, {
      extra: { url: context.url.pathname, method: context.request.method }
    });
    throw error;
  }
});
```

## Logging in Astro Components

You can log directly from your `.astro` files (server-side):

```astro
---
// src/pages/post/[id].astro
import { hub } from '@logtide/core';

const { id } = Astro.params;

hub.captureLog('info', 'Loading post', { postId: id });

const post = await fetchPost(id);
if (!post) {
  hub.captureLog('warn', 'Post not found', { postId: id });
  return Astro.redirect('/404');
}
---
<h1>{post.title}</h1>
```

## Core Web Vitals

The `@logtide/browser` package automatically tracks performance metrics on the client. View these in your LogTide Dashboard under the **Metrics** tab.

## Source Map Uploads

To get readable stack traces in production, upload your source maps during your build:

```bash
# Using @logtide/cli
logtide sourcemaps upload --path ./dist --release 1.0.0 --apiKey YOUR_API_KEY
```

## Next Steps

- [Browser SDK Docs](/docs/sdks/browser/) - Full browser SDK reference
- [Node.js Integration](/integrations/nodejs/) - For SSR and API routes
- [Vite Setup](/integrations/vite/) - Best practices for Vite builds


---

# AWS Lambda Java Logging with Log4j2

> Set up Log4j2 logging in AWS Lambda Java functions and ship structured logs to LogTide. Covers the Lambda appender, JSON layout, and CloudWatch forwarding.

*Source: https://logtide.dev/integrations/aws-lambda-java/*

---

Java is one of the most popular Lambda runtimes for enterprise workloads, and Log4j2 is the de-facto logging framework for it. But getting Log4j2 to behave inside Lambda — and getting those logs somewhere you can actually search them — trips up a lot of teams: plugins disappear during JAR shading, multiline stack traces get split into separate CloudWatch events, and CloudWatch Logs Insights bills you for every query.

This guide covers the correct Log4j2 setup for AWS Lambda Java functions, structured JSON logging, and two ways to ship those logs to LogTide for real search, alerting, and retention without per-query costs.

> **Security note:** use Log4j **2.17.1 or later**. Versions before 2.15.0 are vulnerable to [CVE-2021-44228 (Log4Shell)](https://nvd.nist.gov/vuln/detail/CVE-2021-44228). Pin the version explicitly — don't rely on transitive resolution.

## Why Java Lambda logging is harder than it should be

- **Shading breaks Log4j2 plugins.** Log4j2 discovers appenders via a binary plugin cache (`Log4j2Plugins.dat`). Maven Shade and Gradle Shadow merge JARs without merging this cache, so the Lambda appender silently vanishes.
- **Multiline stack traces fragment.** CloudWatch treats each line as a separate event unless you log JSON, making Java exceptions nearly unreadable.
- **CloudWatch costs scale with usage.** $0.50/GB ingestion plus $0.005/GB scanned per Logs Insights query adds up fast for chatty Java services.
- **No structured querying.** Filtering on MDC values or custom fields in CloudWatch requires fragile filter patterns instead of field-based search.

## Step 1: Correct Log4j2 setup for Lambda

### Dependencies

**Maven:**

```xml
<dependencies>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.23.1</version>
  </dependency>
  <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.23.1</version>
  </dependency>
  <dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-lambda-java-log4j2</artifactId>
    <version>1.6.0</version>
  </dependency>
</dependencies>
```

**Gradle (Kotlin DSL):**

```kotlin
dependencies {
    implementation("org.apache.logging.log4j:log4j-api:2.23.1")
    implementation("org.apache.logging.log4j:log4j-core:2.23.1")
    implementation("com.amazonaws:aws-lambda-java-log4j2:1.6.0")
}
```

### log4j2.xml with the Lambda appender

Place this in `src/main/resources/log4j2.xml`. The `Lambda` appender writes to the Lambda runtime's stdout pipe so each log entry — including multiline stack traces — arrives in CloudWatch as a single event:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.amazonaws.services.lambda.runtime.log4j2">
  <Appenders>
    <Lambda name="Lambda" format="${env:AWS_LAMBDA_LOG_FORMAT:-TEXT}">
      <JsonTemplateLayout eventTemplateUri="classpath:LambdaLayout.json" />
    </Lambda>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="Lambda" />
    </Root>
    <Logger name="software.amazon.awssdk" level="WARN" />
  </Loggers>
</Configuration>
```

### The shading transformer (the step everyone misses)

Without this, your fat JAR loses the Log4j2 plugin cache and the Lambda appender stops working.

**Maven Shade:**

```xml
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>3.5.2</version>
  <dependencies>
    <dependency>
      <groupId>io.github.edwgiz</groupId>
      <artifactId>log4j-maven-shade-plugin-extensions</artifactId>
      <version>2.20.0</version>
    </dependency>
  </dependencies>
  <executions>
    <execution>
      <phase>package</phase>
      <goals><goal>shade</goal></goals>
      <configuration>
        <transformers>
          <transformer implementation="io.github.edwgiz.log4j.maven.plugins.shade.transformer.Log4j2PluginCacheFileTransformer" />
        </transformers>
      </configuration>
    </execution>
  </executions>
</plugin>
```

**Gradle Shadow** users can apply the [`de.sebastianboegl.shadow.transform-log4j2`](https://plugins.gradle.org/plugin/de.sebastianboegl.shadow.transform-log4j2) plugin, which does the same merge.

### Logging from your handler

```java
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.ThreadContext;

public class OrderHandler implements RequestHandler<OrderEvent, String> {
    private static final Logger logger = LogManager.getLogger(OrderHandler.class);

    @Override
    public String handleRequest(OrderEvent event, Context context) {
        ThreadContext.put("requestId", context.getAwsRequestId());
        ThreadContext.put("orderId", event.getOrderId());

        logger.info("Processing order");
        try {
            // business logic
            logger.info("Order processed successfully");
            return "OK";
        } catch (Exception e) {
            logger.error("Order processing failed", e);
            throw e;
        } finally {
            ThreadContext.clearAll();
        }
    }
}
```

At this point you have clean, structured Log4j2 logs in CloudWatch. Now let's make them searchable.

## Step 2, Option A: Forward CloudWatch logs to LogTide

Zero code changes — keep Log4j2 exactly as configured above and forward the function's log group to LogTide with a CloudWatch subscription filter and a small forwarder function.

This is the same pattern used for Node.js functions, so follow the forwarder setup in the [AWS Lambda integration guide](/integrations/aws-lambda/#approach-2-cloudwatch-to-logtide-forwarding). The JSON layout from Step 1 pays off here: the forwarder can parse each event and map `level`, `message`, and your `ThreadContext` fields directly into LogTide's structured metadata instead of shipping opaque text lines.

**Choose this approach when** you have many existing Java functions, you can't modify their code, or you want CloudWatch to remain the primary sink with LogTide as the search and alerting layer.

## Step 2, Option B: LogTide JVM SDK from the handler

For the richest structured data, log directly to LogTide using the [Kotlin/Java SDK](/docs/sdks/kotlin/) (JVM 17+, works from plain Java via interop).

```kotlin
dependencies {
    implementation("io.github.logtide-dev:logtide-core:0.9.8")
}
```

Initialize the client **outside** the handler so it's reused across warm invocations, and flush before returning so no logs are lost when the execution environment freezes:

```kotlin
import dev.logtide.sdk.LogTideClient
import dev.logtide.sdk.models.LogTideClientOptions
import com.amazonaws.services.lambda.runtime.Context
import com.amazonaws.services.lambda.runtime.RequestHandler
import kotlinx.coroutines.runBlocking

// Created once per execution environment (survives warm starts)
private val logtide = LogTideClient(
    LogTideClientOptions(
        apiUrl = System.getenv("LOGTIDE_API_URL"),
        apiKey = System.getenv("LOGTIDE_API_KEY"),
        globalMetadata = mapOf(
            "function" to System.getenv("AWS_LAMBDA_FUNCTION_NAME"),
            "region" to System.getenv("AWS_REGION"),
        )
    )
)

class OrderHandler : RequestHandler<OrderEvent, String> {
    override fun handleRequest(event: OrderEvent, context: Context): String {
        logtide.info("orders", "Processing order", mapOf(
            "requestId" to context.awsRequestId,
            "orderId" to event.orderId,
            "coldStart" to isColdStart(),
        ))

        return try {
            // business logic
            logtide.info("orders", "Order processed", mapOf("orderId" to event.orderId))
            "OK"
        } catch (e: Exception) {
            logtide.error("orders", "Order processing failed", e)
            throw e
        } finally {
            // Flush before the environment freezes — one HTTP round-trip
            runBlocking { logtide.flush() }
        }
    }
}

private var cold = true
private fun isColdStart(): Boolean = cold.also { cold = false }
```

The SDK batches in memory with retry, exponential backoff, and a circuit breaker, so the only synchronous cost is the final `flush()` — typically 10-50 ms in-region.

**Choose this approach when** you want field-based queries on your own metadata, real-time alerting on errors, and cold start tracking — without paying CloudWatch per query.

## Which approach should you pick?

| | Option A: CloudWatch forwarding | Option B: LogTide SDK |
|---|---|---|
| Code changes | None | Handler + dependency |
| Structured metadata | Parsed from JSON layout | Native, arbitrary fields |
| Latency added | None (async forwarding) | One flush per invocation |
| CloudWatch costs | Still paying ingestion | Can reduce retention to minimum |
| Best for | Existing fleets, no-touch migration | New functions, rich telemetry |

Many teams run both: SDK for new services, forwarding for the legacy fleet.

## Troubleshooting

- **`ERROR StatusLogger Unrecognized format specifier`** — your shaded JAR lost the plugin cache. Add the transformer from Step 1.
- **Logs appear in stderr without formatting** — `log4j2.xml` isn't on the classpath; confirm it's in `src/main/resources`.
- **Stack traces split across CloudWatch events** — you're using a pattern layout over stdout instead of the Lambda appender; switch to the `<Lambda>` appender or JSON layout.
- **Nothing in LogTide after `flush()`** — check the function's VPC/egress configuration can reach your LogTide instance, and that the API key has ingest permission for the project.

---

**Next steps:**

- [AWS Lambda integration overview](/integrations/aws-lambda/) — Node.js SDK, extensions, and forwarder setup
- [Kotlin/Java SDK reference](/docs/sdks/kotlin/) — full configuration options
- [Spring Boot integration](/integrations/spring-boot/) — for containerized Java services
- [Cost optimization guide](/use-cases/cost-optimization/) — what you save by moving search off CloudWatch


---

# AWS Lambda Logging Integration

> Send AWS Lambda function logs to LogTide using the JavaScript SDK, CloudWatch forwarding, or Lambda extensions for cost-effective serverless observability.

*Source: https://logtide.dev/integrations/aws-lambda/*

---

AWS Lambda functions generate logs that end up in CloudWatch by default, but searching, alerting, and retaining those logs can be expensive and cumbersome. This guide shows you how to send Lambda logs directly to LogTide using the JavaScript SDK, forward them from CloudWatch, or use Lambda extensions -- giving you structured queries, real-time alerting, and significant cost savings.

> **Running Java on Lambda?** This guide covers the Node.js/TypeScript SDK. For `log4j` on AWS Lambda — plugin cache shading, the Log4j2 Lambda appender, and flush-before-freeze — follow the dedicated [AWS Lambda Log4j2 logging guide](/integrations/aws-lambda-java/).

## Why use LogTide with AWS Lambda?

- **Cost reduction**: CloudWatch Logs charges $0.50/GB ingested and $0.005/GB scanned. LogTide is self-hosted with predictable storage costs
- **Structured queries**: Search Lambda logs with field-based queries instead of CloudWatch's limited filter syntax
- **Cross-service correlation**: Link Lambda logs with API Gateway, ECS, and other service logs in one place
- **Real-time alerting**: Get notified on errors, cold starts, and timeouts without CloudWatch Alarms
- **Cold start tracking**: Automatically detect and measure cold start duration
- **Retention control**: Keep logs as long as you need without escalating CloudWatch storage costs
- **SIEM integration**: LogTide's built-in SIEM capabilities detect security anomalies in Lambda invocations

## Prerequisites

- AWS account with Lambda functions
- Node.js 18+ runtime (for SDK approach)
- LogTide instance with DSN or API key
- AWS SAM CLI or CDK (optional, for deployment)

## Approach 1: LogTide SDK (Recommended)

The LogTide JavaScript SDK provides the most structured approach with automatic batching, cold start detection, and context propagation.

### Installation

Add the SDK to your Lambda function:

```bash
npm install @logtide/core
```

### Basic Lambda Handler

```typescript
// handler.ts
import { hub } from '@logtide/core';

// Initialize outside the handler (persists across warm invocations)
hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: 'order-processor',
  environment: process.env.STAGE || 'production',
  release: process.env.APP_VERSION,
  batchSize: 50,
  flushInterval: 1000,  // Flush quickly for Lambda
});

let isColdStart = true;

export const handler = async (event: any, context: any) => {
  const startTime = Date.now();

  // Track cold starts
  if (isColdStart) {
    hub.captureLog('info', 'Cold start detected', {
      functionName: context.functionName,
      memorySize: context.memoryLimitInMB,
      runtime: process.version,
      coldStart: true,
    });
    isColdStart = false;
  }

  hub.captureLog('info', 'Lambda invocation started', {
    requestId: context.awsRequestId,
    functionName: context.functionName,
    remainingTime: context.getRemainingTimeInMillis(),
  });

  try {
    // Your business logic
    const result = await processOrder(event);

    hub.captureLog('info', 'Lambda invocation completed', {
      requestId: context.awsRequestId,
      duration: Date.now() - startTime,
      result: 'success',
    });

    // Flush before Lambda freezes the execution environment
    await hub.flush();

    return {
      statusCode: 200,
      body: JSON.stringify(result),
    };
  } catch (error) {
    hub.captureError(error as Error, {
      extra: {
        requestId: context.awsRequestId,
        event: JSON.stringify(event),
        remainingTime: context.getRemainingTimeInMillis(),
      },
      tags: {
        functionName: context.functionName,
      },
    });

    // Always flush on error
    await hub.flush();

    return {
      statusCode: 500,
      body: JSON.stringify({ error: 'Internal server error' }),
    };
  }
};

async function processOrder(event: any) {
  // Business logic here
  return { orderId: event.orderId, status: 'processed' };
}
```

### API Gateway Integration

For Lambda functions behind API Gateway:

```typescript
// api-handler.ts
import { hub } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-gateway-handler',
  environment: process.env.STAGE,
});

let isColdStart = true;

export const handler = async (event: any, context: any) => {
  const startTime = Date.now();

  // Parse API Gateway event
  const method = event.httpMethod || event.requestContext?.http?.method;
  const path = event.path || event.requestContext?.http?.path;
  const sourceIp = event.requestContext?.identity?.sourceIp
    || event.requestContext?.http?.sourceIp;
  const userAgent = event.headers?.['User-Agent']
    || event.headers?.['user-agent'];

  hub.captureLog('info', `${method} ${path}`, {
    requestId: context.awsRequestId,
    httpMethod: method,
    path: path,
    sourceIp: sourceIp,
    userAgent: userAgent,
    coldStart: isColdStart,
    queryString: event.queryStringParameters,
  });

  isColdStart = false;

  try {
    const result = await routeRequest(method, path, event);

    hub.captureLog('info', `${method} ${path} ${result.statusCode}`, {
      requestId: context.awsRequestId,
      statusCode: result.statusCode,
      duration: Date.now() - startTime,
    });

    await hub.flush();
    return result;
  } catch (error) {
    hub.captureError(error as Error, {
      extra: { requestId: context.awsRequestId, method, path },
    });
    await hub.flush();

    return {
      statusCode: 500,
      body: JSON.stringify({ error: 'Internal server error' }),
    };
  }
};

async function routeRequest(method: string, path: string, event: any) {
  // Your routing logic
  return { statusCode: 200, body: JSON.stringify({ ok: true }) };
}
```

### Lambda Layer for Shared Configuration

Package the LogTide SDK as a Lambda Layer for reuse across functions:

```bash
# Create layer directory
mkdir -p logtide-layer/nodejs
cd logtide-layer/nodejs

# Install SDK
npm init -y
npm install @logtide/core

# Package the layer
cd ..
zip -r logtide-layer.zip nodejs/
```

Deploy the layer:

```bash
aws lambda publish-layer-version \
  --layer-name logtide-sdk \
  --zip-file fileb://logtide-layer.zip \
  --compatible-runtimes nodejs18.x nodejs20.x \
  --description "LogTide JavaScript SDK for structured logging"
```

Attach to your function:

```bash
aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:123456789:layer:logtide-sdk:1
```

## Approach 2: CloudWatch to LogTide Forwarding

For existing Lambda functions where you cannot modify the code, forward logs from CloudWatch to LogTide using a forwarder Lambda.

### Forwarder Lambda Function

```typescript
// cloudwatch-forwarder.ts
import { gunzipSync } from 'zlib';

const LOGTIDE_API_URL = process.env.LOGTIDE_API_URL!;
const LOGTIDE_API_KEY = process.env.LOGTIDE_API_KEY!;

interface CloudWatchLogsEvent {
  awslogs: {
    data: string;
  };
}

interface LogEvent {
  id: string;
  timestamp: number;
  message: string;
}

interface DecodedPayload {
  logGroup: string;
  logStream: string;
  logEvents: LogEvent[];
}

export const handler = async (event: CloudWatchLogsEvent) => {
  // Decode and decompress CloudWatch log data
  const payload = Buffer.from(event.awslogs.data, 'base64');
  const decompressed = gunzipSync(payload);
  const data: DecodedPayload = JSON.parse(decompressed.toString());

  // Extract function name from log group
  // Format: /aws/lambda/function-name
  const functionName = data.logGroup.replace('/aws/lambda/', '');

  // Transform log events
  const logs = data.logEvents.map((logEvent) => {
    const parsed = parseLogMessage(logEvent.message);
    return {
      timestamp: new Date(logEvent.timestamp).toISOString(),
      service: functionName,
      level: parsed.level,
      message: parsed.message,
      requestId: parsed.requestId,
      source: 'cloudwatch',
      logGroup: data.logGroup,
      logStream: data.logStream,
      ...parsed.metadata,
    };
  });

  // Ship to LogTide
  const response = await fetch(`${LOGTIDE_API_URL}/api/v1/ingest`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Key': LOGTIDE_API_KEY,
    },
    body: JSON.stringify({ logs }),
  });

  if (!response.ok) {
    console.error(`LogTide ingest failed: ${response.status} ${await response.text()}`);
    throw new Error(`LogTide ingest failed: ${response.status}`);
  }

  return { processed: logs.length };
};

function parseLogMessage(message: string) {
  // Parse Lambda platform messages
  if (message.startsWith('START RequestId:')) {
    const requestId = message.match(/RequestId: ([\w-]+)/)?.[1];
    return { level: 'info', message: 'Lambda invocation started', requestId, metadata: {} };
  }

  if (message.startsWith('END RequestId:')) {
    const requestId = message.match(/RequestId: ([\w-]+)/)?.[1];
    return { level: 'info', message: 'Lambda invocation ended', requestId, metadata: {} };
  }

  if (message.startsWith('REPORT RequestId:')) {
    const requestId = message.match(/RequestId: ([\w-]+)/)?.[1];
    const duration = message.match(/Duration: ([\d.]+) ms/)?.[1];
    const billedDuration = message.match(/Billed Duration: (\d+) ms/)?.[1];
    const memoryUsed = message.match(/Max Memory Used: (\d+) MB/)?.[1];
    const memorySize = message.match(/Memory Size: (\d+) MB/)?.[1];
    const initDuration = message.match(/Init Duration: ([\d.]+) ms/)?.[1];

    return {
      level: 'info',
      message: `Lambda report: ${duration}ms, ${memoryUsed}MB used`,
      requestId,
      metadata: {
        duration: parseFloat(duration || '0'),
        billedDuration: parseInt(billedDuration || '0'),
        memoryUsed: parseInt(memoryUsed || '0'),
        memorySize: parseInt(memorySize || '0'),
        coldStart: !!initDuration,
        initDuration: initDuration ? parseFloat(initDuration) : undefined,
      },
    };
  }

  // Try to parse as JSON (structured application logs)
  try {
    const parsed = JSON.parse(message.replace(/^\d{4}-\d{2}-\d{2}T[\d:.Z]+\t[\w-]+\t(INFO|WARN|ERROR|DEBUG)\t/, ''));
    return {
      level: parsed.level || 'info',
      message: parsed.message || message,
      requestId: parsed.requestId,
      metadata: parsed,
    };
  } catch {
    // Plain text log
    const level = message.includes('ERROR') ? 'error'
      : message.includes('WARN') ? 'warn'
      : 'info';
    return { level, message: message.trim(), requestId: undefined, metadata: {} };
  }
}
```

### Deploy the Forwarder

Using AWS SAM:

```yaml
# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  LogTideApiUrl:
    Type: String
  LogTideApiKey:
    Type: String
    NoEcho: true

Resources:
  LogTideForwarder:
    Type: AWS::Serverless::Function
    Properties:
      Handler: cloudwatch-forwarder.handler
      Runtime: nodejs20.x
      Timeout: 60
      MemorySize: 256
      Environment:
        Variables:
          LOGTIDE_API_URL: !Ref LogTideApiUrl
          LOGTIDE_API_KEY: !Ref LogTideApiKey

  # Subscribe to a Lambda function's log group
  LogSubscription:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      LogGroupName: /aws/lambda/my-function
      FilterPattern: ""
      DestinationArn: !GetAtt LogTideForwarder.Arn

  # Permission for CloudWatch to invoke the forwarder
  ForwarderPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref LogTideForwarder
      Action: lambda:InvokeFunction
      Principal: logs.amazonaws.com
      SourceArn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/my-function:*"
```

Deploy:

```bash
sam build && sam deploy \
  --parameter-overrides \
    LogTideApiUrl=https://your-logtide-host \
    LogTideApiKey=your-api-key
```

### Subscribe Multiple Functions

Use a script to subscribe all Lambda log groups:

```bash
#!/bin/bash
FORWARDER_ARN="arn:aws:lambda:us-east-1:123456789:function:LogTideForwarder"

# List all Lambda log groups
aws logs describe-log-groups \
  --log-group-name-prefix "/aws/lambda/" \
  --query 'logGroups[].logGroupName' \
  --output text | tr '\t' '\n' | while read log_group; do

  echo "Subscribing: $log_group"

  aws logs put-subscription-filter \
    --log-group-name "$log_group" \
    --filter-name "logtide-forwarder" \
    --filter-pattern "" \
    --destination-arn "$FORWARDER_ARN"
done
```

## Approach 3: HTTP API Direct (Lightweight)

For minimal overhead without the full SDK:

```typescript
// lightweight-logger.ts
const LOGTIDE_API_URL = process.env.LOGTIDE_API_URL!;
const LOGTIDE_API_KEY = process.env.LOGTIDE_API_KEY!;

const logBuffer: any[] = [];

export function log(level: string, message: string, metadata: Record<string, any> = {}) {
  logBuffer.push({
    timestamp: new Date().toISOString(),
    level,
    message,
    service: process.env.AWS_LAMBDA_FUNCTION_NAME,
    ...metadata,
  });
}

export async function flushLogs() {
  if (logBuffer.length === 0) return;

  const logs = logBuffer.splice(0);

  await fetch(`${LOGTIDE_API_URL}/api/v1/ingest`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Key': LOGTIDE_API_KEY,
    },
    body: JSON.stringify({ logs }),
  });
}
```

Usage:

```typescript
import { log, flushLogs } from './lightweight-logger';

export const handler = async (event: any, context: any) => {
  log('info', 'Processing request', { requestId: context.awsRequestId });

  try {
    const result = await doWork(event);
    log('info', 'Request processed', { requestId: context.awsRequestId });
    await flushLogs();
    return result;
  } catch (error) {
    log('error', (error as Error).message, { requestId: context.awsRequestId });
    await flushLogs();
    throw error;
  }
};
```

## AWS CDK Deployment

### CDK Stack with LogTide Integration

```typescript
// lib/lambda-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as nodejs from 'aws-cdk-lib/aws-lambda-nodejs';
import * as logs from 'aws-cdk-lib/aws-logs';
import { Construct } from 'constructs';

export class LambdaStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // LogTide SDK Layer
    const logtideLayer = new lambda.LayerVersion(this, 'LogTideLayer', {
      code: lambda.Code.fromAsset('layers/logtide'),
      compatibleRuntimes: [lambda.Runtime.NODEJS_20_X],
      description: 'LogTide JavaScript SDK',
    });

    // Application function
    const orderProcessor = new nodejs.NodejsFunction(this, 'OrderProcessor', {
      entry: 'src/handlers/order-processor.ts',
      runtime: lambda.Runtime.NODEJS_20_X,
      memorySize: 512,
      timeout: cdk.Duration.seconds(30),
      layers: [logtideLayer],
      environment: {
        LOGTIDE_DSN: process.env.LOGTIDE_DSN || '',
        STAGE: 'production',
      },
      logRetention: logs.RetentionDays.THREE_DAYS,  // Reduce CloudWatch costs
    });

    // Forwarder function (for functions you can't modify)
    const forwarder = new nodejs.NodejsFunction(this, 'LogTideForwarder', {
      entry: 'src/handlers/cloudwatch-forwarder.ts',
      runtime: lambda.Runtime.NODEJS_20_X,
      memorySize: 256,
      timeout: cdk.Duration.seconds(60),
      environment: {
        LOGTIDE_API_URL: process.env.LOGTIDE_API_URL || '',
        LOGTIDE_API_KEY: process.env.LOGTIDE_API_KEY || '',
      },
    });

    // Subscribe forwarder to a log group
    new logs.SubscriptionFilter(this, 'LogSubscription', {
      logGroup: orderProcessor.logGroup,
      destination: new cdk.aws_logs_destinations.LambdaDestination(forwarder),
      filterPattern: logs.FilterPattern.allEvents(),
    });
  }
}
```

## Cold Start Monitoring

Track cold starts to understand Lambda performance:

### Detection Rules in LogTide

```
service:order-processor AND coldStart:true
```

### Cold Start Dashboard Queries

```
# Cold start count by function
service:* AND coldStart:true | count by service

# Average init duration
service:* AND initDuration:>0 | avg(initDuration) by service

# Cold start rate
service:* AND coldStart:true | count / total_count * 100
```

### Reducing Cold Starts

Log provisioned concurrency usage:

```typescript
hub.captureLog('info', 'Function initialized', {
  coldStart: isColdStart,
  provisionedConcurrency: !!process.env.AWS_LAMBDA_INITIALIZATION_TYPE,
  initType: process.env.AWS_LAMBDA_INITIALIZATION_TYPE,
});
```

## Cost Comparison

### CloudWatch vs LogTide

For a workload generating 100GB of logs per month:

| Cost Component | CloudWatch | LogTide (self-hosted) |
|----------------|------------|----------------------|
| Log ingestion | $50.00/mo ($0.50/GB) | $0 |
| Log storage (3 months) | $9.00/mo ($0.03/GB) | ~$3/mo (EBS) |
| Log queries | $2.50/mo ($0.005/GB scanned) | $0 |
| Alarms (10 alarms) | $1.00/mo | $0 (built-in) |
| Insights queries | $0.005/GB scanned | $0 |
| **Total** | **~$62.50/mo** | **~$3/mo + compute** |

Compute costs for a LogTide instance vary, but a single `t3.medium` ($30/mo) can handle well over 100GB of logs per month.

### Reducing CloudWatch Costs

When using LogTide, reduce CloudWatch retention to minimize storage costs:

```bash
# Set retention to 1 day (minimum for subscription filters)
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 1
```

## Performance Metrics

| Metric | Value | Notes |
|--------|-------|-------|
| SDK overhead | ~2ms per invocation | Batched writes, async flush |
| Flush time | 10-50ms | Depends on network to LogTide |
| Memory overhead | ~3MB | SDK in Lambda memory |
| Cold start impact | ~50ms | One-time SDK initialization |
| Forwarder latency | 1-5 seconds | CloudWatch subscription delay |

## Troubleshooting

### Logs not appearing in LogTide (SDK approach)

1. Verify the DSN is set correctly:
   ```bash
   aws lambda get-function-configuration \
     --function-name my-function \
     --query 'Environment.Variables.LOGTIDE_DSN'
   ```

2. Ensure `hub.flush()` is called before the handler returns:
   ```typescript
   // Always flush before returning
   await hub.flush();
   return response;
   ```

3. Check Lambda timeout -- if the function times out, the flush may not complete. Increase timeout or reduce `flushInterval`.

### Forwarder Lambda errors

1. Check forwarder CloudWatch logs:
   ```bash
   aws logs tail /aws/lambda/LogTideForwarder --follow
   ```

2. Verify the forwarder has network access to your LogTide instance (VPC configuration, security groups).

3. Check subscription filter status:
   ```bash
   aws logs describe-subscription-filters \
     --log-group-name /aws/lambda/my-function
   ```

### High forwarder invocation costs

Batch logs by increasing the subscription filter buffer:

```bash
# The forwarder processes logs in batches automatically.
# If invocation frequency is too high, consider adding a
# Kinesis Data Stream as a buffer between CloudWatch and the forwarder.
```

### Missing cold start data

Cold start detection relies on the `isColdStart` flag. Ensure it is declared outside the handler:

```typescript
// This MUST be outside the handler function
let isColdStart = true;

export const handler = async (event: any, context: any) => {
  // Use isColdStart here, then set to false
  isColdStart = false;
};
```

## Next Steps

- [AWS Lambda Java logging with Log4j2](/integrations/aws-lambda-java/) - `log4j` on AWS Lambda: shading, the Lambda appender, and flush patterns
- [Node.js SDK Reference](/integrations/nodejs/) - Full SDK documentation
- [Docker Integration](/integrations/docker/) - Containerized Lambda alternatives
- [Cost Optimization](/use-cases/cost-optimization/) - Reduce your logging costs
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Alert on Lambda errors and timeouts


---

# Azure Functions Logging Integration

> Send Azure Functions logs to LogTide with the JavaScript SDK: structured logging, cold start tracking, and an alternative to Application Insights per-GB costs.

*Source: https://logtide.dev/integrations/azure-functions/*

---

Azure Functions ship their logs to Application Insights by default — which works, until the Log Analytics bill arrives. At ~$2.76/GB ingested, a chatty function app becomes one of the most expensive log sources in your subscription, and querying it means KQL plus per-GB retention fees.

This guide wires Azure Functions (Node.js v4 programming model) to LogTide for structured, searchable logs with real-time alerting — either replacing verbose App Insights logging or running alongside it.

## Why use LogTide with Azure Functions?

- **Cost control**: Log Analytics charges ~$2.76/GB; LogTide is self-hosted with flat infrastructure costs ([full breakdown](/vs/azure-monitor/))
- **Structured queries**: filter by your own metadata fields instead of writing KQL against `traces`
- **Real-time alerting**: error spikes, timeout patterns, cold start bursts — without Azure Monitor alert rule pricing
- **Cross-cloud correlation**: Functions, on-prem services, and [AWS Lambda](/integrations/aws-lambda/) logs in one place
- **Retention without fees**: keep logs searchable for months instead of paying $0.10/GB/month past the included window

## Prerequisites

- Azure Functions app on the **Node.js v4** programming model (v3 works with minor changes)
- LogTide instance reachable from Azure (public endpoint or VNet)
- A LogTide DSN or API key

## Approach 1: LogTide SDK (recommended)

### Installation

```bash
npm install @logtide/core
```

### HTTP-triggered function

Initialize the hub **outside** the handler — Azure reuses the worker process across invocations, so the client and its batch buffer survive warm starts:

```typescript
// src/functions/processOrder.ts
import { app, HttpRequest, HttpResponseInit, InvocationContext } from '@azure/functions';
import { hub } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: 'order-processor',
  environment: process.env.AZURE_FUNCTIONS_ENVIRONMENT || 'production',
  batchSize: 50,
  flushInterval: 1000, // flush quickly — instances can be recycled anytime
});

let isColdStart = true;

app.http('processOrder', {
  methods: ['POST'],
  authLevel: 'function',
  handler: async (request: HttpRequest, context: InvocationContext): Promise<HttpResponseInit> => {
    const startTime = Date.now();

    if (isColdStart) {
      hub.captureLog('info', 'Cold start detected', {
        functionName: context.functionName,
        coldStart: true,
      });
      isColdStart = false;
    }

    hub.captureLog('info', 'Invocation started', {
      invocationId: context.invocationId,
      functionName: context.functionName,
    });

    try {
      const order = await request.json();
      // business logic

      hub.captureLog('info', 'Order processed', {
        invocationId: context.invocationId,
        duration: Date.now() - startTime,
      });

      await hub.flush();
      return { status: 200, jsonBody: { ok: true } };
    } catch (error) {
      hub.captureError(error as Error, {
        extra: { invocationId: context.invocationId },
        tags: { functionName: context.functionName },
      });

      await hub.flush();
      return { status: 500, jsonBody: { error: 'Internal server error' } };
    }
  },
});
```

### Timer and queue triggers

The same pattern applies — initialize once, log with metadata, flush at the end:

```typescript
import { app, InvocationContext, Timer } from '@azure/functions';
import { hub } from '@logtide/core';

app.timer('nightlyCleanup', {
  schedule: '0 0 3 * * *',
  handler: async (timer: Timer, context: InvocationContext) => {
    hub.captureLog('info', 'Cleanup started', { invocationId: context.invocationId });

    const removed = await runCleanup();

    hub.captureLog('info', 'Cleanup finished', {
      invocationId: context.invocationId,
      removedItems: removed,
    });
    await hub.flush();
  },
});
```

### Configuration

Set the connection in your Function App settings (portal → Environment variables, or `local.settings.json` for local dev):

```json
{
  "Values": {
    "LOGTIDE_DSN": "https://lp_your_key@logtide.example.com/your-project"
  }
}
```

If your LogTide instance is private, use VNet integration on the Function App (Premium and Dedicated plans) so outbound traffic reaches it.

## Approach 2: Keep Application Insights, route the volume to LogTide

You don't have to choose. A pragmatic split that most teams land on:

- **App Insights keeps**: platform metrics, distributed traces, dependency maps, live metrics — the APM features Azure does natively well, with **sampling enabled** to cap cost
- **LogTide gets**: all verbose application logging — request details, business events, debug context — the volume that makes Log Analytics expensive

In `host.json`, keep App Insights log capture at `Warning` and above so duplicated volume stays minimal:

```json
{
  "logging": {
    "logLevel": {
      "default": "Warning"
    },
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "maxTelemetryItemsPerSecond": 5
      }
    }
  }
}
```

Your application code logs everything to LogTide via the SDK; App Insights only records warnings, errors and sampled traces. Typical outcome: 80-95% less Log Analytics ingestion with no loss of searchable detail — it's all in LogTide instead.

## Which approach should you pick?

| | SDK only | SDK + App Insights (split) |
|---|---|---|
| Log Analytics costs | Near zero | Heavily reduced (sampled) |
| Distributed tracing | No (logs only) | Yes (App Insights) |
| Azure-native dashboards | No | Yes |
| Best for | Cost-first, log-centric teams | Teams using APM features |

## Troubleshooting

- **Logs missing from the end of invocations** — the instance was recycled before an interval flush; call `await hub.flush()` before returning.
- **Nothing arrives at LogTide** — check outbound networking: Consumption plan functions need a public LogTide endpoint; VNet-only instances require Premium/Dedicated plans with VNet integration.
- **Duplicate logs in both platforms** — lower the `host.json` default log level so App Insights stops capturing what the SDK already ships.

---

**Next steps:**

- [AWS Lambda integration](/integrations/aws-lambda/) — same pattern on AWS
- [LogTide vs Azure Monitor](/vs/azure-monitor/) — full cost and feature comparison
- [Cloud logging pricing breakdown](/cloud-logging-pricing/) — CloudWatch, GCP and Azure compared
- [Real-time alerting use case](/use-cases/real-time-alerting/) — alert rules on function errors


---

# Bun Runtime Logging Integration

> Run the LogTide JavaScript SDK on Bun for fast structured logging with native TypeScript support and zero-config setup.

*Source: https://logtide.dev/integrations/bun/*

---

Bun is a fast JavaScript runtime that is Node.js API compatible. The LogTide JavaScript SDK works with Bun out of the box — no extra configuration needed. This guide covers setup, framework integration (Hono, Elysia), and Bun-specific patterns.

## Why use LogTide with Bun?

- **Zero config**: The `@logtide/sdk-node` SDK works directly on Bun
- **Native TypeScript**: No build step needed, Bun runs `.ts` files directly
- **Built-in .env**: Bun loads `.env` files automatically
- **Fast startup**: Bun's speed means your logging starts immediately
- **Same API**: If you know the Node.js SDK, you know the Bun SDK

## Prerequisites

- Bun 1.0+ (1.1+ recommended)
- LogTide instance with a DSN or API key

## Installation

```bash
bun add @logtide/sdk-node
```

## Quick Start

```typescript
// index.ts
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  globalMetadata: { environment: process.env.NODE_ENV ?? 'development' },
});

client.info('bun-app', 'Application started', { runtime: 'bun', version: Bun.version });

// Your application logic
const server = Bun.serve({
  port: 3000,
  fetch(req) {
    const url = new URL(req.url);
    client.info('bun-app', `${req.method} ${url.pathname}`);
    return new Response('OK');
  },
});

client.info('bun-app', 'Server listening', { port: server.port });
```

Run directly — no build step:

```bash
bun run index.ts
```

### Environment Variables

Bun loads `.env` files automatically:

```bash
# .env
LOGTIDE_API_URL=http://logtide.internal:8080
LOGTIDE_API_KEY=lp_your_key
NODE_ENV=development
```

No `dotenv` package needed.

## Bun.serve with Logging

### Structured Request Logging

```typescript
// server.ts
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

const server = Bun.serve({
  port: process.env.PORT ?? 3000,

  async fetch(req) {
    const url = new URL(req.url);
    const start = performance.now();
    const traceId = req.headers.get('x-trace-id') ?? crypto.randomUUID();

    try {
      const response = await handleRequest(req, url);
      const durationMs = Math.round(performance.now() - start);

      client.info('api', `${req.method} ${url.pathname} ${response.status}`, {
        method: req.method,
        path: url.pathname,
        statusCode: response.status,
        durationMs,
        traceId,
      });

      response.headers.set('X-Trace-Id', traceId);
      return response;
    } catch (error) {
      const durationMs = Math.round(performance.now() - start);

      client.error('api', `${req.method} ${url.pathname} 500`, {
        method: req.method,
        path: url.pathname,
        statusCode: 500,
        durationMs,
        traceId,
        error: error instanceof Error ? error.message : String(error),
      });

      return new Response('Internal Server Error', { status: 500 });
    }
  },

  error(error) {
    client.error('api', 'Server error', { error: error.message });
    return new Response('Internal Server Error', { status: 500 });
  },
});

async function handleRequest(req: Request, url: URL): Promise<Response> {
  if (url.pathname === '/health') return new Response('OK');
  if (url.pathname === '/api/users') return new Response(JSON.stringify([]), {
    headers: { 'Content-Type': 'application/json' },
  });
  return new Response('Not Found', { status: 404 });
}
```

## Hono Framework Integration

Hono is a lightweight framework that runs on Bun. Use the LogTide middleware:

```typescript
// index.ts
import { Hono } from 'hono';
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

const app = new Hono();

// Logging middleware
app.use('*', async (c, next) => {
  const start = performance.now();
  const traceId = c.req.header('x-trace-id') ?? crypto.randomUUID();
  c.set('traceId', traceId);

  await next();

  const durationMs = Math.round(performance.now() - start);
  const path = new URL(c.req.url).pathname;

  if (path !== '/health') {
    client.info('hono-api', `${c.req.method} ${path} ${c.res.status}`, {
      method: c.req.method,
      path,
      statusCode: c.res.status,
      durationMs,
      traceId,
    });
  }

  c.res.headers.set('X-Trace-Id', traceId);
});

// Error handler
app.onError((err, c) => {
  client.error('hono-api', 'Unhandled error', {
    error: err.message,
    path: new URL(c.req.url).pathname,
    traceId: c.get('traceId'),
  });
  return c.json({ error: 'Internal Server Error' }, 500);
});

app.get('/users/:id', (c) => {
  const id = c.req.param('id');
  client.info('hono-api', 'Fetching user', { userId: id, traceId: c.get('traceId') });
  return c.json({ id, name: 'User' });
});

export default app;
```

## Elysia Framework Integration

Elysia is a Bun-native framework with plugin support:

```typescript
// index.ts
import { Elysia } from 'elysia';
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

const app = new Elysia()
  .onRequest(({ request, set }) => {
    const start = performance.now();
    set.headers['x-start-time'] = String(start);
  })
  .onAfterResponse(({ request, set, response }) => {
    const url = new URL(request.url);
    if (url.pathname === '/health') return;

    const start = Number(set.headers['x-start-time']);
    const durationMs = Math.round(performance.now() - start);

    client.info('elysia-api', `${request.method} ${url.pathname}`, {
      method: request.method,
      path: url.pathname,
      durationMs,
    });
  })
  .onError(({ error, request }) => {
    client.error('elysia-api', 'Request error', {
      error: error.message,
      path: new URL(request.url).pathname,
    });
  })
  .get('/users/:id', ({ params }) => {
    client.info('elysia-api', 'Fetching user', { userId: params.id });
    return { id: params.id, name: 'User' };
  })
  .listen(3000);

client.info('elysia-api', 'Server started', { port: app.server?.port });
```

## Graceful Shutdown

Bun supports `process.on('beforeExit')` and signal handlers:

```typescript
// Flush logs before exit
process.on('SIGINT', async () => {
  client.info('bun-app', 'Shutting down (SIGINT)');
  await client.close();
  process.exit(0);
});

process.on('SIGTERM', async () => {
  client.info('bun-app', 'Shutting down (SIGTERM)');
  await client.close();
  process.exit(0);
});
```

## Docker Deployment

```dockerfile
FROM oven/bun:1-alpine

WORKDIR /app

# Install dependencies
COPY package.json bun.lockb ./
RUN bun install --frozen-lockfile --production

# Copy application
COPY . .

ENV NODE_ENV=production
ENV LOGTIDE_API_URL=""
ENV LOGTIDE_API_KEY=""

EXPOSE 3000
CMD ["bun", "run", "index.ts"]
```

```yaml
# docker-compose.yml
services:
  api:
    build: .
    ports:
      - "3000:3000"
    environment:
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - NODE_ENV=production
```

## Performance

Bun's performance means logging overhead is minimal:

| Metric | Value |
|--------|-------|
| Startup time | ~50ms (vs ~300ms Node.js) |
| Log call overhead | <0.2ms (batched) |
| Memory overhead | ~6MB |
| Throughput | 15,000+ logs/sec |

## Troubleshooting

### Module resolution issues

If you see module resolution errors, ensure your `bunfig.toml` doesn't override Node.js compatibility:

```toml
# bunfig.toml
[install]
peer = false
```

### Bun.serve vs Express

If migrating from Express, note that `Bun.serve` uses Web Standard `Request`/`Response`. The LogTide SDK works with both APIs.

### Hot reload with --watch

```bash
bun --watch run index.ts
```

LogTide re-initializes cleanly on file changes.

## Next Steps

- [Node.js SDK Reference](/integrations/nodejs/) - Full SDK documentation
- [Hono Integration](/integrations/hono/) - Hono-specific guide
- [Elysia Integration](/integrations/elysia/) - Elysia-specific guide
- [Docker Integration](/integrations/docker/) - Container deployments


---

# Deno Runtime Logging Integration

> Use the LogTide JavaScript SDK with Deno via npm specifiers for structured logging with secure-by-default permissions.

*Source: https://logtide.dev/integrations/deno/*

---

Deno supports npm packages via `npm:` specifiers, making the LogTide JavaScript SDK work directly. This guide covers setup, Fresh framework integration, Deno.serve patterns, and Deno Deploy considerations.

## Why use LogTide with Deno?

- **npm: specifier**: Import `@logtide/sdk-node` directly — no import maps needed
- **Secure-by-default**: Deno's permission model means logging requires explicit `--allow-net`
- **Native TypeScript**: No build step, full type checking out of the box
- **Deno Deploy**: Ship logs from edge functions to your LogTide instance
- **Standard APIs**: Uses Web Standard `Request`/`Response` patterns

## Prerequisites

- Deno 1.38+ (2.x recommended for improved npm compatibility)
- LogTide instance with a DSN or API key

## Quick Start

```typescript
// main.ts
import { LogTideClient } from 'npm:@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: Deno.env.get('LOGTIDE_API_URL')!,
  apiKey: Deno.env.get('LOGTIDE_API_KEY')!,
  globalMetadata: { environment: Deno.env.get('DENO_ENV') ?? 'development' },
});

client.info('deno-app', 'Application started', { runtime: 'deno', version: Deno.version.deno });

Deno.serve({ port: 8000 }, async (req) => {
  const url = new URL(req.url);
  client.info('deno-app', `${req.method} ${url.pathname}`);
  return new Response('OK');
});
```

Run with permissions:

```bash
deno run --allow-net --allow-env main.ts
```

### Environment Variables

Create a `.env` file and load it:

```bash
# .env
LOGTIDE_API_URL=http://logtide.internal:8080
LOGTIDE_API_KEY=lp_your_key
DENO_ENV=development
```

```bash
deno run --allow-net --allow-env --allow-read=.env main.ts
```

Or with Deno 2.x, `.env` files load automatically with `--allow-env`.

## Deno.serve with Structured Logging

```typescript
// main.ts
import { LogTideClient } from 'npm:@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: Deno.env.get('LOGTIDE_API_URL')!,
  apiKey: Deno.env.get('LOGTIDE_API_KEY')!,
});

Deno.serve({ port: 8000 }, async (req) => {
  const url = new URL(req.url);
  const start = performance.now();
  const traceId = req.headers.get('x-trace-id') ?? crypto.randomUUID();

  // Skip health checks
  if (url.pathname === '/health') return new Response('OK');

  try {
    const response = await handleRequest(req, url);
    const durationMs = Math.round(performance.now() - start);

    client.info('api', `${req.method} ${url.pathname} ${response.status}`, {
      method: req.method,
      path: url.pathname,
      statusCode: response.status,
      durationMs,
      traceId,
    });

    response.headers.set('X-Trace-Id', traceId);
    return response;
  } catch (error) {
    client.error('api', `${req.method} ${url.pathname} 500`, {
      method: req.method,
      path: url.pathname,
      statusCode: 500,
      traceId,
      error: error instanceof Error ? error.message : String(error),
    });

    return new Response('Internal Server Error', { status: 500 });
  }
});

async function handleRequest(req: Request, url: URL): Promise<Response> {
  switch (url.pathname) {
    case '/api/users':
      return Response.json([]);
    default:
      return new Response('Not Found', { status: 404 });
  }
}
```

## Fresh Framework Integration

Fresh is Deno's full-stack framework. Add logging to middleware and routes:

### Middleware

```typescript
// routes/_middleware.ts
import { FreshContext } from '$fresh/server.ts';
import { LogTideClient } from 'npm:@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: Deno.env.get('LOGTIDE_API_URL')!,
  apiKey: Deno.env.get('LOGTIDE_API_KEY')!,
});

export async function handler(req: Request, ctx: FreshContext) {
  const url = new URL(req.url);
  const start = performance.now();
  const traceId = crypto.randomUUID();

  // Pass trace ID to downstream handlers
  ctx.state.traceId = traceId;
  ctx.state.logtide = client;

  const response = await ctx.next();
  const durationMs = Math.round(performance.now() - start);

  // Skip static assets
  if (!url.pathname.startsWith('/_frsh/')) {
    client.info('fresh-app', `${req.method} ${url.pathname} ${response.status}`, {
      method: req.method,
      path: url.pathname,
      statusCode: response.status,
      durationMs,
      traceId,
    });
  }

  response.headers.set('X-Trace-Id', traceId);
  return response;
}
```

### API Routes

```typescript
// routes/api/users/[id].ts
import { Handlers } from '$fresh/server.ts';

export const handler: Handlers = {
  async GET(_req, ctx) {
    const { logtide, traceId } = ctx.state;
    const { id } = ctx.params;

    logtide.info('fresh-app', 'Fetching user', { userId: id, traceId });

    const user = await db.getUser(id);
    if (!user) {
      logtide.warn('fresh-app', 'User not found', { userId: id, traceId });
      return new Response('Not Found', { status: 404 });
    }

    return Response.json(user);
  },
};
```

### Islands (Client Components)

Islands run in the browser — don't import server logging there. Keep logging in routes and middleware only.

## Oak Framework Integration

If using Oak (Deno's Express-like framework):

```typescript
// main.ts
import { Application, Router } from 'https://deno.land/x/oak/mod.ts';
import { LogTideClient } from 'npm:@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: Deno.env.get('LOGTIDE_API_URL')!,
  apiKey: Deno.env.get('LOGTIDE_API_KEY')!,
});

const app = new Application();

// Logging middleware
app.use(async (ctx, next) => {
  const start = performance.now();
  await next();
  const durationMs = Math.round(performance.now() - start);

  client.info('oak-api', `${ctx.request.method} ${ctx.request.url.pathname} ${ctx.response.status}`, {
    method: ctx.request.method,
    path: ctx.request.url.pathname,
    statusCode: ctx.response.status,
    durationMs,
  });
});

// Error handler
app.use(async (ctx, next) => {
  try {
    await next();
  } catch (error) {
    client.error('oak-api', 'Unhandled error', {
      error: error instanceof Error ? error.message : String(error),
      path: ctx.request.url.pathname,
    });
    ctx.response.status = 500;
    ctx.response.body = { error: 'Internal Server Error' };
  }
});

const router = new Router();
router.get('/users/:id', (ctx) => {
  client.info('oak-api', 'Fetching user', { userId: ctx.params.id });
  ctx.response.body = { id: ctx.params.id };
});

app.use(router.routes());
app.use(router.allowedMethods());

await app.listen({ port: 8000 });
```

## Deno Deploy

Deno Deploy runs your code at the edge. LogTide works there with some considerations:

```typescript
// main.ts (for Deno Deploy)
import { LogTideClient } from 'npm:@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: Deno.env.get('LOGTIDE_API_URL')!,
  apiKey: Deno.env.get('LOGTIDE_API_KEY')!,
  // Smaller batches for edge (shorter-lived processes)
  batchSize: 10,
  flushInterval: 2000,
});

Deno.serve(async (req) => {
  const url = new URL(req.url);
  const start = performance.now();

  const response = await handleRequest(req, url);
  const durationMs = Math.round(performance.now() - start);

  client.info('edge-api', `${req.method} ${url.pathname} ${response.status}`, {
    method: req.method,
    path: url.pathname,
    statusCode: response.status,
    durationMs,
    region: Deno.env.get('DENO_REGION') ?? 'unknown',
  });

  return response;
});
```

**Notes for Deploy:**
- Set `LOGTIDE_API_URL` and `LOGTIDE_API_KEY` in the Deno Deploy dashboard environment variables
- Use smaller batch sizes since edge functions may terminate quickly
- Network permissions are granted automatically on Deploy

## Permissions Reference

Deno requires explicit permissions. For LogTide:

```bash
# Minimum permissions
deno run \
  --allow-net=your-logtide-host:8080 \                # Network to LogTide
  --allow-env=LOGTIDE_API_URL,LOGTIDE_API_KEY,DENO_ENV \  # Environment variables
  main.ts

# Development (more permissive)
deno run --allow-net --allow-env --allow-read main.ts
```

## Docker Deployment

```dockerfile
FROM denoland/deno:2.0.0

WORKDIR /app

# Cache dependencies
COPY deno.json deno.lock ./
RUN deno install

# Copy application
COPY . .

# Cache the main module
RUN deno cache main.ts

ENV LOGTIDE_API_URL=""
ENV LOGTIDE_API_KEY=""

EXPOSE 8000
CMD ["deno", "run", "--allow-net", "--allow-env", "main.ts"]
```

## Performance

| Metric | Value |
|--------|-------|
| Startup time | ~100ms |
| Log call overhead | <0.3ms (batched) |
| Memory overhead | ~8MB |
| npm: specifier load | First run only (cached after) |

## Troubleshooting

### "Requires net access"

Run with `--allow-net` or specify your LogTide host:

```bash
deno run --allow-net=api.logtide.dev main.ts
```

### npm package not found

Ensure you're using the `npm:` prefix:

```typescript
// Correct
import { LogTideClient } from 'npm:@logtide/sdk-node';

// Wrong (won't resolve)
import { LogTideClient } from '@logtide/sdk-node';
```

### Slow first run

Deno downloads and caches npm packages on first run. Subsequent runs are fast. In Docker, use `RUN deno cache main.ts` to pre-cache.

## Next Steps

- [Node.js SDK Reference](/integrations/nodejs/) - Full SDK documentation
- [Docker Integration](/integrations/docker/) - Container deployments
- [Bun Integration](/integrations/bun/) - Alternative JS runtime


---

# Django Application Logging Integration

> Add structured logging to Django applications with the LogTide Python SDK — built-in middleware, Celery support, admin audit trail, and stdlib logging bridge.

*Source: https://logtide.dev/integrations/django/*

---

LogTide's Python SDK ships a built-in Django middleware that hooks into Django's middleware system for automatic request logging, per-request context, and structured output. This guide covers middleware setup, Celery task logging, admin audit trails, and stdlib logging integration.

## Why use LogTide with Django?

- **Built-in middleware**: `LogTideDjangoMiddleware` handles request/response logging automatically
- **Per-request context**: Attach trace IDs to all logs in a request lifecycle
- **Celery support**: Trace async tasks back to the originating request
- **Admin audit trail**: Log all Django admin actions with user and object context
- **Stdlib logging bridge**: Use `LogTideHandler` with Django's `LOGGING` config
- **Non-blocking**: Background batching keeps request latency unaffected

## Prerequisites

- Python 3.8+ (3.11+ recommended)
- Django 3.2+
- LogTide instance with API key

## Installation

```bash
pip install logtide-sdk[django]
```

## Quick Start

### 1. Configure Settings

```python
# settings.py
import os
from logtide_sdk import LogTideClient, ClientOptions

LOGTIDE_CLIENT = LogTideClient(
    ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
        global_metadata={
            "environment": os.environ.get("DJANGO_ENV", "production"),
            "version": os.environ.get("APP_VERSION", "unknown"),
        },
    )
)

LOGTIDE_SERVICE_NAME = "django-app"

# Optional settings (all have defaults)
LOGTIDE_LOG_REQUESTS = True         # Log incoming requests
LOGTIDE_LOG_RESPONSES = True        # Log responses with status and duration
LOGTIDE_LOG_ERRORS = True           # Log unhandled exceptions
LOGTIDE_INCLUDE_HEADERS = False     # Include request headers in metadata
LOGTIDE_SKIP_HEALTH_CHECK = True    # Auto-skip /health and /healthz
LOGTIDE_SKIP_PATHS = ["/metrics"]   # Additional paths to skip

MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    "logtide_sdk.middleware.LogTideDjangoMiddleware",  # early in the chain
    # ... other middleware
    "django.middleware.common.CommonMiddleware",
]
```

### 2. Use in Views

```python
# views.py
from django.conf import settings
from django.shortcuts import get_object_or_404, render

def user_profile(request, user_id):
    client = settings.LOGTIDE_CLIENT

    client.info("django-app", "Viewing user profile", {"user_id": user_id})

    user = get_object_or_404(User, pk=user_id)
    return render(request, "profile.html", {"user": user})
```

### Environment Variables

```bash
export LOGTIDE_API_URL="http://your-logtide-instance:8080"
export LOGTIDE_API_KEY="lp_your_api_key_here"
```

## Request Logging Output

Each request automatically generates a structured log:

```json
{
  "level": "info",
  "message": "GET /api/users/123 200",
  "service": "django-app",
  "metadata": {
    "method": "GET",
    "path": "/api/users/123",
    "status_code": 200,
    "duration_ms": 34.5,
    "trace_id": "abc123-def456"
  }
}
```

## Trace ID Context

Use context managers to correlate all logs within a request:

```python
# views.py
from django.conf import settings
import uuid

def process_order(request, order_id):
    client = settings.LOGTIDE_CLIENT
    trace_id = request.META.get("HTTP_X_TRACE_ID", str(uuid.uuid4()))

    with client.with_trace_id(trace_id):
        client.info("orders", "Processing order", {"order_id": order_id})
        # ... all logs inside this block share the same trace_id
```

## Django REST Framework Integration

```python
# views.py
from rest_framework.views import APIView
from rest_framework.response import Response
from django.conf import settings

class OrderView(APIView):
    def post(self, request):
        client = settings.LOGTIDE_CLIENT

        client.info(
            "api",
            "Creating order",
            {"user_id": str(request.user.id), "items": len(request.data.get("items", []))},
        )

        try:
            order = create_order(request.user, request.data)
            client.info("api", "Order created", {"order_id": str(order.id)})
            return Response({"id": order.id}, status=201)

        except InsufficientStockError as e:
            client.warn(
                "api",
                "Order failed: insufficient stock",
                {"product_id": str(e.product_id), "requested": e.requested},
            )
            return Response({"error": str(e)}, status=400)
```

## Celery Task Logging

### Configure Celery with LogTide

```python
# celery.py
import os
from celery import Celery
from celery.signals import task_prerun, task_postrun, task_failure

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

app = Celery("myproject")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

@task_prerun.connect
def task_started(sender=None, task_id=None, args=None, kwargs=None, **kw):
    from django.conf import settings
    settings.LOGTIDE_CLIENT.info(
        "celery",
        f"Task started: {sender.name}",
        {"task_id": task_id, "task_name": sender.name},
    )

@task_postrun.connect
def task_completed(sender=None, task_id=None, retval=None, state=None, **kw):
    from django.conf import settings
    settings.LOGTIDE_CLIENT.info(
        "celery",
        f"Task completed: {sender.name}",
        {"task_id": task_id, "task_name": sender.name, "state": state},
    )

@task_failure.connect
def task_failed(sender=None, task_id=None, exception=None, traceback=None, **kw):
    from django.conf import settings
    settings.LOGTIDE_CLIENT.error(
        "celery",
        f"Task failed: {sender.name}",
        exception,
    )
```

### Propagate Trace Context to Tasks

```python
# tasks.py
from celery import shared_task
from django.conf import settings

@shared_task(bind=True)
def process_order(self, order_id: int, trace_id: str = None):
    client = settings.LOGTIDE_CLIENT

    with client.with_trace_id(trace_id or self.request.id):
        client.info("celery", "Processing order", {"order_id": order_id})

        order = Order.objects.get(pk=order_id)
        # ... process order ...

        client.info("celery", "Order processed", {"order_id": order_id})


# In your view, pass the trace ID
def create_order_view(request):
    order = Order.objects.create(...)
    trace_id = request.META.get("HTTP_X_TRACE_ID", str(uuid.uuid4()))
    process_order.delay(order.id, trace_id=trace_id)
    return JsonResponse({"id": order.id})
```

## Admin Audit Trail

Log all Django admin actions with full context:

```python
# admin.py
from django.contrib import admin
from django.conf import settings

class AuditedModelAdmin(admin.ModelAdmin):
    def save_model(self, request, obj, form, change):
        action = "updated" if change else "created"
        settings.LOGTIDE_CLIENT.info(
            "admin",
            f"Admin {action} {obj.__class__.__name__}",
            {
                "action": action,
                "model": obj.__class__.__name__,
                "object_id": str(obj.pk),
                "admin_user": request.user.username,
                "changed_fields": list(form.changed_data) if change else [],
            },
        )
        super().save_model(request, obj, form, change)

    def delete_model(self, request, obj):
        settings.LOGTIDE_CLIENT.warn(
            "admin",
            f"Admin deleted {obj.__class__.__name__}",
            {
                "action": "deleted",
                "model": obj.__class__.__name__,
                "object_id": str(obj.pk),
                "admin_user": request.user.username,
            },
        )
        super().delete_model(request, obj)
```

## stdlib `logging` Integration

Bridge Django's built-in logging framework to LogTide via `LogTideHandler`:

```python
# settings.py
from logtide_sdk import LogTideClient, ClientOptions, LogTideHandler
import logging

LOGTIDE_CLIENT = LogTideClient(ClientOptions(
    api_url=os.environ["LOGTIDE_API_URL"],
    api_key=os.environ["LOGTIDE_API_KEY"],
))

# Add handler programmatically
_handler = LogTideHandler(client=LOGTIDE_CLIENT, service="django-app")
_handler.setLevel(logging.WARNING)

logging.getLogger("django.request").addHandler(_handler)
logging.getLogger("myapp").addHandler(_handler)
```

Or via Django's `LOGGING` dict (requires a custom handler class in your codebase):

```python
# myapp/logging.py
from logtide_sdk import LogTideHandler as _LogTideHandler

class DjangoLogTideHandler(_LogTideHandler):
    """Wrapper so it can be referenced from LOGGING config."""
    pass
```

```python
# settings.py
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "logtide": {
            "()": "myapp.logging.DjangoLogTideHandler",
            "client": LOGTIDE_CLIENT,
            "service": "django-app",
            "level": "WARNING",
        },
        "console": {"class": "logging.StreamHandler"},
    },
    "loggers": {
        "django.request": {
            "handlers": ["logtide", "console"],
            "level": "ERROR",
            "propagate": False,
        },
        "myapp": {
            "handlers": ["logtide", "console"],
            "level": "INFO",
        },
    },
}
```

## Graceful Shutdown

Ensure logs are flushed when Django stops:

```python
# apps.py
import atexit
from django.apps import AppConfig

class MyAppConfig(AppConfig):
    name = "myapp"

    def ready(self):
        from django.conf import settings
        atexit.register(settings.LOGTIDE_CLIENT.close)
```

## Docker Deployment

```yaml
# docker-compose.yml
services:
  web:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DJANGO_ENV=production
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - DATABASE_URL=postgres://user:pass@db:5432/myapp
    command: gunicorn myproject.wsgi:application --bind 0.0.0.0:8000 --workers 4
    depends_on:
      - db

  celery:
    build: .
    environment:
      - DJANGO_ENV=production
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    command: celery -A myproject worker -l info
    depends_on:
      - db
      - redis

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass

  redis:
    image: redis:7-alpine
```

## Performance

| Metric | Value |
|--------|-------|
| Middleware overhead | <1ms per request |
| Memory overhead | ~10MB |
| Network calls | 1 per batch (100 logs default) |
| Celery task overhead | <0.5ms |

## Troubleshooting

### Logs not appearing

1. Enable debug mode: `ClientOptions(..., debug=True)`
2. Check middleware order — `LogTideDjangoMiddleware` should be near the top of `MIDDLEWARE`
3. Ensure `LOGTIDE_CLIENT.close()` is called on exit (see Graceful Shutdown section)
4. Check circuit breaker state: `print(settings.LOGTIDE_CLIENT.get_circuit_breaker_state())`

### Duplicate request logs

If you see duplicate logs, check that you are not capturing requests both via `LogTideDjangoMiddleware` and via the stdlib `LOGGING` config. Use one approach.

### Celery tasks missing logs

Ensure the Celery worker process has access to the LogTide environment variables:

```bash
celery -A myproject inspect conf
```

## Next Steps

- [Python SDK Reference](/integrations/python/) - Full SDK documentation
- [Docker Integration](/integrations/docker/) - Container deployments
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging
- [Security Monitoring](/use-cases/security-monitoring/) - Admin audit alerts


---

# Docker Compose Logging Integration

> Centralized logging for Docker Compose stacks using Fluent Bit to collect and ship logs from all services to LogTide.

*Source: https://logtide.dev/integrations/docker-compose/*

---

Docker Compose makes it easy to run multi-container applications, but logs are scattered across containers. This guide shows you how to centralize all your Docker Compose logs in LogTide without modifying your application code.

## Why centralize Docker Compose logs?

- **Zero code changes**: Collect logs from any container without SDK integration
- **Automatic service tagging**: Logs are tagged with service name, container ID, and compose project
- **Unified view**: See all your services' logs in one place
- **Persistent logging**: Logs survive container restarts and redeployments
- **Production-ready**: Battle-tested configuration for high-volume logging

## Prerequisites

- Docker Engine 20.10+
- Docker Compose v2 (comes with Docker Desktop)
- LogTide instance with API key
- Basic understanding of Docker Compose

## Quick Start (5 minutes)

Here's a minimal working example with nginx and a Node.js app:

### docker-compose.yml

```yaml
services:
  # Your application services
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "docker.{{.Name}}"
    depends_on:
      - fluent-bit
      - app

  app:
    build: .
    environment:
      - NODE_ENV=production
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "docker.{{.Name}}"
    depends_on:
      - fluent-bit

  # Log collector
  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    ports:
      - "24224:24224"
    environment:
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
```

### fluent-bit.conf

```ini
[SERVICE]
    Flush         1
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf

[INPUT]
    Name          forward
    Listen        0.0.0.0
    Port          24224

[FILTER]
    Name          record_modifier
    Match         docker.*
    Record        hostname ${HOSTNAME}
    Record        environment production

[OUTPUT]
    Name          http
    Match         *
    Host          ${LOGTIDE_API_URL}
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

### Start the Stack

```bash
# Set environment variables
export LOGTIDE_API_URL=api.logtide.dev
export LOGTIDE_API_KEY=your-api-key

# Start all services
docker compose up -d

# Check logs are flowing
docker compose logs fluent-bit
```

## Production Configuration

For production deployments, use this enhanced configuration:

### docker-compose.production.yml

```yaml
services:
  # Reverse proxy
  nginx:
    image: nginx:alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "{{.Name}}"
        fluentd-async: "true"
        fluentd-buffer-limit: "1048576"
    depends_on:
      fluent-bit:
        condition: service_healthy
    networks:
      - frontend
      - backend

  # Application
  app:
    build:
      context: .
      dockerfile: Dockerfile.production
    restart: unless-stopped
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgres://user:pass@postgres:5432/app
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "{{.Name}}"
        fluentd-async: "true"
    depends_on:
      fluent-bit:
        condition: service_healthy
      postgres:
        condition: service_healthy
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    networks:
      - backend

  # Background worker
  worker:
    build:
      context: .
      dockerfile: Dockerfile.worker
    restart: unless-stopped
    environment:
      - QUEUE_URL=redis://redis:6379
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "{{.Name}}"
        fluentd-async: "true"
    depends_on:
      - fluent-bit
      - redis
    networks:
      - backend

  # Database
  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
      - POSTGRES_DB=app
    volumes:
      - postgres_data:/var/lib/postgresql/data
    secrets:
      - db_password
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "{{.Name}}"
        fluentd-async: "true"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d app"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - backend

  # Cache
  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "{{.Name}}"
        fluentd-async: "true"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - backend

  # Log collector
  fluent-bit:
    image: fluent/fluent-bit:latest
    restart: unless-stopped
    volumes:
      - ./config/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./config/parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - fluent_buffer:/var/log/fluent-bit
    ports:
      - "24224:24224"
    environment:
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - COMPOSE_PROJECT=${COMPOSE_PROJECT_NAME:-myapp}
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:2020/api/v1/health"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - frontend
      - backend

networks:
  frontend:
  backend:

volumes:
  postgres_data:
  redis_data:
  fluent_buffer:

secrets:
  db_password:
    file: ./secrets/db_password.txt
```

### Production Fluent Bit Configuration

```ini
# config/fluent-bit.conf
[SERVICE]
    Flush             5
    Grace             30
    Log_Level         info
    Daemon            off
    Parsers_File      parsers.conf
    HTTP_Server       On
    HTTP_Listen       0.0.0.0
    HTTP_Port         2020
    Health_Check      On
    HC_Errors_Count   5
    HC_Retry_Failure_Count  5
    HC_Period         60
    storage.path      /var/log/fluent-bit/
    storage.sync      normal
    storage.checksum  off
    storage.backlog.mem_limit 50M

[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M

# Parse JSON logs from applications
[FILTER]
    Name              parser
    Match             *
    Key_Name          log
    Parser            docker_json
    Reserve_Data      On
    Preserve_Key      Off

# Add metadata
[FILTER]
    Name              record_modifier
    Match             *
    Record            project ${COMPOSE_PROJECT}
    Record            environment production

# Extract service name from container name
[FILTER]
    Name              lua
    Match             *
    script            /fluent-bit/etc/extract_service.lua
    call              extract_service

# Retry on failure with exponential backoff
[OUTPUT]
    Name              http
    Match             *
    Host              ${LOGTIDE_API_URL}
    Port              443
    URI               /api/v1/ingest/single
    Format            json
    Header            X-API-Key ${LOGTIDE_API_KEY}
    Header            Content-Type application/json
    tls               On
    tls.verify        On
    Retry_Limit       5
    storage.total_limit_size  100M
```

### Lua Script for Service Extraction

```lua
-- config/extract_service.lua
function extract_service(tag, timestamp, record)
    -- Container name format: project-service-1
    local container = record["container_name"] or ""
    local service = container:match("^[^-]+-([^-]+)")
    if service then
        record["service"] = service
    end
    return 1, timestamp, record
end
```

### Parsers Configuration

```ini
# config/parsers.conf
[PARSER]
    Name        docker_json
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

[PARSER]
    Name        nginx_access
    Format      regex
    Regex       ^(?<remote>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^"]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name        postgres
    Format      regex
    Regex       ^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} \w+) \[(?<pid>\d+)\] (?<level>\w+):  (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L %Z
```

## Logging Driver Options

The fluentd logging driver supports several useful options:

```yaml
logging:
  driver: fluentd
  options:
    # Fluent Bit address
    fluentd-address: localhost:24224

    # Tag for routing
    tag: "{{.Name}}"  # or "docker.{{.ImageName}}.{{.ID}}"

    # Async mode (recommended for production)
    fluentd-async: "true"

    # Buffer settings
    fluentd-buffer-limit: "1048576"  # 1MB buffer
    fluentd-retry-wait: "1s"
    fluentd-max-retries: "3"

    # Sub-second precision
    fluentd-sub-second-precision: "true"
```

### Available Tag Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `{{.ID}}` | Container ID (12 chars) | `abc123def456` |
| `{{.FullID}}` | Full container ID | `abc123...xyz` |
| `{{.Name}}` | Container name | `myapp-web-1` |
| `{{.ImageID}}` | Image ID | `sha256:abc...` |
| `{{.ImageName}}` | Image name | `nginx:alpine` |
| `{{.ImageFullID}}` | Full image ID | `sha256:abc...xyz` |
| `{{.DaemonName}}` | Docker daemon name | `docker` |

## Handling JSON Logs

If your applications output JSON logs, configure Fluent Bit to parse them:

### Application Outputting JSON

```javascript
// app.js
const log = (level, message, data = {}) => {
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    level,
    message,
    ...data
  }));
};

log('info', 'Server started', { port: 3000 });
log('error', 'Database connection failed', { error: 'timeout' });
```

### Fluent Bit Parser Filter

```ini
[FILTER]
    Name          parser
    Match         myapp-app-*
    Key_Name      log
    Parser        docker_json
    Reserve_Data  On
    Preserve_Key  Off
```

This extracts JSON fields from the `log` key and merges them into the record.

## Multiple Environments

Use Compose profiles for different environments:

### docker-compose.yml

```yaml
services:
  fluent-bit:
    image: fluent/fluent-bit:latest
    profiles: ["logging"]
    volumes:
      - ./config/${ENVIRONMENT:-development}/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    environment:
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
```

### Development Config

```ini
# config/development/fluent-bit.conf
[SERVICE]
    Flush     1
    Log_Level debug
    Daemon    off

[INPUT]
    Name   forward
    Port   24224

[OUTPUT]
    Name   stdout
    Match  *
    Format json_lines
```

### Production Config

```ini
# config/production/fluent-bit.conf
[SERVICE]
    Flush     5
    Log_Level info
    Daemon    off

[INPUT]
    Name   forward
    Port   24224

[OUTPUT]
    Name   http
    Match  *
    # ... production settings
```

### Start with Profile

```bash
# Development (logs to stdout)
docker compose up

# Production (logs to LogTide)
docker compose --profile logging up -d
```

## Scaling Considerations

### Horizontal Scaling

When scaling services, each replica sends logs to Fluent Bit:

```yaml
services:
  app:
    deploy:
      replicas: 5
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "{{.Name}}"  # Includes replica number: myapp-app-1, myapp-app-2, etc.
```

### High Availability Fluent Bit

For mission-critical logging, run multiple Fluent Bit instances:

```yaml
services:
  fluent-bit-1:
    image: fluent/fluent-bit:latest
    ports:
      - "24224:24224"
    # ...

  fluent-bit-2:
    image: fluent/fluent-bit:latest
    ports:
      - "24225:24224"
    # ...

  app:
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224,localhost:24225"
        fluentd-async: "true"
```

## Verification

### Check Fluent Bit Status

```bash
# Health check
curl http://localhost:2020/api/v1/health

# Metrics
curl http://localhost:2020/api/v1/metrics

# Storage status (if using buffering)
curl http://localhost:2020/api/v1/storage
```

### View Collected Logs

```bash
# Follow Fluent Bit logs
docker compose logs -f fluent-bit

# Check specific service logs in LogTide
# Filter by: service = "app" AND project = "myapp"
```

### Test Log Delivery

```bash
# Send a test log
docker compose exec app sh -c 'echo "{\"level\":\"info\",\"message\":\"Test log\"}"'

# Check LogTide dashboard for the test log
```

## Performance Tuning

### Memory Usage

Control buffer sizes to manage memory:

```ini
[INPUT]
    Name              forward
    Buffer_Chunk_Size 512K   # Reduce for memory-constrained environments
    Buffer_Max_Size   2M

[SERVICE]
    storage.backlog.mem_limit 25M  # Limit backlog memory
```

### Throughput

For high-volume logging:

```ini
[SERVICE]
    Flush    1            # More frequent flushes
    Workers  4            # Parallel output workers

[OUTPUT]
    Name     http
    Workers  2            # Parallel HTTP connections
```

### Disk Buffering

Enable disk buffering for reliability:

```ini
[SERVICE]
    storage.path              /var/log/fluent-bit/
    storage.sync              normal
    storage.checksum          off

[INPUT]
    Name                      forward
    storage.type              filesystem  # Enable disk buffering

[OUTPUT]
    Name                      http
    storage.total_limit_size  500M  # Max disk usage per output
```

## Troubleshooting

### Logs not reaching LogTide

1. **Check Fluent Bit is running:**
```bash
docker compose ps fluent-bit
docker compose logs fluent-bit
```

2. **Verify connectivity:**
```bash
docker compose exec fluent-bit curl -v https://api.logtide.dev/health
```

3. **Check logging driver:**
```bash
docker inspect myapp-app-1 --format '{{.HostConfig.LogConfig}}'
```

### Container startup fails with logging driver

If containers fail because Fluent Bit isn't ready:

```yaml
services:
  app:
    depends_on:
      fluent-bit:
        condition: service_healthy
    logging:
      driver: fluentd
      options:
        fluentd-async: "true"  # Don't block on logging
```

### High memory usage

Reduce buffer sizes and enable disk buffering:

```ini
[INPUT]
    Name              forward
    Buffer_Chunk_Size 256K
    Buffer_Max_Size   1M
    storage.type      filesystem
```

## Next Steps

- **[Docker Integration](/integrations/docker/)** - Single container logging patterns
- **[nginx Integration](/integrations/nginx/)** - Detailed nginx access log parsing
- **[Node.js Integration](/integrations/nodejs/)** - Application-level logging
- **[GDPR Compliance](/use-cases/gdpr-compliance/)** - Privacy-first logging setup


---

# Docker Container Logging Integration

> Send Docker container logs to LogTide using Fluent Bit. Complete setup guide with Docker Compose and Kubernetes examples.

*Source: https://logtide.dev/integrations/docker/*

---

Collecting logs from Docker containers is essential for debugging, monitoring, and security. This guide shows you how to send container logs to LogTide using Fluent Bit - no code changes required.

## Why send Docker logs to LogTide?

- **Centralized visibility**: All container logs in one place with powerful search
- **No agent per container**: Single Fluent Bit instance collects from all containers
- **Automatic service detection**: Container names become service names in LogTide
- **Privacy-first**: Logs stay in your infrastructure with GDPR-compliant storage
- **Real-time tailing**: Watch logs stream in with live tail feature

## Prerequisites

- Docker 19.03+ or Docker Compose v2+
- LogTide instance running (self-hosted or cloud)
- API key from LogTide dashboard

## Quick Start (5 minutes)

The fastest way to get Docker logs into LogTide is using Fluent Bit as a sidecar or system-level collector.

### Step 1: Create Fluent Bit Configuration

Create a file named `fluent-bit.conf`:

```ini
[SERVICE]
    Flush         1
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          forward
    Listen        0.0.0.0
    Port          24224

[OUTPUT]
    Name          http
    Match         *
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json_lines
    Header        X-API-Key YOUR_API_KEY
    Header        Content-Type application/x-ndjson
    Json_date_key time
    Json_date_format iso8601
    tls           On
```

Replace `YOUR_LOGTIDE_HOST` with your LogTide API endpoint (e.g., `api.logtide.dev` or your self-hosted domain) and `YOUR_API_KEY` with your project API key.

### Step 2: Run Fluent Bit with Docker

```bash
docker run -d \
  --name fluent-bit \
  -p 24224:24224 \
  -v $(pwd)/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \
  fluent/fluent-bit:latest
```

### Step 3: Configure Docker to use Fluent Bit

Run containers with the Fluentd log driver:

```bash
docker run -d \
  --name my-app \
  --log-driver=fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt tag="{{.Name}}" \
  your-image:latest
```

That's it! Your container logs should now appear in LogTide within seconds.

## Production Setup with Docker Compose

For production, use Docker Compose to manage both Fluent Bit and your application containers.

### docker-compose.yml

```yaml
version: "3.8"

services:
  fluent-bit:
    image: fluent/fluent-bit:latest
    container_name: fluent-bit
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - ./fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf
    ports:
      - "24224:24224"
    restart: unless-stopped

  web:
    image: nginx:alpine
    container_name: web
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        tag: "web"
    depends_on:
      - fluent-bit
    ports:
      - "80:80"

  api:
    build: ./api
    container_name: api
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        tag: "api"
    depends_on:
      - fluent-bit
    environment:
      - NODE_ENV=production
```

### Enhanced Fluent Bit Configuration

For production, use a more robust configuration with parsing and buffering:

```ini
[SERVICE]
    Flush         1
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020

# Accept logs from Docker containers
[INPUT]
    Name          forward
    Listen        0.0.0.0
    Port          24224
    Buffer_Chunk_Size 1M
    Buffer_Max_Size   6M

# Parse JSON logs if your app outputs structured logs
[FILTER]
    Name          parser
    Match         *
    Key_Name      log
    Parser        json
    Reserve_Data  On

# Add container metadata
[FILTER]
    Name          modify
    Match         *
    Add           source docker
    Add           environment production

# Add required fields for LogTide
[FILTER]
    Name          modify
    Match         *
    Add           level info
    Rename        log message
    Copy          container_name service

# Send to LogTide
[OUTPUT]
    Name          http
    Match         *
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json_lines
    Header        X-API-Key YOUR_API_KEY
    Header        Content-Type application/x-ndjson
    Json_date_key time
    Json_date_format iso8601
    tls           On
    tls.verify    On
    Retry_Limit   5
```

### parsers.conf

```ini
[PARSER]
    Name        json
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
```

## Kubernetes Setup

For Kubernetes deployments, use a DaemonSet to collect logs from all nodes.

### fluent-bit-daemonset.yaml

```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        ports:
        - containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config
          mountPath: /fluent-bit/etc/
        env:
        - name: LOGTIDE_HOST
          valueFrom:
            secretKeyRef:
              name: logtide-credentials
              key: host
        - name: LOGTIDE_API_KEY
          valueFrom:
            secretKeyRef:
              name: logtide-credentials
              key: api-key
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config
        configMap:
          name: fluent-bit-config
```

Create the secret for your API key:

```bash
kubectl create secret generic logtide-credentials \
  --namespace=logging \
  --from-literal=host=api.logtide.dev \
  --from-literal=api-key=YOUR_API_KEY
```

## Verification

Check that logs are arriving in LogTide:

1. Open LogTide dashboard and navigate to your project
2. Use the search bar to filter by service name (container name)
3. Enable live tail to see new logs in real-time

### Debugging

If logs aren't appearing, check Fluent Bit status:

```bash
# Check Fluent Bit logs
docker logs fluent-bit

# Verify Fluent Bit is receiving logs
curl http://localhost:2020/api/v1/metrics/prometheus
```

Common issues:
- **Connection refused**: Ensure LogTide host and port are correct
- **401 Unauthorized**: Check your API key is valid and has ingestion permissions
- **Logs delayed**: Increase `Flush` interval or check network latency

## Performance Considerations

| Metric | Value | Notes |
|--------|-------|-------|
| Memory | ~50MB | Per Fluent Bit instance |
| CPU | <1% | At 1000 logs/second |
| Network | ~1KB per log | Average with metadata |
| Latency | <100ms | From container to LogTide |

For high-volume environments (>10,000 logs/second), consider:
- Running multiple Fluent Bit instances behind a load balancer
- Enabling compression in the HTTP output
- Using the batch endpoint (`/api/v1/ingest`) with buffering

## Detection Rules

Once your Docker logs are in LogTide, create detection rules for common issues:

**Container restart detection:**
```
service:* AND message:"container restart"
```

**Out of memory errors:**
```
level:error AND message:"out of memory"
```

**Failed health checks:**
```
service:healthcheck AND level:error
```

## Next Steps

- [nginx Integration](/integrations/nginx/) - Detailed nginx log parsing
- [Node.js SDK](/integrations/nodejs/) - Structured logging from Node.js apps
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging


---

# Elysia Framework Logging Integration

> Add structured logging to Elysia applications running on Bun with lifecycle hooks, scoped context, and trace propagation.

*Source: https://logtide.dev/integrations/elysia/*

---

LogTide's Elysia SDK provides a Bun-optimized plugin for automatic request logging, scoped context, lifecycle hooks, and W3C trace propagation.

## Why use LogTide with Elysia?

- **Bun-optimized**: Takes advantage of Bun's performance
- **Plugin architecture**: Use `.as('global')` for app-wide coverage
- **Lifecycle hooks**: onRequest, onAfterHandle, onError
- **Scoped context**: Per-request scope via store decorators

## Prerequisites

- Bun 1.x
- Elysia 1.x
- LogTide instance with a DSN

## Installation

```bash
bun add @logtide/elysia
```

## Quick Start

```typescript
import { Elysia } from 'elysia';
import { hub } from '@logtide/core';
import { logtide } from '@logtide/elysia';

const app = new Elysia()
  .use(logtide({
    dsn: process.env.LOGTIDE_DSN,
    service: 'elysia-api',
    environment: process.env.NODE_ENV,
  }))
  .get('/users/:id', ({ params }) => {
    hub.captureLog('info', 'Fetching user', { userId: params.id });
    return { id: params.id };
  })
  .listen(3000);
```

Use `.as('global')` to apply the plugin to all routes, including those registered after.

## Plugin Options

```typescript
app.use(logtide({
  dsn: process.env.LOGTIDE_DSN,
  service: 'elysia-api',
  environment: process.env.NODE_ENV,
  release: '1.0.0',
  tracesSampleRate: 1.0,
}));
```

## Scoped Context

```typescript
app
  .use(logtide({
    dsn: process.env.LOGTIDE_DSN,
    service: 'elysia-api',
  }))
  .get('/orders', () => {
    hub.captureLog('info', 'Listing orders');
    return { orders: [] };
  });
```

## Lifecycle Hooks

| Hook | Behavior |
|------|----------|
| `onRequest` | Creates scope, extracts traceparent, starts span |
| `onAfterHandle` | Logs completion with duration |
| `onError` | Captures errors with request context |

## Error Handling

```typescript
app
  .use(logtide({
    dsn: process.env.LOGTIDE_DSN,
    service: 'elysia-api',
  }))
  .onError(({ code, error }) => {
    // Error is already captured by the plugin
    return new Response(JSON.stringify({
      error: error.message,
    }), {
      status: code === 'NOT_FOUND' ? 404 : 500,
    });
  });
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Elysia SDK Reference](/docs/sdks/elysia/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# Express.js Logging Integration

> Add structured logging to Express.js applications with automatic request tracing, scoped context, and error capture.

*Source: https://logtide.dev/integrations/express/*

---

LogTide's Express SDK provides drop-in middleware for automatic request logging, per-request context scoping, and distributed tracing with W3C Trace Context.

## Why use LogTide with Express?

- **Zero-config request logging**: Every HTTP request/response logged automatically
- **Per-request scoping**: Attach user, tenant, and custom tags to all logs in a request
- **Distributed tracing**: W3C traceparent headers propagated automatically
- **Error capture**: Unhandled errors captured with full request context
- **Non-blocking**: Logging never slows down your request handling

## Prerequisites

- Node.js 18+
- Express 4.x or 5.x
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/express
```

This includes `@logtide/core` as a dependency.

## Quick Start

```typescript
import express from 'express';
import { hub } from '@logtide/core';
import { logtide } from '@logtide/express';

const app = express();

// Add LogTide middleware (initializes LogTide automatically)
app.use(logtide({
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
  environment: process.env.NODE_ENV,
}));

app.get('/users/:id', (req, res) => {
  req.logtideScope.setTag('userId', req.params.id);
  res.json({ id: req.params.id });
});

app.listen(3000);
```

## Middleware Configuration

```typescript
app.use(logtide({
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
  environment: process.env.NODE_ENV,
  release: '1.0.0',
  tracesSampleRate: 1.0,
}));
```

## Per-Request Context

Attach context to all logs within a request:

```typescript
app.use((req, res, next) => {
  if (req.user) {
    req.logtideScope.setExtra('userId', req.user.id);
    req.logtideScope.setExtra('email', req.user.email);
  }
  req.logtideScope.setTag('tenant', req.headers['x-tenant-id']);
  next();
});
```

## Trace Propagation

The middleware extracts incoming `traceparent` headers and makes the trace ID available:

```typescript
app.get('/api/data', async (req, res) => {
  // Include trace ID in responses for debugging
  res.set('X-Trace-Id', req.logtideTraceId);
  res.json({ data: [] });
});
```

## Error Handling

```typescript
// The middleware automatically captures 5xx errors.
// Your error response handler:
app.use((err, req, res, next) => {
  res.status(err.status || 500).json({
    error: err.message,
    traceId: req.logtideTraceId,
  });
});
```

## Request Logging Output

Each request generates a structured log:

```json
{
  "level": "info",
  "message": "GET /users/123 200",
  "service": "api-server",
  "metadata": {
    "method": "GET",
    "path": "/users/123",
    "statusCode": 200,
    "duration": 45,
    "ip": "192.168.1.1",
    "userAgent": "Mozilla/5.0..."
  }
}
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Express SDK Reference](/docs/sdks/express/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# FastAPI Application Logging Integration

> Add structured logging to FastAPI apps with the LogTide Python SDK — async middleware, dependency injection, WebSocket support, and Uvicorn deployment.

*Source: https://logtide.dev/integrations/fastapi/*

---

LogTide's Python SDK ships a built-in FastAPI/Starlette middleware for automatic structured logging with timing and trace IDs. This guide covers middleware setup, dependency-based logging, WebSocket tracing, background tasks, and production deployment with Uvicorn.

## Why use LogTide with FastAPI?

- **Built-in async middleware**: `LogTideFastAPIMiddleware` handles request/response logging automatically
- **Async-native**: Uses `AsyncLogTideClient` for non-blocking log shipping
- **Dependency injection**: Inject a request-scoped logger into any endpoint via `Depends()`
- **WebSocket support**: Log WebSocket connection lifecycle and message events
- **Background tasks**: Trace background task execution back to the originating request
- **Zero overhead**: Background batching keeps endpoint latency unaffected

## Prerequisites

- Python 3.8+ (3.11+ recommended for performance)
- FastAPI 0.100+
- LogTide instance with API key

## Installation

```bash
pip install logtide-sdk[fastapi]
```

For the async client:

```bash
pip install logtide-sdk[fastapi,async]
```

## Quick Start

```python
# main.py
import os
from contextlib import asynccontextmanager
from fastapi import FastAPI
from logtide_sdk import LogTideClient, ClientOptions
from logtide_sdk.middleware import LogTideFastAPIMiddleware

# Create client at module level so it is available for add_middleware()
client = LogTideClient(
    ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
        global_metadata={
            "environment": os.environ.get("APP_ENV", "production"),
            "version": os.environ.get("APP_VERSION", "unknown"),
        },
    )
)


@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.logtide = client
    yield
    client.close()


app = FastAPI(title="My API", lifespan=lifespan)
app.add_middleware(LogTideFastAPIMiddleware, client=client, service_name="fastapi-app")


@app.get("/")
async def root():
    return {"message": "Hello, World!"}
```

### Simpler Setup (Sync Client)

```python
import os
from fastapi import FastAPI
from logtide_sdk import LogTideClient, ClientOptions
from logtide_sdk.middleware import LogTideFastAPIMiddleware

app = FastAPI()
client = LogTideClient(ClientOptions(
    api_url=os.environ["LOGTIDE_API_URL"],
    api_key=os.environ["LOGTIDE_API_KEY"],
))

app.add_middleware(LogTideFastAPIMiddleware, client=client, service_name="fastapi-app")
```

### Environment Variables

```bash
export LOGTIDE_API_URL="http://your-logtide-instance:8080"
export LOGTIDE_API_KEY="lp_your_api_key_here"
uvicorn main:app --reload
```

## Skipped Paths

When `skip_health_check=True` (the default), the middleware skips logging for `/health`, `/healthz`, `/docs`, `/redoc`, and `/openapi.json`. Use `skip_paths` for any additional paths:

```python
app.add_middleware(
    LogTideFastAPIMiddleware,
    client=client,
    service_name="fastapi-app",
    skip_paths=["/metrics", "/ready"],
)
```

## Request Logging Output

Each request automatically generates a structured log:

```json
{
  "level": "info",
  "message": "GET /api/users/42 200",
  "service": "fastapi-app",
  "metadata": {
    "method": "GET",
    "path": "/api/users/42",
    "status_code": 200,
    "duration_ms": 12.4,
    "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  }
}
```

## Dependency Injection for Logging

Use FastAPI's dependency injection to provide a request-scoped logger with automatic trace context:

```python
# dependencies.py
from fastapi import Request, Depends
from logtide_sdk import LogTideClient


class RequestLogger:
    """Request-scoped logger that automatically includes trace context."""

    def __init__(self, client: LogTideClient, trace_id: str, service: str = "api"):
        self.client = client
        self._service = service
        self._trace_id = trace_id

    def info(self, message: str, metadata: dict = None):
        self.client.info(self._service, message, {**(metadata or {}), "trace_id": self._trace_id})

    def warn(self, message: str, metadata: dict = None):
        self.client.warn(self._service, message, {**(metadata or {}), "trace_id": self._trace_id})

    def error(self, message: str, metadata=None):
        self.client.error(self._service, message, metadata or {"trace_id": self._trace_id})

    def debug(self, message: str, metadata: dict = None):
        self.client.debug(self._service, message, {**(metadata or {}), "trace_id": self._trace_id})


async def get_logger(request: Request) -> RequestLogger:
    return RequestLogger(
        client=request.app.state.logtide,
        trace_id=request.headers.get("X-Trace-ID", "unknown"),
    )
```

### Using the Dependency in Routes

```python
# routes/users.py
from fastapi import APIRouter, Depends, HTTPException
from dependencies import RequestLogger, get_logger

router = APIRouter(prefix="/api/users", tags=["users"])

@router.get("/{user_id}")
async def get_user(user_id: int, logger: RequestLogger = Depends(get_logger)):
    logger.info("Fetching user profile", {"user_id": user_id})

    user = await user_repository.get(user_id)
    if not user:
        logger.warn("User not found", {"user_id": user_id})
        raise HTTPException(status_code=404, detail="User not found")

    return user

@router.post("/")
async def create_user(data: CreateUserRequest, logger: RequestLogger = Depends(get_logger)):
    logger.info("Creating user", {"email": data.email})

    try:
        user = await user_repository.create(data)
        logger.info("User created", {"user_id": user.id})
        return user
    except DuplicateEmailError:
        logger.warn("Duplicate email", {"email": data.email})
        raise HTTPException(status_code=409, detail="Email already exists")
```

## Background Tasks Logging

Trace background tasks back to the originating request by passing the trace ID explicitly:

```python
# routes/orders.py
from fastapi import APIRouter, BackgroundTasks, Depends, Request
from logtide_sdk import LogTideClient

router = APIRouter(prefix="/api/orders", tags=["orders"])

async def process_order_background(order_id: int, trace_id: str, client: LogTideClient):
    with client.with_trace_id(trace_id):
        client.info("worker", "Processing order", {"order_id": order_id})
        try:
            await charge_payment(order_id)
            await send_confirmation_email(order_id)
            client.info("worker", "Order processed", {"order_id": order_id})
        except Exception as e:
            client.error("worker", "Order processing failed", e)


@router.post("/")
async def create_order(
    data: CreateOrderRequest,
    background_tasks: BackgroundTasks,
    request: Request,
):
    client = request.app.state.logtide
    trace_id = request.headers.get("X-Trace-ID", "unknown")

    order = await order_repository.create(data)
    client.info("api", "Order created, queuing processing", {"order_id": order.id, "trace_id": trace_id})

    # Pass trace_id explicitly — background tasks run outside the request scope
    background_tasks.add_task(process_order_background, order.id, trace_id, client)

    return {"id": order.id, "status": "processing"}
```

## WebSocket Logging

Log WebSocket connection lifecycle and message events:

```python
# routes/websocket.py
from fastapi import APIRouter, WebSocket, WebSocketDisconnect, Request
import uuid

router = APIRouter()

@router.websocket("/ws/{channel}")
async def websocket_endpoint(websocket: WebSocket, channel: str):
    client = websocket.app.state.logtide
    connection_id = str(uuid.uuid4())

    await websocket.accept()
    client.info("websocket", "Connection opened", {"connection_id": connection_id, "channel": channel})

    try:
        while True:
            data = await websocket.receive_text()
            client.debug("websocket", "Message received", {
                "connection_id": connection_id,
                "channel": channel,
                "size": len(data),
            })
            await websocket.send_text(f"Echo: {data}")

    except WebSocketDisconnect as e:
        client.info("websocket", "Connection closed", {
            "connection_id": connection_id,
            "channel": channel,
            "close_code": e.code,
        })
    except Exception as e:
        client.error("websocket", "Connection error", e)
```

## Exception Handlers

Register global exception handlers with structured logging:

```python
# main.py
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from logtide_sdk import serialize_exception

def register_exception_handlers(app: FastAPI):
    @app.exception_handler(Exception)
    async def unhandled_exception_handler(request: Request, exc: Exception):
        trace_id = request.headers.get("X-Trace-ID", "unknown")
        app.state.logtide.error(
            "api",
            f"Unhandled exception: {type(exc).__name__}",
            {**serialize_exception(exc), "path": request.url.path, "trace_id": trace_id},
        )
        return JSONResponse(
            status_code=500,
            content={"detail": "Internal server error", "trace_id": trace_id},
        )
```

## Async Client (High-Throughput)

For maximum async performance, use `AsyncLogTideClient`:

```python
import os
from contextlib import asynccontextmanager
from fastapi import FastAPI
from logtide_sdk import AsyncLogTideClient, ClientOptions
from logtide_sdk.middleware import LogTideFastAPIMiddleware


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with AsyncLogTideClient(ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
    )) as client:
        app.state.logtide = client
        yield


app = FastAPI(lifespan=lifespan)
app.add_middleware(LogTideFastAPIMiddleware, client=..., service_name="fastapi-app")
```

## Uvicorn Deployment

### Run with Uvicorn

```bash
# Development
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Production
uvicorn main:app \
    --host 0.0.0.0 \
    --port 8000 \
    --workers 4 \
    --loop uvloop \
    --http httptools \
    --no-access-log
```

## Docker Deployment

```dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser
USER appuser

EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "4"]
```

```yaml
# docker-compose.yml
services:
  api:
    build: .
    ports:
      - "8000:8000"
    environment:
      - APP_ENV=production
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - DATABASE_URL=postgresql+asyncpg://user:pass@db:5432/myapp
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 10s
      timeout: 5s
      retries: 5
```

## Performance

| Metric | Value |
|--------|-------|
| Middleware overhead | <0.5ms per request |
| Memory overhead | ~8MB |
| Network calls | 1 per batch (100 logs default) |
| WebSocket event overhead | <0.2ms per message |
| Uvicorn worker compatibility | uvloop, asyncio |

## Troubleshooting

### Logs not appearing

1. Enable debug mode: `ClientOptions(..., debug=True)`
2. Verify the lifespan handler initializes the client before routes are called
3. Check circuit breaker state: `print(app.state.logtide.get_circuit_breaker_state())`

### Middleware not capturing requests

Add the middleware **after** creating the `FastAPI` instance but **before** including routers:

```python
app = FastAPI(lifespan=lifespan)
app.add_middleware(LogTideFastAPIMiddleware, client=client, service_name="fastapi-app")
app.include_router(users_router)
```

### Trace ID missing in background tasks

Background tasks run outside the request scope. Always pass `trace_id` explicitly:

```python
background_tasks.add_task(my_task, trace_id=request.headers.get("X-Trace-ID"))
```

### WebSocket logs flooding

For high-traffic WebSocket connections, use `debug` level for individual messages and `info` only for connection events. Set `debug=False` in production `ClientOptions`.

## Next Steps

- [Python SDK Reference](/integrations/python/) - Full SDK documentation
- [Docker Integration](/integrations/docker/) - Container deployment patterns
- [PostgreSQL Integration](/integrations/postgresql/) - Database logging correlation
- [API Monitoring](/use-cases/api-monitoring/) - Endpoint performance tracking
- [Security Monitoring](/use-cases/security-monitoring/) - Threat detection and alerting


---

# Fastify Logging Integration

> Add structured logging to Fastify applications with plugin-based integration, lifecycle hooks, and trace propagation.

*Source: https://logtide.dev/integrations/fastify/*

---

LogTide's Fastify SDK provides a native plugin that hooks into Fastify's lifecycle for automatic request logging, per-request scoping, and distributed tracing.

## Why use LogTide with Fastify?

- **Native plugin architecture**: Follows Fastify plugin conventions
- **Lifecycle hooks**: Automatic instrumentation via onRequest, onSend, onResponse, onError
- **Per-request scoping**: Isolated context per request through decorators
- **Zero overhead**: Asynchronous batching keeps Fastify fast

## Prerequisites

- Node.js 18+
- Fastify 4.x or 5.x
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/fastify
```

## Quick Start

```typescript
import Fastify from 'fastify';
import { hub } from '@logtide/core';
import { logtide } from '@logtide/fastify';

const fastify = Fastify();

// Register the plugin (initializes LogTide automatically)
await fastify.register(logtide, {
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
  environment: process.env.NODE_ENV,
});

fastify.get('/users/:id', async (request) => {
  request.logtideScope.setTag('userId', request.params.id);

  hub.captureLog('info', 'Fetching user', {
    userId: request.params.id,
  });

  return { id: request.params.id };
});

await fastify.listen({ port: 3000 });
```

## Plugin Options

```typescript
await fastify.register(logtide, {
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
  environment: process.env.NODE_ENV,
  release: '1.0.0',
  tracesSampleRate: 1.0,
});
```

## Request Scoping

Each request gets an isolated scope:

```typescript
fastify.addHook('onRequest', async (request) => {
  request.logtideScope.setExtra('userId', request.user?.id);
  request.logtideScope.setTag('tenant', request.headers['x-tenant-id']);
});
```

## Lifecycle Integration

The plugin hooks into Fastify's lifecycle automatically:

| Hook | Behavior |
|------|----------|
| `onRequest` | Creates scope, extracts traceparent, starts span |
| `onSend` | Optionally captures the response body |
| `onResponse` | Logs completion with duration and status |
| `onError` | Captures errors with request context |

## Error Handling

```typescript
fastify.setErrorHandler((error, request, reply) => {
  // Error already captured by LogTide plugin
  reply.status(error.statusCode || 500).send({
    error: error.message,
    traceId: request.logtideTraceId,
  });
});
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Fastify SDK Reference](/docs/sdks/fastify/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# Flask Application Logging Integration

> Add structured logging to Flask apps with the LogTide Python SDK — request middleware, blueprint support, error handlers, and Gunicorn deployment.

*Source: https://logtide.dev/integrations/flask/*

---

LogTide's Python SDK ships a built-in Flask middleware that auto-logs every request and response with timing, status code, and trace IDs. This guide covers the app factory pattern, blueprint-scoped logging, error handling, and production deployment with Gunicorn.

## Why use LogTide with Flask?

- **Built-in middleware**: `LogTideFlaskMiddleware` handles request/response logging automatically
- **App factory support**: Initialize inside `create_app()` for clean configuration
- **Blueprint logging**: Scoped logging per blueprint with independent service names
- **Error handlers**: Capture unhandled exceptions with full request context
- **Non-blocking**: Background batching keeps response latency unaffected
- **Gunicorn ready**: Works correctly with pre-fork workers

## Prerequisites

- Python 3.8+ (3.11+ recommended)
- Flask 2.0+
- LogTide instance with API key

## Installation

```bash
pip install logtide-sdk[flask]
```

## Quick Start

```python
# app.py
import os
from flask import Flask
from logtide_sdk import LogTideClient, ClientOptions
from logtide_sdk.middleware import LogTideFlaskMiddleware

app = Flask(__name__)

client = LogTideClient(
    ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
    )
)

LogTideFlaskMiddleware(
    app,
    client=client,
    service_name="flask-app",
    log_requests=True,
    log_responses=True,
    skip_paths=["/health", "/healthz", "/metrics"],
)

@app.route("/")
def index():
    return "Hello, World!"

if __name__ == "__main__":
    app.run()
```

### Environment Variables

```bash
export LOGTIDE_API_URL="http://your-logtide-instance:8080"
export LOGTIDE_API_KEY="lp_your_api_key_here"
flask run
```

## App Factory Pattern

Initialize LogTide inside `create_app()` for clean configuration:

```python
# app/__init__.py
import os
import atexit
from flask import Flask
from logtide_sdk import LogTideClient, ClientOptions
from logtide_sdk.middleware import LogTideFlaskMiddleware


def create_app(config_name=None):
    app = Flask(__name__)
    app.config.from_object(get_config(config_name))

    # Initialize LogTide client
    client = LogTideClient(
        ClientOptions(
            api_url=app.config["LOGTIDE_API_URL"],
            api_key=app.config["LOGTIDE_API_KEY"],
            global_metadata={
                "environment": app.config.get("ENV", "development"),
                "version": app.config.get("APP_VERSION", "unknown"),
            },
        )
    )
    app.logtide = client

    # Register middleware
    LogTideFlaskMiddleware(
        app,
        client=client,
        service_name="flask-app",
        skip_paths=["/health", "/healthz"],
    )

    # Register blueprints
    from app.routes.api import api_bp
    from app.routes.auth import auth_bp
    app.register_blueprint(api_bp, url_prefix="/api")
    app.register_blueprint(auth_bp, url_prefix="/auth")

    # Register error handlers
    register_error_handlers(app)

    # Graceful shutdown
    atexit.register(client.close)

    return app
```

### Configuration

```python
# config.py
import os

class Config:
    LOGTIDE_API_URL = os.environ.get("LOGTIDE_API_URL")
    LOGTIDE_API_KEY = os.environ.get("LOGTIDE_API_KEY")

class ProductionConfig(Config):
    ENV = "production"

class DevelopmentConfig(Config):
    ENV = "development"

def get_config(name=None):
    configs = {
        "production": ProductionConfig,
        "development": DevelopmentConfig,
    }
    return configs.get(name or os.environ.get("FLASK_ENV", "development"))
```

## Request Logging Output

Each request automatically generates a structured log:

```json
{
  "level": "info",
  "message": "GET /api/users/42 200",
  "service": "flask-app",
  "metadata": {
    "method": "GET",
    "path": "/api/users/42",
    "status_code": 200,
    "duration_ms": 18.3,
    "trace_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  }
}
```

## Error Handling

Register centralized error handlers that capture exceptions with full context:

```python
# app/errors.py
from flask import Flask, request, g, jsonify
from logtide_sdk import serialize_exception

def register_error_handlers(app: Flask):
    @app.errorhandler(404)
    def not_found(error):
        app.logtide.warn(
            "flask-app",
            "Resource not found",
            {"path": request.path, "method": request.method},
        )
        return jsonify({"error": "Not found"}), 404

    @app.errorhandler(500)
    def internal_error(error):
        app.logtide.error(
            "flask-app",
            "Internal server error",
            serialize_exception(error),
        )
        return jsonify({"error": "Internal server error"}), 500

    @app.errorhandler(Exception)
    def unhandled_exception(error):
        app.logtide.error(
            "flask-app",
            f"Unhandled exception: {type(error).__name__}",
            error,
        )
        return jsonify({"error": "Internal server error"}), 500
```

## Blueprint Logging

Log directly in blueprint handlers with scoped service names:

```python
# app/routes/api.py
from flask import Blueprint, request, g, jsonify, current_app

api_bp = Blueprint("api", __name__)

@api_bp.route("/users/<int:user_id>")
def get_user(user_id):
    client = current_app.logtide
    client.info("api", "Fetching user profile", {"user_id": user_id})

    user = User.query.get_or_404(user_id)
    return jsonify(user.to_dict())

@api_bp.route("/users", methods=["POST"])
def create_user():
    client = current_app.logtide
    data = request.get_json()

    client.info("api", "Creating user", {"email": data.get("email")})

    try:
        user = User(email=data["email"], name=data["name"])
        db.session.add(user)
        db.session.commit()
        client.info("api", "User created", {"user_id": user.id})
        return jsonify(user.to_dict()), 201
    except IntegrityError:
        db.session.rollback()
        client.warn("api", "Duplicate email", {"email": data.get("email")})
        return jsonify({"error": "Email already exists"}), 409
```

## stdlib `logging` Integration

If you prefer to use Python's stdlib `logging`, use `LogTideHandler`:

```python
import logging
from logtide_sdk import LogTideHandler

handler = LogTideHandler(client=app.logtide, service="flask-app")
handler.setLevel(logging.WARNING)

logging.getLogger("app").addHandler(handler)
```

## Trace ID Context

Use context managers to correlate all logs within a request:

```python
@api_bp.route("/orders/<int:order_id>")
def process_order(order_id):
    client = current_app.logtide
    with client.with_new_trace_id():
        client.info("orders", "Processing order", {"order_id": order_id})
        # ... all logs inside this block share the same trace_id
```

## SQLAlchemy Query Logging

Log slow queries using SQLAlchemy event listeners:

```python
# app/extensions.py
import time
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import event

db = SQLAlchemy()

def init_query_logging(app):
    engine = db.engine
    slow_threshold_ms = app.config.get("SLOW_QUERY_THRESHOLD_MS", 100)

    @event.listens_for(engine, "before_cursor_execute")
    def before_execute(conn, cursor, statement, parameters, context, executemany):
        conn.info.setdefault("query_start_time", []).append(time.monotonic())

    @event.listens_for(engine, "after_cursor_execute")
    def after_execute(conn, cursor, statement, parameters, context, executemany):
        total_ms = (time.monotonic() - conn.info["query_start_time"].pop()) * 1000
        if total_ms >= slow_threshold_ms:
            app.logtide.warn(
                "database",
                "Slow SQL query",
                {"query": statement[:500], "duration_ms": round(total_ms, 2)},
            )
```

## Gunicorn Deployment

```python
# gunicorn.conf.py
import multiprocessing
import os

workers = int(os.environ.get("GUNICORN_WORKERS", multiprocessing.cpu_count() * 2 + 1))
worker_class = "gthread"
threads = int(os.environ.get("GUNICORN_THREADS", 4))
timeout = 120
bind = "0.0.0.0:8000"

def worker_exit(server, worker):
    """Flush pending logs when a worker exits."""
    from app import create_app
    app = create_app()
    if hasattr(app, "logtide"):
        app.logtide.close()
```

```bash
gunicorn "app:create_app()" --config gunicorn.conf.py --preload
```

## Docker Deployment

```dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser
USER appuser

EXPOSE 8000
CMD ["gunicorn", "app:create_app()", "--config", "gunicorn.conf.py"]
```

```yaml
# docker-compose.yml
services:
  web:
    build: .
    ports:
      - "8000:8000"
    environment:
      - FLASK_ENV=production
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - DATABASE_URL=postgresql://user:pass@db:5432/myapp
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 10s
      timeout: 5s
      retries: 5
```

## Performance

| Metric | Value |
|--------|-------|
| Middleware overhead | <1ms per request |
| Memory overhead | ~8MB |
| Network calls | 1 per batch (100 logs default) |
| Gunicorn worker compatibility | gthread, gevent, sync |

## Troubleshooting

### Logs not appearing

1. Enable debug mode to see what is happening:
   ```python
   ClientOptions(..., debug=True)
   ```
2. Ensure `client.close()` is called on shutdown (see Gunicorn section).
3. Check the circuit breaker state: `print(client.get_circuit_breaker_state())`

### Duplicate request logs

Ensure you have not registered the middleware twice (e.g., both manually and via an extension). Use `LogTideFlaskMiddleware` only once.

### SQLAlchemy events not firing

Call `init_query_logging(app)` after `db.init_app(app)` and inside an app context:

```python
with app.app_context():
    init_query_logging(app)
```

### Gunicorn workers losing logs

With the `--preload` flag, the client is shared across forked workers. Ensure each worker calls `client.close()` on exit via the `worker_exit` hook in `gunicorn.conf.py`.

## Next Steps

- [Python SDK Reference](/integrations/python/) - Full SDK documentation
- [Docker Integration](/integrations/docker/) - Container deployment patterns
- [PostgreSQL Integration](/integrations/postgresql/) - Database logging correlation
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging
- [Security Monitoring](/use-cases/security-monitoring/) - Threat detection and alerting


---

# Go Integration

> Send structured logs from Go applications to LogTide using the official Go SDK.

*Source: https://logtide.dev/integrations/go/*

---

Send structured logs from any Go application to LogTide using the official SDK. It handles batching, retries, and circuit breaking automatically.

## Installation

```bash
go get github.com/logtide-dev/logtide-sdk-go
```

Requires Go 1.23+.

## Quick Start

```go
package main

import (
    "context"
    logtide "github.com/logtide-dev/logtide-sdk-go"
)

func main() {
    flush := logtide.Init(logtide.ClientOptions{
        DSN:         "https://lp_your_api_key@api.logtide.dev",
        Service:     "my-service",
        Environment: "production",
    })
    defer flush()

    logtide.Info(context.Background(), "Application started", map[string]any{
        "version": "1.2.3",
    })
}
```

## Configuration

```go
opts := logtide.NewClientOptions()
opts.DSN         = "https://lp_your_api_key@api.logtide.dev"
opts.Service     = "my-service"
opts.Release     = "v1.2.3"
opts.Environment = "production"
opts.Tags        = map[string]string{"region": "eu-west-1"}

client, err := logtide.NewClient(opts)
if err != nil {
    log.Fatal(err)
}
defer client.Close()
```

## HTTP Middleware

The built-in `net/http` middleware handles scope isolation, HTTP tagging, and breadcrumbs per request:

```go
import lnethttp "github.com/logtide-dev/logtide-sdk-go/integrations/nethttp"

mux := http.NewServeMux()
http.ListenAndServe(":8080", lnethttp.Middleware(mux))
```

## OpenTelemetry

Trace and span IDs are extracted automatically from any active OTel span in the context:

```go
ctx, span := tracer.Start(ctx, "process-order")
defer span.End()

// trace_id and span_id included automatically
client.Info(ctx, "Order processed", map[string]any{"order_id": id})
```

## Graceful Shutdown

```go
// Global Init pattern
flush := logtide.Init(opts)
defer flush() // flushes all buffered entries

// Explicit client pattern
client, _ := logtide.NewClient(opts)
defer client.Close()
```

## Next Steps

- [Go SDK reference](/docs/sdks/go/) — full API documentation
- [Docker Integration](/integrations/docker/) — container deployment
- [Kubernetes Integration](/integrations/kubernetes/) — cluster logging


---

# Google Cloud Run Logging Integration

> Send Google Cloud Run logs to LogTide with the JavaScript SDK or a Fluent Bit sidecar — structured queries and alerting beyond Cloud Logging's free tier.

*Source: https://logtide.dev/integrations/gcp-cloud-run/*

---

Cloud Run is the easiest way to run containers on GCP, and its logging starts equally easy: anything on stdout lands in Cloud Logging automatically. The catch arrives with scale — after the free 50 GiB/month, you pay $0.50/GiB ingested, retention past 30 days is metered, and structured queries mean BigQuery-backed Log Analytics with its own costs.

This guide ships Cloud Run logs to LogTide for field-based search, real-time alerting, and retention measured in months — with the JavaScript SDK or a zero-code-change Fluent Bit sidecar.

## Why use LogTide with Cloud Run?

- **Cost beyond the free tier**: $0.50/GiB adds up for chatty services; LogTide is flat-cost infrastructure ([full comparison](/vs/gcp-cloud-logging/))
- **Structured metadata queries**: filter by `tenant`, `endpoint` or `revision` directly instead of writing Log Analytics SQL
- **Real-time alerting**: SSE-backed alerts on error patterns without log-based metric + alerting policy plumbing
- **Unified view**: Cloud Run, GKE, on-prem and other clouds in one searchable timeline
- **Retention control**: keep everything searchable without per-GiB retention fees

## Prerequisites

- A Cloud Run service (Node.js for the SDK approach; any language for the sidecar)
- LogTide instance reachable from Cloud Run (public HTTPS or internal via VPC connector)
- LogTide DSN or API key

## Approach 1: LogTide SDK (recommended for Node.js)

### Installation

```bash
npm install @logtide/core
```

### Service setup with graceful shutdown

Cloud Run instances are long-lived between requests but can be stopped anytime (scale-down sends `SIGTERM` with a 10-second grace window). Initialize once at startup, flush on shutdown:

```typescript
// server.ts
import express from 'express';
import { hub } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: process.env.K_SERVICE || 'cloud-run-app',   // injected by Cloud Run
  environment: process.env.NODE_ENV || 'production',
  release: process.env.K_REVISION,                      // revision name, e.g. app-00042-xyz
  batchSize: 50,
  flushInterval: 2000,
});

const app = express();
app.use(express.json());

app.post('/orders', async (req, res) => {
  const startTime = Date.now();

  hub.captureLog('info', 'Order received', {
    orderId: req.body.orderId,
    revision: process.env.K_REVISION,
  });

  try {
    // business logic
    hub.captureLog('info', 'Order processed', {
      orderId: req.body.orderId,
      duration: Date.now() - startTime,
    });
    res.json({ ok: true });
  } catch (error) {
    hub.captureError(error as Error, {
      extra: { orderId: req.body.orderId },
      tags: { endpoint: '/orders' },
    });
    res.status(500).json({ error: 'Internal server error' });
  }
});

const server = app.listen(Number(process.env.PORT) || 8080);

// Cloud Run sends SIGTERM before stopping the instance (10s grace period)
process.on('SIGTERM', async () => {
  await hub.flush();
  server.close(() => process.exit(0));
});
```

The `K_SERVICE` and `K_REVISION` environment variables are injected by Cloud Run automatically — using them as `service` and `release` gives you per-revision filtering in LogTide for free, which makes *"did the new revision introduce these errors?"* a one-click query.

### Deploy with the DSN

```bash
gcloud run deploy my-service \
  --image gcr.io/my-project/my-service \
  --set-secrets LOGTIDE_DSN=logtide-dsn:latest \
  --region europe-west1
```

Store the DSN in Secret Manager rather than a plain environment variable.

## Approach 2: Fluent Bit sidecar (any language, no code changes)

Cloud Run supports multi-container services: run Fluent Bit as a sidecar that tails your app's stdout (via a shared volume) or receives forward-protocol logs, and ships them to LogTide.

The simplest robust pattern is logging to a file on an in-memory shared volume:

```yaml
# service.yaml (Cloud Run multicontainer)
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-service
spec:
  template:
    spec:
      containers:
        - name: app
          image: gcr.io/my-project/my-service
          volumeMounts:
            - name: logs
              mountPath: /var/log/app
        - name: fluentbit
          image: cr.fluentbit.io/fluent/fluent-bit:3.0
          volumeMounts:
            - name: logs
              mountPath: /var/log/app
      volumes:
        - name: logs
          emptyDir:
            medium: Memory
            sizeLimit: 64Mi
```

Fluent Bit configuration:

```ini
[INPUT]
    Name tail
    Path /var/log/app/*.log
    Parser json
    Tag cloudrun.app

[OUTPUT]
    Name http
    Match *
    Host logtide.example.com
    Port 443
    URI /api/v1/ingest/single
    Format json_lines
    tls On
    Header X-API-Key ${LOGTIDE_API_KEY}
    Header Content-Type application/json
```

Your app writes JSON lines to `/var/log/app/app.log` (or you keep stdout for Cloud Logging *and* a file for LogTide). No SDK, works for Go, Python, Java, Rust — anything.

## Which approach should you pick?

| | SDK | Fluent Bit sidecar |
|---|---|---|
| Code changes | Yes (logging calls) | None |
| Structured metadata | Native, arbitrary fields | Whatever your JSON logs carry |
| Languages | Node.js (others via [SDKs](/docs/sdks/)) | Any |
| Extra container cost | None | Small sidecar (~50 MB RAM) |
| Best for | New/Node services, rich telemetry | Existing images, polyglot fleets |

## Keeping Cloud Logging in the loop

Cloud Run always writes stdout/stderr to Cloud Logging — that's not disableable, and the first 50 GiB/month is free anyway. The cost-effective split:

1. Log **concise** lines to stdout (request summary, errors) → Cloud Logging, within the free tier, short retention
2. Send **verbose** structured logs to LogTide (full context, business events) → searchable for months
3. Set Cloud Logging retention to the minimum and drop `_Default` sink noise with [exclusion filters](https://cloud.google.com/logging/docs/routing/overview)

You keep GCP-native quick debugging, and the volume that would blow past the free tier lives in LogTide instead.

## Troubleshooting

- **Logs lost on scale-down** — flush in the `SIGTERM` handler; Cloud Run gives you up to 10 seconds.
- **Sidecar can't reach LogTide** — if LogTide is internal, attach a Serverless VPC Access connector to the service and use the internal hostname.
- **Duplicate entries in both platforms** — expected if you log the same lines to stdout and the SDK; split concise vs verbose as above.

---

**Next steps:**

- [LogTide vs GCP Cloud Logging](/vs/gcp-cloud-logging/) — full cost and feature comparison
- [Docker integration](/integrations/docker/) — the same sidecar pattern outside Cloud Run
- [Cloud logging pricing breakdown](/cloud-logging-pricing/) — CloudWatch, GCP and Azure compared
- [Microservices observability](/use-cases/microservices-observability/) — correlating services in one timeline


---

# Hono Framework Logging Integration

> Add structured logging to Hono applications across Node.js, Bun, Deno, and Cloudflare Workers with trace propagation.

*Source: https://logtide.dev/integrations/hono/*

---

LogTide's Hono SDK provides lightweight middleware for automatic request logging and distributed tracing across all Hono-supported runtimes: Node.js, Bun, Deno, and Cloudflare Workers.

## Why use LogTide with Hono?

- **Multi-runtime**: Same code on Node.js, Bun, Deno, and Workers
- **Lightweight**: Minimal overhead, fetch-based transport
- **Context-based**: Uses Hono's `c.get()`/`c.set()` for scoping
- **Trace propagation**: W3C traceparent headers across services
- **Automatic**: Request logging without manual instrumentation

## Prerequisites

- Hono 4.x
- Node.js 18+, Bun, Deno, or Cloudflare Workers
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/hono
# or
bun add @logtide/hono
```

## Quick Start

```typescript
import { Hono } from 'hono';
import { hub } from '@logtide/core';
import { logtide } from '@logtide/hono';

const app = new Hono();
app.use('*', logtide({
  dsn: process.env.LOGTIDE_DSN,
  service: 'hono-api',
  environment: process.env.NODE_ENV,
}));

app.get('/users/:id', (c) => {
  c.get('logtideScope').setTag('userId', c.req.param('id'));

  hub.captureLog('info', 'Fetching user', {
    userId: c.req.param('id'),
  });

  return c.json({ id: c.req.param('id') });
});

export default app;
```

## Middleware Options

```typescript
app.use('*', logtide({
  dsn: process.env.LOGTIDE_DSN,
  service: 'hono-api',
  environment: process.env.NODE_ENV,
  release: '1.0.0',
  tracesSampleRate: 1.0,
}));
```

## Context-Based Scoping

```typescript
app.get('/orders', (c) => {
  const scope = c.get('logtideScope');
  scope.setExtra('userId', 'user-123');
  scope.setTag('module', 'orders');

  const traceId = c.get('logtideTraceId');

  return c.json({ orders: [], traceId });
});
```

## Runtime Examples

### Node.js

```typescript
import { serve } from '@hono/node-server';
serve(app, { port: 3000 });
```

### Bun

```typescript
export default { port: 3000, fetch: app.fetch };
```

### Cloudflare Workers

```typescript
export default app;
```

### Deno

```typescript
Deno.serve(app.fetch);
```

## Error Handling

```typescript
app.onError((err, c) => {
  // Errors are captured automatically by the middleware
  return c.json({
    error: err.message,
    traceId: c.get('logtideTraceId'),
  }, 500);
});
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Hono SDK Reference](/docs/sdks/hono/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# Host System Monitoring Integration

> Collect CPU, memory, disk I/O, and network metrics from your host machine using OTLP-compatible tools and visualize them in the LogTide Metrics Explorer.

*Source: https://logtide.dev/integrations/host-monitoring/*

---

Monitor your host machine's health by collecting system metrics and viewing them in the LogTide Metrics Explorer. LogTide accepts metrics via the standard **OpenTelemetry (OTLP)** endpoint, so you can use Fluent Bit (included in Docker Compose), or any OTLP-compatible tool you already have.

## Standard OTLP Endpoint

LogTide exposes a standard OpenTelemetry metrics endpoint:

```
POST /v1/otlp/metrics
Content-Type: application/json
X-API-Key: <your-api-key>
```

This means you are **not limited to Fluent Bit**. Any tool that speaks OTLP can send metrics to LogTide:

| Tool | Description |
|------|-------------|
| **Fluent Bit** | Included in Docker Compose (`--profile metrics`), zero config |
| **OpenTelemetry Collector** | The standard OTel collector with `otlphttp` exporter |
| **Grafana Alloy / Agent** | Grafana's collector, supports OTLP export |
| **Vector** | Datadog's collector, supports OTLP export |
| **Prometheus + OTLP remote write** | Export Prometheus metrics to LogTide |
| **Application OTel SDKs** | Node.js, Python, Go, Java — send custom metrics directly |

## Option A: Fluent Bit (included, zero config)

The Docker Compose setup includes a pre-configured Fluent Bit metrics container that collects CPU, memory, disk, and network metrics automatically.

### 1. Download configuration files

```bash
curl -O https://raw.githubusercontent.com/logtide-dev/logtide/main/docker/fluent-bit-metrics.conf
curl -O https://raw.githubusercontent.com/logtide-dev/logtide/main/docker/format_metrics.lua
```

Place these in the same directory as your `docker-compose.yml`.

### 2. Set your API key

```bash
# Add to your .env file (if not already set)
echo "FLUENT_BIT_API_KEY=lp_your_api_key_here" >> .env
```

### 3. Start with metrics profile

```bash
# Metrics only
docker compose --profile metrics up -d

# Or combine with Docker log collection
docker compose --profile logging --profile metrics up -d
```

## Option B: OpenTelemetry Collector

If you already run an OpenTelemetry Collector, add LogTide as an OTLP HTTP exporter:

```yaml
# otel-collector-config.yaml
receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      disk:
      network:

exporters:
  otlphttp/logtide:
    endpoint: http://<logtide-host>:8080
    headers:
      X-API-Key: "lp_your_api_key_here"

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      exporters: [otlphttp/logtide]
```

## Option C: Grafana Alloy

```hcl
otelcol.receiver.hostmetrics "default" {
  collection_interval = "30s"
  scrapers {
    cpu {}
    memory {}
    disk {}
    network {}
  }
}

otelcol.exporter.otlphttp "logtide" {
  client {
    endpoint = "http://<logtide-host>:8080"
    headers  = { "X-API-Key" = "lp_your_api_key_here" }
  }
}
```

## Option D: Any OTLP-compatible tool

Point any OTLP HTTP exporter at:

```
http://<logtide-host>:8080/v1/otlp/metrics
```

With header `X-API-Key: <your-api-key>`. Both JSON and Protobuf content types are supported.

## What gets collected (Fluent Bit)

The included Fluent Bit configuration sends OTLP gauge metrics every 30-60 seconds:

| Metric Name | Interval | Description |
|------------|----------|-------------|
| `system.cpu.utilization` | 30s | Total CPU usage % |
| `system.cpu.user` | 30s | User CPU % |
| `system.cpu.system` | 30s | System CPU % |
| `system.memory.utilization` | 30s | Memory usage % |
| `system.memory.usage` | 30s | Memory used (MB) |
| `system.memory.total` | 30s | Total memory (MB) |
| `system.disk.read` | 60s | Disk read throughput (KB) |
| `system.disk.write` | 60s | Disk write throughput (KB) |
| `system.network.rx` | 30s | Network received per interval (KB) |
| `system.network.tx` | 30s | Network transmitted per interval (KB) |

All metrics appear under service `host-system` in the Metrics Explorer.

## Auto-detection (Fluent Bit)

The Lua script automatically detects the primary disk device and network interface by reading /proc:

- **Disk**: Reads /proc/diskstats, selects the device with the most I/O activity. Skips partitions (sda1), loop devices, and device-mapper entries.
- **Network**: Reads /proc/net/dev, selects the interface with the most traffic. Skips lo (loopback).

Detection runs once and caches the result for the lifetime of the container. If your primary device changes, restart the metrics container:

```bash
docker compose restart fluent-bit-metrics
```

## Viewing metrics

Metrics appear in the **Metrics Explorer** in the LogTide dashboard (`/dashboard/metrics`). You can:

- Browse all metric names
- Filter by service (`host-system` for Fluent Bit, or your custom service name)
- View time-series charts for any metric
- Compare metrics across services

## Configuration

### Adjusting collection intervals (Fluent Bit)

Edit `fluent-bit-metrics.conf` to change how often metrics are collected:

```ini
[INPUT]
    Name         cpu
    Tag          metrics.cpu
    Interval_Sec 10   # Every 10 seconds instead of 30
```

### Flush interval

The Flush setting in the [SERVICE] section controls how often metrics are sent to LogTide. Default is 30 seconds.

## Troubleshooting

### Metrics not appearing

1. Check the container is running:
```bash
docker compose ps fluent-bit-metrics
```

2. Check logs for errors:
```bash
docker compose logs fluent-bit-metrics
```

3. Verify API key is set:
```bash
grep FLUENT_BIT_API_KEY .env
```

4. Look for HTTP status codes in the logs — `200` means metrics are being accepted.

### Platform limitations

System metrics via Fluent Bit require Linux /proc filesystem. They won't work on macOS or Windows Docker Desktop (the container sees the VM's /proc, not the host's). Use the OpenTelemetry Collector with hostmetrics receiver as an alternative on those platforms.

## Next Steps

- [Docker Log Collection](/integrations/docker/) - Collect container logs alongside metrics
- [OpenTelemetry](/docs/opentelemetry/) - Send traces and logs via OTLP
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Set up alerts for metric thresholds
- [systemd Journal](/integrations/systemd/) - Collect system service logs


---

# Kubernetes Centralized Logging Integration

> Collect and ship logs from Kubernetes clusters to LogTide using Fluent Bit DaemonSet or sidecar patterns.

*Source: https://logtide.dev/integrations/kubernetes/*

---

Kubernetes generates logs from multiple sources: application containers, system components, and the control plane. This guide shows you how to collect all these logs and ship them to LogTide using Fluent Bit.

## Why centralized logging for Kubernetes?

- **Ephemeral containers**: Pods die, logs disappear. Centralized logging persists everything
- **Multi-node visibility**: See logs from all nodes in one place
- **Namespace isolation**: Filter and route logs by namespace, labels, or annotations
- **Correlation**: Link application logs with Kubernetes events and metrics
- **Compliance**: Audit trails and retention policies for regulated environments

## Architecture Options

### Option 1: DaemonSet (Recommended)

Deploy Fluent Bit as a DaemonSet to collect logs from all nodes:

**Kubernetes Cluster with DaemonSet:**
- Each node runs a Fluent Bit pod
- Fluent Bit reads from `/var/log/containers/`
- All logs are shipped to LogTide centrally
- Pods can come and go, logs persist

### Option 2: Sidecar

Deploy Fluent Bit as a sidecar container for specific applications:

**Sidecar Pattern:**
- App container writes logs to a shared volume
- Fluent Bit sidecar reads from the same volume
- Useful for custom log parsing per application
- Higher resource usage but more flexibility

## Prerequisites

- Kubernetes 1.21+ cluster
- kubectl configured with cluster access
- Helm 3.x (optional, for Helm deployment)
- LogTide instance with API key

## Quick Start: DaemonSet Deployment

### 1. Create Namespace and Secret

```bash
kubectl create namespace logtide

kubectl create secret generic logtide-credentials \
  --namespace logtide \
  --from-literal=api-key=YOUR_LOGTIDE_API_KEY
```

### 2. Deploy Fluent Bit ConfigMap

```yaml
# fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logtide
  labels:
    app.kubernetes.io/name: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              On
        Annotations         Off

    # Extract service name from nested kubernetes metadata
    [FILTER]
        Name          nest
        Match         kube.*
        Operation     lift
        Nested_under  kubernetes
        Add_prefix    k8s_

    [FILTER]
        Name          modify
        Match         kube.*
        Rename        k8s_container_name service

    [FILTER]
        Name    modify
        Match   kube.*
        Add     cluster ${CLUSTER_NAME}
        Add     level info
        Rename  log message

    [OUTPUT]
        Name              http
        Match             kube.*
        Host              ${LOGTIDE_HOST}
        Port              443
        URI               /api/v1/ingest/single
        Format            json_lines
        Header            X-API-Key ${LOGTIDE_API_KEY}
        Header            Content-Type application/x-ndjson
        Json_date_key     time
        Json_date_format  iso8601
        tls               On
        tls.verify        On
        Retry_Limit       5

  parsers.conf: |
    [PARSER]
        Name        cri
        Format      regex
        Regex       ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

    [PARSER]
        Name        json
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
```

### 3. Deploy Fluent Bit DaemonSet

```yaml
# fluent-bit-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logtide
  labels:
    app.kubernetes.io/name: fluent-bit
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:2.2
          env:
            - name: LOGTIDE_HOST
              value: "api.logtide.dev"
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: api-key
            - name: CLUSTER_NAME
              value: "production"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          ports:
            - containerPort: 2020
              name: metrics
          resources:
            limits:
              cpu: 200m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logtide
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logtide
```

### 4. Apply Configuration

```bash
kubectl apply -f fluent-bit-configmap.yaml
kubectl apply -f fluent-bit-daemonset.yaml
```

### 5. Verify Deployment

```bash
# Check DaemonSet status
kubectl get daemonset fluent-bit -n logtide

# Check pod logs
kubectl logs -n logtide -l app.kubernetes.io/name=fluent-bit --tail=100

# Check metrics endpoint
kubectl port-forward -n logtide ds/fluent-bit 2020:2020
curl http://localhost:2020/api/v1/metrics
```

## Helm Deployment

For production deployments, use the official Fluent Bit Helm chart:

```bash
helm repo add fluent https://fluent.github.io/helm-charts

helm install fluent-bit fluent/fluent-bit \
  --namespace logtide \
  --create-namespace \
  --set env[0].name=LOGTIDE_API_KEY \
  --set env[0].valueFrom.secretKeyRef.name=logtide-credentials \
  --set env[0].valueFrom.secretKeyRef.key=api-key \
  --values values.yaml
```

### values.yaml

```yaml
# values.yaml
config:
  outputs: |
    [OUTPUT]
        Name              http
        Match             *
        Host              api.logtide.dev
        Port              443
        URI               /api/v1/ingest/single
        Format            json_lines
        Header            X-API-Key ${LOGTIDE_API_KEY}
        Header            Content-Type application/x-ndjson
        Json_date_key     time
        Json_date_format  iso8601
        tls               On

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        Labels              On

    # Extract service name from nested kubernetes metadata
    [FILTER]
        Name          nest
        Match         kube.*
        Operation     lift
        Nested_under  kubernetes
        Add_prefix    k8s_

    [FILTER]
        Name          modify
        Match         kube.*
        Rename        k8s_container_name service

    [FILTER]
        Name    modify
        Match   kube.*
        Add     level info
        Rename  log message

resources:
  limits:
    cpu: 200m
    memory: 256Mi
  requests:
    cpu: 100m
    memory: 128Mi

tolerations:
  - key: node-role.kubernetes.io/master
    effect: NoSchedule
  - key: node-role.kubernetes.io/control-plane
    effect: NoSchedule
```

## Namespace-Based Filtering

### Exclude Namespaces

Exclude system namespaces from logging:

```ini
[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Exclude_Path      /var/log/containers/*_kube-system_*.log,/var/log/containers/*_kube-public_*.log

[FILTER]
    Name    grep
    Match   kube.*
    Exclude $kubernetes['namespace_name'] ^(kube-system|kube-public)$
```

### Route by Namespace

Send different namespaces to different LogTide projects:

```ini
[OUTPUT]
    Name              http
    Match             kube.var.log.containers.*_production_*
    Host              api.logtide.dev
    Port              443
    URI               /api/v1/ingest/single
    Format            json_lines
    Header            X-API-Key ${LOGTIDE_PRODUCTION_KEY}
    Header            Content-Type application/x-ndjson
    Json_date_key     time
    Json_date_format  iso8601
    tls               On

[OUTPUT]
    Name              http
    Match             kube.var.log.containers.*_staging_*
    Host              api.logtide.dev
    Port              443
    URI               /api/v1/ingest/single
    Format            json_lines
    Header            X-API-Key ${LOGTIDE_STAGING_KEY}
    Header            Content-Type application/x-ndjson
    Json_date_key     time
    Json_date_format  iso8601
    tls               On
```

## Sidecar Pattern

For applications that need custom log parsing or isolation:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest
          volumeMounts:
            - name: logs
              mountPath: /var/log/app

        - name: fluent-bit
          image: fluent/fluent-bit:2.2
          env:
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: api-key
          volumeMounts:
            - name: logs
              mountPath: /var/log/app
              readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
          resources:
            limits:
              cpu: 50m
              memory: 64Mi

      volumes:
        - name: logs
          emptyDir: {}
        - name: fluent-bit-config
          configMap:
            name: my-app-fluent-bit-config
```

## Log Enrichment

### Add Kubernetes Metadata

The Kubernetes filter automatically adds:

```json
{
  "kubernetes": {
    "pod_name": "my-app-7d8f9b6c5-x2j4k",
    "namespace_name": "production",
    "container_name": "app",
    "container_id": "abc123...",
    "labels": {
      "app": "my-app",
      "version": "1.2.3"
    }
  }
}
```

### Add Custom Labels

Use annotations to add custom metadata:

```yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    logtide.dev/service: "payment-service"
    logtide.dev/team: "payments"
```

Then parse in Fluent Bit. Since the Kubernetes metadata is nested, you need to lift the annotations first:

```ini
# First, lift the kubernetes object (if not already done)
[FILTER]
    Name          nest
    Match         kube.*
    Operation     lift
    Nested_under  kubernetes
    Add_prefix    k8s_

# Then lift the annotations
[FILTER]
    Name          nest
    Match         kube.*
    Operation     lift
    Nested_under  k8s_annotations
    Add_prefix    annotation_

# Now rename the annotation fields
[FILTER]
    Name    modify
    Match   kube.*
    Rename  annotation_logtide.dev/service service
    Rename  annotation_logtide.dev/team team
```

Alternatively, use a Lua script for more control:

```ini
[FILTER]
    Name          lua
    Match         kube.*
    Script        /fluent-bit/scripts/extract_metadata.lua
    Call          extract_metadata
```

```lua
-- extract_metadata.lua
function extract_metadata(tag, timestamp, record)
    if record["kubernetes"] then
        local k8s = record["kubernetes"]

        -- Extract container name as service
        if k8s["container_name"] then
            record["service"] = k8s["container_name"]
        end

        -- Extract custom annotations
        if k8s["annotations"] then
            if k8s["annotations"]["logtide.dev/service"] then
                record["service"] = k8s["annotations"]["logtide.dev/service"]
            end
            if k8s["annotations"]["logtide.dev/team"] then
                record["team"] = k8s["annotations"]["logtide.dev/team"]
            end
        end
    end
    return 1, timestamp, record
end
```

## Multi-Cluster Setup

For multi-cluster environments, add cluster identification:

```yaml
env:
  - name: CLUSTER_NAME
    value: "us-east-1-prod"
  - name: CLUSTER_REGION
    value: "us-east-1"
```

```ini
[FILTER]
    Name    modify
    Match   *
    Add     cluster ${CLUSTER_NAME}
    Add     region ${CLUSTER_REGION}
```

## Monitoring Fluent Bit

### Prometheus Metrics

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: fluent-bit
  namespace: logtide
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit
  endpoints:
    - port: metrics
      interval: 30s
```

### Key Metrics to Watch

| Metric | Description | Alert Threshold |
|--------|-------------|-----------------|
| `fluentbit_input_records_total` | Records ingested | Sudden drops |
| `fluentbit_output_retries_total` | Retry attempts | >10/min |
| `fluentbit_output_errors_total` | Output errors | >0 |
| `fluentbit_filter_records_total` | Records filtered | Compare with input |

## Troubleshooting

### All logs show "unknown" service

If all your logs appear with service name "unknown" in LogTide, this means the `service` field isn't being extracted properly from the Kubernetes metadata.

The Kubernetes filter nests container information under `kubernetes.container_name`, but LogTide expects `service` at the top level. You need to lift this nested field using the `nest` filter:

```ini
# Add these filters AFTER the kubernetes filter
[FILTER]
    Name          nest
    Match         kube.*
    Operation     lift
    Nested_under  kubernetes
    Add_prefix    k8s_

[FILTER]
    Name          modify
    Match         kube.*
    Rename        k8s_container_name service
```

This lifts all fields from the `kubernetes` object to the top level (with `k8s_` prefix) and then renames `k8s_container_name` to `service`.

### HTTP 400 "body must be object" error

If you see this error in Fluent Bit logs:
```
[error] [output:http:http.0] HTTP status=400
```

This means the output configuration is incorrect. Make sure you're using:
- `URI /api/v1/ingest/single` (not `/api/v1/ingest`)
- `Format json_lines` (not `json`)
- `Header Content-Type application/x-ndjson`

### No logs appearing

1. Check Fluent Bit pods are running:
   ```bash
   kubectl get pods -n logtide -l app.kubernetes.io/name=fluent-bit
   ```

2. Check Fluent Bit logs:
   ```bash
   kubectl logs -n logtide -l app.kubernetes.io/name=fluent-bit
   ```

3. Verify log files exist on nodes:
   ```bash
   kubectl debug node/NODE_NAME -it --image=busybox -- ls /var/log/containers/
   ```

### High memory usage

Reduce memory buffer limit:

```ini
[INPUT]
    Name              tail
    Mem_Buf_Limit     5MB  # Reduce from default
```

### Logs being dropped

Check for backpressure and increase retry limits:

```ini
[OUTPUT]
    Name              http
    Retry_Limit       10
    storage.total_limit_size 5G
```

## Resource Recommendations

| Cluster Size | CPU Request | Memory Request | Memory Limit |
|--------------|-------------|----------------|--------------|
| Small (<20 nodes) | 50m | 64Mi | 128Mi |
| Medium (20-100) | 100m | 128Mi | 256Mi |
| Large (100+) | 200m | 256Mi | 512Mi |

## Next Steps

- [Docker Integration](/integrations/docker/) - Container logging basics
- [nginx Integration](/integrations/nginx/) - Web server logs in Kubernetes
- [Node.js SDK](/integrations/nodejs/) - Application-level logging


---

# Laravel and PHP Logging Integration

> Send structured logs from Laravel applications to LogTide with zero-config auto-discovery, middleware, log channel, and breadcrumb integrations.

*Source: https://logtide.dev/integrations/laravel/*

---

The LogTide Laravel package provides a drop-in integration for Laravel 10, 11, and 12 with zero-configuration needed. It auto-registers middleware, a log channel, a Facade, and breadcrumb integrations for database queries, cache events, and queue jobs.

## Why use LogTide with Laravel?

- **Zero-config**: Auto-discovery registers the service provider and facade automatically
- **Automatic middleware**: Request tracing with distributed tracing (W3C traceparent) out of the box
- **Log channel**: Use Laravel's standard `Log::info()` to send logs to LogTide
- **Breadcrumbs**: Automatic breadcrumbs for DB queries, cache events, and queue jobs
- **Scoped context**: Per-request isolation with the Hub/Scope architecture
- **GDPR compliance**: Self-hosted option for EU data residency

## Prerequisites

- PHP 8.1+
- Laravel 10.x, 11.x, or 12.x
- Composer
- LogTide instance with DSN

## Installation

```bash
composer require logtide/logtide-laravel
```

The service provider and facade are registered automatically via Laravel's package auto-discovery.

Publish the configuration:

```bash
php artisan vendor:publish --tag=logtide-config
```

## Configuration

### Environment Variables

Add to your `.env` file:

```env
LOGTIDE_DSN=https://lp_your_key@api.logtide.dev
```

That's it! The SDK is now active with sensible defaults.

Optional environment variables:

```env
LOGTIDE_SERVICE=my-laravel-app        # default: APP_NAME
LOGTIDE_ENVIRONMENT=production         # default: APP_ENV
LOGTIDE_RELEASE=1.2.3
LOGTIDE_TRACES_SAMPLE_RATE=1.0
LOGTIDE_DEBUG=false
```

### config/logtide.php

```php
<?php

return [
    'dsn' => env('LOGTIDE_DSN'),

    'service' => env('LOGTIDE_SERVICE', env('APP_NAME', 'laravel')),
    'environment' => env('LOGTIDE_ENVIRONMENT', env('APP_ENV', 'production')),
    'release' => env('LOGTIDE_RELEASE'),

    // Batching
    'batch_size' => (int) env('LOGTIDE_BATCH_SIZE', 100),
    'flush_interval' => (int) env('LOGTIDE_FLUSH_INTERVAL', 5000),
    'max_buffer_size' => (int) env('LOGTIDE_MAX_BUFFER_SIZE', 10000),
    'max_retries' => (int) env('LOGTIDE_MAX_RETRIES', 3),

    // Tracing
    'traces_sample_rate' => (float) env('LOGTIDE_TRACES_SAMPLE_RATE', 1.0),

    // Privacy
    'send_default_pii' => (bool) env('LOGTIDE_SEND_DEFAULT_PII', false),

    // Breadcrumb integrations
    'breadcrumbs' => [
        'db_queries' => true,
        'cache' => true,
        'queue' => true,
        'http_client' => true,
    ],

    // Paths to skip in the HTTP middleware
    'skip_paths' => ['/health', '/healthz'],

    'debug' => (bool) env('LOGTIDE_DEBUG', false),
];
```

### config/logging.php

Add the LogTide channel to your logging stack:

```php
<?php

return [
    'default' => env('LOG_CHANNEL', 'stack'),

    'channels' => [
        'stack' => [
            'driver' => 'stack',
            'channels' => ['daily', 'logtide'],
            'ignore_exceptions' => false,
        ],

        // The 'logtide' channel is auto-registered by the service provider.
        // No manual configuration needed.

        // ... other channels
    ],
];
```

## Basic Usage

### Using the Logger

```php
use Illuminate\Support\Facades\Log;

// Simple messages
Log::info('User logged in');
Log::warning('Rate limit approaching');
Log::error('Payment failed');

// With structured context
Log::info('Order placed', [
    'order_id' => $order->id,
    'amount' => $order->total,
    'currency' => $order->currency,
    'user_id' => $order->user_id,
]);

// With exception
try {
    $this->processPayment($order);
} catch (PaymentException $e) {
    Log::error('Payment processing failed', [
        'order_id' => $order->id,
        'exception' => $e,
    ]);
}
```

### Using the Facade

```php
use LogTide\Laravel\LogtideFacade as Logtide;
use LogTide\Enum\LogLevel;

// Direct Hub API access
Logtide::captureLog(LogLevel::INFO, 'Custom event', ['key' => 'value']);

// Capture exceptions
Logtide::captureException($exception);

// Scoped context
Logtide::withScope(function () {
    Logtide::getScope()->setTag('handler', 'checkout');
    Logtide::captureLog(LogLevel::INFO, 'Checkout started');
});

// Tracing
$span = Logtide::startSpan('process.order');
// ... do work ...
if ($span !== null) {
    Logtide::finishSpan($span);
}

// Flush
Logtide::flush();
```

## Request Middleware

The HTTP middleware is **automatically registered** by the service provider. For each request it:

- Creates an isolated scope (tags and breadcrumbs don't leak between requests)
- Extracts incoming `traceparent` header for distributed tracing
- Starts a SERVER span with HTTP attributes (method, URL, user agent)
- Captures 5xx responses as error log events
- Catches and reports uncaught exceptions
- Injects `traceparent` into the response for trace correlation

Paths listed in `config('logtide.skip_paths')` are skipped (default: `/health`, `/healthz`).

## Breadcrumb Integrations

The service provider automatically registers breadcrumb integrations:

- **Database Queries** — Records `QueryExecuted` events with query, bindings, and duration
- **Cache Events** — Records cache hits, misses, writes, and deletes
- **Queue Jobs** — Records job processing events with job class, queue name, and attempt

Toggle them in `config/logtide.php` under the `breadcrumbs` key.

## Queue Job Logging

Jobs automatically have access to LogTide:

```php
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Support\Facades\Log;

class ProcessOrder implements ShouldQueue
{
    use Queueable;

    public function __construct(
        public Order $order
    ) {}

    public function handle()
    {
        Log::info('Processing order in background', [
            'order_id' => $this->order->id,
        ]);

        // Process order...

        Log::info('Order processed successfully');
    }
}
```

## Docker Deployment

```yaml
# docker-compose.yml
services:
  app:
    build: .
    environment:
      - APP_ENV=production
      - LOGTIDE_DSN=${LOGTIDE_DSN}
    depends_on:
      - mysql
      - redis

  queue:
    build: .
    command: php artisan queue:work --tries=3
    environment:
      - APP_ENV=production
      - LOGTIDE_DSN=${LOGTIDE_DSN}
    depends_on:
      - mysql
      - redis

  scheduler:
    build: .
    command: php artisan schedule:work
    environment:
      - APP_ENV=production
      - LOGTIDE_DSN=${LOGTIDE_DSN}
```

## Troubleshooting

### Logs not appearing

1. Check DSN is set:
   ```bash
   php artisan tinker --execute="dump(config('logtide.dsn'));"
   ```

2. Enable debug mode:
   ```env
   LOGTIDE_DEBUG=true
   ```

3. Check Laravel logs for errors:
   ```bash
   tail -f storage/logs/laravel.log
   ```

### Queue logs not appearing

Queue workers need the same environment variables:

```bash
LOGTIDE_DSN=xxx php artisan queue:work
```

## Next Steps

- [PHP SDK Reference](/docs/sdks/php/) - Core SDK documentation
- [Symfony Integration](/docs/sdks/symfony/) - Symfony Bundle integration
- [nginx Integration](/integrations/nginx/) - Web server logs
- [Docker Integration](/integrations/docker/) - Container deployment
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging


---

# MongoDB Log Monitoring Integration

> Collect MongoDB profiler output, slow operation logs, and replica set events in LogTide for database observability.

*Source: https://logtide.dev/integrations/mongodb/*

---

MongoDB 4.4+ outputs structured JSON logs by default, making it straightforward to ship to LogTide. This guide covers log collection with Fluent Bit, profiler configuration for slow operations, and replica set event monitoring.

## Why monitor MongoDB logs?

- **Slow operation detection**: Find expensive queries and aggregations before they cascade
- **Replica set health**: Track elections, step-downs, and replication lag
- **Connection monitoring**: Detect connection storms and pool exhaustion
- **Security auditing**: Track authentication failures and unauthorized access
- **Capacity planning**: Understand query patterns, collection scans, and index usage

## Prerequisites

- MongoDB 4.4+ (5.0+ recommended for structured logging)
- Fluent Bit for log collection
- LogTide instance with API key

## MongoDB Configuration

### Enable Profiling

```javascript
// Connect to MongoDB and enable profiler
use admin;

// Profile slow operations (over 100ms)
db.setProfilingLevel(1, { slowms: 100 });

// Verify
db.getProfilingStatus();
// { "was" : 1, "slowms" : 100, "sampleRate" : 1 }
```

### Configure Log Output

```yaml
# /etc/mongod.conf
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
  # MongoDB 4.4+ outputs structured JSON by default

# Log verbosity (0-5, default 0)
  component:
    accessControl:
      verbosity: 1     # Auth events
    command:
      verbosity: 0     # Command execution
    query:
      verbosity: 0     # Query planning
    replication:
      verbosity: 1     # Replica set events
    storage:
      verbosity: 0     # Storage engine

# Profiler settings
operationProfiling:
  mode: slowOp
  slowOpThresholdMs: 100
  slowOpSampleRate: 1.0
```

Restart MongoDB:

```bash
sudo systemctl restart mongod
```

### Log Output Example

MongoDB 4.4+ structured log format:

```json
{
  "t": {"$date": "2025-01-31T10:00:00.000+00:00"},
  "s": "I",
  "c": "COMMAND",
  "id": 51803,
  "ctx": "conn123",
  "msg": "Slow query",
  "attr": {
    "type": "command",
    "ns": "mydb.users",
    "command": {"find": "users", "filter": {"email": "user@example.com"}},
    "planSummary": "COLLSCAN",
    "keysExamined": 0,
    "docsExamined": 50000,
    "nreturned": 1,
    "durationMillis": 234
  }
}
```

## Fluent Bit Configuration

### Collect MongoDB Logs

```ini
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

# MongoDB structured logs (JSON by default since 4.4)
[INPUT]
    Name          tail
    Path          /var/log/mongodb/mongod.log
    Tag           mongodb.log
    Parser        json
    Refresh_Interval 5

# Add metadata
[FILTER]
    Name          record_modifier
    Match         mongodb.*
    Record        hostname ${HOSTNAME}
    Record        service mongodb
    Record        environment production

# Map MongoDB severity to standard levels
[FILTER]
    Name          lua
    Match         mongodb.*
    script        /etc/fluent-bit/mongo_level.lua
    call          map_severity

# Output to LogTide
[OUTPUT]
    Name          http
    Match         *
    Host          api.logtide.dev
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

### Severity Mapping Script

```lua
-- /etc/fluent-bit/mongo_level.lua
local severity_map = {
    F = "critical",  -- Fatal
    E = "error",     -- Error
    W = "warning",   -- Warning
    I = "info",      -- Informational
    D = "debug",     -- Debug
}

function map_severity(tag, timestamp, record)
    local s = record["s"]
    if s and severity_map[s] then
        record["level"] = severity_map[s]
    else
        record["level"] = "info"
    end
    return 1, timestamp, record
end
```

## Profiler Data Collection

### Script-Based Profiler Collection

```python
#!/usr/bin/env python3
# /opt/mongodb-monitor/profiler.py

import json
import os
from datetime import datetime, timedelta
from pymongo import MongoClient

MONGO_URI = os.environ.get('MONGO_URI', 'mongodb://localhost:27017')

client = MongoClient(MONGO_URI)

# Collect from all databases
for db_name in client.list_database_names():
    if db_name in ('admin', 'config', 'local'):
        continue

    db = client[db_name]

    # Get recent profiler entries
    since = datetime.utcnow() - timedelta(minutes=5)
    cursor = db.system.profile.find(
        {'ts': {'$gte': since}},
        limit=100,
    ).sort('ts', -1)

    for entry in cursor:
        log = {
            'log_type': 'profiler',
            'database': db_name,
            'collection': entry.get('ns', '').split('.')[-1],
            'operation': entry.get('op', 'unknown'),
            'duration_ms': entry.get('millis', 0),
            'plan_summary': entry.get('planSummary', ''),
            'keys_examined': entry.get('keysExamined', 0),
            'docs_examined': entry.get('docsExamined', 0),
            'nreturned': entry.get('nreturned', 0),
            'response_length': entry.get('responseLength', 0),
        }

        # Flag collection scans
        if 'COLLSCAN' in log['plan_summary']:
            log['warning'] = 'collection_scan'

        print(json.dumps(log, default=str))

client.close()
```

Install dependency: `pip install pymongo`

### Fluent Bit Exec Input

```ini
[INPUT]
    Name          exec
    Tag           mongodb.profiler
    Command       python3 /opt/mongodb-monitor/profiler.py
    Interval_Sec  300
    Parser        json

[FILTER]
    Name          record_modifier
    Match         mongodb.profiler
    Record        service mongodb
```

## Docker Setup

```yaml
# docker-compose.yml
services:
  mongodb:
    image: mongo:7
    command: >
      mongod
      --profile 1
      --slowms 100
    ports:
      - "27017:27017"
    volumes:
      - mongo_data:/data/db
      - mongo_logs:/var/log/mongodb
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}

  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./mongo_level.lua:/fluent-bit/etc/mongo_level.lua:ro
      - mongo_logs:/var/log/mongodb:ro
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      - mongodb

volumes:
  mongo_data:
  mongo_logs:
```

## Monitoring Queries

### Slow Operations

```sql
SELECT * FROM logs
WHERE service = 'mongodb'
  AND log_type = 'profiler'
  AND duration_ms > 500
ORDER BY duration_ms DESC
LIMIT 50
```

### Collection Scans (Missing Indexes)

```sql
SELECT * FROM logs
WHERE service = 'mongodb'
  AND warning = 'collection_scan'
ORDER BY docs_examined DESC
LIMIT 50
```

### Replica Set Events

```sql
SELECT * FROM logs
WHERE service = 'mongodb'
  AND (message LIKE '%election%'
       OR message LIKE '%stepDown%'
       OR message LIKE '%replSetReconfig%')
ORDER BY timestamp DESC
```

### Authentication Failures

```sql
SELECT * FROM logs
WHERE service = 'mongodb'
  AND level = 'warning'
  AND (message LIKE '%authentication%'
       OR message LIKE '%auth%')
ORDER BY timestamp DESC
```

## Detection Rules

### Sustained Slow Operations

```
service:mongodb AND log_type:profiler AND duration_ms > 1000
| window 5m
| count > 10
```

Alert: "MongoDB experiencing sustained slow operations"

### Collection Scan on Large Collection

```
service:mongodb AND warning:collection_scan AND docs_examined > 10000
```

Alert: "Collection scan on large collection — missing index?"

### Replica Set Election

```
service:mongodb AND message:"election"
```

Alert: "MongoDB replica set election triggered"

### Connection Spike

```
service:mongodb AND message:"connection accepted"
| window 1m
| count > 100
```

Alert: "MongoDB connection spike detected"

## Performance Impact

| Metric | Without Profiling | With Profiling (slowOp) | Overhead |
|--------|-------------------|-------------------------|----------|
| QPS | 50,000 | 49,800 | -0.4% |
| Avg latency | 1ms | 1.01ms | +1% |
| Disk I/O | Baseline | +2-5 MB/day | Minimal |

**Notes:**
- `slowOp` profiling only logs slow operations — minimal overhead
- Level 2 profiling (all operations) has significant overhead — avoid in production
- Structured JSON logs have negligible overhead vs legacy format

## Security

### Read-Only Monitor User

```javascript
use admin;
db.createUser({
  user: "logtide_monitor",
  pwd: "SECURE_PASSWORD",
  roles: [
    { role: "clusterMonitor", db: "admin" },
    { role: "read", db: "admin" },
  ]
});
```

### Redacting Sensitive Data

MongoDB's `--redactClientLogData` flag removes field values from logs:

```yaml
# mongod.conf
security:
  redactClientLogData: true
```

This replaces values with `"###"` in log output, preventing PII exposure.

## Troubleshooting

### Profiler entries not appearing

1. Check profiling level:
   ```javascript
   db.getProfilingStatus()
   ```
2. Check profiler collection:
   ```javascript
   db.system.profile.find().limit(5).sort({ts: -1})
   ```
3. Generate a test slow query:
   ```javascript
   db.collection.find({nonIndexedField: "test"}).explain("executionStats")
   ```

### Log file not updating

Check MongoDB has write permissions:

```bash
ls -la /var/log/mongodb/
# mongodb user needs write access
```

### Fluent Bit JSON parse errors

MongoDB logs may contain nested objects. Ensure Fluent Bit's JSON parser handles nested fields:

```ini
[PARSER]
    Name        json
    Format      json
    Time_Key    t
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container deployment
- [Kubernetes Integration](/integrations/kubernetes/) - StatefulSet patterns
- [Node.js Integration](/integrations/nodejs/) - Application-level MongoDB logging
- [Incident Response](/use-cases/incident-response/) - Debug database issues


---

# MySQL and MariaDB Log Integration

> Monitor MySQL slow queries, error logs, and connection events in LogTide with structured parsing and real-time alerting.

*Source: https://logtide.dev/integrations/mysql/*

---

MySQL and MariaDB generate slow query logs, error logs, and general query logs that are critical for performance monitoring and security. This guide shows you how to collect, parse, and ship these logs to LogTide using Fluent Bit.

## Why monitor MySQL logs?

- **Slow query detection**: Find unoptimized queries before they cause outages
- **Error tracking**: Catch connection failures, table corruption, and replication issues
- **Security monitoring**: Detect unauthorized access, privilege escalation, and unusual patterns
- **Capacity planning**: Track query patterns and growth over time
- **Compliance**: Maintain audit trail for database access (SOC2, HIPAA)

## Prerequisites

- MySQL 8.0+ or MariaDB 10.6+
- Fluent Bit for log collection
- LogTide instance with API key
- Root or admin access to MySQL

## MySQL Configuration

### Enable Slow Query Log

```sql
-- Run in MySQL client
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL slow_query_log_file = '/var/log/mysql/slow.log';
SET GLOBAL long_query_time = 1;          -- Log queries over 1 second
SET GLOBAL log_queries_not_using_indexes = 'ON';
SET GLOBAL min_examined_row_limit = 1000; -- Skip trivial queries
```

Or in the configuration file:

```ini
# /etc/mysql/mysql.conf.d/mysqld.cnf (MySQL)
# /etc/mysql/mariadb.conf.d/50-server.cnf (MariaDB)

[mysqld]
# Slow query log
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1
log_queries_not_using_indexes = 1
min_examined_row_limit = 1000

# Error log
log_error = /var/log/mysql/error.log
log_error_verbosity = 2    # 1=errors, 2=+warnings, 3=+notes

# General query log (enable only for debugging, high overhead)
# general_log = 1
# general_log_file = /var/log/mysql/general.log
```

Restart MySQL:

```bash
sudo systemctl restart mysql
```

### Verify Configuration

```sql
SHOW VARIABLES LIKE 'slow_query%';
SHOW VARIABLES LIKE 'long_query_time';
SHOW VARIABLES LIKE 'log_error%';
```

## Fluent Bit Configuration

### Collect MySQL Logs

```ini
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

# MySQL error log
[INPUT]
    Name          tail
    Path          /var/log/mysql/error.log
    Tag           mysql.error
    Parser        mysql_error
    Refresh_Interval 5

# MySQL slow query log
[INPUT]
    Name          tail
    Path          /var/log/mysql/slow.log
    Tag           mysql.slow
    Parser        mysql_slow
    Multiline     On
    Parser_Firstline mysql_slow_firstline
    Refresh_Interval 5

# Add metadata
[FILTER]
    Name          record_modifier
    Match         mysql.*
    Record        hostname ${HOSTNAME}
    Record        service mysql
    Record        environment production

# Output to LogTide
[OUTPUT]
    Name          http
    Match         *
    Host          api.logtide.dev
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

### Parser Configuration

```ini
# /etc/fluent-bit/parsers.conf

# MySQL error log format
[PARSER]
    Name        mysql_error
    Format      regex
    Regex       ^(?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+(?<thread>\d+)\s+\[(?<level>\w+)\]\s+\[(?<error_code>\w+)\]\s+\[(?<subsystem>\w+)\]\s+(?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ

# MySQL slow query first line
[PARSER]
    Name        mysql_slow_firstline
    Format      regex
    Regex       ^# Time: (?<time>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})

# MySQL slow query content
[PARSER]
    Name        mysql_slow
    Format      regex
    Regex       ^# User@Host: (?<user>\S+)\s+@\s+(?<host>\S+)\s+\[(?<ip>[^\]]*)\].*Query_time:\s+(?<query_time>[\d.]+)\s+Lock_time:\s+(?<lock_time>[\d.]+)\s+Rows_sent:\s+(?<rows_sent>\d+)\s+Rows_examined:\s+(?<rows_examined>\d+)
```

## Slow Query Collector Script

For more structured output, use a Python script that queries the `performance_schema`:

```python
#!/usr/bin/env python3
# /opt/mysql-monitor/slow_queries.py

import json
import os
import mysql.connector

config = {
    'host': os.environ.get('MYSQL_HOST', 'localhost'),
    'port': int(os.environ.get('MYSQL_PORT', 3306)),
    'user': os.environ.get('MYSQL_USER', 'monitor'),
    'password': os.environ.get('MYSQL_PASSWORD'),
    'database': 'performance_schema',
}

conn = mysql.connector.connect(**config)
cursor = conn.cursor(dictionary=True)

# Get top slow queries from performance_schema
cursor.execute("""
    SELECT
        DIGEST_TEXT AS query_pattern,
        COUNT_STAR AS exec_count,
        ROUND(AVG_TIMER_WAIT / 1e9, 2) AS avg_ms,
        ROUND(MAX_TIMER_WAIT / 1e9, 2) AS max_ms,
        SUM_ROWS_SENT AS total_rows_sent,
        SUM_ROWS_EXAMINED AS total_rows_examined,
        FIRST_SEEN,
        LAST_SEEN
    FROM events_statements_summary_by_digest
    WHERE AVG_TIMER_WAIT > 1e9  -- Over 1ms average
    ORDER BY AVG_TIMER_WAIT DESC
    LIMIT 50
""")

for row in cursor.fetchall():
    log = {
        'log_type': 'slow_query_digest',
        'query_pattern': row['query_pattern'][:500],
        'exec_count': row['exec_count'],
        'avg_ms': float(row['avg_ms']),
        'max_ms': float(row['max_ms']),
        'total_rows_sent': row['total_rows_sent'],
        'total_rows_examined': row['total_rows_examined'],
        'first_seen': str(row['FIRST_SEEN']),
        'last_seen': str(row['LAST_SEEN']),
    }
    print(json.dumps(log))

cursor.close()
conn.close()
```

Run periodically via cron or Fluent Bit exec:

```ini
[INPUT]
    Name          exec
    Tag           mysql.slowdigest
    Command       python3 /opt/mysql-monitor/slow_queries.py
    Interval_Sec  300
    Parser        json
```

## Docker Setup

```yaml
# docker-compose.yml
services:
  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: myapp
    command: >
      --slow-query-log=ON
      --slow-query-log-file=/var/log/mysql/slow.log
      --long-query-time=1
      --log-queries-not-using-indexes=ON
      --log-error=/var/log/mysql/error.log
    volumes:
      - mysql_data:/var/lib/mysql
      - mysql_logs:/var/log/mysql
    ports:
      - "3306:3306"

  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - mysql_logs:/var/log/mysql:ro
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      - mysql

volumes:
  mysql_data:
  mysql_logs:
```

## Kubernetes Deployment

```yaml
# mysql-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.0
          args:
            - --slow-query-log=ON
            - --long-query-time=1
            - --log-queries-not-using-indexes=ON
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: root-password
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
            - name: logs
              mountPath: /var/log/mysql

        - name: log-collector
          image: fluent/fluent-bit:latest
          volumeMounts:
            - name: logs
              mountPath: /var/log/mysql
              readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
          env:
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: api-key

      volumes:
        - name: logs
          emptyDir: {}
        - name: fluent-bit-config
          configMap:
            name: mysql-fluent-bit-config

  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 50Gi
```

## Monitoring Queries

### Slow Queries Over Time

```sql
SELECT * FROM logs
WHERE service = 'mysql'
  AND log_type = 'slow_query_digest'
  AND avg_ms > 100
ORDER BY avg_ms DESC
LIMIT 50
```

### Error Log Analysis

```sql
SELECT * FROM logs
WHERE service = 'mysql'
  AND level IN ('error', 'warning')
ORDER BY timestamp DESC
LIMIT 100
```

### Connection Tracking

```sql
SELECT * FROM logs
WHERE service = 'mysql'
  AND (message LIKE '%Access denied%'
       OR message LIKE '%Too many connections%'
       OR message LIKE '%Aborted connection%')
ORDER BY timestamp DESC
```

### Replication Lag

```sql
SELECT * FROM logs
WHERE service = 'mysql'
  AND (message LIKE '%Slave%'
       OR message LIKE '%Replica%'
       OR message LIKE '%replication%')
ORDER BY timestamp DESC
```

## Detection Rules

### Slow Query Spike

```
service:mysql AND log_type:slow_query_digest AND avg_ms > 5000
```

Alert: "MySQL query averaging over 5 seconds"

### Access Denied Burst

```
service:mysql AND message:"Access denied"
| window 5m
| count > 10
```

Alert: "Multiple failed MySQL login attempts"

### Replication Failure

```
service:mysql AND level:error AND message:"replication"
```

Alert: "MySQL replication error detected"

### Connection Pool Exhaustion

```
service:mysql AND message:"Too many connections"
```

Alert: "MySQL connection limit reached"

## Performance Impact

| Metric | Without Logging | With Slow Query Log | Overhead |
|--------|-----------------|---------------------|----------|
| QPS | 10,000 | 9,950 | -0.5% |
| Avg query latency | 2ms | 2.01ms | +0.5% |
| Disk I/O | Baseline | +5-10 MB/day | Minimal |
| CPU usage | Baseline | +0.1% | Negligible |

**Notes:**
- Slow query log has negligible impact (only logs slow queries)
- General query log has significant overhead — use only for debugging
- Fluent Bit runs out-of-band, no MySQL impact

## Security Considerations

### Create a Read-Only Monitor User

```sql
CREATE USER 'logtide_monitor'@'localhost' IDENTIFIED BY 'SECURE_PASSWORD';
GRANT SELECT ON performance_schema.* TO 'logtide_monitor'@'localhost';
GRANT PROCESS ON *.* TO 'logtide_monitor'@'localhost';
FLUSH PRIVILEGES;
```

### Sensitive Query Filtering

Avoid logging queries with sensitive data:

```ini
# Fluent Bit filter to redact passwords
[FILTER]
    Name          lua
    Match         mysql.slow
    script        /etc/fluent-bit/sanitize.lua
    call          redact_sensitive
```

```lua
-- /etc/fluent-bit/sanitize.lua
function redact_sensitive(tag, timestamp, record)
    local query = record["query_pattern"] or ""
    query = query:gsub("'[^']*'", "'***'")
    query = query:gsub("password%s*=%s*%S+", "password=***")
    record["query_pattern"] = query
    return 1, timestamp, record
end
```

## Troubleshooting

### Slow query log not capturing queries

1. Verify slow log is enabled:
   ```sql
   SHOW VARIABLES LIKE 'slow_query%';
   ```
2. Check threshold:
   ```sql
   SHOW VARIABLES LIKE 'long_query_time';
   ```
3. Generate a test slow query:
   ```sql
   SELECT SLEEP(2);
   ```
4. Check file permissions:
   ```bash
   ls -la /var/log/mysql/
   ```

### MariaDB differences

MariaDB uses the same configuration options but may store logs in different default locations. Check:

```bash
mariadb --help --verbose | grep -A1 'log-error'
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log collection
- [Kubernetes Integration](/integrations/kubernetes/) - Cluster-level monitoring
- [PostgreSQL Integration](/integrations/postgresql/) - Compare with PostgreSQL logging
- [Security Monitoring](/use-cases/security-monitoring/) - Database access auditing


---

# Next.js Application Logging Integration

> Send structured logs from Next.js applications to LogTide with instrumentation hooks, error boundaries, and navigation tracking.

*Source: https://logtide.dev/integrations/nextjs/*

---

LogTide's Next.js SDK (`@logtide/nextjs`) provides server-side instrumentation, client-side error boundaries, and navigation tracking for Next.js App Router applications.

## Why use LogTide with Next.js?

- **Instrumentation hook**: Initialize once in `instrumentation.ts`, capture all server errors
- **Error boundaries**: Client-side React error capture with `LogtideErrorBoundary`
- **Navigation tracking**: Track client-side route changes as breadcrumbs
- **Server + client**: Separate DSNs for server and client-side logging
- **Self-hosted option**: Keep logs in your infrastructure

## Prerequisites

- Next.js 14+ (App Router)
- Node.js 18+
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/nextjs
```

## Server-Side Setup

Initialize LogTide via the instrumentation hook:

```typescript
// instrumentation.ts
import { registerLogtide } from '@logtide/nextjs/server';

export async function register() {
  registerLogtide({
    dsn: process.env.LOGTIDE_DSN!,
    service: 'nextjs-app',
    environment: process.env.NODE_ENV,
    release: process.env.NEXT_PUBLIC_APP_VERSION,
  });
}
```

Enable the hook in `next.config.ts`:

```typescript
const nextConfig = {
  experimental: {
    instrumentationHook: true,
  },
};

export default nextConfig;
```

## Client-Side Setup

Initialize LogTide on the client:

```tsx
// app/layout.tsx
import { initLogtide } from '@logtide/nextjs/client';

initLogtide({
  dsn: process.env.NEXT_PUBLIC_LOGTIDE_DSN!,
  environment: process.env.NODE_ENV,
});

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>{children}</body>
    </html>
  );
}
```

## Environment Variables

```env
# .env.local
LOGTIDE_DSN=https://lp_abc123@api.logtide.dev
NEXT_PUBLIC_LOGTIDE_DSN=https://lp_client456@api.logtide.dev
NEXT_PUBLIC_APP_VERSION=1.0.0
```

## Error Capture

Server-side errors in API routes, Server Components, and Server Actions are captured automatically via `captureRequestError`.

## Error Boundary

Use `LogtideErrorBoundary` for client-side React error capture:

`LogtideErrorBoundary` wraps your component tree and reports any rendering
error it catches to LogTide. It accepts an optional `fallback` (a node, or a
`(error) => node` function) to render in place of the crashed subtree:

```tsx
// app/providers.tsx
'use client';

import { LogtideErrorBoundary } from '@logtide/nextjs/client';

export function Providers({ children }: { children: React.ReactNode }) {
  return (
    <LogtideErrorBoundary fallback={<div>Something went wrong</div>}>
      {children}
    </LogtideErrorBoundary>
  );
}
```

Then wrap your app with it in `app/layout.tsx`:

```tsx
// app/layout.tsx
import { Providers } from './providers';

export default function RootLayout({ children }: { children: React.ReactNode }) {
  return (
    <html lang="en">
      <body>
        <Providers>{children}</Providers>
      </body>
    </html>
  );
}
```

## Navigation Tracking

Track client-side navigation as breadcrumbs:

```tsx
// app/providers.tsx
'use client';

import { trackNavigation } from '@logtide/nextjs/client';
import { usePathname } from 'next/navigation';
import { useEffect } from 'react';

export function LogtideNavigationTracker() {
  const pathname = usePathname();
  useEffect(() => {
    trackNavigation(pathname);
  }, [pathname]);
  return null;
}
```

## API Route Logging

```typescript
// app/api/users/[id]/route.ts
import { hub } from '@logtide/core';

export async function GET(
  request: Request,
  { params }: { params: { id: string } }
) {
  hub.captureLog('info', 'Fetching user', { userId: params.id });

  try {
    const user = await getUser(params.id);
    return Response.json(user);
  } catch (error) {
    hub.captureError(error, {
      extra: { userId: params.id },
    });
    return Response.json({ error: 'Internal error' }, { status: 500 });
  }
}
```

## Server Actions

```typescript
// app/actions/user.ts
'use server';

import { hub } from '@logtide/core';

export async function updateUser(formData: FormData) {
  const userId = formData.get('userId') as string;

  hub.addBreadcrumb({
    type: 'custom',
    category: 'action',
    message: 'Update user action triggered',
    timestamp: Date.now(),
  });

  try {
    await db.user.update({ where: { id: userId }, data: { name: formData.get('name') } });
    hub.captureLog('info', 'User updated', { userId });
    return { success: true };
  } catch (error) {
    hub.captureError(error, { extra: { userId } });
    return { success: false };
  }
}
```

## Docker Deployment

```dockerfile
FROM node:20-alpine AS base

FROM base AS deps
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable pnpm && pnpm install --frozen-lockfile

FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN corepack enable pnpm && pnpm build

FROM base AS runner
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/public ./public
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
EXPOSE 3000
CMD ["node", "server.js"]
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Next.js SDK Reference](/docs/sdks/nextjs-sdk/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployment
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging


---

# nginx Access and Error Log Integration

> Send nginx access and error logs to LogTide with structured parsing. Complete setup for Docker, systemd, and Kubernetes.

*Source: https://logtide.dev/integrations/nginx/*

---

nginx is one of the most popular web servers. This guide shows you how to send nginx access and error logs to LogTide with full structured parsing, enabling powerful queries and alerting.

## Why send nginx logs to LogTide?

- **Request analytics**: Analyze response times, status codes, and traffic patterns
- **Error detection**: Get alerted on 5xx errors, slow requests, and security issues
- **Compliance**: Audit trail for GDPR and security requirements
- **Correlation**: Link nginx logs with application logs using request IDs
- **Real-time visibility**: Live tail for debugging production issues

## Prerequisites

- nginx 1.18+ (any version with JSON log format support)
- LogTide instance with API key
- Fluent Bit for log forwarding (or systemd-journal)

## Quick Start (5 minutes)

### Step 1: Configure nginx JSON Log Format

Edit your nginx configuration (`/etc/nginx/nginx.conf` or site config):

```nginx
http {
    # JSON log format for structured logging
    log_format json_combined escape=json
    '{'
        '"time":"$time_iso8601",'
        '"remote_addr":"$remote_addr",'
        '"request_method":"$request_method",'
        '"request_uri":"$request_uri",'
        '"status":$status,'
        '"body_bytes_sent":$body_bytes_sent,'
        '"request_time":$request_time,'
        '"upstream_response_time":"$upstream_response_time",'
        '"http_referrer":"$http_referer",'
        '"http_user_agent":"$http_user_agent",'
        '"http_x_forwarded_for":"$http_x_forwarded_for",'
        '"request_id":"$request_id"'
    '}';

    access_log /var/log/nginx/access.log json_combined;
    error_log /var/log/nginx/error.log warn;
}
```

Reload nginx:

```bash
sudo nginx -t && sudo nginx -s reload
```

### Step 2: Set Up Fluent Bit

Create `/etc/fluent-bit/fluent-bit.conf`:

```ini
[SERVICE]
    Flush         1
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Path          /var/log/nginx/access.log
    Parser        nginx_json
    Tag           nginx.access
    Refresh_Interval 5

[INPUT]
    Name          tail
    Path          /var/log/nginx/error.log
    Parser        nginx_error
    Tag           nginx.error
    Refresh_Interval 5

[FILTER]
    Name          modify
    Match         nginx.*
    Add           service nginx

[FILTER]
    Name          modify
    Match         nginx.access
    Add           level info

[FILTER]
    Name          modify
    Match         nginx.error
    Add           level error
    Rename        message log_message

# Create message field from request data (for access logs)
[FILTER]
    Name          lua
    Match         nginx.access
    script        /fluent-bit/etc/nginx_message.lua
    call          create_message

[OUTPUT]
    Name          http
    Match         nginx.*
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json_lines
    Header        X-API-Key YOUR_API_KEY
    Header        Content-Type application/x-ndjson
    Json_date_key time
    Json_date_format iso8601
    tls           On
```

Create `/etc/fluent-bit/nginx_message.lua`:

```lua
-- Create message from nginx access log fields
function create_message(tag, timestamp, record)
    local method = record["request_method"] or "GET"
    local uri = record["request_uri"] or "/"
    local status = record["status"] or 0
    record["message"] = string.format("%s %s %d", method, uri, status)
    return 1, timestamp, record
end
```

Create `/etc/fluent-bit/parsers.conf`:

```ini
[PARSER]
    Name        nginx_json
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S%z

[PARSER]
    Name        nginx_error
    Format      regex
    Regex       ^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<level>\w+)\] (?<pid>\d+)#(?<tid>\d+): (?<message>.*)$
    Time_Key    time
    Time_Format %Y/%m/%d %H:%M:%S
```

Start Fluent Bit:

```bash
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit
```

## Production Configuration

### Enhanced nginx Log Format

For production, include additional fields for debugging and security:

```nginx
log_format json_production escape=json
'{'
    '"time":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"request_method":"$request_method",'
    '"request_uri":"$request_uri",'
    '"server_protocol":"$server_protocol",'
    '"status":$status,'
    '"body_bytes_sent":$body_bytes_sent,'
    '"request_time":$request_time,'
    '"upstream_response_time":"$upstream_response_time",'
    '"upstream_addr":"$upstream_addr",'
    '"http_referrer":"$http_referer",'
    '"http_user_agent":"$http_user_agent",'
    '"http_x_forwarded_for":"$http_x_forwarded_for",'
    '"http_x_real_ip":"$http_x_real_ip",'
    '"request_id":"$request_id",'
    '"ssl_protocol":"$ssl_protocol",'
    '"ssl_cipher":"$ssl_cipher",'
    '"request_length":$request_length,'
    '"connection":$connection,'
    '"connection_requests":$connection_requests'
'}';
```

### Request ID Generation

Enable request ID generation for correlation with application logs:

```nginx
server {
    # Generate unique request ID
    set $request_id $request_id;
    if ($request_id = "") {
        set $request_id $pid-$msec-$remote_addr-$connection;
    }

    # Pass to upstream applications
    proxy_set_header X-Request-ID $request_id;

    # Include in response for debugging
    add_header X-Request-ID $request_id;
}
```

## Docker Setup

### docker-compose.yml

```yaml
version: "3.8"

services:
  nginx:
    image: nginx:alpine
    container_name: nginx
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/logs:/var/log/nginx
    ports:
      - "80:80"
      - "443:443"

  fluent-bit:
    image: fluent/fluent-bit:latest
    container_name: fluent-bit
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - ./nginx/logs:/var/log/nginx:ro
    depends_on:
      - nginx
    restart: unless-stopped
```

### Alternative: Docker Log Driver

If you prefer not to mount log files, configure nginx to log to stdout/stderr:

```nginx
error_log /dev/stderr warn;
access_log /dev/stdout json_combined;
```

Then use Docker's Fluentd log driver as shown in the [Docker integration guide](/integrations/docker/).

## Kubernetes Setup

### ConfigMap for nginx

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {
        worker_connections 1024;
    }
    http {
        log_format json_combined escape=json
        '{'
            '"time":"$time_iso8601",'
            '"remote_addr":"$remote_addr",'
            '"request_method":"$request_method",'
            '"request_uri":"$request_uri",'
            '"status":$status,'
            '"request_time":$request_time,'
            '"request_id":"$request_id"'
        '}';

        access_log /var/log/nginx/access.log json_combined;
        error_log /var/log/nginx/error.log warn;

        server {
            listen 80;
            location / {
                root /usr/share/nginx/html;
            }
        }
    }
```

### Deployment with Sidecar

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
        - name: logs
          mountPath: /var/log/nginx

      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: logs
          mountPath: /var/log/nginx
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/

      volumes:
      - name: config
        configMap:
          name: nginx-config
      - name: logs
        emptyDir: {}
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
```

## Verification

Test that logs are arriving in LogTide:

```bash
# Generate some traffic
curl -I http://localhost/

# Check LogTide for nginx logs
# Filter by: service:nginx
```

In LogTide, you should see structured fields like:

| Field | Example Value |
|-------|---------------|
| service | nginx |
| status | 200 |
| request_method | GET |
| request_uri | /api/users |
| request_time | 0.045 |
| remote_addr | 192.168.1.100 |

## Detection Rules

Create alerts for common nginx issues:

### High Error Rate

```
service:nginx AND status:>=500
```

Set threshold: >10 errors in 5 minutes

### Slow Requests

```
service:nginx AND request_time:>2
```

Requests taking longer than 2 seconds.

### 404 Spam (Potential Scan)

```
service:nginx AND status:404
```

Set threshold: >100 in 1 minute from same IP

### Unusual User Agents

```
service:nginx AND http_user_agent:(*curl* OR *wget* OR *python*)
```

Detect automated access patterns.

## Performance Metrics

| Metric | Value | Notes |
|--------|-------|-------|
| Log overhead | <1ms | Per request with JSON format |
| Disk I/O | ~500 bytes/request | With full metadata |
| Memory (Fluent Bit) | ~30MB | For nginx log tailing |

## Privacy Considerations

For GDPR compliance, consider:

1. **IP anonymization**: Truncate or hash IP addresses
2. **User agent filtering**: Remove or mask identifying information
3. **Request body**: Never log POST body data
4. **Retention**: Configure LogTide retention policies

Example IP anonymization in nginx:

```nginx
map $remote_addr $remote_addr_anon {
    ~(?P<ip>\d+\.\d+\.\d+)\.\d+ $ip.0;
    default 0.0.0.0;
}
```

## Troubleshooting

### Logs not appearing in LogTide

1. Check nginx is writing to the configured log file:
   ```bash
   tail -f /var/log/nginx/access.log
   ```

2. Verify Fluent Bit can read the file:
   ```bash
   docker logs fluent-bit
   ```

3. Test LogTide connectivity:
   ```bash
   curl -X POST https://YOUR_LOGTIDE_HOST/api/v1/ingest \
     -H "X-API-Key: YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"logs":[{"service":"test","level":"info","message":"test"}]}'
   ```

### JSON parse errors

If you see parse errors, check that nginx JSON escaping is enabled:

```nginx
log_format json_combined escape=json ...
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log collection
- [Node.js SDK](/integrations/nodejs/) - Application logging with request correlation
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging setup


---

# Node.js SDK Integration

> Send structured logs from Node.js applications to LogTide with automatic batching, retries, and console interception.

*Source: https://logtide.dev/integrations/nodejs/*

---

The LogTide JavaScript SDK (`@logtide/core`) provides structured logging with DSN-based configuration, automatic batching, distributed tracing (W3C Trace Context), breadcrumbs, and scopes.

## Why use the LogTide JavaScript SDK?

- **DSN-based config**: Single connection string for all settings
- **Distributed tracing**: W3C Trace Context propagation across services
- **Breadcrumbs**: Trail of events leading up to errors
- **Scopes**: Per-request context isolation
- **Console interception**: Capture `console.log()` calls automatically
- **TypeScript**: Full type definitions included

## Prerequisites

- Node.js 18+ (LTS recommended)
- npm, yarn, or pnpm
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/core
```

For framework-specific integration, install the corresponding package instead:

```bash
npm install @logtide/express    # Express.js
npm install @logtide/fastify    # Fastify
npm install @logtide/nextjs     # Next.js
npm install @logtide/nuxt       # Nuxt
npm install @logtide/sveltekit  # SvelteKit
npm install @logtide/hono       # Hono
npm install @logtide/angular    # Angular
npm install @logtide/elysia     # Elysia
```

## Quick Start (5 minutes)

### Basic Setup

```typescript
import { hub } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
  environment: process.env.NODE_ENV,
  release: '1.0.0',
});

// Capture logs
hub.captureLog('info', 'Application started', {
  version: '1.0.0',
  environment: process.env.NODE_ENV,
});

// Different log levels
hub.captureLog('debug', 'Debug information');
hub.captureLog('info', 'User logged in', { userId: '123' });
hub.captureLog('warn', 'Rate limit approaching', { current: 90, max: 100 });
hub.captureLog('error', 'Failed to process payment', { orderId: '456' });
hub.captureLog('critical', 'Database connection lost');

// Graceful shutdown - flush remaining logs
process.on('SIGTERM', async () => {
  await hub.close();
  process.exit(0);
});
```

### Environment Variables

Store your DSN in an environment variable:

```bash
# .env
LOGTIDE_DSN=https://lp_abc123@api.logtide.dev
```

## Configuration Options

```typescript
hub.init({
  // Required
  dsn: process.env.LOGTIDE_DSN,

  // Recommended
  service: 'api-server',
  environment: process.env.NODE_ENV,
  release: process.env.npm_package_version,

  // Batching (optional)
  batchSize: 100,        // Flush after N logs (default: 100)
  flushInterval: 5000,   // Flush every N ms (default: 5000)

  // Reliability (optional)
  maxRetries: 3,
  circuitBreakerThreshold: 5,
  circuitBreakerResetMs: 30000,

  // Tracing (optional)
  tracesSampleRate: 1.0,

  // Integrations (optional)
  integrations: [
    new ConsoleIntegration(),
    new GlobalErrorIntegration(),
  ],
});
```

## Express.js Integration

For Express apps, use the dedicated `@logtide/express` package:

```typescript
import express from 'express';
import { hub } from '@logtide/core';
import { logtide } from '@logtide/express';

const app = express();
app.use(logtide({
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
}));

app.get('/users/:id', (req, res) => {
  req.logtideScope.setTag('userId', req.params.id);
  res.json({ id: req.params.id });
});
```

See the [Express integration guide](/integrations/express/) for full details.

## Fastify Integration

For Fastify apps, use the dedicated `@logtide/fastify` package:

```typescript
import Fastify from 'fastify';
import { logtide } from '@logtide/fastify';

const fastify = Fastify();
await fastify.register(logtide, {
  dsn: process.env.LOGTIDE_DSN,
  service: 'api-server',
});

fastify.get('/users/:id', async (request) => {
  request.logtideScope.setTag('userId', request.params.id);
  return { id: request.params.id };
});
```

See the [Fastify integration guide](/integrations/fastify/) for full details.

## Error Capture

```typescript
try {
  await processPayment(orderId);
} catch (error) {
  hub.captureError(error, {
    extra: { orderId },
    tags: { module: 'payments' },
  });
}
```

## Scopes & Breadcrumbs

```typescript
// Add breadcrumbs for debugging context
hub.addBreadcrumb({
  type: 'custom',
  category: 'auth',
  message: 'User authenticated',
  level: 'info',
  timestamp: Date.now(),
});

// Framework SDKs create per-request scopes automatically.
// For manual scopes:
const client = hub.getClient();
const scope = client.createScope();
scope.setTag('handler', 'payment');
scope.setExtra('userId', 'user-123');
client.captureError(new Error('Payment failed'), {}, scope);
```

## Console Interception

```typescript
import { hub, ConsoleIntegration } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  integrations: [
    new ConsoleIntegration(),
  ],
});

// Automatically captured:
console.warn('Disk space low');
console.error('Connection timeout');
```

## Production Best Practices

### 1. Always Close on Shutdown

```typescript
const shutdown = async () => {
  await hub.close();
  process.exit(0);
};

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
```

### 2. Set Environment and Release

```typescript
hub.init({
  dsn: process.env.LOGTIDE_DSN,
  environment: process.env.NODE_ENV,
  release: process.env.npm_package_version,
});
```

### 3. Don't Log Sensitive Data

```typescript
// Avoid logging sensitive data in metadata
hub.captureLog('info', 'User action', {
  userId: user.id,
  // Don't include: passwords, tokens, credit cards
});
```

### 4. Use Appropriate Log Levels

| Level | Use Case |
|-------|----------|
| `debug` | Detailed debugging info (disabled in production) |
| `info` | Normal operations: requests, user actions |
| `warn` | Recoverable issues: rate limits, retries |
| `error` | Failures that need attention |
| `critical` | System failures requiring immediate action |

## Performance

| Metric | Value |
|--------|-------|
| Memory overhead | ~5MB |
| Latency (batched) | <1ms per log |
| Network calls | 1 per batch (100 logs default) |
| CPU impact | <0.1% |

## Framework Packages

| Package | Framework |
|---------|-----------|
| [`@logtide/express`](/integrations/express/) | Express.js |
| [`@logtide/fastify`](/integrations/fastify/) | Fastify |
| [`@logtide/nextjs`](/integrations/nextjs/) | Next.js |
| [`@logtide/nuxt`](/integrations/nuxt/) | Nuxt |
| [`@logtide/sveltekit`](/integrations/sveltekit/) | SvelteKit |
| [`@logtide/hono`](/integrations/hono/) | Hono |
| [`@logtide/angular`](/integrations/angular/) | Angular |
| [`@logtide/elysia`](/integrations/elysia/) | Elysia |

## Next Steps

- [JavaScript SDK Reference](/docs/sdks/nodejs/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Containerized deployments
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging


---

# Nuxt Application Logging Integration

> Add structured logging to Nuxt applications with a zero-config module, runtime config injection, and server/client support.

*Source: https://logtide.dev/integrations/nuxt/*

---

LogTide's Nuxt module provides zero-config structured logging for Nuxt applications with automatic server and client initialization, runtime config injection, and Vue composables.

## Why use LogTide with Nuxt?

- **Zero-config**: Add module to nuxt.config.ts and it works
- **Runtime config**: DSN injected via Nuxt runtime config, overridable at deploy time
- **Server + client**: Auto-initializes on both sides
- **Composable API**: `useLogtide()` composable for Vue components
- **Error capture**: Automatic on server and client

## Prerequisites

- Nuxt 3.x
- Node.js 18+
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/nuxt
```

## Quick Start

```typescript
// nuxt.config.ts
export default defineNuxtConfig({
  modules: ['@logtide/nuxt'],

  logtide: {
    dsn: process.env.LOGTIDE_DSN,
    service: 'nuxt-app',
    environment: process.env.NODE_ENV,
  },
});
```

That's all you need. The module handles initialization, error capture, and provides composables.

## Configuration

```typescript
// nuxt.config.ts
export default defineNuxtConfig({
  modules: ['@logtide/nuxt'],

  logtide: {
    dsn: process.env.LOGTIDE_DSN,
    // Or, instead of a dsn:
    // apiUrl: process.env.LOGTIDE_API_URL,
    // apiKey: process.env.LOGTIDE_API_KEY,

    service: 'nuxt-app',
    environment: process.env.NODE_ENV,
    release: '1.0.0',

    // Tracing
    tracesSampleRate: 1.0,
  },
});
```

The same configuration is applied to both the server (Nitro) and the client (Vue) — there is no separate client/server toggle.

## Server-Side Usage

```typescript
// server/api/users/[id].get.ts
import { hub } from '@logtide/core';

export default defineEventHandler(async (event) => {
  const id = getRouterParam(event, 'id');

  hub.captureLog('info', 'Fetching user', { userId: id });

  const user = await getUserById(id);
  return user;
});
```

## Client-Side Usage

```vue
<script setup>
import { useLogtide } from '#imports';

const logtide = useLogtide();

async function handleSubmit() {
  logtide.addBreadcrumb({
    type: 'ui',
    category: 'ui',
    message: 'Form submitted',
    timestamp: Date.now(),
  });

  try {
    await $fetch('/api/submit', { method: 'POST', body: formData });
    logtide.captureLog('info', 'Form submitted');
  } catch (error) {
    logtide.captureError(error);
  }
}
</script>
```

## Environment Variables

Override configuration at deploy time:

```bash
LOGTIDE_DSN=https://lp_abc123@api.logtide.dev
NUXT_PUBLIC_LOGTIDE_DSN=https://lp_client456@api.logtide.dev
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [Nuxt SDK Reference](/docs/sdks/nuxt-sdk/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# PHP Logging Integration

> Send structured logs from PHP applications to LogTide using the official SDK with Hub/Scope architecture, tracing, and Monolog handlers.

*Source: https://logtide.dev/integrations/php/*

---

The LogTide PHP SDK provides structured logging for any PHP application, whether you're using a framework or plain PHP. With DSN-based configuration, Hub/Scope architecture, and built-in Monolog handlers, it fits naturally into your existing logging setup.

## Why send PHP logs to LogTide?

- **Hub/Scope architecture**: Per-request context isolation with tags, breadcrumbs, and user data
- **Built-in Monolog handlers**: Drop-in `LogtideHandler` and `BreadcrumbHandler` for existing Monolog setups
- **Distributed tracing**: W3C Trace Context (traceparent) propagation across services
- **Automatic batching**: Reduces network overhead by batching log entries
- **Zero framework lock-in**: Works with plain PHP, Laravel, Symfony, Slim, or WordPress
- **Privacy-first**: Self-hosted option keeps logs in your infrastructure

## Prerequisites

- PHP 8.1 or higher
- Composer for dependency management
- LogTide instance (self-hosted or cloud)
- DSN from your LogTide project settings

## Installation

Install the LogTide PHP SDK via Composer:

```bash
composer require logtide/logtide
```

For framework-specific integrations, install the corresponding package instead:

```bash
composer require logtide/logtide-laravel    # Laravel 10/11/12
composer require logtide/logtide-symfony    # Symfony 6.4/7.x
composer require logtide/logtide-slim       # Slim 4
composer require logtide/logtide-wordpress  # WordPress
```

## Quick Start (5 minutes)

### Basic Setup

The simplest way to start logging:

```php
<?php

require_once __DIR__ . '/vendor/autoload.php';

use function LogTide\{init, info, error, warn, debug, critical, flush};

// Initialize the SDK with your DSN
init([
    'dsn' => getenv('LOGTIDE_DSN') ?: 'https://lp_your_key@api.logtide.dev',
    'service' => 'my-php-app',
    'environment' => getenv('APP_ENV') ?: 'production',
]);

// Send logs at different levels
info('Application started', [
    'php_version' => PHP_VERSION,
    'environment' => getenv('APP_ENV') ?: 'production',
]);

debug('Processing request', ['uri' => $_SERVER['REQUEST_URI'] ?? '/']);
warn('High memory usage', ['memory_mb' => memory_get_usage(true) / 1024 / 1024]);
error('Failed to connect to external service', ['service' => 'payment-gateway']);

// Shutdown is auto-handled via register_shutdown_function
```

### Environment Variables

Store your credentials securely:

```bash
# .env
LOGTIDE_DSN=https://lp_your_key@api.logtide.dev
APP_ENV=production
```

Load with `vlucas/phpdotenv` or your framework's environment handling.

## Configuration Options

```php
<?php

use function LogTide\init;

init([
    // Required
    'dsn' => 'https://lp_your_key@api.logtide.dev',

    // Recommended
    'service' => 'my-php-app',
    'environment' => 'production',
    'release' => '1.2.3',

    // Batching
    'batch_size' => 100,              // Logs per batch (default: 100)
    'flush_interval' => 5000,         // Flush interval in ms (default: 5000)
    'max_buffer_size' => 10000,       // Max logs in buffer (default: 10000)

    // Reliability
    'max_retries' => 3,               // Retry attempts on failure (default: 3)
    'retry_delay_ms' => 1000,         // Initial retry delay in ms (default: 1000)
    'circuit_breaker_threshold' => 5, // Failures before circuit opens (default: 5)

    // Tracing
    'traces_sample_rate' => 1.0,      // 0.0-1.0, sample rate for traces

    // Debug
    'debug' => false,
]);
```

## Monolog Integration

The SDK includes built-in Monolog handlers. No separate package needed:

```php
<?php

use Monolog\Logger;
use Monolog\Level;
use LogTide\Monolog\LogtideHandler;
use LogTide\Monolog\BreadcrumbHandler;

// Initialize the SDK first
\LogTide\init(['dsn' => getenv('LOGTIDE_DSN')]);

$logger = new Logger('my-app');

// Low-level logs become breadcrumbs (trail of events)
$logger->pushHandler(new BreadcrumbHandler(Level::Debug));

// Important logs are sent as events to LogTide
$logger->pushHandler(new LogtideHandler(Level::Warning));

// Use standard Monolog methods
$logger->info('Cache hit', ['key' => 'user:123']);       // → breadcrumb
$logger->error('Payment failed', ['order_id' => 456]);   // → LogTide event
```

### Combining with Other Handlers

```php
<?php

use Monolog\Logger;
use Monolog\Handler\StreamHandler;
use Monolog\Handler\RotatingFileHandler;
use LogTide\Monolog\LogtideHandler;

$logger = new Logger('my-app');

// Local file logging (for debugging)
if (getenv('APP_ENV') === 'development') {
    $logger->pushHandler(new StreamHandler('php://stdout', Level::Debug));
}

// Rotating file for persistent local logs
$logger->pushHandler(new RotatingFileHandler(
    '/var/log/myapp/app.log',
    14, // Keep 14 days
    Level::Warning
));

// LogTide for centralized logging
$logger->pushHandler(new LogtideHandler(Level::Info));
```

## Scopes & Context

Track requests across your application with scoped context:

```php
<?php

use function LogTide\{withScope, configureScope, info, captureException};

// Set global context
configureScope(function (\LogTide\State\Scope $scope): void {
    $scope->setTag('region', 'eu-west-1');
    $scope->setUser(['id' => $_SESSION['user_id'] ?? null]);
});

// Isolated scope for a block of code
withScope(function () {
    \LogTide\LogtideSdk::getCurrentHub()->getScope()->setTag('handler', 'checkout');

    info('Processing order', ['order_id' => $orderId]);
    // This event includes the 'handler' tag
});

info('Back to global scope');
// This event does NOT include the 'handler' tag
```

## Distributed Tracing

Propagate trace context across services using W3C traceparent:

```php
<?php

use function LogTide\{continueTrace, getTraceparent, startSpan, finishSpan};

// Continue a trace from an incoming request
$traceparent = $_SERVER['HTTP_TRACEPARENT'] ?? '';
if (!empty($traceparent)) {
    continueTrace($traceparent);
}

// Create spans for operations
$span = startSpan('db.query', ['kind' => \LogTide\Enum\SpanKind::CLIENT]);
// ... do work ...
if ($span !== null) {
    finishSpan($span);
}

// Propagate to outgoing requests
$outgoingTraceparent = getTraceparent();
```

## Docker Setup

### PHP-FPM Container

```dockerfile
# Dockerfile
FROM php:8.3-fpm-alpine

# Install Composer
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer

# Install dependencies
WORKDIR /var/www/html
COPY composer.json composer.lock ./
RUN composer install --no-dev --optimize-autoloader

# Copy application
COPY . .

EXPOSE 9000
CMD ["php-fpm"]
```

### Docker Compose with nginx

```yaml
# docker-compose.yml
services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./public:/var/www/html/public:ro
    depends_on:
      - php

  php:
    build: .
    environment:
      - LOGTIDE_DSN=${LOGTIDE_DSN}
      - APP_ENV=production
    volumes:
      - ./:/var/www/html
```

## Verification

Check logs are arriving in LogTide:

```php
<?php
// test-logging.php
require_once __DIR__ . '/vendor/autoload.php';

\LogTide\init([
    'dsn' => getenv('LOGTIDE_DSN'),
    'service' => 'test',
]);

\LogTide\info('Test log from PHP', [
    'timestamp' => date('c'),
    'php_version' => PHP_VERSION,
    'test' => true,
]);

\LogTide\flush();

echo "Log sent! Check LogTide dashboard.\n";
```

Run:

```bash
LOGTIDE_DSN=https://lp_your_key@api.logtide.dev php test-logging.php
```

## Troubleshooting

### Logs not appearing in LogTide

1. **Check DSN:**
```php
\LogTide\init(['dsn' => '...', 'debug' => true]);
// Look for error messages in stdout
```

2. **Verify network connectivity:**
```bash
curl -I https://api.logtide.dev/health
```

3. **Flush is automatic** &mdash; the SDK registers a shutdown function. For long-running processes, call `\LogTide\flush()` manually.

### Memory issues with high volume

Reduce buffer size:

```php
\LogTide\init([
    'dsn' => '...',
    'batch_size' => 50,
    'max_buffer_size' => 5000,
]);
```

## Next Steps

- **[Laravel Integration](/docs/sdks/laravel/)** - Auto-discovery, Facade, log channel, breadcrumbs
- **[Symfony Integration](/docs/sdks/symfony/)** - Bundle with semantic config and event subscribers
- **[Slim Integration](/docs/sdks/slim/)** - PSR-15 middleware for Slim 4
- **[WordPress Integration](/docs/sdks/wordpress/)** - WordPress hooks and database monitoring
- **[Docker Setup](/integrations/docker/)** - Container deployment best practices
- **[GDPR Compliance](/use-cases/gdpr-compliance/)** - Privacy-first logging implementation


---

# PostgreSQL Log Analysis Integration

> Monitor PostgreSQL queries, slow logs, and errors with LogTide. Complete setup for Docker, Kubernetes, and bare metal.

*Source: https://logtide.dev/integrations/postgresql/*

---

PostgreSQL generates valuable operational data: slow queries, connection errors, and checkpoint statistics. This guide shows you how to collect PostgreSQL logs and ship them to LogTide for analysis and alerting.

## Why monitor PostgreSQL with LogTide?

- **Slow query detection**: Find queries that degrade performance
- **Error alerting**: Get notified about connection failures and crashes
- **Audit logging**: Track who did what for compliance
- **Correlation**: Link database issues with application errors
- **Historical analysis**: Understand patterns over time

## Prerequisites

- PostgreSQL 12+ (14+ recommended)
- LogTide instance with API key
- Fluent Bit or Vector for log shipping

## Quick Start

### 1. Configure PostgreSQL Logging

Edit `postgresql.conf`:

```ini
# Enable logging
logging_collector = on
log_destination = 'csvlog'
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_rotation_age = 1d
log_rotation_size = 100MB

# Log format (CSV for structured parsing)
log_min_messages = warning
log_min_error_statement = error
log_min_duration_statement = 500  # Log queries slower than 500ms

# What to log
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 0
log_autovacuum_min_duration = 0

# Query logging
log_statement = 'ddl'  # none, ddl, mod, all
log_duration = off     # Set to on for all query durations

# Error verbosity
log_error_verbosity = default

# Line prefix for non-CSV logs
log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h '
```

Reload configuration:

```bash
sudo -u postgres psql -c "SELECT pg_reload_conf();"
```

### 2. Set Up Fluent Bit

Create `/etc/fluent-bit/fluent-bit.conf`:

```ini
[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Tag           postgresql
    Path          /var/lib/postgresql/*/data/pg_log/*.csv
    Parser        postgresql_csv
    DB            /var/log/flb_postgresql.db
    Refresh_Interval 10

[FILTER]
    Name          modify
    Match         postgresql
    Add           service postgresql

[FILTER]
    Name          lua
    Match         postgresql
    script        /etc/fluent-bit/parse_pg.lua
    call          parse_postgresql

[OUTPUT]
    Name          http
    Match         postgresql
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json_lines
    Header        X-API-Key YOUR_API_KEY
    Header        Content-Type application/x-ndjson
    Json_date_key time
    Json_date_format iso8601
    tls           On
```

Create `/etc/fluent-bit/parsers.conf`:

```ini
[PARSER]
    Name        postgresql_csv
    Format      regex
    Regex       ^(?<log_time>[^,]*),(?<user_name>[^,]*),(?<database_name>[^,]*),(?<process_id>[^,]*),(?<connection_from>[^,]*),(?<session_id>[^,]*),(?<session_line_num>[^,]*),(?<command_tag>[^,]*),(?<session_start_time>[^,]*),(?<virtual_transaction_id>[^,]*),(?<transaction_id>[^,]*),(?<error_severity>[^,]*),(?<sql_state_code>[^,]*),(?<message>[^,]*),(?<detail>[^,]*),(?<hint>[^,]*),(?<internal_query>[^,]*),(?<internal_query_pos>[^,]*),(?<context>[^,]*),(?<query>[^,]*),(?<query_pos>[^,]*),(?<location>[^,]*),(?<application_name>[^,]*)
    Time_Key    log_time
    Time_Format %Y-%m-%d %H:%M:%S.%L %z
```

Create `/etc/fluent-bit/parse_pg.lua`:

```lua
function parse_postgresql(tag, timestamp, record)
    -- Parse duration from message
    local duration_match = string.match(record["message"] or "", "duration: ([%d.]+) ms")
    if duration_match then
        record["duration_ms"] = tonumber(duration_match)
    end

    -- Determine log level from error_severity
    local severity = record["error_severity"] or ""
    if severity == "ERROR" or severity == "FATAL" or severity == "PANIC" then
        record["level"] = "error"
    elseif severity == "WARNING" then
        record["level"] = "warn"
    else
        record["level"] = "info"
    end

    -- Clean up empty fields
    for k, v in pairs(record) do
        if v == "" then
            record[k] = nil
        end
    end

    return 1, timestamp, record
end
```

### 3. Start Fluent Bit

```bash
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit
```

## Slow Query Monitoring

### Configuration for Slow Queries

```ini
# postgresql.conf
log_min_duration_statement = 500  # Log queries > 500ms
log_statement = 'none'            # Don't log all statements
```

### Analyzing Slow Queries in LogTide

```
service:postgresql AND duration_ms:>1000
```

### Alert on Slow Queries

Create an alert rule:
- Filter: `service:postgresql AND duration_ms:>5000`
- Threshold: >5 queries in 5 minutes
- Severity: Warning

## Connection Monitoring

### Track Connection Events

```ini
# postgresql.conf
log_connections = on
log_disconnections = on
```

Example log:

```json
{
  "service": "postgresql",
  "level": "info",
  "message": "connection authorized: user=app database=production",
  "user_name": "app",
  "database_name": "production",
  "connection_from": "192.168.1.100",
  "application_name": "api-server"
}
```

### Alert on Connection Failures

```
service:postgresql AND level:error AND message:*connection*
```

## Docker Setup

### docker-compose.yml

```yaml
version: "3.8"

services:
  postgres:
    image: postgres:16-alpine
    container_name: postgres
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: production
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./postgresql.conf:/etc/postgresql/postgresql.conf:ro
      - postgres_logs:/var/lib/postgresql/data/pg_log
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 10s
      timeout: 5s
      retries: 5

  fluent-bit:
    image: fluent/fluent-bit:latest
    container_name: fluent-bit-postgres
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro
      - ./fluent-bit/parse_pg.lua:/etc/fluent-bit/parse_pg.lua:ro
      - postgres_logs:/var/log/postgresql:ro
    depends_on:
      - postgres
    restart: unless-stopped

volumes:
  postgres_data:
  postgres_logs:
```

### postgresql.conf for Docker

```ini
# Logging configuration for Docker
listen_addresses = '*'
logging_collector = on
log_destination = 'csvlog'
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d.log'

# Slow queries
log_min_duration_statement = 500
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on

# Performance
shared_buffers = 256MB
effective_cache_size = 768MB
maintenance_work_mem = 64MB
```

## Kubernetes Setup

### ConfigMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
data:
  postgresql.conf: |
    listen_addresses = '*'
    logging_collector = on
    log_destination = 'csvlog'
    log_directory = 'pg_log'
    log_filename = 'postgresql-%Y-%m-%d.log'
    log_min_duration_statement = 500
    log_checkpoints = on
    log_connections = on
    log_disconnections = on
    log_lock_waits = on
```

### StatefulSet with Sidecar

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: config
              mountPath: /etc/postgresql/postgresql.conf
              subPath: postgresql.conf
            - name: logs
              mountPath: /var/lib/postgresql/data/pg_log

        - name: fluent-bit
          image: fluent/fluent-bit:latest
          volumeMounts:
            - name: logs
              mountPath: /var/log/postgresql
              readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
          env:
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: api-key

      volumes:
        - name: config
          configMap:
            name: postgres-config
        - name: logs
          emptyDir: {}
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-postgres-config

  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
```

## pgAudit for Compliance

For detailed audit logging, install pgAudit:

### Install Extension

```sql
CREATE EXTENSION pgaudit;
```

### Configure

```ini
# postgresql.conf
shared_preload_libraries = 'pgaudit'
pgaudit.log = 'ddl, role, write'
pgaudit.log_catalog = off
pgaudit.log_client = on
pgaudit.log_level = log
```

### Audit Log Output

```json
{
  "service": "postgresql",
  "level": "info",
  "message": "AUDIT: SESSION,1,1,DDL,CREATE TABLE,TABLE,public.users,CREATE TABLE users (id serial PRIMARY KEY)",
  "user_name": "admin",
  "database_name": "production"
}
```

## Detection Rules

### High Error Rate

```
service:postgresql AND level:error
```
Threshold: >10 errors in 5 minutes

### Very Slow Queries

```
service:postgresql AND duration_ms:>10000
```
Alert on any query >10 seconds

### Connection Exhaustion

```
service:postgresql AND message:"too many connections"
```

### Deadlocks

```
service:postgresql AND message:*deadlock*
```

### Checkpoint Warnings

```
service:postgresql AND message:"checkpoints are occurring too frequently"
```

## Performance Impact

| Configuration | CPU Overhead | I/O Overhead |
|---------------|--------------|--------------|
| Basic logging | <1% | ~5% |
| Slow query (500ms) | <1% | ~2% |
| All statements | 5-10% | 20-30% |
| pgAudit (DDL+write) | 2-5% | 10-15% |

## Troubleshooting

### Logs not appearing

1. Check logging is enabled:
   ```sql
   SHOW logging_collector;
   SHOW log_directory;
   SHOW log_destination;
   ```

2. Verify log files exist:
   ```bash
   ls -la /var/lib/postgresql/*/data/pg_log/
   ```

3. Check Fluent Bit can read files:
   ```bash
   docker logs fluent-bit-postgres
   ```

### CSV parsing errors

Ensure `log_destination = 'csvlog'` is set. Standard log format won't parse correctly with the CSV parser.

### Missing slow queries

Check `log_min_duration_statement`:
```sql
SHOW log_min_duration_statement;
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log collection
- [Node.js SDK](/integrations/nodejs/) - Application logging
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant setup


---

# Python SDK Integration

> Send structured logs from Python applications to LogTide using the official SDK with sync & async clients, stdlib logging integration, and middleware.

*Source: https://logtide.dev/integrations/python/*

---

The LogTide Python SDK provides structured logging with automatic batching, sync and async clients, stdlib `logging` integration, payload limits, and middleware for Flask, Django, FastAPI, and Starlette.

## Why use the LogTide Python SDK?

- **Sync & async**: `LogTideClient` (requests) and `AsyncLogTideClient` (aiohttp) for every use case
- **stdlib `logging` integration**: Drop in `LogTideHandler` to any existing logging setup
- **Automatic batching**: Reduces network overhead with intelligent batching and configurable flush interval
- **Circuit breaker**: Prevents app crashes if LogTide is unreachable — logs are silently dropped, never blocking
- **Payload limits**: Truncate oversized fields, remove base64 blobs, exclude sensitive fields
- **Trace correlation**: Context managers for request-scoped trace IDs
- **Type hints**: Full typing support for Python 3.8+

## Prerequisites

- Python 3.8+
- pip or poetry
- LogTide instance with API key

## Installation

```bash
pip install logtide-sdk
```

With optional extras:

```bash
# Async client (AsyncLogTideClient)
pip install logtide-sdk[async]

# Flask middleware
pip install logtide-sdk[flask]

# Django middleware
pip install logtide-sdk[django]

# FastAPI middleware
pip install logtide-sdk[fastapi]

# Starlette middleware (standalone)
pip install logtide-sdk[starlette]
```

## Quick Start

```python
import os
from logtide_sdk import LogTideClient, ClientOptions

client = LogTideClient(
    ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
    )
)

# Send logs — args: service, message, metadata (dict or Exception)
client.info("api-gateway", "Server started", {"port": 3000})
client.error("database", "Connection failed", Exception("Timeout"))

# Graceful shutdown — also registered automatically via atexit
client.close()
```

### Environment Variables

```bash
# .env
LOGTIDE_API_URL=http://localhost:8080
LOGTIDE_API_KEY=lp_your_api_key_here
```

## Configuration Options

```python
import os
from logtide_sdk import LogTideClient, ClientOptions, PayloadLimitsOptions

client = LogTideClient(
    ClientOptions(
        # Required
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],

        # Performance
        batch_size=100,              # Flush after N logs (default: 100)
        flush_interval=5000,         # Auto-flush interval in ms (default: 5000)
        max_buffer_size=10000,       # Max buffered logs; excess silently dropped (default: 10000)

        # Reliability
        max_retries=3,               # Retry failed requests (default: 3)
        retry_delay_ms=1000,         # Initial retry delay in ms, doubles each attempt (default: 1000)
        circuit_breaker_threshold=5, # Open circuit after N consecutive failures (default: 5)
        circuit_breaker_reset_ms=30000, # Try half-open after N ms (default: 30000)

        # Metadata merged into every log
        global_metadata={
            "environment": os.environ.get("APP_ENV", "production"),
            "version": os.environ.get("APP_VERSION", "unknown"),
        },

        # Auto-generate a UUID trace ID for every log that has none (default: False)
        auto_trace_id=False,

        # Disable internal metrics collection if not needed (default: True)
        enable_metrics=True,

        # Payload safety
        payload_limits=PayloadLimitsOptions(
            max_field_size=10 * 1024,        # Truncate strings > 10 KB
            exclude_fields=["password"],     # Replace value with "[EXCLUDED]"
        ),

        debug=False,
    )
)
```

## Logging Methods

```python
# Log levels: debug, info, warn, error, critical
# Signature: client.<level>(service, message, metadata=None)

client.debug("worker", "Processing item", {"item_id": 42})
client.info("api-gateway", "Request received", {"method": "GET", "path": "/users"})
client.warn("cache", "Cache miss", {"key": "user:123"})
client.error("database", "Query failed", {"query": "SELECT *"})
client.critical("system", "Out of memory", {"used": "95%"})
```

### Exception Auto-Serialization

Pass an `Exception` directly as metadata — it is serialized automatically with full stack frames:

```python
try:
    raise RuntimeError("Database timeout")
except Exception as e:
    client.error("database", "Query failed", e)

# Produces metadata:
# {
#   "exception": {
#     "type": "RuntimeError",
#     "message": "Database timeout",
#     "language": "python",
#     "stacktrace": [{"file": "app.py", "function": "run_query", "line": 42}],
#     "raw": "Traceback (most recent call last): ..."
#   }
# }
```

## Async Client

Install aiohttp first: `pip install logtide-sdk[async]`

```python
import asyncio
import os
from logtide_sdk import AsyncLogTideClient, ClientOptions

async def main():
    async with AsyncLogTideClient(ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
    )) as client:
        await client.info("my-service", "Hello from async!")
        await client.error("my-service", "Something failed", Exception("oops"))

asyncio.run(main())
```

Manual lifecycle (without context manager):

```python
client = AsyncLogTideClient(options)
await client.start()   # starts background flush loop
try:
    await client.info("svc", "message")
finally:
    await client.close()
```

## stdlib `logging` Integration

`LogTideHandler` is a standard `logging.Handler` — drop it into any existing setup:

```python
import logging
import os
from logtide_sdk import LogTideClient, ClientOptions, LogTideHandler

client = LogTideClient(ClientOptions(
    api_url=os.environ["LOGTIDE_API_URL"],
    api_key=os.environ["LOGTIDE_API_KEY"],
))

handler = LogTideHandler(client=client, service="my-service")
handler.setLevel(logging.WARNING)

logger = logging.getLogger(__name__)
logger.addHandler(handler)

# These are forwarded to LogTide automatically
logger.warning("Low disk space")
logger.error("Unhandled exception", exc_info=True)
```

Exception info is serialized with full structured stack frames when `exc_info=True` is used.

## Trace ID Context

### Manual Trace ID

```python
client.set_trace_id("request-123")

client.info("api", "Request received")
client.info("db", "Querying users")
client.info("api", "Response sent")

client.set_trace_id(None)  # clear
```

### Scoped Trace ID (Context Manager)

```python
with client.with_trace_id("request-456"):
    client.info("api", "Processing in context")
    client.warn("cache", "Cache miss")
# Trace ID automatically restored after block
```

### Auto-Generated Trace ID

```python
with client.with_new_trace_id():
    client.info("worker", "Background job started")
    client.info("worker", "Job completed")
```

## Payload Limits

```python
from logtide_sdk import PayloadLimitsOptions

# Configure in ClientOptions
payload_limits=PayloadLimitsOptions(
    max_field_size=5 * 1024,           # Truncate strings longer than 5 KB
    max_log_size=100 * 1024,           # Drop entries larger than 100 KB
    exclude_fields=["password", "token"],  # Replace with "[EXCLUDED]"
    truncation_marker="...[TRUNCATED]", # Appended to truncated strings
)

# Base64 strings (data URIs, long blobs) are replaced automatically with:
# "[BASE64 DATA REMOVED]"
```

## Internal Metrics

Monitor the health of the SDK at runtime:

```python
import threading
import time

def monitor():
    while True:
        m = client.get_metrics()
        if m.logs_dropped > 0:
            print(f"WARNING: {m.logs_dropped} logs dropped (buffer full)")
        if m.circuit_breaker_trips > 0:
            print("ERROR: Circuit breaker tripped")
        time.sleep(60)

threading.Thread(target=monitor, daemon=True).start()

# Get current circuit breaker state
print(client.get_circuit_breaker_state())  # CLOSED | OPEN | HALF_OPEN

# Available metrics
m = client.get_metrics()
print(m.logs_sent, m.logs_dropped, m.errors, m.retries, m.avg_latency_ms, m.circuit_breaker_trips)
client.reset_metrics()
```

## Production Best Practices

### 1. Always Flush on Shutdown

`client.close()` is registered automatically via `atexit`, but call it explicitly in signal handlers:

```python
import signal
import sys

def shutdown_handler(signum, frame):
    client.close()
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown_handler)
signal.signal(signal.SIGINT, shutdown_handler)
```

### 2. Use Environment-Specific Configuration

```python
import os

environment = os.environ.get("APP_ENV", "development")

client = LogTideClient(
    ClientOptions(
        api_url=os.environ["LOGTIDE_API_URL"],
        api_key=os.environ["LOGTIDE_API_KEY"],
        global_metadata={
            "environment": environment,
            "version": os.environ.get("APP_VERSION", "unknown"),
        },
        # Smaller batches in development for faster feedback
        batch_size=10 if environment == "development" else 100,
        debug=(environment == "development"),
    )
)
```

### 3. Don't Log Sensitive Data

Use `exclude_fields` in `PayloadLimitsOptions`, or mask data before logging:

```python
def mask_email(email: str) -> str:
    local, domain = email.split("@")
    return f"{local[0]}***@{domain}"

# Bad
client.info("auth", "User login", {"email": user.email, "password": password})

# Good
client.info("auth", "User login", {"user_id": user.id, "email": mask_email(user.email)})
```

### 4. Use Appropriate Log Levels

| Level | Use Case |
|-------|----------|
| `debug` | Detailed debugging info (disable in production) |
| `info` | Normal operations: requests, user actions |
| `warn` | Recoverable issues: rate limits, retries |
| `error` | Failures needing attention |
| `critical` | System failures requiring immediate action |

## Troubleshooting

### Logs not appearing

1. Enable debug mode to see what is happening:
   ```python
   ClientOptions(..., debug=True)
   ```

2. Ensure shutdown is called before process exit:
   ```python
   client.close()
   ```

3. Check circuit breaker state:
   ```python
   print(client.get_circuit_breaker_state())
   ```

### High memory usage

Reduce batch size or lower the buffer cap:

```python
ClientOptions(
    batch_size=50,
    flush_interval=2000,  # flush every 2s
    max_buffer_size=1000,
)
```

### Async event loop issues

Always use `async with` or call `await client.start()` before logging:

```python
async def main():
    async with AsyncLogTideClient(options) as client:
        await your_app(client)
```

## Next Steps

- [Flask Integration](/integrations/flask/) - Flask middleware and request logging
- [Django Integration](/integrations/django/) - Django middleware and Celery support
- [FastAPI Integration](/integrations/fastapi/) - Async middleware and dependency injection
- [Docker Integration](/integrations/docker/) - Containerized deployments
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy-compliant logging


---

# RabbitMQ Log and Monitoring Integration

> Forward RabbitMQ server logs, queue metrics, and consumer events to LogTide for centralized message broker observability.

*Source: https://logtide.dev/integrations/rabbitmq/*

---

RabbitMQ is a widely used message broker that sits at the heart of event-driven architectures. When queues back up, consumers crash, or messages land in dead letter exchanges, you need centralized visibility. This guide shows you how to forward RabbitMQ server logs, queue metrics, consumer patterns, and dead letter events to LogTide.

## Why use LogTide with RabbitMQ?

- **Queue visibility**: Monitor queue depths, consumer counts, and message rates from a centralized dashboard
- **Dead letter tracking**: Get alerted when messages fail processing and land in dead letter queues
- **Consumer health**: Detect crashed, stuck, or slow consumers before message backlogs grow
- **Cluster monitoring**: Track node health, network partitions, and failover events across your cluster
- **Incident response**: Correlate message broker issues with application errors during outages
- **Audit trail**: Record who published what, when, and which consumers processed each message

## Prerequisites

- RabbitMQ 3.12+ (with management plugin enabled)
- LogTide instance with API key
- Fluent Bit or Vector for log forwarding
- `rabbitmq_management` plugin (for metrics collection)

## Quick Start (10 minutes)

### Step 1: Configure RabbitMQ JSON Logging

RabbitMQ 3.9+ supports structured JSON logging natively. Configure in `rabbitmq.conf`:

```ini
# /etc/rabbitmq/rabbitmq.conf

# Enable JSON log format
log.console = true
log.console.level = info
log.console.formatter = json

# File logging with JSON format
log.file = /var/log/rabbitmq/rabbit.log
log.file.level = info
log.file.formatter = json
log.file.rotation.date = $D0
log.file.rotation.size = 104857600
log.file.rotation.count = 7
```

Or using the advanced config format (`advanced.config`):

```erlang
% /etc/rabbitmq/advanced.config
[
  {rabbit, [
    {log, [
      {file, [{file, "/var/log/rabbitmq/rabbit.log"},
              {level, info},
              {formatter, {rabbit_logger_json_fmt, #{}}},
              {date, "$D0"},
              {size, 104857600},
              {count, 7}]}
    ]}
  ]}
].
```

Restart RabbitMQ:

```bash
sudo systemctl restart rabbitmq-server
```

### Step 2: Set Up Fluent Bit

Create `/etc/fluent-bit/fluent-bit.conf`:

```ini
[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

# RabbitMQ server logs
[INPUT]
    Name          tail
    Path          /var/log/rabbitmq/rabbit.log
    Tag           rabbitmq.server
    Refresh_Interval 5

[FILTER]
    Name          parser
    Match         rabbitmq.server
    Key_Name      log
    Parser        json
    Reserve_Data  On

[FILTER]
    Name          modify
    Match         rabbitmq.*
    Add           service rabbitmq

[FILTER]
    Name          lua
    Match         rabbitmq.server
    script        /etc/fluent-bit/rabbitmq_level.lua
    call          map_level

[OUTPUT]
    Name          http
    Match         rabbitmq.*
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

Create `/etc/fluent-bit/rabbitmq_level.lua`:

```lua
function map_level(tag, timestamp, record)
    local level = record["level"] or "info"
    -- RabbitMQ uses Erlang log levels
    local level_map = {
        emergency = "critical",
        alert = "critical",
        critical = "critical",
        error = "error",
        warning = "warn",
        notice = "info",
        info = "info",
        debug = "debug",
    }
    record["level"] = level_map[level] or "info"
    return 1, timestamp, record
end
```

Start Fluent Bit:

```bash
sudo systemctl enable fluent-bit
sudo systemctl start fluent-bit
```

### Step 3: Verify

```bash
# Generate a log event
rabbitmqctl status

# Check in LogTide with filter: service:rabbitmq
```

## Queue Metrics Collection

### Management API Metrics Collector

Use the RabbitMQ Management HTTP API to collect queue metrics and send them to LogTide:

```python
#!/usr/bin/env python3
# /opt/rabbitmq-metrics/collect.py

import json
import os
import time
import requests
from datetime import datetime, timezone

RABBITMQ_HOST = os.environ.get('RABBITMQ_HOST', 'localhost')
RABBITMQ_PORT = os.environ.get('RABBITMQ_MGMT_PORT', '15672')
RABBITMQ_USER = os.environ.get('RABBITMQ_USER', 'guest')
RABBITMQ_PASS = os.environ.get('RABBITMQ_PASS', 'guest')
LOGTIDE_API_URL = os.environ.get('LOGTIDE_API_URL')
LOGTIDE_API_KEY = os.environ.get('LOGTIDE_API_KEY')

BASE_URL = f"http://{RABBITMQ_HOST}:{RABBITMQ_PORT}/api"
AUTH = (RABBITMQ_USER, RABBITMQ_PASS)


def collect_queue_metrics():
    """Collect metrics from all queues."""
    response = requests.get(f"{BASE_URL}/queues", auth=AUTH)
    response.raise_for_status()
    queues = response.json()

    logs = []
    for queue in queues:
        log = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "service": "rabbitmq",
            "log_type": "queue_metrics",
            "level": "info",
            "message": f"Queue metrics: {queue['name']} - {queue.get('messages', 0)} messages",
            "queue_name": queue["name"],
            "vhost": queue["vhost"],
            "messages": queue.get("messages", 0),
            "messages_ready": queue.get("messages_ready", 0),
            "messages_unacked": queue.get("messages_unacknowledged", 0),
            "consumers": queue.get("consumers", 0),
            "publish_rate": queue.get("message_stats", {}).get("publish_details", {}).get("rate", 0),
            "deliver_rate": queue.get("message_stats", {}).get("deliver_get_details", {}).get("rate", 0),
            "ack_rate": queue.get("message_stats", {}).get("ack_details", {}).get("rate", 0),
            "memory": queue.get("memory", 0),
            "state": queue.get("state", "unknown"),
            "durable": queue.get("durable", False),
        }

        # Set level based on queue health
        if queue.get("messages", 0) > 10000:
            log["level"] = "warn"
            log["message"] = f"Queue backlog: {queue['name']} - {queue['messages']} messages"
        if queue.get("consumers", 0) == 0 and queue.get("messages", 0) > 0:
            log["level"] = "error"
            log["message"] = f"Queue has no consumers: {queue['name']} - {queue['messages']} messages"

        logs.append(log)

    return logs


def collect_node_metrics():
    """Collect cluster node health metrics."""
    response = requests.get(f"{BASE_URL}/nodes", auth=AUTH)
    response.raise_for_status()
    nodes = response.json()

    logs = []
    for node in nodes:
        log = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "service": "rabbitmq",
            "log_type": "node_metrics",
            "level": "info",
            "message": f"Node health: {node['name']} - running={node.get('running', False)}",
            "node_name": node["name"],
            "node_type": node.get("type", "disc"),
            "running": node.get("running", False),
            "mem_used": node.get("mem_used", 0),
            "mem_limit": node.get("mem_limit", 0),
            "mem_alarm": node.get("mem_alarm", False),
            "disk_free": node.get("disk_free", 0),
            "disk_free_limit": node.get("disk_free_limit", 0),
            "disk_free_alarm": node.get("disk_free_alarm", False),
            "fd_used": node.get("fd_used", 0),
            "fd_total": node.get("fd_total", 0),
            "sockets_used": node.get("sockets_used", 0),
            "sockets_total": node.get("sockets_total", 0),
            "uptime": node.get("uptime", 0),
        }

        # Alert on resource issues
        if node.get("mem_alarm", False):
            log["level"] = "critical"
            log["message"] = f"Memory alarm on node: {node['name']}"
        elif node.get("disk_free_alarm", False):
            log["level"] = "critical"
            log["message"] = f"Disk free alarm on node: {node['name']}"
        elif not node.get("running", False):
            log["level"] = "error"
            log["message"] = f"Node not running: {node['name']}"

        logs.append(log)

    return logs


def collect_connection_metrics():
    """Collect connection metrics."""
    response = requests.get(f"{BASE_URL}/connections", auth=AUTH)
    response.raise_for_status()
    connections = response.json()

    log = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "service": "rabbitmq",
        "log_type": "connection_metrics",
        "level": "info",
        "message": f"Active connections: {len(connections)}",
        "total_connections": len(connections),
        "connections_by_user": {},
        "connections_by_vhost": {},
    }

    for conn in connections:
        user = conn.get("user", "unknown")
        vhost = conn.get("vhost", "/")
        log["connections_by_user"][user] = log["connections_by_user"].get(user, 0) + 1
        log["connections_by_vhost"][vhost] = log["connections_by_vhost"].get(vhost, 0) + 1

    return [log]


def ship_to_logtide(logs):
    """Send collected metrics to LogTide."""
    if not logs:
        return

    response = requests.post(
        f"{LOGTIDE_API_URL}/api/v1/ingest",
        headers={
            "Content-Type": "application/json",
            "X-API-Key": LOGTIDE_API_KEY,
        },
        json={"logs": logs},
    )

    if not response.ok:
        print(f"LogTide ingest failed: {response.status_code} {response.text}")


if __name__ == "__main__":
    all_logs = []
    all_logs.extend(collect_queue_metrics())
    all_logs.extend(collect_node_metrics())
    all_logs.extend(collect_connection_metrics())
    ship_to_logtide(all_logs)
    print(f"Shipped {len(all_logs)} metric events to LogTide")
```

### Schedule the Collector

Run every 30 seconds via cron or systemd timer:

```ini
# /etc/systemd/system/rabbitmq-metrics.service
[Unit]
Description=RabbitMQ metrics collector for LogTide
After=network.target rabbitmq-server.service

[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /opt/rabbitmq-metrics/collect.py
EnvironmentFile=/etc/rabbitmq-metrics/environment
```

```ini
# /etc/systemd/system/rabbitmq-metrics.timer
[Unit]
Description=Run RabbitMQ metrics collector every 30 seconds

[Timer]
OnBootSec=10
OnUnitActiveSec=30

[Install]
WantedBy=timers.target
```

```bash
sudo systemctl enable rabbitmq-metrics.timer
sudo systemctl start rabbitmq-metrics.timer
```

## Dead Letter Queue Monitoring

### Configure Dead Letter Exchange

Set up dead letter exchanges so failed messages are routed to a monitored queue:

```bash
# Create the dead letter exchange
rabbitmqadmin declare exchange name=dlx type=direct

# Create the dead letter queue
rabbitmqadmin declare queue name=dead-letters durable=true

# Bind DLQ to the exchange
rabbitmqadmin declare binding source=dlx destination=dead-letters routing_key=dead-letter

# Configure your application queue with DLX
rabbitmqadmin declare queue name=orders \
  durable=true \
  arguments='{"x-dead-letter-exchange":"dlx","x-dead-letter-routing-key":"dead-letter","x-message-ttl":300000}'
```

### Dead Letter Consumer for LogTide

```typescript
// dead-letter-monitor.ts
import amqplib from 'amqplib';

const RABBITMQ_URL = process.env.RABBITMQ_URL || 'amqp://localhost';
const LOGTIDE_API_URL = process.env.LOGTIDE_API_URL!;
const LOGTIDE_API_KEY = process.env.LOGTIDE_API_KEY!;

async function startDeadLetterMonitor() {
  const connection = await amqplib.connect(RABBITMQ_URL);
  const channel = await connection.createChannel();

  // Ensure the dead letter queue exists
  await channel.assertQueue('dead-letters', { durable: true });

  console.log('Dead letter monitor started. Waiting for messages...');

  channel.consume('dead-letters', async (msg) => {
    if (!msg) return;

    const headers = msg.properties.headers || {};
    const deathHistory = headers['x-death'] || [];
    const firstDeath = deathHistory[0] || {};

    const logEvent = {
      timestamp: new Date().toISOString(),
      service: 'rabbitmq',
      log_type: 'dead_letter',
      level: 'error',
      message: `Dead letter from queue: ${firstDeath.queue || 'unknown'}`,
      original_queue: firstDeath.queue,
      original_exchange: firstDeath.exchange,
      death_reason: firstDeath.reason,
      death_count: firstDeath.count || 1,
      routing_key: msg.fields.routingKey,
      content_type: msg.properties.contentType,
      message_id: msg.properties.messageId,
      correlation_id: msg.properties.correlationId,
      app_id: msg.properties.appId,
      payload_size: msg.content.length,
    };

    // Try to parse the message body for additional context
    try {
      const body = JSON.parse(msg.content.toString());
      logEvent.message = `Dead letter: ${firstDeath.reason || 'unknown'} - ${body.type || body.action || 'message'} from ${firstDeath.queue || 'unknown'}`;
    } catch {
      // Binary or non-JSON message
    }

    // Ship to LogTide
    try {
      await fetch(`${LOGTIDE_API_URL}/api/v1/ingest`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'X-API-Key': LOGTIDE_API_KEY,
        },
        body: JSON.stringify({ logs: [logEvent] }),
      });
    } catch (error) {
      console.error('Failed to ship dead letter to LogTide:', error);
    }

    // Acknowledge the dead letter
    channel.ack(msg);
  });

  // Handle connection errors
  connection.on('error', (err) => {
    console.error('RabbitMQ connection error:', err);
    process.exit(1);
  });
}

startDeadLetterMonitor().catch(console.error);
```

## Consumer Logging Patterns

### Node.js Consumer with LogTide

```typescript
// consumer.ts
import amqplib from 'amqplib';
import { hub } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: 'order-consumer',
  environment: process.env.NODE_ENV,
});

async function startConsumer() {
  const connection = await amqplib.connect(process.env.RABBITMQ_URL!);
  const channel = await connection.createChannel();

  // Prefetch for controlled consumption
  await channel.prefetch(10);
  await channel.assertQueue('orders', { durable: true });

  hub.captureLog('info', 'Consumer started', {
    queue: 'orders',
    prefetch: 10,
  });

  channel.consume('orders', async (msg) => {
    if (!msg) return;

    const startTime = Date.now();
    const messageId = msg.properties.messageId || 'unknown';

    hub.captureLog('debug', 'Message received', {
      queue: 'orders',
      messageId,
      contentType: msg.properties.contentType,
      redelivered: msg.fields.redelivered,
    });

    try {
      const body = JSON.parse(msg.content.toString());
      await processOrder(body);

      hub.captureLog('info', `Order processed: ${body.orderId}`, {
        queue: 'orders',
        messageId,
        orderId: body.orderId,
        processingTime: Date.now() - startTime,
      });

      channel.ack(msg);
    } catch (error) {
      const err = error as Error;

      hub.captureLog('error', `Order processing failed: ${err.message}`, {
        queue: 'orders',
        messageId,
        error: err.message,
        processingTime: Date.now() - startTime,
        redelivered: msg.fields.redelivered,
      });

      // Requeue once, then dead letter
      if (!msg.fields.redelivered) {
        channel.nack(msg, false, true);  // Requeue
      } else {
        channel.nack(msg, false, false); // Send to DLX
      }
    }
  });

  // Monitor consumer health
  setInterval(() => {
    hub.captureLog('info', 'Consumer heartbeat', {
      queue: 'orders',
      uptime: process.uptime(),
      memoryUsage: process.memoryUsage().heapUsed,
    });
  }, 60000);
}

async function processOrder(order: any) {
  // Business logic here
}

startConsumer().catch(console.error);
```

## Docker Setup

### docker-compose.yml

```yaml
services:
  rabbitmq:
    image: rabbitmq:3.13-management-alpine
    container_name: rabbitmq
    hostname: rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
      RABBITMQ_DEFAULT_VHOST: /
    ports:
      - "5672:5672"
      - "15672:15672"
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
      - ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf:ro
      - rabbitmq_logs:/var/log/rabbitmq
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "check_running"]
      interval: 30s
      timeout: 10s
      retries: 5

  fluent-bit:
    image: fluent/fluent-bit:latest
    container_name: fluent-bit
    volumes:
      - ./fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
      - ./fluent-bit/rabbitmq_level.lua:/etc/fluent-bit/rabbitmq_level.lua:ro
      - rabbitmq_logs:/var/log/rabbitmq:ro
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: unless-stopped

  rabbitmq-metrics:
    image: python:3.12-slim
    container_name: rabbitmq-metrics
    volumes:
      - ./metrics/collect.py:/opt/collect.py:ro
      - ./metrics/requirements.txt:/opt/requirements.txt:ro
    command: >
      bash -c "pip install -r /opt/requirements.txt &&
               while true; do
                 python /opt/collect.py;
                 sleep 30;
               done"
    environment:
      - RABBITMQ_HOST=rabbitmq
      - RABBITMQ_MGMT_PORT=15672
      - RABBITMQ_USER=admin
      - RABBITMQ_PASS=${RABBITMQ_PASSWORD}
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      rabbitmq:
        condition: service_healthy

  dead-letter-monitor:
    image: node:20-alpine
    container_name: dead-letter-monitor
    volumes:
      - ./dead-letter-monitor:/app
    working_dir: /app
    command: node dead-letter-monitor.js
    environment:
      - RABBITMQ_URL=amqp://admin:${RABBITMQ_PASSWORD}@rabbitmq:5672
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      rabbitmq:
        condition: service_healthy
    restart: unless-stopped

volumes:
  rabbitmq_data:
  rabbitmq_logs:
```

### RabbitMQ Config for Docker

```ini
# rabbitmq.conf
log.console = true
log.console.level = info
log.console.formatter = json

log.file = /var/log/rabbitmq/rabbit.log
log.file.level = info
log.file.formatter = json
log.file.rotation.size = 104857600
log.file.rotation.count = 5

management.tcp.port = 15672
```

## Kubernetes Deployment

### RabbitMQ Cluster Operator

For Kubernetes, the RabbitMQ Cluster Operator is recommended:

```yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
  namespace: messaging
spec:
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: "2"
      memory: 2Gi
  rabbitmq:
    additionalConfig: |
      log.console.level = info
      log.console.formatter = json
  override:
    statefulSet:
      spec:
        template:
          metadata:
            annotations:
              fluentbit.io/parser: json
```

## Detection Rules

Create alerts for common RabbitMQ issues:

### Queue Backlog

```
service:rabbitmq AND log_type:queue_metrics AND messages:>10000
```

Threshold: any occurrence indicates growing backlog.

### No Consumers

```
service:rabbitmq AND log_type:queue_metrics AND consumers:0 AND messages:>0
```

Queues with messages but no consumers.

### Dead Letters

```
service:rabbitmq AND log_type:dead_letter
```

Any occurrence -- failed message processing needs investigation.

### Memory Alarm

```
service:rabbitmq AND mem_alarm:true
```

Critical -- RabbitMQ will block publishing when memory alarm triggers.

### Disk Space Alarm

```
service:rabbitmq AND disk_free_alarm:true
```

Critical -- RabbitMQ will block publishing when disk space is low.

### Node Down

```
service:rabbitmq AND log_type:node_metrics AND running:false
```

Any occurrence -- cluster degradation.

### Connection Spike

```
service:rabbitmq AND log_type:connection_metrics AND total_connections:>500
```

Threshold depends on your baseline.

## Management Plugin Metrics

### Key Metrics to Monitor

| Metric | Healthy | Warning | Critical |
|--------|---------|---------|----------|
| Queue depth | <1000 | 1000-10000 | >10000 |
| Consumer count | >0 | - | 0 |
| Publish rate | Stable | 2x baseline | 5x baseline |
| Memory usage | <80% limit | 80-90% | >90% (alarm) |
| Disk free | >2x limit | 1-2x limit | <limit (alarm) |
| File descriptors | <80% total | 80-90% | >90% |
| Unacked messages | <prefetch * consumers | 2x expected | >5x expected |

### Overview API Endpoint

```bash
# Get cluster overview
curl -u admin:password http://localhost:15672/api/overview | jq '{
  rabbitmq_version: .rabbitmq_version,
  erlang_version: .erlang_version,
  message_stats: .message_stats,
  queue_totals: .queue_totals,
  object_totals: .object_totals
}'
```

## Performance Metrics

| Metric | Value | Notes |
|--------|-------|-------|
| JSON log overhead | <1% | Compared to default format |
| Metrics collector CPU | <1% | Python script every 30s |
| Metrics collector memory | ~30MB | Python with requests library |
| Fluent Bit memory | ~30MB | Tailing server logs |
| Dead letter monitor | ~20MB | Node.js AMQP consumer |

## Troubleshooting

### RabbitMQ logs not in JSON format

1. Verify the formatter configuration:
   ```bash
   rabbitmqctl environment | grep log
   ```

2. Ensure you are running RabbitMQ 3.9+ (JSON formatter was introduced in 3.9).

3. Check for syntax errors in `rabbitmq.conf`:
   ```bash
   rabbitmq-diagnostics check_running
   ```

### Metrics collector failing

1. Verify the management plugin is enabled:
   ```bash
   rabbitmq-plugins list | grep management
   # If not enabled:
   rabbitmq-plugins enable rabbitmq_management
   ```

2. Test the management API:
   ```bash
   curl -u admin:password http://localhost:15672/api/queues | jq length
   ```

3. Check credentials and network access from the collector.

### Dead letter monitor not receiving messages

1. Verify the dead letter exchange binding:
   ```bash
   rabbitmqadmin list bindings | grep dlx
   ```

2. Check that the source queue has the DLX argument set:
   ```bash
   rabbitmqadmin list queues name arguments
   ```

3. Test by publishing and rejecting a message:
   ```bash
   rabbitmqadmin publish exchange=amq.default routing_key=orders payload='{"test": true}'
   # Then nack the message in your consumer
   ```

### High memory usage

1. Check which queues are consuming the most memory:
   ```bash
   rabbitmqctl list_queues name messages memory --sort-by memory --reverse
   ```

2. Enable lazy queues for large backlogs:
   ```bash
   rabbitmqctl set_policy lazy-queues "^lazy\." '{"queue-mode":"lazy"}' --apply-to queues
   ```

3. Check for message TTL to prevent unbounded growth:
   ```bash
   rabbitmqadmin list queues name arguments | grep ttl
   ```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log collection patterns
- [Docker Compose Integration](/integrations/docker-compose/) - Multi-service logging
- [Node.js Integration](/integrations/nodejs/) - Application-level consumer logging
- [Incident Response](/use-cases/incident-response/) - Build runbooks for RabbitMQ outages
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Alert on queue backlogs and dead letters


---

# React Error Tracking & Monitoring Integration

> Professional React error tracking with automatic capture, Error Boundaries, Web Vitals, and session-based breadcrumbs.

*Source: https://logtide.dev/integrations/react/*

---

LogTide's React integration provides professional-grade error tracking and performance monitoring for React applications. It includes automatic unhandled error capture, React Error Boundaries, and Core Web Vitals collection.

## Why use LogTide with React?

- **Automatic error capture**: Catches all unhandled exceptions and promise rejections.
- **Error Boundaries**: Component-level error catching with full stack traces.
- **Core Web Vitals**: Monitor LCP, FID, CLS, and INP to ensure a smooth UX.
- **Breadcrumbs**: Automatically track user clicks and network requests leading up to an error.
- **Source Maps**: See the original source code in your stack traces instead of minified production code.

## Prerequisites

- React 16.8+ (Hooks support)
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/browser
```

## Quick Setup

Initialize LogTide as early as possible in your `main.tsx` or `index.tsx`:

```typescript
// main.tsx
import { initLogtide } from '@logtide/browser';

initLogtide({
  dsn: 'https://lp_abc123@api.logtide.dev',
  service: 'react-frontend',
  environment: 'production',
  release: '1.0.0',
});

// ... bootstrap your React app
```

## Using Error Boundaries

Wrap your application or specific components with an Error Boundary to capture rendering errors:

```tsx
import React from 'react';
import { LogtideErrorBoundary } from '@logtide/browser';

function MyErrorPage({ error }: { error: Error }) {
  return (
    <div className="error-container">
      <h2>Something went wrong!</h2>
      <pre>{error.message}</pre>
    </div>
  );
}

export default function App() {
  return (
    <LogtideErrorBoundary fallback={(error) => <MyErrorPage error={error} />}>
      <Dashboard />
    </LogtideErrorBoundary>
  );
}
```

## Capture Errors Manually

You can use the `hub` API to capture errors from `try/catch` blocks or log custom events:

```tsx
import { hub } from '@logtide/core';

function CheckoutButton() {
  const handleCheckout = async () => {
    hub.addBreadcrumb({
      type: 'ui',
      category: 'ui',
      message: 'Checkout button clicked',
      timestamp: Date.now(),
    });

    try {
      await processPayment();
      hub.captureLog('info', 'Payment successful', { amount: 99.99 });
    } catch (error) {
      hub.captureError(error, {
        tags: { flow: 'checkout' },
        extra: { cartId: 'abc-123' }
      });
    }
  };

  return <button onClick={handleCheckout}>Pay Now</button>;
}
```

## Core Web Vitals

LogTide automatically tracks performance metrics. You can view these in your LogTide Dashboard under the **Metrics** tab to monitor your app's health in the real world.

## Source Map Uploads

To get readable stack traces, upload your source maps during your build process:

```bash
# Using @logtide/cli
logtide sourcemaps upload --path ./dist --release 1.0.0 --apiKey YOUR_API_KEY
```

## Next Steps

- [Browser SDK Docs](/docs/sdks/browser/) - Full browser SDK reference
- [Next.js Integration](/integrations/nextjs/) - For SSR and App Router
- [Remix Integration](/integrations/remix/) - Full-stack React monitoring


---

# Redis Logging and Monitoring

> Monitor Redis slow queries, operational events, and cluster health in LogTide with structured parsing for performance analysis.

*Source: https://logtide.dev/integrations/redis/*

---

Redis is often a critical component in your stack, but its logs are frequently overlooked. This guide shows you how to capture Redis slow queries, operational events, and cluster health metrics in LogTide for proactive monitoring.

## Why monitor Redis logs?

- **Slow query detection**: Find performance bottlenecks before they impact users
- **Connection tracking**: Monitor client connections and authentication attempts
- **Memory alerts**: Catch memory issues before OOM kills your Redis
- **Cluster health**: Track failovers, replication lag, and node status
- **Security monitoring**: Detect unauthorized access attempts
- **Capacity planning**: Understand usage patterns over time

## Prerequisites

- Redis 6.0+ (7.x recommended)
- Fluent Bit for log collection
- LogTide instance with API key
- Basic understanding of Redis configuration

## Redis Configuration

### Enable Logging

Configure Redis to log to a file and capture slow queries:

```conf
# /etc/redis/redis.conf

# Log file location
logfile /var/log/redis/redis.log

# Log level: debug, verbose, notice, warning
loglevel notice

# Slow log threshold (in microseconds)
# Log commands taking longer than 10ms
slowlog-log-slower-than 10000

# Keep last 1024 slow queries
slowlog-max-len 1024

# Log client connections/disconnections
# Available in Redis 7.0+
# set-tcp-keepalive 300
```

Restart Redis:

```bash
sudo systemctl restart redis
```

### Verify Logging

```bash
# Check log file
tail -f /var/log/redis/redis.log

# Check slow log
redis-cli SLOWLOG GET 10
```

## Fluent Bit Configuration

### Basic Redis Log Collection

```ini
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf

# Collect Redis logs
[INPUT]
    Name          tail
    Path          /var/log/redis/redis.log
    Tag           redis.log
    Parser        redis
    Refresh_Interval 5

# Collect slow queries via exec
[INPUT]
    Name          exec
    Tag           redis.slowlog
    Command       redis-cli SLOWLOG GET 10 RESET
    Interval_Sec  60
    Parser        redis_slowlog

# Add metadata
[FILTER]
    Name          record_modifier
    Match         redis.*
    Record        hostname ${HOSTNAME}
    Record        service redis
    Record        environment production

# Output to LogTide
[OUTPUT]
    Name          http
    Match         *
    Host          api.logtide.dev
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

### Redis Log Parser

```ini
# /etc/fluent-bit/parsers.conf
[PARSER]
    Name        redis
    Format      regex
    Regex       ^(?<pid>\d+):(?<role>[XCSM]) (?<time>\d+ \w+ \d{4} \d{2}:\d{2}:\d{2}\.\d{3}) (?<level>[.\-*#]) (?<message>.*)$
    Time_Key    time
    Time_Format %d %b %Y %H:%M:%S.%L

[PARSER]
    Name        redis_slowlog
    Format      json
```

### Role Mapping

Redis log format includes role indicators:

| Character | Role |
|-----------|------|
| `X` | Sentinel |
| `C` | RDB/AOF writing child |
| `S` | Replica |
| `M` | Master |

## Slow Query Monitoring

### Dedicated Slow Log Collector

The simplest approach is to use a Python script that properly parses Redis slow log output:

```python
#!/usr/bin/env python3
# /opt/redis-slowlog/collect.py

import json
import os
import redis
from datetime import datetime

REDIS_HOST = os.environ.get('REDIS_HOST', 'localhost')
REDIS_PORT = int(os.environ.get('REDIS_PORT', 6379))
REDIS_PASSWORD = os.environ.get('REDIS_PASSWORD', None)

r = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD,
    decode_responses=True
)

# Get slow log entries
entries = r.slowlog_get(100)

for entry in entries:
    log = {
        'id': entry['id'],
        'timestamp': entry['start_time'],
        'duration_us': entry['duration'],
        'duration_ms': entry['duration'] / 1000,
        'command': ' '.join(entry['command']) if isinstance(entry['command'], list) else entry['command'],
        'client_address': entry.get('client_address', ''),
        'client_name': entry.get('client_name', ''),
    }
    print(json.dumps(log))

# Reset to avoid duplicates
r.slowlog_reset()
```

Install the dependency: `pip install redis`

Alternatively, use the simpler Lua approach via Fluent Bit for basic parsing:

```bash
#!/bin/bash
# /opt/redis-slowlog/collect.sh (simple version)
# Outputs raw slow log for Fluent Bit to process

redis-cli ${REDIS_PASSWORD:+-a $REDIS_PASSWORD} \
    -h ${REDIS_HOST:-localhost} \
    -p ${REDIS_PORT:-6379} \
    SLOWLOG GET 100

redis-cli ${REDIS_PASSWORD:+-a $REDIS_PASSWORD} \
    -h ${REDIS_HOST:-localhost} \
    -p ${REDIS_PORT:-6379} \
    SLOWLOG RESET > /dev/null
```

### Fluent Bit Exec Input

```ini
[INPUT]
    Name          exec
    Tag           redis.slowlog
    Command       /opt/redis-slowlog/collect.sh
    Interval_Sec  30
    Parser        json

[FILTER]
    Name          record_modifier
    Match         redis.slowlog
    Record        log_type slowlog
    Record        hostname ${HOSTNAME}
```

## Docker Setup

### Docker Compose

```yaml
# docker-compose.yml
services:
  redis:
    image: redis:7-alpine
    command: >
      redis-server
      --loglevel notice
      --slowlog-log-slower-than 10000
      --slowlog-max-len 1024
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
      - ./redis-logs:/var/log/redis
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: redis
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis-slowlog-collector:
    image: redis:7-alpine
    command: >
      sh -c 'while true; do
        redis-cli -h redis SLOWLOG GET 100 | head -500;
        redis-cli -h redis SLOWLOG RESET;
        sleep 30;
      done'
    depends_on:
      - redis
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: redis.slowlog

  fluent-bit:
    image: fluent/fluent-bit:latest
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    ports:
      - "24224:24224"
    environment:
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}

volumes:
  redis_data:
```

### Kubernetes StatefulSet

```yaml
# redis-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
      annotations:
        fluentbit.io/parser: redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          command:
            - redis-server
            - --loglevel
            - notice
            - --slowlog-log-slower-than
            - "10000"
            - --slowlog-max-len
            - "1024"
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
          resources:
            requests:
              memory: "256Mi"
              cpu: "100m"
            limits:
              memory: "1Gi"
              cpu: "500m"
          livenessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 5

        - name: slowlog-collector
          image: redis:7-alpine
          command:
            - sh
            - -c
            - |
              while true; do
                redis-cli -h localhost SLOWLOG GET 100 2>/dev/null
                redis-cli -h localhost SLOWLOG RESET 2>/dev/null
                sleep 30
              done
          resources:
            requests:
              memory: "32Mi"
              cpu: "10m"
            limits:
              memory: "64Mi"
              cpu: "50m"

  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
```

## Monitoring Queries

### Slow Query Detection

In LogTide, create alerts for slow queries:

```sql
-- Find queries slower than 100ms
SELECT * FROM logs
WHERE service = 'redis'
  AND log_type = 'slowlog'
  AND duration_ms > 100
ORDER BY timestamp DESC
LIMIT 100
```

### Connection Issues

```sql
-- Find connection-related events
SELECT * FROM logs
WHERE service = 'redis'
  AND (message LIKE '%Connection%'
       OR message LIKE '%Client closed%'
       OR message LIKE '%Accepted%')
ORDER BY timestamp DESC
```

### Memory Warnings

```sql
-- Find memory-related warnings
SELECT * FROM logs
WHERE service = 'redis'
  AND level IN ('warning', 'error')
  AND (message LIKE '%memory%'
       OR message LIKE '%OOM%'
       OR message LIKE '%maxmemory%')
ORDER BY timestamp DESC
```

### Replication Status

```sql
-- Monitor replica sync events
SELECT * FROM logs
WHERE service = 'redis'
  AND (message LIKE '%MASTER%'
       OR message LIKE '%replica%'
       OR message LIKE '%sync%')
ORDER BY timestamp DESC
```

## Redis Sentinel Monitoring

For high availability setups with Sentinel:

### Sentinel Log Collection

```ini
[INPUT]
    Name          tail
    Path          /var/log/redis/sentinel.log
    Tag           redis.sentinel
    Parser        redis

[FILTER]
    Name          record_modifier
    Match         redis.sentinel
    Record        log_type sentinel
```

### Key Sentinel Events

Monitor these events:

| Event | Description |
|-------|-------------|
| `+sdown` | Instance is subjectively down |
| `+odown` | Instance is objectively down |
| `-sdown` | Instance is no longer subjectively down |
| `-odown` | Instance is no longer objectively down |
| `+failover-state-*` | Failover progress |
| `+switch-master` | Master changed |
| `+slave` | New replica detected |
| `-slave` | Replica removed |

### Failover Alert Query

```sql
-- Detect failover events
SELECT * FROM logs
WHERE service = 'redis'
  AND log_type = 'sentinel'
  AND (message LIKE '%+failover%'
       OR message LIKE '%+switch-master%'
       OR message LIKE '%+odown%')
ORDER BY timestamp DESC
```

## Redis Cluster Monitoring

### Cluster Log Events

```ini
[INPUT]
    Name          tail
    Path          /var/log/redis/redis-cluster-*.log
    Tag           redis.cluster
    Parser        redis

[FILTER]
    Name          lua
    Match         redis.cluster
    script        /etc/fluent-bit/redis_cluster.lua
    call          parse_cluster_events
```

### Cluster Health Script

```bash
#!/bin/bash
# /opt/redis-cluster/health.sh

REDIS_NODES="${REDIS_NODES:-localhost:7000,localhost:7001,localhost:7002}"

for node in ${REDIS_NODES//,/ }; do
    host="${node%:*}"
    port="${node#*:}"

    info=$(redis-cli -h $host -p $port CLUSTER INFO 2>/dev/null)

    if [ $? -eq 0 ]; then
        state=$(echo "$info" | grep cluster_state | cut -d: -f2 | tr -d '\r')
        slots_ok=$(echo "$info" | grep cluster_slots_ok | cut -d: -f2 | tr -d '\r')
        known_nodes=$(echo "$info" | grep cluster_known_nodes | cut -d: -f2 | tr -d '\r')

        echo "{\"node\":\"$node\",\"state\":\"$state\",\"slots_ok\":$slots_ok,\"known_nodes\":$known_nodes,\"timestamp\":$(date +%s)}"
    else
        echo "{\"node\":\"$node\",\"state\":\"unreachable\",\"timestamp\":$(date +%s)}"
    fi
done
```

## Performance Metrics

Benchmark from production Redis (100k ops/sec):

| Metric | Without Logging | With Logging | Overhead |
|--------|-----------------|--------------|----------|
| Ops/sec | 100,000 | 99,500 | -0.5% |
| Avg latency | 0.5ms | 0.52ms | +4% |
| P99 latency | 2ms | 2.1ms | +5% |
| Memory usage | 1GB | 1.02GB | +2% |

**Notes:**
- Slowlog has minimal impact on performance
- File logging adds ~2% overhead
- Fluent Bit runs out-of-band, no Redis impact

## Security Considerations

### Authentication

When Redis requires authentication:

```bash
# Set password in environment
export REDIS_PASSWORD="your-redis-password"

# Update collect script
redis-cli -a $REDIS_PASSWORD SLOWLOG GET 100
```

### Sensitive Command Logging

Avoid logging commands with sensitive data:

```lua
-- /etc/fluent-bit/redis_sanitize.lua
local sensitive_commands = {
    "AUTH", "CONFIG SET", "ACL SETUSER"
}

function sanitize_command(tag, timestamp, record)
    local cmd = record["command"] or ""

    for _, pattern in ipairs(sensitive_commands) do
        if cmd:upper():find(pattern) then
            record["command"] = pattern .. " [REDACTED]"
            break
        end
    end

    return 1, timestamp, record
end
```

## Troubleshooting

### No logs appearing

1. **Check Redis is logging:**
```bash
redis-cli CONFIG GET loglevel
redis-cli CONFIG GET logfile
```

2. **Verify file permissions:**
```bash
ls -la /var/log/redis/
# Fluent Bit needs read access
```

3. **Check Fluent Bit:**
```bash
sudo systemctl status fluent-bit
sudo journalctl -u fluent-bit -f
```

### Slow log not capturing queries

1. **Check threshold:**
```bash
redis-cli CONFIG GET slowlog-log-slower-than
# Result is in microseconds (10000 = 10ms)
```

2. **Lower threshold for testing:**
```bash
redis-cli CONFIG SET slowlog-log-slower-than 1000  # 1ms
```

3. **Verify slow log has entries:**
```bash
redis-cli SLOWLOG LEN
redis-cli SLOWLOG GET 5
```

### High memory usage from logging

Limit slow log size:

```conf
# Keep fewer entries
slowlog-max-len 256

# Or reset periodically
redis-cli SLOWLOG RESET
```

## Detection Rules

Create LogTide alerts for:

### Slow Query Alert

```sql
-- Alert: Queries over 100ms
WHERE service = 'redis'
  AND log_type = 'slowlog'
  AND duration_ms > 100
-- Threshold: 10 in 5 minutes
```

### Memory Warning

```sql
-- Alert: Memory pressure
WHERE service = 'redis'
  AND level = 'warning'
  AND message LIKE '%memory%'
-- Threshold: Any occurrence
```

### Cluster Failover

```sql
-- Alert: Cluster failover started
WHERE service = 'redis'
  AND (message LIKE '%+failover%'
       OR message LIKE '%+switch-master%')
-- Threshold: Any occurrence
```

## Next Steps

- **[Docker Integration](/integrations/docker/)** - Container deployment patterns
- **[Kubernetes Integration](/integrations/kubernetes/)** - StatefulSet best practices
- **[Node.js Integration](/integrations/nodejs/)** - Application-level Redis logging
- **[Security Monitoring](/use-cases/security-monitoring/)** - Build Redis security alerts


---

# Remix Application Logging Integration

> Send structured logs from Remix loaders and actions to LogTide with server-side request tracing and error boundary capture.

*Source: https://logtide.dev/integrations/remix/*

---

Remix runs loaders and actions on the server, making it a natural fit for structured logging. This guide shows you how to add LogTide to Remix for automatic request tracing, loader/action logging, and error boundary capture.

## Why use LogTide with Remix?

- **Server-side first**: Loaders and actions run on the server — log them with full context
- **Request tracing**: Correlate logs across nested route loaders in a single page load
- **Error boundaries**: Capture errors with the route context that triggered them
- **Action audit trail**: Log form submissions and mutations with user context
- **Non-blocking**: Async batching keeps loader response times fast

## Prerequisites

- Node.js 18+ or 20+
- Remix 2.x
- LogTide instance with a DSN or API key

## Installation

```bash
npm install @logtide/sdk-node
```

## Quick Start

### Initialize LogTide

```typescript
// app/lib/logtide.server.ts
import { LogTideClient } from '@logtide/sdk-node';

export const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  globalMetadata: { environment: process.env.NODE_ENV },
});
```

The `.server.ts` suffix ensures this module is never bundled into client code.

### Use in Loaders

```typescript
// app/routes/users.$id.tsx
import { json, type LoaderFunctionArgs } from '@remix-run/node';
import { logtide } from '~/lib/logtide.server';

export async function loader({ params, request }: LoaderFunctionArgs) {
  const traceId = request.headers.get('x-trace-id') ?? crypto.randomUUID();

  logtide.info('remix-app', 'Loading user profile', {
    userId: params.id,
    traceId,
  });

  const user = await db.user.findUnique({ where: { id: params.id } });

  if (!user) {
    logtide.warn('remix-app', 'User not found', { userId: params.id, traceId });
    throw new Response('Not found', { status: 404 });
  }

  return json({ user }, { headers: { 'X-Trace-Id': traceId } });
}
```

### Use in Actions

```typescript
// app/routes/settings.tsx
import { redirect, type ActionFunctionArgs } from '@remix-run/node';
import { logtide } from '~/lib/logtide.server';

export async function action({ request }: ActionFunctionArgs) {
  const session = await getSession(request);
  const userId = session.get('userId');
  const formData = await request.formData();

  logtide.info('remix-app', 'Settings updated', {
    userId,
    fields: [...formData.keys()],
  });

  await updateSettings(userId, Object.fromEntries(formData));
  return redirect('/settings');
}
```

## Request Tracing Middleware

For automatic logging of all requests, create a middleware using Remix's `entry.server.tsx`:

```typescript
// app/entry.server.tsx
import { PassThrough } from 'node:stream';
import { createReadableStreamFromReadable } from '@remix-run/node';
import { RemixServer } from '@remix-run/react';
import { renderToPipeableStream } from 'react-dom/server';
import { logtide } from '~/lib/logtide.server';

const ABORT_DELAY = 5_000;

export default function handleRequest(
  request: Request,
  responseStatusCode: number,
  responseHeaders: Headers,
  remixContext: any,
) {
  const url = new URL(request.url);
  const start = performance.now();
  const traceId = request.headers.get('x-trace-id') ?? crypto.randomUUID();

  // Skip static assets
  if (url.pathname.startsWith('/build/') || url.pathname === '/favicon.ico') {
    return defaultHandler(request, responseStatusCode, responseHeaders, remixContext);
  }

  return new Promise((resolve, reject) => {
    const { pipe, abort } = renderToPipeableStream(
      <RemixServer context={remixContext} url={request.url} abortDelay={ABORT_DELAY} />,
      {
        onShellReady() {
          const body = new PassThrough();
          const stream = createReadableStreamFromReadable(body);

          responseHeaders.set('Content-Type', 'text/html');
          responseHeaders.set('X-Trace-Id', traceId);

          const durationMs = Math.round(performance.now() - start);

          logtide.info('remix-app', `${request.method} ${url.pathname} ${responseStatusCode}`, {
            method: request.method,
            path: url.pathname,
            statusCode: responseStatusCode,
            durationMs,
            traceId,
          });

          resolve(new Response(stream, {
            headers: responseHeaders,
            status: responseStatusCode,
          }));

          pipe(body);
        },
        onShellError(error) {
          logtide.error('remix-app', 'Remix shell render error', {
            error: String(error),
            path: url.pathname,
            traceId,
          });
          reject(error);
        },
        onError(error) {
          responseStatusCode = 500;
          logtide.error('remix-app', 'Remix render error', {
            error: String(error),
            path: url.pathname,
            traceId,
          });
        },
      },
    );

    setTimeout(abort, ABORT_DELAY);
  });
}
```

## Error Boundary Logging

### Root Error Boundary

```typescript
// app/root.tsx
import { isRouteErrorResponse, useRouteError } from '@remix-run/react';

export function ErrorBoundary() {
  const error = useRouteError();

  // Server-side logging happens in entry.server.tsx
  // This component handles the client-side rendering

  if (isRouteErrorResponse(error)) {
    return (
      <div className="error-page">
        <h1>{error.status} {error.statusText}</h1>
        <p>{error.data}</p>
      </div>
    );
  }

  return (
    <div className="error-page">
      <h1>Something went wrong</h1>
      <p>Please try again later.</p>
    </div>
  );
}
```

### Server-Side Error Logging

Catch errors in loaders/actions with a utility:

```typescript
// app/lib/with-logging.server.ts
import { logtide } from '~/lib/logtide.server';

export function withLogging<T>(
  routeName: string,
  handler: (args: any) => Promise<T>,
) {
  return async (args: any) => {
    const start = performance.now();
    const traceId = args.request.headers.get('x-trace-id') ?? crypto.randomUUID();

    try {
      const result = await handler(args);
      const durationMs = Math.round(performance.now() - start);

      logtide.info('remix-app', `${routeName} completed`, { durationMs, traceId });
      return result;
    } catch (error) {
      logtide.error('remix-app', `${routeName} failed`, {
        error: error instanceof Error ? error.message : String(error),
        errorType: error instanceof Error ? error.constructor.name : 'Unknown',
        traceId,
      });
      throw error;
    }
  };
}

// Usage
export const loader = withLogging('users.$id', async ({ params }: LoaderFunctionArgs) => {
  const user = await db.user.findUnique({ where: { id: params.id } });
  if (!user) throw new Response('Not found', { status: 404 });
  return json({ user });
});
```

## Session and Auth Context

```typescript
// app/lib/logtide.server.ts (extended)
import { type Session } from '@remix-run/node';

export function logWithSession(session: Session) {
  const userId = session.get('userId');
  const role = session.get('role');

  return {
    info(message: string, metadata?: Record<string, unknown>) {
      logtide.info('remix-app', message, { userId, role, ...metadata });
    },
    warn(message: string, metadata?: Record<string, unknown>) {
      logtide.warn('remix-app', message, { userId, role, ...metadata });
    },
    error(message: string, metadata?: Record<string, unknown>) {
      logtide.error('remix-app', message, { userId, role, ...metadata });
    },
  };
}

// Usage in loader/action
export async function action({ request }: ActionFunctionArgs) {
  const session = await getSession(request);
  const log = logWithSession(session);

  log.info('Deleting account', { reason: 'user_requested' });
  // ...
}
```

## Docker Deployment

```dockerfile
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/build ./build
COPY --from=build /app/public ./public
COPY --from=build /app/package*.json ./
RUN npm ci --omit=dev

ENV NODE_ENV=production
ENV LOGTIDE_API_URL=""
ENV LOGTIDE_API_KEY=""

EXPOSE 3000
CMD ["npm", "start"]
```

```yaml
# docker-compose.yml
services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - SESSION_SECRET=${SESSION_SECRET}
```

## Performance

| Metric | Value |
|--------|-------|
| Loader overhead | <0.5ms per log call |
| Memory overhead | ~8MB |
| Batch network calls | 1 per 100 logs |
| SSR impact | Negligible (async shipping) |

## Troubleshooting

### Logs not appearing

1. Ensure `logtide.server.ts` has the `.server.ts` suffix — without it, the client bundle may try to import server-only code
2. Check credentials are set: `echo $LOGTIDE_API_URL` and `echo $LOGTIDE_API_KEY`
3. Add `await logtide.close()` to your server's graceful shutdown handler

### "Cannot use import statement outside a module"

Ensure your `tsconfig.json` has `"module": "ESNext"` and your Remix config supports ESM.

### Logs from nested loaders appear out of order

Remix runs nested route loaders in parallel. Use the `traceId` to correlate them:

```
traceId:abc123 → root loader
traceId:abc123 → users layout loader
traceId:abc123 → users.$id loader
```

## Next Steps

- [Node.js SDK Reference](/integrations/nodejs/) - Full SDK documentation
- [Docker Integration](/integrations/docker/) - Container deployments
- [Incident Response](/use-cases/incident-response/) - Production debugging with traces


---

# Ruby on Rails Logging Integration

> Send structured logs, exceptions, and traces from Ruby on Rails to LogTide with the logtide gem — a Railtie installs per-request middleware automatically.

*Source: https://logtide.dev/integrations/rails/*

---

The LogTide Ruby SDK (the `logtide` gem) provides a drop-in integration for Ruby on Rails. A Railtie installs the Rack middleware automatically, so each request gets an isolated scope, HTTP tags, `traceparent` parsing, and request/response breadcrumbs with no manual wiring. The SDK uses the standard library only — no runtime dependencies.

## Why use LogTide with Rails?

- **Zero-wiring middleware**: the Railtie installs the Rack middleware automatically
- **Per-request isolation**: tags, user, and breadcrumbs don't leak between requests
- **Structured exception capture**: cause chains and stack frames, not stringified messages
- **Distributed tracing**: W3C `traceparent` in and out, with log/trace correlation
- **stdlib `Logger` bridge**: keep your existing logging calls
- **Self-hosted**: keep log data on your own infrastructure, with no per-GB fees

## Prerequisites

- Ruby 3.1+
- Rails 6.1, 7.x, or 8.x
- Bundler
- A LogTide instance with a DSN

## Installation

Add the gem to your `Gemfile`:

```ruby
gem "logtide"
```

Then install it:

```bash
bundle install
```

## Configuration

Initialise the SDK in an initializer. The Railtie picks up the configured client and
installs the Rack middleware for you:

```ruby
# config/initializers/logtide.rb
Logtide.init(
  dsn: ENV["LOGTIDE_DSN"],
  service: "my-app",
  environment: Rails.env,
  release: ENV["GIT_SHA"]
)
```

Set the DSN in your environment:

```bash
LOGTIDE_DSN=https://lp_your_key@logs.example.com
```

That's it — the middleware is active with sensible defaults. See the
[Ruby SDK reference](/docs/sdks/ruby/) for the full list of configuration options
(batching, retry, circuit breaker, sampling, breadcrumbs, and PII handling).

## Basic Usage

### Logging

```ruby
Logtide.info("user signed in", { user_id: current_user.id })
Logtide.warn("rate limit approaching", { ip: request.remote_ip })
Logtide.error("payment failed", { order_id: order.id })
```

### Capturing exceptions

Use `capture_exception` so the cause chain and stack frames are preserved for grouping:

```ruby
class OrdersController < ApplicationController
  def create
    @order = Order.create!(order_params)
    charge_card!(@order)
  rescue Stripe::CardError => e
    Logtide.capture_exception(e, { order_id: @order&.id })
    redirect_to new_order_path, alert: "Payment failed"
  end
end
```

For app-wide error reporting, capture from `rescue_from`:

```ruby
class ApplicationController < ActionController::Base
  rescue_from StandardError do |e|
    Logtide.capture_exception(e)
    raise
  end
end
```

### stdlib Logger bridge

`Logtide::LoggerBridge` is a drop-in replacement for the stdlib `Logger`. Wire it as
the Rails logger to forward framework and application logs to LogTide:

```ruby
# config/initializers/logtide.rb
Rails.logger = Logtide::LoggerBridge.new
```

`fatal` calls are delivered as `critical` entries.

## Scope and breadcrumbs

The Rack middleware already isolates a scope per request. Add request-specific tags,
user, and breadcrumbs to enrich every entry within that request:

```ruby
class ApplicationController < ActionController::Base
  before_action do
    Logtide.configure_scope do |scope|
      scope.set_user(id: current_user&.id, email: current_user&.email)
      scope.set_tag("tenant", current_tenant.slug)
    end
  end
end
```

## Background jobs

The web middleware only covers HTTP requests. Wrap job execution in `with_scope` so
each job runs in its own isolated scope, and flush long-lived workers on shutdown:

```ruby
class ProcessOrderJob < ApplicationJob
  def perform(order_id)
    Logtide.with_scope do |scope|
      scope.set_tag("job", self.class.name)
      Logtide.info("processing order", { order_id: order_id })
      # ... work ...
    end
  end
end
```

For short-lived processes (rake tasks, one-off scripts), call `Logtide.flush` (or
`Logtide.close`) before exit so buffered entries are delivered. A best-effort
`at_exit` hook is installed, but an explicit flush is the safe choice.

## Distributed tracing

The middleware starts a server span per request and reads inbound `traceparent`
headers automatically. Logs emitted during the request carry the active
`trace_id` and `span_id`. To propagate the trace to a downstream service:

```ruby
headers = Logtide.trace_propagation_headers
# => { "traceparent" => "00-..." }

Net::HTTP.post(uri, payload.to_json, headers.merge("content-type" => "application/json"))
```

## Docker Deployment

```yaml
# docker-compose.yml
services:
  web:
    build: .
    environment:
      - RAILS_ENV=production
      - LOGTIDE_DSN=${LOGTIDE_DSN}
      - GIT_SHA=${GIT_SHA}
    depends_on:
      - postgres

  worker:
    build: .
    command: bundle exec sidekiq
    environment:
      - RAILS_ENV=production
      - LOGTIDE_DSN=${LOGTIDE_DSN}
    depends_on:
      - postgres
      - redis
```

## Troubleshooting

### Logs not appearing

1. Confirm the DSN is set: `bin/rails runner 'p ENV["LOGTIDE_DSN"]'`
2. Enable SDK debug output to log internals to stderr:

   ```ruby
   Logtide.init(dsn: ENV["LOGTIDE_DSN"], service: "my-app", debug: true)
   ```

3. Check `Logtide.get_metrics` — a rising `logs_dropped` means the buffer hit
   `max_buffer_size`; a non-zero `circuit_breaker_trips` means delivery is failing.

### Worker logs missing

Background workers need the same `LOGTIDE_DSN` in their environment, and long-lived
workers should not rely solely on `at_exit` — flush on shutdown where possible.

## Next Steps

- [Ruby SDK Reference](/docs/sdks/ruby/) — full SDK documentation
- [PostgreSQL Integration](/integrations/postgresql/) — database logs
- [Docker Integration](/integrations/docker/) — container deployment
- [Incident Response](/use-cases/incident-response/) — from log to root cause


---

# SolidJS Error Tracking & Monitoring Integration

> Professional SolidJS error tracking with automatic capture, Error Boundaries, Web Vitals, and breadcrumbs.

*Source: https://logtide.dev/integrations/solidjs/*

---

LogTide's SolidJS integration provides professional-grade error tracking and performance monitoring for Solid applications. It includes automatic unhandled error capture, Solid Error Boundaries, and Core Web Vitals collection.

## Why use LogTide with SolidJS?

- **Automatic error capture**: Catches all unhandled exceptions and promise rejections.
- **Error Boundaries**: Component-level error catching with Solid's native `ErrorBoundary`.
- **Core Web Vitals**: Monitor your app's performance metrics (LCP, FID, CLS, INP) in the real world.
- **Breadcrumbs**: Track clicks and network requests leading up to an error.
- **Fine-grained Tracing**: Correlate all logs and errors for a single user session.

## Prerequisites

- SolidJS 1.0+
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/browser
```

## Quick Setup

Initialize LogTide as early as possible in your `index.tsx`:

```typescript
// index.tsx
import { render } from 'solid-js/web';
import { initLogtide } from '@logtide/browser';
import App from './App';

// Initialize LogTide
initLogtide({
  dsn: 'https://lp_abc123@api.logtide.dev',
  service: 'solid-frontend',
  environment: 'production',
  release: '1.0.0',
});

render(() => <App />, document.getElementById('root') as HTMLElement);
```

## Using Error Boundaries

Wrap your application or specific components with Solid's native `ErrorBoundary` to capture rendering errors and report them to LogTide:

```tsx
import { ErrorBoundary } from 'solid-js';
import { hub } from '@logtide/core';

function App() {
  return (
    <ErrorBoundary
      fallback={(err, reset) => {
        // Capture the error manually in the fallback
        hub.captureError(err, { tags: { boundary: 'main' } });

        return (
          <div>
            <h2>Something went wrong!</h2>
            <button onClick={reset}>Try again</button>
          </div>
        );
      }}
    >
      <MyDashboard />
    </ErrorBoundary>
  );
}
```

## Capture Errors Manually

You can use the `hub` API to capture errors from `try/catch` blocks or log custom events:

```tsx
import { hub } from '@logtide/core';

function CheckoutButton() {
  const handleCheckout = async () => {
    hub.addBreadcrumb({
      type: 'ui',
      category: 'ui',
      message: 'Checkout button clicked',
      timestamp: Date.now(),
    });

    try {
      await processPayment();
      hub.captureLog('info', 'Payment successful');
    } catch (err) {
      hub.captureError(err, {
        tags: { flow: 'checkout' },
        extra: { cartId: 'abc-123' }
      });
    }
  };

  return <button onClick={handleCheckout}>Pay Now</button>;
}
```

## Core Web Vitals

LogTide automatically tracks performance metrics. View the results in your LogTide Dashboard under the **Metrics** tab.

## Source Map Uploads

To get readable stack traces in production, upload your source maps during your Vite build:

```bash
# Using @logtide/cli
logtide sourcemaps upload --path ./dist --release 1.0.0 --apiKey YOUR_API_KEY
```

## Next Steps

- [Browser SDK Docs](/docs/sdks/browser/) - Full browser SDK reference
- [React Integration](/integrations/react/) - For React developers
- [Node.js Integration](/integrations/nodejs/) - For server-side monitoring


---

# Spring Boot and Kotlin Integration

> Send structured logs from Spring Boot applications to LogTide with the native Kotlin SDK, middleware support, and coroutines.

*Source: https://logtide.dev/integrations/spring-boot/*

---

LogTide integrates with Spring Boot through the native Kotlin SDK, providing structured logging with automatic batching, retry logic, circuit breaker pattern, and seamless integration with Spring's ecosystem.

## Why use LogTide with Spring Boot?

- **Native Kotlin SDK**: Full coroutines support with suspend functions
- **Spring middleware**: Automatic HTTP request/response logging
- **Automatic batching**: Configurable batch size and flush intervals
- **Actuator ready**: Health checks and metrics integration out of the box
- **Kotlin-first**: Full Kotlin support with coroutines-aware logging
- **Zero code changes**: Configure in XML, no application code modifications

## Prerequisites

- Java 17+ or Kotlin 1.9+
- Spring Boot 3.x
- Gradle or Maven build system
- LogTide instance with API key

## Installation

### Gradle (Kotlin DSL)

```kotlin
// build.gradle.kts
dependencies {
    implementation("io.github.logtide-dev:logtide-core:0.9.8")
}
```

### Gradle (Groovy)

```groovy
// build.gradle
dependencies {
    implementation 'io.github.logtide-dev:logtide-core:0.9.8'
}
```

### Maven

```xml
<!-- pom.xml -->
<dependency>
    <groupId>io.github.logtide-dev</groupId>
    <artifactId>logtide-core</artifactId>
    <version>0.9.8</version>
</dependency>
```

## Quick Start (5 minutes)

### 1. Configure the LogTide Client

Create a Spring configuration bean:

```kotlin
// LogTideConfig.kt
import dev.logtide.sdk.LogTideClient
import dev.logtide.sdk.models.LogTideClientOptions
import kotlinx.coroutines.runBlocking
import org.springframework.beans.factory.annotation.Value
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import jakarta.annotation.PreDestroy
import kotlin.time.Duration.Companion.seconds

@Configuration
class LogTideConfig {

    @Value("\${logtide.api-url:https://api.logtide.dev}")
    private lateinit var apiUrl: String

    @Value("\${logtide.api-key}")
    private lateinit var apiKey: String

    private var client: LogTideClient? = null

    @Bean
    fun logTideClient(): LogTideClient {
        client = LogTideClient(
            LogTideClientOptions(
                apiUrl = apiUrl,
                apiKey = apiKey,
                batchSize = 50,
                flushInterval = 5.seconds,
                service = "spring-app"
            )
        )
        return client!!
    }

    @PreDestroy
    fun cleanup() {
        runBlocking { client?.close() }
    }
}
```

### 2. Configure Environment Variables

```yaml
# application.yml
logtide:
  api-url: https://api.logtide.dev
  api-key: ${LOGTIDE_API_KEY}
```

Or in `application.properties`:

```properties
logtide.api-url=https://api.logtide.dev
logtide.api-key=${LOGTIDE_API_KEY}
```

### 3. Start Logging

```kotlin
// UserService.kt
import dev.logtide.sdk.LogTideClient
import org.springframework.stereotype.Service

@Service
class UserService(private val logtide: LogTideClient) {

    fun createUser(email: String): User {
        logtide.info("user-service", "Creating user", mapOf("email" to email))

        val user = userRepository.save(User(email = email))

        logtide.info("user-service", "User created successfully", mapOf(
            "userId" to user.id,
            "email" to email
        ))

        return user
    }

    fun handleError(e: Exception) {
        logtide.error("user-service", "Operation failed", e)
    }
}
```

```java
// Java
import dev.logtide.sdk.LogTideClient;
import org.springframework.stereotype.Service;
import java.util.Map;

@Service
public class UserService {
    private final LogTideClient logtide;

    public UserService(LogTideClient logtide) {
        this.logtide = logtide;
    }

    public User createUser(String email) {
        logtide.info("user-service", "Creating user", Map.of("email", email));

        User user = userRepository.save(new User(email));

        logtide.info("user-service", "User created successfully", Map.of(
            "userId", user.getId(),
            "email", email
        ));

        return user;
    }
}
```

## MDC Context Propagation

MDC (Mapped Diagnostic Context) allows you to attach contextual data to all log messages within a request:

### Request Filter for Trace IDs

```kotlin
// TraceIdFilter.kt
import jakarta.servlet.FilterChain
import jakarta.servlet.http.HttpServletRequest
import jakarta.servlet.http.HttpServletResponse
import org.slf4j.MDC
import org.springframework.stereotype.Component
import org.springframework.web.filter.OncePerRequestFilter
import java.util.UUID

@Component
class TraceIdFilter : OncePerRequestFilter() {

    override fun doFilterInternal(
        request: HttpServletRequest,
        response: HttpServletResponse,
        filterChain: FilterChain
    ) {
        val traceId = request.getHeader("X-Trace-ID")
            ?: UUID.randomUUID().toString()

        try {
            MDC.put("traceId", traceId)
            MDC.put("method", request.method)
            MDC.put("uri", request.requestURI)
            MDC.put("remoteAddr", request.remoteAddr)

            response.setHeader("X-Trace-ID", traceId)
            filterChain.doFilter(request, response)
        } finally {
            MDC.clear()
        }
    }
}
```

### Controller with MDC

```kotlin
// UserController.kt
import org.slf4j.LoggerFactory
import org.slf4j.MDC
import org.springframework.web.bind.annotation.*

@RestController
@RequestMapping("/api/users")
class UserController(private val userService: UserService) {

    private val logger = LoggerFactory.getLogger(javaClass)

    @PostMapping
    fun createUser(@RequestBody request: CreateUserRequest): User {
        // Add user context to MDC
        MDC.put("action", "createUser")

        logger.info("Creating new user")

        val user = userService.createUser(request.email)

        // Add user ID after creation
        MDC.put("userId", user.id.toString())

        logger.info("User created successfully")

        return user
    }
}
```

All logs will now include:
```json
{
  "level": "INFO",
  "message": "User created successfully",
  "traceId": "abc-123-def",
  "method": "POST",
  "uri": "/api/users",
  "action": "createUser",
  "userId": "42"
}
```

## WebFlux Reactive Logging

For Spring WebFlux applications, use reactor context propagation:

### Reactive MDC Filter

```kotlin
// ReactiveTraceFilter.kt
import org.slf4j.MDC
import org.springframework.stereotype.Component
import org.springframework.web.server.ServerWebExchange
import org.springframework.web.server.WebFilter
import org.springframework.web.server.WebFilterChain
import reactor.core.publisher.Mono
import reactor.util.context.Context
import java.util.UUID

@Component
class ReactiveTraceFilter : WebFilter {

    override fun filter(exchange: ServerWebExchange, chain: WebFilterChain): Mono<Void> {
        val traceId = exchange.request.headers.getFirst("X-Trace-ID")
            ?: UUID.randomUUID().toString()

        val contextMap = mapOf(
            "traceId" to traceId,
            "method" to exchange.request.method.name(),
            "uri" to exchange.request.path.value()
        )

        return chain.filter(exchange)
            .contextWrite { ctx ->
                ctx.put("logtide-mdc", contextMap)
            }
            .doOnEach { signal ->
                signal.contextView.getOrEmpty<Map<String, String>>("logtide-mdc")
                    .ifPresent { mdc ->
                        mdc.forEach { (key, value) -> MDC.put(key, value) }
                    }
            }
            .doFinally {
                MDC.clear()
            }
    }
}
```

### Reactive Controller

```kotlin
// ReactiveUserController.kt
import kotlinx.coroutines.slf4j.MDCContext
import kotlinx.coroutines.withContext
import org.slf4j.LoggerFactory
import org.springframework.web.bind.annotation.*
import reactor.core.publisher.Mono

@RestController
@RequestMapping("/api/users")
class ReactiveUserController(private val userService: UserService) {

    private val logger = LoggerFactory.getLogger(javaClass)

    @GetMapping("/{id}")
    suspend fun getUser(@PathVariable id: Long): User = withContext(MDCContext()) {
        logger.info("Fetching user")
        userService.findById(id)
    }

    @PostMapping
    fun createUser(@RequestBody request: CreateUserRequest): Mono<User> {
        return Mono.fromCallable {
            logger.info("Creating user")
            userService.createUser(request.email)
        }.doOnSuccess { user ->
            logger.info("User created: ${user.id}")
        }
    }
}
```

## Structured Logging with Kotlin

Use structured arguments for better queryability:

```kotlin
// Using kotlin-logging library
import io.github.oshai.kotlinlogging.KotlinLogging

private val logger = KotlinLogging.logger {}

class OrderService {
    fun processOrder(order: Order) {
        logger.info {
            "Processing order" to mapOf(
                "orderId" to order.id,
                "amount" to order.total,
                "currency" to order.currency,
                "items" to order.items.size
            )
        }

        try {
            paymentService.charge(order)
            logger.info { "Order payment successful" }
        } catch (e: PaymentException) {
            logger.error(e) {
                "Payment failed" to mapOf(
                    "orderId" to order.id,
                    "errorCode" to e.code
                )
            }
            throw e
        }
    }
}
```

## Actuator Integration

Expose logging configuration via Spring Boot Actuator:

```yaml
# application.yml
management:
  endpoints:
    web:
      exposure:
        include: health,info,loggers,metrics
  endpoint:
    health:
      show-details: when_authorized

logging:
  level:
    root: INFO
    com.yourcompany: DEBUG
    dev.logtide: DEBUG
```

### Custom Health Indicator

```kotlin
// LogTideHealthIndicator.kt
import org.springframework.boot.actuate.health.Health
import org.springframework.boot.actuate.health.HealthIndicator
import org.springframework.stereotype.Component

@Component
class LogTideHealthIndicator : HealthIndicator {

    override fun health(): Health {
        return try {
            // Check LogTide connectivity
            val healthy = checkLogTideConnection()
            if (healthy) {
                Health.up()
                    .withDetail("status", "Connected")
                    .build()
            } else {
                Health.down()
                    .withDetail("status", "Disconnected")
                    .build()
            }
        } catch (e: Exception) {
            Health.down(e).build()
        }
    }

    private fun checkLogTideConnection(): Boolean {
        // Implementation depends on LogTide SDK
        return true
    }
}
```

## Docker Setup

### Multi-Stage Build

```dockerfile
# Dockerfile
FROM gradle:8-jdk21 AS build
WORKDIR /app
COPY build.gradle.kts settings.gradle.kts ./
COPY src ./src
RUN gradle build -x test --no-daemon

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app

# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

COPY --from=build /app/build/libs/*.jar app.jar

EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
    CMD wget -q --spider http://localhost:8080/actuator/health || exit 1

ENTRYPOINT ["java", "-jar", "app.jar"]
```

### Docker Compose

```yaml
# docker-compose.yml
services:
  app:
    build: .
    ports:
      - "8080:8080"
    environment:
      - SPRING_PROFILES_ACTIVE=production
      - LOGTIDE_API_URL=${LOGTIDE_API_URL}
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
      - JAVA_OPTS=-Xmx512m -Xms256m
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/actuator/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    depends_on:
      - postgres

  postgres:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_USER=myapp
      - POSTGRES_PASSWORD=secret
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:
```

## Kubernetes Deployment

```yaml
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: spring-app
  template:
    metadata:
      labels:
        app: spring-app
    spec:
      containers:
        - name: app
          image: your-registry/spring-app:latest
          ports:
            - containerPort: 8080
          env:
            - name: SPRING_PROFILES_ACTIVE
              value: "kubernetes"
            - name: LOGTIDE_API_URL
              value: "https://api.logtide.dev"
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-secrets
                  key: api-key
            - name: JAVA_OPTS
              value: "-Xmx512m -Xms256m -XX:+UseG1GC"
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "1000m"
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 20
            periodSeconds: 5
---
apiVersion: v1
kind: Secret
metadata:
  name: logtide-secrets
type: Opaque
data:
  api-key: <base64-encoded-api-key>
```

## Verification

Test logging is working:

```kotlin
// HealthController.kt
@RestController
class HealthController {

    private val logger = LoggerFactory.getLogger(javaClass)

    @GetMapping("/test-logging")
    fun testLogging(): Map<String, String> {
        logger.debug("Debug message from test endpoint")
        logger.info("Info message from test endpoint")
        logger.warn("Warning message from test endpoint")
        logger.error("Error message from test endpoint")

        return mapOf("status" to "Logs sent to LogTide")
    }
}
```

```bash
curl http://localhost:8080/test-logging
```

Check LogTide dashboard for logs with your application context.

## Performance Impact

Benchmarks from production Spring Boot application (1000 RPS):

| Metric | Without LogTide | With LogTide (Async) | Overhead |
|--------|-----------------|---------------------|----------|
| Avg latency | 25ms | 25.5ms | +2% |
| P99 latency | 85ms | 87ms | +2.4% |
| Memory usage | 450MB | 470MB | +4.4% |
| CPU usage | 35% | 36% | +2.9% |

**Notes:**
- AsyncAppender ensures logging never blocks request processing
- Queue size of 1024 handles burst traffic without drops
- G1GC recommended for predictable pause times

## Troubleshooting

### Logs not appearing in LogTide

1. **Check client initialization:**
```kotlin
// Enable debug logging
val client = LogTideClient(
    LogTideClientOptions(
        apiUrl = apiUrl,
        apiKey = apiKey,
        debug = true  // Enable internal logging
    )
)
```

2. **Test connectivity:**
```bash
curl -H "X-API-Key: $LOGTIDE_API_KEY" \
     https://api.logtide.dev/api/v1/ingest \
     -d '{"logs":[{"service":"test","level":"info","message":"test"}]}'
```

3. **Verify API key format:**
API keys should start with `lp_` prefix.

### Trace context not propagating

For async/reactive applications, ensure context propagation is configured:

```kotlin
// For Kotlin coroutines - use withContext to propagate trace ID
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.7.3")
```

### High memory usage

Reduce batch size and buffer limits in your configuration:

```kotlin
val client = LogTideClient(
    LogTideClientOptions(
        apiUrl = apiUrl,
        apiKey = apiKey,
        batchSize = 20,           // Smaller batches
        flushInterval = 2.seconds, // More frequent flushes
        maxBufferSize = 500       // Limit buffer size
    )
)
```

## Next Steps

- **[Docker Integration](/integrations/docker/)** - Container deployment patterns
- **[Kubernetes Integration](/integrations/kubernetes/)** - Production K8s deployment
- **[PostgreSQL Integration](/integrations/postgresql/)** - Database logging correlation
- **[Multi-tenant SaaS](/use-cases/multi-tenant-saas/)** - Tenant isolation patterns
- **[GDPR Compliance](/use-cases/gdpr-compliance/)** - Privacy-first logging


---

# SvelteKit Application Logging Integration

> Add structured logging to SvelteKit applications with server hooks, error handling, fetch instrumentation, and client init.

*Source: https://logtide.dev/integrations/sveltekit/*

---

LogTide's SvelteKit SDK integrates through SvelteKit's hooks system — `handle`, `handleError`, and `handleFetch` — with support for both server-side and client-side error capture.

## Why use LogTide with SvelteKit?

- **Hook-based**: Integrates through SvelteKit's native hooks
- **Fetch instrumentation**: Trace propagation on server-side fetches
- **Form actions**: Capture errors in form action handlers
- **Client + server**: Error capture on both sides
- **Load functions**: Context available in server load functions

## Prerequisites

- SvelteKit 2.x
- Node.js 18+
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/sveltekit
```

## Quick Start

### Server Hooks

```typescript
// src/hooks.server.ts
import { logtideHandle, logtideHandleError } from '@logtide/sveltekit';

export const handle = logtideHandle({
  dsn: import.meta.env.LOGTIDE_DSN,
  service: 'sveltekit-app',
  environment: import.meta.env.MODE,
});
export const handleError = logtideHandleError();
```

### Client Hooks

```typescript
// src/hooks.client.ts
import { logtideHandleError } from '@logtide/sveltekit';

export const handleError = logtideHandleError();
```

## Composing with Other Hooks

```typescript
import { sequence } from '@sveltejs/kit/hooks';
import { logtideHandle } from '@logtide/sveltekit';

export const handle = sequence(
  logtideHandle({
    dsn: import.meta.env.LOGTIDE_DSN,
    service: 'sveltekit-app',
    environment: import.meta.env.MODE,
  }),
  ({ event, resolve }) => {
    // Your custom logic
    return resolve(event);
  }
);
```

## Fetch Instrumentation

```typescript
// src/hooks.server.ts
import { logtideHandleFetch } from '@logtide/sveltekit';

export const handleFetch = logtideHandleFetch();
// Outgoing fetches from load functions include traceparent headers
```

## Load Functions

```typescript
// src/routes/users/[id]/+page.server.ts
import { hub } from '@logtide/core';

export async function load({ params, fetch }) {
  hub.captureLog('info', 'Loading user', { userId: params.id });

  const res = await fetch(`/api/users/${params.id}`);
  if (!res.ok) {
    hub.captureLog('warn', 'User not found', { userId: params.id });
    throw error(404, 'Not found');
  }

  return { user: await res.json() };
}
```

## Form Actions

```typescript
// src/routes/settings/+page.server.ts
import { hub } from '@logtide/core';

export const actions = {
  update: async ({ request }) => {
    const data = await request.formData();

    hub.addBreadcrumb({
      type: 'ui',
      category: 'form',
      message: 'Settings form submitted',
      timestamp: Date.now(),
    });

    try {
      await updateSettings(Object.fromEntries(data));
      hub.captureLog('info', 'Settings updated');
      return { success: true };
    } catch (error) {
      hub.captureError(error);
      return fail(500, { message: 'Update failed' });
    }
  },
};
```

## Next Steps

- [JavaScript SDK](/docs/sdks/nodejs/) - Core SDK reference
- [SvelteKit SDK Reference](/docs/sdks/sveltekit/) - Full API documentation
- [Docker Integration](/integrations/docker/) - Container deployments


---

# systemd Journal Integration

> Stream systemd journal logs to LogTide with structured metadata from services, containers, and system components.

*Source: https://logtide.dev/integrations/systemd/*

---

systemd's journal provides rich structured logging for Linux services. This guide shows you how to forward journal logs to LogTide while preserving all the valuable metadata systemd captures automatically.

## Why forward systemd journal logs?

- **Rich metadata**: Unit names, PIDs, UIDs, SELinux contexts, and more
- **Structured by default**: Journal entries are inherently structured
- **System-wide visibility**: Collect from kernel, services, and containers
- **Boot-persistent**: Track issues across reboots
- **Priority filtering**: Route logs by severity level
- **No code changes**: Capture logs from any systemd service

## Prerequisites

- Linux system with systemd 245+ (Ubuntu 20.04+, RHEL 8+, Debian 11+)
- Fluent Bit 2.0+ installed
- LogTide instance with API key
- Root or sudo access for configuration

## Installation

### Install Fluent Bit

```bash
# Ubuntu/Debian
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

# RHEL/CentOS/Fedora
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

# Or via package manager
# Ubuntu/Debian
sudo apt-get install fluent-bit

# RHEL/CentOS (enable EPEL first)
sudo yum install fluent-bit
```

### Verify Installation

```bash
fluent-bit --version
# Fluent Bit v2.x.x
```

## Quick Start (10 minutes)

### 1. Configure Fluent Bit

Create `/etc/fluent-bit/fluent-bit.conf`:

```ini
[SERVICE]
    Flush         5
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf

[INPUT]
    Name            systemd
    Tag             systemd.*
    Read_From_Tail  On
    Strip_Underscores On

[OUTPUT]
    Name          http
    Match         *
    Host          api.logtide.dev
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
```

### 2. Set API Key

```bash
# Create environment file
sudo mkdir -p /etc/fluent-bit
echo "LOGTIDE_API_KEY=your-api-key" | sudo tee /etc/fluent-bit/environment
sudo chmod 600 /etc/fluent-bit/environment
```

### 3. Configure systemd Service

Create `/etc/systemd/system/fluent-bit.service.d/override.conf`:

```ini
[Service]
EnvironmentFile=/etc/fluent-bit/environment
```

### 4. Start Fluent Bit

```bash
sudo systemctl daemon-reload
sudo systemctl enable fluent-bit
sudo systemctl start fluent-bit
sudo systemctl status fluent-bit
```

## Production Configuration

### Full Fluent Bit Configuration

```ini
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush             5
    Grace             30
    Log_Level         info
    Daemon            off
    Parsers_File      parsers.conf
    HTTP_Server       On
    HTTP_Listen       0.0.0.0
    HTTP_Port         2020
    Health_Check      On
    storage.path      /var/log/fluent-bit/
    storage.sync      normal
    storage.checksum  off
    storage.backlog.mem_limit 50M

# Read from systemd journal
[INPUT]
    Name              systemd
    Tag               systemd.*
    Path              /var/log/journal
    Read_From_Tail    On
    Strip_Underscores On
    DB                /var/log/fluent-bit/systemd.db

# Filter to specific units (optional)
[INPUT]
    Name                     systemd
    Tag                      systemd.apps.*
    Systemd_Filter           _SYSTEMD_UNIT=nginx.service
    Systemd_Filter           _SYSTEMD_UNIT=myapp.service
    Systemd_Filter           _SYSTEMD_UNIT=postgresql.service
    Read_From_Tail           On
    Strip_Underscores        On
    DB                       /var/log/fluent-bit/systemd-apps.db

# Add hostname and environment
[FILTER]
    Name          record_modifier
    Match         systemd.*
    Record        hostname ${HOSTNAME}
    Record        environment production
    Record        source systemd

# Map journal fields to LogTide fields
[FILTER]
    Name          modify
    Match         systemd.*
    Rename        SYSTEMD_UNIT unit
    Rename        PRIORITY priority
    Rename        MESSAGE message
    Rename        SYSLOG_IDENTIFIER service
    Rename        _PID pid
    Rename        _UID uid
    Rename        _GID gid
    Rename        _CMDLINE cmdline
    Rename        _EXE exe

# Convert priority to level
[FILTER]
    Name          lua
    Match         systemd.*
    script        /etc/fluent-bit/priority_to_level.lua
    call          priority_to_level

# Output to LogTide
[OUTPUT]
    Name                  http
    Match                 *
    Host                  api.logtide.dev
    Port                  443
    URI                   /api/v1/ingest/single
    Format                json
    Header                X-API-Key ${LOGTIDE_API_KEY}
    Header                Content-Type application/json
    tls                   On
    tls.verify            On
    Retry_Limit           5
    storage.total_limit_size 100M
```

### Priority to Level Mapping

Create the Lua script for converting syslog priorities to log levels:

```bash
# Create the Lua script
sudo tee /etc/fluent-bit/priority_to_level.lua << 'LUAEOF'

```lua
-- Map syslog priority to log level
local priority_map = {
    ["0"] = "critical",  -- EMERG
    ["1"] = "critical",  -- ALERT
    ["2"] = "critical",  -- CRIT
    ["3"] = "error",     -- ERR
    ["4"] = "warning",   -- WARNING
    ["5"] = "info",      -- NOTICE
    ["6"] = "info",      -- INFO
    ["7"] = "debug"      -- DEBUG
}

function priority_to_level(tag, timestamp, record)
    local priority = record["priority"]
    if priority then
        record["level"] = priority_map[tostring(priority)] or "info"
    else
        record["level"] = "info"
    end
    return 1, timestamp, record
end
LUAEOF

# Set proper permissions
sudo chmod 644 /etc/fluent-bit/priority_to_level.lua
```

## Filtering by Unit

### Filter Specific Services

Only collect logs from specific units:

```ini
[INPUT]
    Name                     systemd
    Tag                      systemd.web.*
    Systemd_Filter           _SYSTEMD_UNIT=nginx.service
    Systemd_Filter           _SYSTEMD_UNIT=apache2.service
    Read_From_Tail           On
```

### Filter by Priority

Only collect warnings and above:

```ini
[INPUT]
    Name                     systemd
    Tag                      systemd.important.*
    Systemd_Filter           PRIORITY=0
    Systemd_Filter           PRIORITY=1
    Systemd_Filter           PRIORITY=2
    Systemd_Filter           PRIORITY=3
    Systemd_Filter           PRIORITY=4
    Read_From_Tail           On
```

### Exclude Noisy Units

```ini
[FILTER]
    Name     grep
    Match    systemd.*
    Exclude  unit ^(systemd-timesyncd|snapd)\.service$
```

## Creating Application Services

### Example: Node.js Application Service

Create `/etc/systemd/system/myapp.service`:

```ini
[Unit]
Description=My Node.js Application
After=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=always
RestartSec=10

# Environment
Environment=NODE_ENV=production
Environment=PORT=3000
EnvironmentFile=/opt/myapp/.env

# Logging (goes to journal)
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/opt/myapp/data

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable myapp
sudo systemctl start myapp
```

### Application Logging Best Practices

Structure your application logs for better parsing:

```javascript
// Node.js example
const log = (level, message, data = {}) => {
  console.log(JSON.stringify({
    timestamp: new Date().toISOString(),
    level,
    message,
    ...data
  }));
};

// Usage
log('info', 'Server started', { port: 3000 });
log('error', 'Database connection failed', { error: err.message });
```

Fluent Bit can parse this JSON:

```ini
[FILTER]
    Name          parser
    Match         systemd.apps.myapp
    Key_Name      MESSAGE
    Parser        json
    Reserve_Data  On
```

## Docker with journald Driver

Use journald as Docker's logging driver:

### Configure Docker Daemon

Edit `/etc/docker/daemon.json`:

```json
{
  "log-driver": "journald",
  "log-opts": {
    "tag": "docker/{{.Name}}"
  }
}
```

Restart Docker:

```bash
sudo systemctl restart docker
```

### Collect Docker Logs via Journal

```ini
[INPUT]
    Name                     systemd
    Tag                      docker.*
    Systemd_Filter           CONTAINER_NAME=*
    Read_From_Tail           On
    Strip_Underscores        On

[FILTER]
    Name          modify
    Match         docker.*
    Rename        CONTAINER_NAME container
    Rename        CONTAINER_ID container_id
    Rename        IMAGE_NAME image
```

## Journal Metadata Fields

systemd journal provides rich metadata automatically:

| Journal Field | Description | Example |
|---------------|-------------|---------|
| `_SYSTEMD_UNIT` | Service unit name | `nginx.service` |
| `_PID` | Process ID | `1234` |
| `_UID` | User ID | `1000` |
| `_GID` | Group ID | `1000` |
| `_COMM` | Command name | `nginx` |
| `_EXE` | Executable path | `/usr/sbin/nginx` |
| `_CMDLINE` | Full command line | `nginx: master process` |
| `PRIORITY` | Syslog priority (0-7) | `6` |
| `SYSLOG_IDENTIFIER` | Service identifier | `nginx` |
| `_HOSTNAME` | Hostname | `web-server-1` |
| `_MACHINE_ID` | Machine UUID | `abc123...` |
| `_BOOT_ID` | Boot UUID | `def456...` |
| `__REALTIME_TIMESTAMP` | Timestamp (microseconds) | `1706972400000000` |

## Persistent Journal Configuration

Ensure journal logs persist across reboots:

```bash
# Create journal directory
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal

# Configure journald
sudo tee /etc/systemd/journald.conf.d/persistent.conf << EOF
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=2G
SystemKeepFree=1G
MaxRetentionSec=30day
ForwardToSyslog=no
EOF

# Restart journald
sudo systemctl restart systemd-journald
```

## Verification

### Check Fluent Bit Status

```bash
# Service status
sudo systemctl status fluent-bit

# Fluent Bit logs
sudo journalctl -u fluent-bit -f

# HTTP health check
curl http://localhost:2020/api/v1/health
```

### Verify Journal Reading

```bash
# Check Fluent Bit is reading from journal
sudo journalctl -u fluent-bit | grep -i "systemd"

# Send a test log
logger -p user.info "Test message from $(hostname)"

# Check in LogTide for the test message
```

### View Journal Entries

```bash
# Recent entries
journalctl -n 50

# Specific unit
journalctl -u nginx.service -f

# JSON output (see what Fluent Bit sees)
journalctl -o json -n 1 | jq .
```

## Performance Tuning

### Memory Usage

Control Fluent Bit memory usage:

```ini
[SERVICE]
    storage.backlog.mem_limit 25M

[INPUT]
    Name       systemd
    Mem_Buf_Limit 10M
```

### Read Position Tracking

Use a database to track read position:

```ini
[INPUT]
    Name    systemd
    DB      /var/log/fluent-bit/systemd.db
```

### Disk Buffering

Enable disk buffering for reliability:

```ini
[SERVICE]
    storage.path     /var/log/fluent-bit/
    storage.sync     normal
    storage.checksum off

[OUTPUT]
    Name    http
    storage.total_limit_size 200M
```

## Security Considerations

### Run Fluent Bit with Minimal Permissions

```ini
# /etc/systemd/system/fluent-bit.service.d/security.conf
[Service]
# Read-only access to journal
ReadOnlyPaths=/var/log/journal
ReadWritePaths=/var/log/fluent-bit

# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
```

### Filter Sensitive Data

Remove or hash sensitive information:

```ini
[FILTER]
    Name          modify
    Match         *
    Remove        _SELINUX_CONTEXT
    Remove        _SOURCE_REALTIME_TIMESTAMP

[FILTER]
    Name          lua
    Match         *
    script        /etc/fluent-bit/sanitize.lua
    call          sanitize_logs
```

```lua
-- /etc/fluent-bit/sanitize.lua
function sanitize_logs(tag, timestamp, record)
    -- Hash user IDs
    if record["uid"] then
        record["uid_hash"] = hash(record["uid"])
        record["uid"] = nil
    end

    -- Remove command line (may contain secrets)
    record["cmdline"] = nil

    return 1, timestamp, record
end

function hash(value)
    -- Simple hash for demonstration
    local h = 0
    for i = 1, #value do
        h = (h * 31 + string.byte(value, i)) % 2147483647
    end
    return string.format("%x", h)
end
```

## Troubleshooting

### Fluent Bit not reading from journal

1. **Check permissions:**
```bash
# Fluent Bit user needs access to journal
sudo usermod -a -G systemd-journal fluent-bit
sudo systemctl restart fluent-bit
```

2. **Verify journal path:**
```bash
ls -la /var/log/journal/
# Or for volatile journal:
ls -la /run/log/journal/
```

3. **Check Fluent Bit logs:**
```bash
sudo journalctl -u fluent-bit -n 100 --no-pager
```

### High CPU usage

Reduce processing with filters:

```ini
[INPUT]
    Name       systemd
    # Skip old logs
    Read_From_Tail On
    # Only specific units
    Systemd_Filter _SYSTEMD_UNIT=myapp.service
```

### Missing logs after reboot

Enable position tracking:

```ini
[INPUT]
    Name    systemd
    DB      /var/log/fluent-bit/systemd.db
    DB.Sync Full
```

## Next Steps

- **[Docker Integration](/integrations/docker/)** - Container logging patterns
- **[nginx Integration](/integrations/nginx/)** - Web server log parsing
- **[PostgreSQL Integration](/integrations/postgresql/)** - Database logging
- **[Security Monitoring](/use-cases/security-monitoring/)** - Build alerting on journal events


---

# Traefik Reverse Proxy Log Integration

> Forward Traefik access logs and middleware metrics to LogTide with structured JSON parsing for Docker and Kubernetes environments.

*Source: https://logtide.dev/integrations/traefik/*

---

Traefik is a modern, cloud-native reverse proxy and ingress controller designed for microservices. This guide shows you how to forward Traefik access logs, middleware metrics, and error events to LogTide for centralized visibility across your infrastructure.

## Why use LogTide with Traefik?

- **Request visibility**: Analyze traffic patterns, status codes, and latencies across all your services
- **Middleware insights**: Track rate limiting, authentication, and circuit breaker events in real time
- **Security monitoring**: Detect suspicious traffic, brute-force attempts, and unauthorized access
- **Centralized logging**: Correlate Traefik routing decisions with backend application logs
- **Dynamic routing audit**: Understand which routes are active and how traffic flows through your infrastructure
- **Multi-environment**: Consistent log collection whether Traefik runs in Docker, Kubernetes, or bare metal

## Prerequisites

- Traefik v2.10+ or v3.x
- LogTide instance with API key
- One of the following log forwarding tools:
  - Vector (recommended for high throughput)
  - Fluent Bit
  - Or direct file tailing via HTTP API

## Quick Start (10 minutes)

### Step 1: Enable Traefik JSON Access Logs

Traefik supports structured JSON access logs out of the box. Configure via static configuration.

**Using `traefik.yml`:**

```yaml
# /etc/traefik/traefik.yml
accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  bufferingSize: 100
  fields:
    defaultMode: keep
    names:
      ClientUsername: drop
    headers:
      defaultMode: drop
      names:
        User-Agent: keep
        Authorization: redact
        X-Forwarded-For: keep
        X-Real-Ip: keep
        X-Request-Id: keep

log:
  filePath: "/var/log/traefik/traefik.log"
  level: WARN
  format: json
```

**Using CLI flags:**

```bash
traefik \
  --accesslog=true \
  --accesslog.filepath=/var/log/traefik/access.log \
  --accesslog.format=json \
  --accesslog.bufferingsize=100 \
  --log.filepath=/var/log/traefik/traefik.log \
  --log.level=WARN \
  --log.format=json
```

**Using environment variables:**

```bash
TRAEFIK_ACCESSLOG=true
TRAEFIK_ACCESSLOG_FILEPATH=/var/log/traefik/access.log
TRAEFIK_ACCESSLOG_FORMAT=json
TRAEFIK_LOG_FILEPATH=/var/log/traefik/traefik.log
TRAEFIK_LOG_LEVEL=WARN
TRAEFIK_LOG_FORMAT=json
```

### Step 2: Set Up Vector for Log Forwarding

Vector is the recommended forwarder for Traefik logs due to its high throughput and native JSON support.

Install Vector:

```bash
# Ubuntu/Debian
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | sudo bash
sudo apt-get install vector

# RHEL/CentOS
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.rpm.sh' | sudo bash
sudo yum install vector
```

Create `/etc/vector/vector.toml`:

```toml
# /etc/vector/vector.toml

# Traefik access logs
[sources.traefik_access]
type = "file"
include = ["/var/log/traefik/access.log"]
read_from = "end"

[transforms.parse_traefik_access]
type = "remap"
inputs = ["traefik_access"]
source = '''
. = parse_json!(.message)
.service = "traefik"
.log_type = "access"
.level = if .DownstreamStatus >= 500 { "error" } else if .DownstreamStatus >= 400 { "warn" } else { "info" }
.message = join!(
  [.RequestMethod, .RequestPath, to_string!(.DownstreamStatus)],
  separator: " "
)
'''

# Traefik application logs (errors, warnings)
[sources.traefik_app]
type = "file"
include = ["/var/log/traefik/traefik.log"]
read_from = "end"

[transforms.parse_traefik_app]
type = "remap"
inputs = ["traefik_app"]
source = '''
. = parse_json!(.message)
.service = "traefik"
.log_type = "application"
'''

# Ship to LogTide
[sinks.logtide]
type = "http"
inputs = ["parse_traefik_access", "parse_traefik_app"]
uri = "https://YOUR_LOGTIDE_HOST/api/v1/ingest"
method = "post"
encoding.codec = "json"

[sinks.logtide.request.headers]
X-API-Key = "${LOGTIDE_API_KEY}"
Content-Type = "application/json"

[sinks.logtide.batch]
max_bytes = 1048576
timeout_secs = 5

[sinks.logtide.buffer]
type = "disk"
max_size = 268435456
when_full = "block"
```

Start Vector:

```bash
sudo systemctl enable vector
sudo systemctl start vector
```

### Step 3: Verify Logs in LogTide

Generate some traffic and check LogTide:

```bash
curl -I http://localhost/
# Filter in LogTide: service:traefik
```

## Docker Setup

### docker-compose.yml with Labels

Traefik integrates natively with Docker using labels for dynamic routing. Here is a complete setup with log forwarding:

```yaml
# docker-compose.yml
services:
  traefik:
    image: traefik:v3.1
    container_name: traefik
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--accesslog=true"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--accesslog.format=json"
      - "--accesslog.bufferingsize=100"
      - "--accesslog.fields.defaultmode=keep"
      - "--accesslog.fields.headers.defaultmode=drop"
      - "--accesslog.fields.headers.names.User-Agent=keep"
      - "--accesslog.fields.headers.names.X-Forwarded-For=keep"
      - "--accesslog.fields.headers.names.X-Request-Id=keep"
      - "--log.filepath=/var/log/traefik/traefik.log"
      - "--log.level=WARN"
      - "--log.format=json"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik_logs:/var/log/traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.local`)"
      - "traefik.http.routers.dashboard.service=api@internal"

  vector:
    image: timberio/vector:latest-alpine
    container_name: vector
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - traefik_logs:/var/log/traefik:ro
      - vector_data:/var/lib/vector
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
    depends_on:
      - traefik
    restart: unless-stopped

  # Example backend service
  whoami:
    image: traefik/whoami
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.local`)"
      - "traefik.http.routers.whoami.entrypoints=web"

volumes:
  traefik_logs:
  vector_data:
```

### Docker Log Driver Alternative

Instead of file-based logging, capture Traefik stdout/stderr with Fluent Bit:

```yaml
services:
  traefik:
    image: traefik:v3.1
    command:
      - "--accesslog=true"
      - "--accesslog.format=json"
      - "--log.level=WARN"
      - "--log.format=json"
      # Logs go to stdout/stderr (default when no filepath set)
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: traefik
        fluentd-async: "true"

  fluent-bit:
    image: fluent/fluent-bit:latest
    ports:
      - "24224:24224"
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro
    environment:
      - LOGTIDE_API_KEY=${LOGTIDE_API_KEY}
```

## Kubernetes IngressRoute Setup

### Traefik CRD with Logging

For Kubernetes, Traefik is typically deployed via Helm with custom values:

```yaml
# values.yaml for Traefik Helm chart
additionalArguments:
  - "--accesslog=true"
  - "--accesslog.format=json"
  - "--accesslog.fields.defaultmode=keep"
  - "--accesslog.fields.headers.defaultmode=drop"
  - "--accesslog.fields.headers.names.User-Agent=keep"
  - "--accesslog.fields.headers.names.X-Forwarded-For=keep"
  - "--accesslog.fields.headers.names.X-Request-Id=keep"
  - "--log.level=WARN"
  - "--log.format=json"

logs:
  access:
    enabled: true
    format: json
  general:
    level: WARN
    format: json
```

Deploy with Helm:

```bash
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm install traefik traefik/traefik -f values.yaml -n traefik --create-namespace
```

### IngressRoute Example

```yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: my-app
  namespace: default
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: Host(`app.example.com`)
      kind: Rule
      services:
        - name: my-app-service
          port: 80
      middlewares:
        - name: rate-limit
        - name: security-headers
```

### Fluent Bit DaemonSet for Traefik Pods

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-traefik
  namespace: traefik
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info

    [INPUT]
        Name              tail
        Path              /var/log/containers/traefik-*.log
        Parser            cri
        Tag               traefik.*
        Refresh_Interval  5
        DB                /var/log/fluent-bit/traefik.db

    [FILTER]
        Name          parser
        Match         traefik.*
        Key_Name      log
        Parser        json
        Reserve_Data  On

    [FILTER]
        Name          modify
        Match         traefik.*
        Add           service traefik
        Add           environment production
        Add           cluster ${CLUSTER_NAME}

    [OUTPUT]
        Name          http
        Match         traefik.*
        Host          YOUR_LOGTIDE_HOST
        Port          443
        URI           /api/v1/ingest/single
        Format        json
        Header        X-API-Key ${LOGTIDE_API_KEY}
        Header        Content-Type application/json
        tls           On
        tls.verify    On
```

## Middleware Logging

### Rate Limiting Middleware

Configure rate limiting and log rejected requests:

```yaml
# Docker labels
labels:
  - "traefik.http.middlewares.rate-limit.ratelimit.average=100"
  - "traefik.http.middlewares.rate-limit.ratelimit.burst=50"
  - "traefik.http.middlewares.rate-limit.ratelimit.period=1m"
  - "traefik.http.routers.my-app.middlewares=rate-limit"
```

Rate-limited requests appear in access logs with `429` status codes. Create a LogTide alert:

```
service:traefik AND DownstreamStatus:429
```

### Authentication Middleware

Track authentication events:

```yaml
# Kubernetes middleware
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: auth-forward
spec:
  forwardAuth:
    address: http://auth-service:4180/verify
    trustForwardHeader: true
    authResponseHeaders:
      - X-Auth-User
      - X-Auth-Groups
```

Monitor failed authentication:

```
service:traefik AND DownstreamStatus:401
```

### Circuit Breaker Middleware

```yaml
labels:
  - "traefik.http.middlewares.cb.circuitbreaker.expression=LatencyAtQuantileMS(50.0) > 1000"
  - "traefik.http.middlewares.cb.circuitbreaker.checkperiod=10s"
  - "traefik.http.middlewares.cb.circuitbreaker.fallbackduration=30s"
```

Monitor circuit breaker trips in LogTide:

```
service:traefik AND DownstreamStatus:503 AND message:*circuit*
```

## Access Log Fields Reference

Traefik JSON access logs include these fields:

| Field | Description | Example |
|-------|-------------|---------|
| `StartUTC` | Request start time | `2025-01-15T10:30:00Z` |
| `Duration` | Total request duration (ns) | `45000000` |
| `RouterName` | Matched router name | `my-app@docker` |
| `ServiceName` | Backend service name | `my-app@docker` |
| `ServiceURL` | Backend URL | `http://172.18.0.3:8080` |
| `DownstreamStatus` | Response status code | `200` |
| `DownstreamContentSize` | Response body size | `1234` |
| `RequestMethod` | HTTP method | `GET` |
| `RequestPath` | Request path | `/api/users` |
| `RequestProtocol` | HTTP version | `HTTP/2.0` |
| `ClientAddr` | Client IP and port | `192.168.1.100:54321` |
| `ClientHost` | Client IP only | `192.168.1.100` |
| `RequestCount` | Request count on connection | `5` |
| `entryPointName` | Traefik entrypoint | `websecure` |
| `TLSVersion` | TLS version used | `1.3` |
| `TLSCipher` | TLS cipher suite | `TLS_AES_128_GCM_SHA256` |

## Detection Rules

Create alerts for common Traefik issues:

### High Error Rate

```
service:traefik AND DownstreamStatus:>=500
```

Threshold: >20 errors in 5 minutes.

### Slow Backend Responses

```
service:traefik AND Duration:>2000000000
```

Requests where the backend took longer than 2 seconds (duration is in nanoseconds).

### Potential DDoS or Scan

```
service:traefik AND DownstreamStatus:404
```

Threshold: >200 in 1 minute from the same `ClientHost`.

### TLS Errors

```
service:traefik AND log_type:application AND msg:*TLS*
```

Catch certificate issues and TLS handshake failures.

### Service Down

```
service:traefik AND DownstreamStatus:502
```

Backend service unreachable -- often indicates a crashed container or misconfigured service discovery.

## Fluent Bit Alternative

If you prefer Fluent Bit over Vector:

```ini
# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name          tail
    Path          /var/log/traefik/access.log
    Tag           traefik.access
    Refresh_Interval 5

[INPUT]
    Name          tail
    Path          /var/log/traefik/traefik.log
    Tag           traefik.app
    Refresh_Interval 5

[FILTER]
    Name          parser
    Match         traefik.*
    Key_Name      log
    Parser        json
    Reserve_Data  On

[FILTER]
    Name          modify
    Match         traefik.*
    Add           service traefik

[FILTER]
    Name          lua
    Match         traefik.access
    script        /etc/fluent-bit/traefik_level.lua
    call          set_level

[OUTPUT]
    Name          http
    Match         traefik.*
    Host          YOUR_LOGTIDE_HOST
    Port          443
    URI           /api/v1/ingest/single
    Format        json
    Header        X-API-Key ${LOGTIDE_API_KEY}
    Header        Content-Type application/json
    tls           On
    tls.verify    On
    Retry_Limit   5
```

Create `/etc/fluent-bit/traefik_level.lua`:

```lua
function set_level(tag, timestamp, record)
    local status = tonumber(record["DownstreamStatus"]) or 0
    if status >= 500 then
        record["level"] = "error"
    elseif status >= 400 then
        record["level"] = "warn"
    else
        record["level"] = "info"
    end
    record["message"] = string.format(
        "%s %s %d",
        record["RequestMethod"] or "GET",
        record["RequestPath"] or "/",
        status
    )
    return 1, timestamp, record
end
```

## Performance Metrics

| Metric | Value | Notes |
|--------|-------|-------|
| Access log overhead | <1ms per request | JSON format with buffering |
| Disk I/O | ~400 bytes/request | With default field set |
| Vector memory | ~50MB | Tailing two log files |
| Fluent Bit memory | ~30MB | Alternative forwarder |
| Buffering size | 100 (recommended) | Flush every 100 entries |

## Troubleshooting

### Access logs not being written

1. Verify access logging is enabled:
   ```bash
   # Check Traefik dashboard or API
   curl http://localhost:8080/api/rawdata/overview | jq .
   ```

2. Check Traefik logs for configuration errors:
   ```bash
   docker logs traefik 2>&1 | grep -i "access"
   ```

3. Ensure the log directory exists and is writable:
   ```bash
   docker exec traefik ls -la /var/log/traefik/
   ```

### Logs arriving but fields are missing

Verify your field configuration allows the fields you need:

```yaml
accessLog:
  fields:
    defaultMode: keep  # Change from "drop" to "keep"
```

### Vector not shipping logs

1. Check Vector status:
   ```bash
   vector top
   # Or check systemd
   sudo systemctl status vector
   ```

2. Verify Vector can read the log file:
   ```bash
   docker exec vector cat /var/log/traefik/access.log | head -5
   ```

3. Test LogTide connectivity:
   ```bash
   curl -X POST https://YOUR_LOGTIDE_HOST/api/v1/ingest \
     -H "X-API-Key: YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"logs":[{"service":"test","level":"info","message":"traefik test"}]}'
   ```

### High memory usage with buffering

Reduce the buffering size if memory is a concern:

```yaml
accessLog:
  bufferingSize: 50  # Default is 0 (synchronous), 100 recommended
```

## Next Steps

- [Docker Integration](/integrations/docker/) - Container log collection patterns
- [Docker Compose Integration](/integrations/docker-compose/) - Multi-service logging
- [Kubernetes Integration](/integrations/kubernetes/) - Cluster-wide log aggregation
- [Security Monitoring](/use-cases/security-monitoring/) - Build alerting rules for Traefik events
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Get notified on traffic anomalies


---

# Vue.js Error Tracking & Monitoring Integration

> Professional Vue.js error tracking with automatic capture, Vue Error Handlers, Core Web Vitals, and breadcrumbs.

*Source: https://logtide.dev/integrations/vue/*

---

LogTide's Vue.js integration provides professional-grade error tracking and performance monitoring for Vue applications. It includes automatic unhandled error capture, Vue global error handlers, and Core Web Vitals collection.

## Why use LogTide with Vue?

- **Global error handler**: Automatically captures all errors in components, watchers, and lifecycle hooks.
- **Core Web Vitals**: Monitor your app's performance metrics (LCP, FID, CLS) in the real world.
- **Breadcrumbs**: Track the clicks and network requests leading up to an error.
- **Session context**: Correlate all logs and errors for a single user session.
- **Source Maps**: Map production errors back to your original Vue SFC files.

## Prerequisites

- Vue 3.0+ (Composition or Options API)
- LogTide instance with a DSN

## Installation

```bash
npm install @logtide/browser
```

## Quick Setup

Initialize LogTide and register the Vue error handler in your `main.ts` or `main.js`:

```typescript
// main.ts
import { createApp } from 'vue';
import { initLogtide } from '@logtide/browser';
import App from './App.vue';

const app = createApp(App);

// Initialize LogTide
initLogtide({
  dsn: 'https://lp_abc123@api.logtide.dev',
  service: 'vue-frontend',
  environment: 'production',
  release: '1.0.0',
});

// Register the Vue global error handler
app.config.errorHandler = (err, instance, info) => {
  // Capture the error with Vue component context
  hub.captureError(err, {
    tags: { info, component: instance?.$options?.name },
  });
};

app.mount('#app');
```

## Manual Logging in Components

Using the Composition API:

```vue
<script setup>
import { hub } from '@logtide/core';

async function handleAction() {
  hub.addBreadcrumb({
    type: 'ui',
    category: 'ui',
    message: 'User clicked action',
    timestamp: Date.now(),
  });

  try {
    await performAction();
    hub.captureLog('info', 'Action success');
  } catch (err) {
    hub.captureError(err);
  }
}
</script>
```

Using the Options API:

```javascript
export default {
  methods: {
    handleAction() {
      this.$hub.addBreadcrumb({ ... });
      // or directly use 'hub' from '@logtide/core'
    }
  }
}
```

## Core Web Vitals

LogTide automatically tracks performance metrics. No additional configuration is required. View the metrics in your LogTide Dashboard under **Metrics**.

## Source Map Uploads

To get readable stack traces in production, upload your source maps during your Vite or Webpack build:

```bash
# Using @logtide/cli
logtide sourcemaps upload --path ./dist --release 1.0.0 --apiKey YOUR_API_KEY
```

## Next Steps

- [Browser SDK Docs](/docs/sdks/browser/) - Full browser SDK reference
- [Nuxt Integration](/integrations/nuxt/) - For SSR and full-stack Vue monitoring
- [Vite Setup](/integrations/vite/) - Best practices for Vite builds



# Use Cases

---

# API Monitoring and Analytics

> Monitor API performance, track latency percentiles, detect error spikes, and analyze consumer usage patterns with LogTide.

*Source: https://logtide.dev/use-cases/api-monitoring/*

---

Your API is your product. When response times creep up, error rates spike, or a specific consumer hammers your endpoints, you need to know immediately -- not when customers start complaining. This guide shows how to build comprehensive API monitoring with LogTide, from basic request logging to latency percentile tracking and consumer analytics.

## The Problem

Most teams start with no API monitoring at all. The first sign of trouble is a customer support ticket: "Your API is slow."

```
❌ API monitoring anti-patterns:

1. No request logging     → "How many requests do we actually get?"
2. Average-only metrics   → P99 is 10x worse than average, but you can't see it
3. No error breakdown     → "500 errors are up" -- which endpoint? which consumer?
4. No rate limit tracking → Abusive consumers degrade everyone's experience
5. No version tracking    → Can't tell if v2 is faster than v1
```

| Problem | Business Impact |
|---------|----------------|
| Undetected latency spikes | Users abandon requests, revenue drops |
| Silent error rate increase | Data corruption, broken integrations |
| No consumer visibility | One bad actor degrades service for everyone |
| Missing audit trail | Cannot debug partner integration issues |

## The LogTide Approach

Turn every API request into a structured, queryable event:

```d2
direction: down
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

mobile: Mobile { class: src }
web: Web { class: src }
partners: Partners { class: src }

api: "Express / Fastify API" {
  class: group
  mw: "LogTide Request Middleware\nmethod · path · status · latency\nconsumer identity · API version\nrequest / response sizes" { class: src }
}

lt: LogTide {
  class: group
  grid-rows: 3
  dashboards: Dashboards { class: hub }
  alerts: Alerts { class: hub }
  analytics: Analytics { class: hub }
}

mobile -> api.mw { class: flow }
web -> api.mw { class: flow }
partners -> api.mw { class: flow }
api.mw -> lt: "Batched, async" { class: flow }
```

## Implementation

### 1. Express Request Logging Middleware

A production-ready middleware that captures everything you need:

```typescript
// middleware/api-logger.ts
import { LogTideClient } from '@logtide/sdk-node';
import { Request, Response, NextFunction } from 'express';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  batchSize: 200,
  flushInterval: 3000,
});

export function apiLogger() {
  return (req: Request, res: Response, next: NextFunction) => {
    const startTime = process.hrtime.bigint();
    const requestSize = parseInt(req.headers['content-length'] || '0', 10);

    // Track response size
    let responseSize = 0;
    const originalEnd = res.end.bind(res);
    res.end = function (chunk: any, ...args: any[]) {
      if (chunk) responseSize += Buffer.byteLength(chunk);
      return originalEnd(chunk, ...args);
    };

    res.on('finish', () => {
      const durationMs = Number((process.hrtime.bigint() - startTime) / 1_000_000n);
      const consumerId = extractConsumerId(req);
      const apiVersion = extractApiVersion(req);

      const metrics = {
        method: req.method,
        path: req.path,
        route: req.route?.path || req.path,  // Route pattern, not actual path
        statusCode: res.statusCode,
        durationMs,
        requestSize,
        responseSize,
        consumerId,
        apiVersion,
        userAgent: req.headers['user-agent'] || null,
        ip: req.ip,
      };

      if (res.statusCode >= 500) {
        client.error('api', `${req.method} ${req.path} ${res.statusCode}`, metrics);
      } else if (res.statusCode >= 400) {
        client.warn('api', `${req.method} ${req.path} ${res.statusCode}`, metrics);
      } else {
        client.info('api', `${req.method} ${req.path} ${res.statusCode}`, metrics);
      }
    });

    next();
  };
}

function extractConsumerId(req: Request): string | null {
  const apiKey = req.headers['x-api-key'] as string;
  if (apiKey) return `key:${apiKey.slice(0, 8)}...`;

  const auth = req.headers.authorization;
  if (auth?.startsWith('Bearer ')) {
    try {
      const payload = JSON.parse(Buffer.from(auth.split('.')[1], 'base64').toString());
      return payload.client_id || payload.sub || null;
    } catch { return null; }
  }
  return null;
}

function extractApiVersion(req: Request): string | null {
  const pathMatch = req.path.match(/^\/(v\d+)\//);
  if (pathMatch) return pathMatch[1];
  return req.headers['x-api-version'] as string || null;
}
```

### 2. Fastify Request Logging Plugin

```typescript
// plugins/api-logger.ts
import { FastifyPluginAsync } from 'fastify';
import fp from 'fastify-plugin';
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  batchSize: 200,
  flushInterval: 3000,
});

const apiLoggerPlugin: FastifyPluginAsync = async (fastify) => {
  fastify.addHook('onRequest', async (request) => {
    request.startTime = process.hrtime.bigint();
  });

  fastify.addHook('onResponse', async (request, reply) => {
    const durationMs = Number((process.hrtime.bigint() - request.startTime) / 1_000_000n);

    const metrics = {
      method: request.method,
      route: request.routeOptions?.url || request.url,
      statusCode: reply.statusCode,
      durationMs,
      consumerId: request.headers['x-api-key'] || null,
      ip: request.ip,
    };

    if (reply.statusCode >= 500) {
      client.error('api', `${request.method} ${request.url} ${reply.statusCode}`, metrics);
    } else if (reply.statusCode >= 400) {
      client.warn('api', `${request.method} ${request.url} ${reply.statusCode}`, metrics);
    } else {
      client.info('api', `${request.method} ${request.url} ${reply.statusCode}`, metrics);
    }
  });

  fastify.addHook('onClose', async () => { await client.flush(); });
};

export default fp(apiLoggerPlugin, { name: 'api-logger' });
```

### 3. Latency Percentile Tracking

Average latency hides problems. A p50 of 50ms with a p99 of 5000ms means 1 in 100 users waits 100x longer:

```typescript
// middleware/latency-tracker.ts
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({ apiUrl: process.env.LOGTIDE_API_URL!, apiKey: process.env.LOGTIDE_API_KEY! });

const latencyBuffer: Map<string, number[]> = new Map();

export function trackLatency(route: string, durationMs: number) {
  if (!latencyBuffer.has(route)) latencyBuffer.set(route, []);
  latencyBuffer.get(route)!.push(durationMs);
}

// Report percentiles every 60 seconds
setInterval(() => {
  for (const [route, latencies] of latencyBuffer.entries()) {
    if (latencies.length === 0) continue;
    latencies.sort((a, b) => a - b);

    const pct = (p: number) => latencies[Math.max(0, Math.ceil((p / 100) * latencies.length) - 1)];

    client.info('api-metrics', 'API latency report', {
      event: 'api.latency_report',
      route,
      request_count: latencies.length,
      p50_ms: pct(50),
      p95_ms: pct(95),
      p99_ms: pct(99),
      max_ms: latencies[latencies.length - 1],
      avg_ms: Math.round(latencies.reduce((a, b) => a + b, 0) / latencies.length),
    });

    latencyBuffer.set(route, []);
  }
}, 60_000);
```

### 4. Rate Limit Logging

When you enforce rate limits, log the events for analysis:

```typescript
// middleware/rate-limiter.ts
import { LogTideClient } from '@logtide/sdk-node';
import { Request, Response, NextFunction } from 'express';

const client = new LogTideClient({ apiUrl: process.env.LOGTIDE_API_URL!, apiKey: process.env.LOGTIDE_API_KEY! });

const requestCounts: Map<string, { count: number; resetAt: number }> = new Map();

export function rateLimiter(windowMs: number, maxRequests: number) {
  return (req: Request, res: Response, next: NextFunction) => {
    const key = (req.headers['x-api-key'] as string) || req.ip || 'anonymous';
    const now = Date.now();

    let entry = requestCounts.get(key);
    if (!entry || now > entry.resetAt) {
      entry = { count: 0, resetAt: now + windowMs };
      requestCounts.set(key, entry);
    }
    entry.count++;

    res.setHeader('X-RateLimit-Limit', maxRequests);
    res.setHeader('X-RateLimit-Remaining', Math.max(0, maxRequests - entry.count));

    if (entry.count > maxRequests) {
      client.warn('api', 'Rate limit exceeded', {
        event: 'api.rate_limit_exceeded',
        consumer: key,
        path: req.path,
        request_count: entry.count,
        limit: maxRequests,
      });
      return res.status(429).json({ error: 'Rate limit exceeded' });
    }

    // Warn when approaching limit (80%)
    if (entry.count > maxRequests * 0.8) {
      client.info('api', 'Consumer approaching rate limit', {
        event: 'api.rate_limit_warning',
        consumer: key,
        request_count: entry.count,
        limit: maxRequests,
      });
    }

    next();
  };
}
```

### 5. Consumer Analytics

Track which consumers use which endpoints and how much:

```typescript
// analytics/consumer-tracker.ts
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({ apiUrl: process.env.LOGTIDE_API_URL!, apiKey: process.env.LOGTIDE_API_KEY! });

const activity: Map<string, {
  requests: number; errors: number;
  totalLatency: number; endpoints: Record<string, number>;
}> = new Map();

export function trackConsumer(consumerId: string, route: string, status: number, durationMs: number) {
  if (!consumerId) return;
  if (!activity.has(consumerId)) {
    activity.set(consumerId, { requests: 0, errors: 0, totalLatency: 0, endpoints: {} });
  }
  const a = activity.get(consumerId)!;
  a.requests++;
  if (status >= 500) a.errors++;
  a.totalLatency += durationMs;
  a.endpoints[route] = (a.endpoints[route] || 0) + 1;
}

// Report every 5 minutes
setInterval(() => {
  for (const [id, a] of activity.entries()) {
    if (a.requests === 0) continue;
    client.info('api-analytics', 'Consumer activity report', {
      event: 'api.consumer_report',
      consumer_id: id,
      request_count: a.requests,
      error_rate_pct: Math.round((a.errors / a.requests) * 10000) / 100,
      avg_latency_ms: Math.round(a.totalLatency / a.requests),
      top_endpoints: Object.fromEntries(
        Object.entries(a.endpoints).sort((x, y) => y[1] - x[1]).slice(0, 10)
      ),
    });
    activity.set(id, { requests: 0, errors: 0, totalLatency: 0, endpoints: {} });
  }
}, 300_000);
```

### 6. nginx Access Log Integration

Capture nginx access logs upstream of your application for a complete picture:

```nginx
# nginx.conf - Structured JSON access log
log_format logtide_json escape=json
  '{'
    '"timestamp":"$time_iso8601",'
    '"method":"$request_method",'
    '"path":"$uri",'
    '"status":$status,'
    '"body_bytes_sent":$body_bytes_sent,'
    '"request_time":$request_time,'
    '"upstream_response_time":"$upstream_response_time",'
    '"remote_addr":"$remote_addr",'
    '"http_user_agent":"$http_user_agent",'
    '"http_x_api_key":"$http_x_api_key"'
  '}';

access_log /var/log/nginx/api-access.log logtide_json;
```

Ship nginx JSON logs to LogTide with the [nginx integration](/integrations/nginx/) or a log shipper container.

## Real-World Example: Fintech API Platform

A fintech company exposes a payments API to 200+ partners, processing 2 million requests per day.

**Before LogTide:**
- Latency SLAs measured only by synthetic checks every 60s
- Consumer complaints about slowness took days to investigate
- Rate limit violations discovered after the fact
- No per-consumer usage analytics for billing

**After LogTide:**

```
1. Alert: "p99 latency > 2000ms on POST /v1/payments"

2. Query: Recent slow requests
   route:/v1/payments AND durationMs:>2000 AND time:>30m

3. Discovery: 80% of slow requests from consumer "key:a1b2c3d4..."
   They're sending 500-item batch requests (normal is 10-20)

4. Resolution: Contact consumer about batch size limits.
   Add request body size validation.

   Total investigation time: 8 minutes
```

**Results:**
- Latency issues detected in minutes, not days
- Per-consumer SLA tracking automated
- Rate limit abusers identified proactively
- API usage reports generated for billing

## Query Patterns for API Monitoring

```
# Top endpoints by request volume
service:api AND time:>1h | group by route | count | sort desc

# Endpoints with highest error rate
service:api AND time:>1h
  | group by route | ratio(statusCode:>=500) | sort desc

# Slowest endpoints by p99
event:api.latency_report AND time:>1h | sort by p99_ms desc

# Consumers hitting rate limits most
event:api.rate_limit_exceeded AND time:>24h
  | group by consumer | count | sort desc

# API version adoption
service:api AND time:>7d | group by apiVersion | count

# Large response payloads (optimization targets)
service:api AND responseSize:>1000000 AND time:>24h
  | group by route | avg(responseSize) | sort desc
```

## Alerting Configuration

```yaml
# High error rate on any endpoint
- name: api-error-rate
  query: 'service:api AND statusCode:>=500 AND time:>5m'
  threshold: 50
  window: 5m
  severity: critical

# Latency spike
- name: api-latency-spike
  query: 'event:api.latency_report AND p99_ms:>2000'
  threshold: 1
  window: 5m
  severity: warning

# Rate limit abuse
- name: api-rate-limit-abuse
  query: 'event:api.rate_limit_exceeded AND time:>10m'
  threshold: 100
  window: 10m
  severity: warning

# Zero traffic (API may be down)
- name: api-zero-traffic
  query: 'service:api AND time:>5m'
  threshold: 0
  condition: equals
  window: 5m
  severity: critical
```

## API Monitoring Checklist

### Request Logging
- [ ] Every request logged with method, path, status, and latency
- [ ] Route patterns used (not actual paths with IDs) for grouping
- [ ] Request and response sizes captured
- [ ] Consumer identity extracted from API key or JWT
- [ ] API version tracked (header or path-based)

### Latency Tracking
- [ ] High-resolution timing with `process.hrtime.bigint()`
- [ ] Percentile reports generated (p50, p95, p99)
- [ ] Slow request threshold alerts configured
- [ ] Upstream (nginx) and application latency correlated

### Error and Rate Limit Monitoring
- [ ] Error rates calculated per endpoint
- [ ] Status code breakdown tracked
- [ ] Rate limit events logged with consumer identity
- [ ] Approaching-limit warnings generated
- [ ] Abuse patterns detectable through queries

## Common Pitfalls

### 1. "Using the actual path instead of the route pattern"

Logging `/v1/products/abc123` instead of `/v1/products/:id` creates thousands of unique "endpoints" that cannot be grouped or analyzed.

**Solution:** Always use the route pattern from your framework (`req.route.path` in Express, `request.routeOptions.url` in Fastify).

### 2. "Setting alerts on averages"

Average latency of 100ms sounds great. But if p99 is 5000ms, 1% of your users are having a terrible experience.

**Solution:** Alert on percentiles (p95 or p99), not averages. A p99 spike is often the first sign of a systemic issue.

### 3. "Only monitoring from the application layer"

Your application may report 200ms latency, but the user experiences 800ms because of TLS handshake, nginx buffering, and network overhead.

**Solution:** Monitor at both the nginx layer and the application layer. Compare upstream vs. downstream latency to identify infrastructure bottlenecks.

## Next Steps

- [Express Integration](/integrations/express/) - Full Express SDK setup
- [Fastify Integration](/integrations/fastify/) - Fastify plugin configuration
- [nginx Integration](/integrations/nginx/) - Upstream access log shipping
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Configure alert rules
- [Incident Response](/use-cases/incident-response/) - Debug API issues fast

---

**Ready to monitor your API?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Share your API monitoring setup


---

# DevOps CI/CD Pipeline Logging

> Centralize and audit CI/CD pipeline logs from GitHub Actions, GitLab CI and others with LogTide: release visibility, audit trail and pipeline analytics.

*Source: https://logtide.dev/use-cases/devops-ci-cd/*

---

Your CI/CD pipeline generates some of the most valuable operational data in your organization: what was built, when it was deployed, whether tests passed, who approved the release. Yet most teams let these logs rot in GitHub Actions or GitLab CI with a 30-90 day retention limit. When a production incident traces back to a bad deploy three weeks ago, the evidence is already gone. This guide shows how to ship all pipeline logs to LogTide for permanent, searchable retention.

## The Problem with Pipeline Logs

CI/CD platforms treat logs as ephemeral artifacts:

```
❌ Pipeline log problems:

1. Retention limits    → GitHub Actions: 90 days, GitLab CI: 30 days
2. No search           → Can't query across builds or pipelines
3. Siloed by platform  → GitHub Actions + ArgoCD + Terraform = 3 log stores
4. No correlation      → Can't connect a deploy to a production incident
5. No alerting         → Build failures go unnoticed until someone checks
6. No audit trail      → Who deployed what, when? Scroll through pages of UI
```

| Scenario | Without Centralized Pipeline Logs |
|----------|----------------------------------|
| Production incident | "When was the last deploy?" -- check 3 different UIs |
| Flaky test investigation | No way to see test failure patterns over time |
| Compliance audit | Manually screenshot pipeline runs to prove controls |
| Build time regression | No historical data to compare build durations |

## The LogTide Approach

Ship every pipeline event to LogTide and treat CI/CD as a first-class data source:

```d2
direction: down
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

gh: GitHub Actions { class: src }
gl: GitLab CI { class: src }
jenkins: "Jenkins / Other CI" { class: src }

lt: LogTide {
  class: group
  grid-columns: 3
  pipeline: Pipeline Dashboard { class: hub }
  deploy: Deploy Tracker { class: hub }
  audit: Release Audit Trail { class: hub }
}

gh -> lt: HTTP POST { class: flow }
gl -> lt: HTTP POST { class: flow }
jenkins -> lt: HTTP POST { class: flow }
```

## Implementation

### 1. LogTide CI Helper Script

Create a reusable script that any CI platform can call:

```bash
#!/bin/bash
# scripts/logtide-ci.sh

LOGTIDE_API_URL="${LOGTIDE_API_URL:?LOGTIDE_API_URL is required}"
LOGTIDE_API_KEY="${LOGTIDE_API_KEY:?LOGTIDE_API_KEY is required}"

logtide_log() {
  local level="$1" message="$2"
  shift 2

  local metadata="{"
  local first=true
  while [ $# -gt 0 ]; do
    [ "$first" = true ] && first=false || metadata+=","
    metadata+="\"${1%%=*}\": \"${1#*=}\""
    shift
  done
  metadata+="}"

  curl -s -X POST "${LOGTIDE_API_URL}/api/v1/ingest" \
    -H "X-API-Key: ${LOGTIDE_API_KEY}" \
    -H "Content-Type: application/json" \
    -d "{\"logs\":[{\"level\":\"${level}\",\"message\":\"${message}\",\"service\":\"ci-pipeline\",\"metadata\":${metadata}}]}"
}

logtide_build_start() {
  logtide_log "info" "Build started" \
    "event=build.started" "commit=${COMMIT_SHA}" "branch=${BRANCH}" \
    "pipeline_id=${PIPELINE_ID}" "triggered_by=${TRIGGERED_BY}"
}

logtide_build_success() {
  logtide_log "info" "Build succeeded" \
    "event=build.success" "commit=${COMMIT_SHA}" "branch=${BRANCH}" \
    "pipeline_id=${PIPELINE_ID}" "duration_seconds=${BUILD_DURATION}"
}

logtide_build_failure() {
  logtide_log "error" "Build failed" \
    "event=build.failure" "commit=${COMMIT_SHA}" "branch=${BRANCH}" \
    "pipeline_id=${PIPELINE_ID}" "failure_step=${FAILURE_STEP}"
}

logtide_deploy() {
  logtide_log "info" "Deployment completed" \
    "event=deploy.completed" "version=${VERSION}" "environment=${ENVIRONMENT}" \
    "commit=${COMMIT_SHA}" "deployed_by=${DEPLOYED_BY}" "image_tag=${IMAGE_TAG}"
}

logtide_test_results() {
  logtide_log "info" "Test results" \
    "event=tests.completed" "passed=${TESTS_PASSED}" "failed=${TESTS_FAILED}" \
    "skipped=${TESTS_SKIPPED}" "coverage=${COVERAGE}" "pipeline_id=${PIPELINE_ID}"
}
```

### 2. GitHub Actions Integration

```yaml
# .github/workflows/ci-cd.yml
name: CI/CD Pipeline
on:
  push:
    branches: [main, develop]

env:
  LOGTIDE_API_URL: ${{ secrets.LOGTIDE_API_URL }}
  LOGTIDE_API_KEY: ${{ secrets.LOGTIDE_API_KEY }}

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Log build start
        run: |
          source ./scripts/logtide-ci.sh
          COMMIT_SHA="${{ github.sha }}" BRANCH="${{ github.ref_name }}" \
          PIPELINE_ID="${{ github.run_id }}" TRIGGERED_BY="${{ github.actor }}" \
          logtide_build_start

      - name: Install and test
        run: |
          npm ci
          npm test -- --coverage --json --outputFile=test-results.json

      - name: Log test results
        if: always()
        run: |
          source ./scripts/logtide-ci.sh
          TESTS_PASSED=$(jq '.numPassedTests' test-results.json 2>/dev/null || echo 0)
          TESTS_FAILED=$(jq '.numFailedTests' test-results.json 2>/dev/null || echo 0)
          TESTS_SKIPPED=$(jq '.numPendingTests' test-results.json 2>/dev/null || echo 0)
          PIPELINE_ID="${{ github.run_id }}" COVERAGE="unknown" \
          logtide_test_results

      - name: Log build result
        if: always()
        run: |
          source ./scripts/logtide-ci.sh
          export COMMIT_SHA="${{ github.sha }}"
          export BRANCH="${{ github.ref_name }}"
          export PIPELINE_ID="${{ github.run_id }}"
          export BUILD_DURATION="$SECONDS"
          if [ "${{ job.status }}" == "success" ]; then
            logtide_build_success
          else
            FAILURE_STEP="${{ github.action }}" logtide_build_failure
          fi

  deploy-production:
    needs: build-and-test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - name: Deploy and log
        run: |
          kubectl set image deployment/myapp \
            myapp=myregistry/myapp:${{ github.sha }} -n production

          source ./scripts/logtide-ci.sh
          VERSION="${{ github.sha }}" ENVIRONMENT="production" \
          COMMIT_SHA="${{ github.sha }}" DEPLOYED_BY="${{ github.actor }}" \
          IMAGE_TAG="myregistry/myapp:${{ github.sha }}" \
          logtide_deploy
```

### 3. GitLab CI Integration

```yaml
# .gitlab-ci.yml
stages: [build, test, deploy]

build:
  stage: build
  script:
    - source ./scripts/logtide-ci.sh
    - COMMIT_SHA="$CI_COMMIT_SHA" BRANCH="$CI_COMMIT_REF_NAME"
      PIPELINE_ID="$CI_PIPELINE_ID" TRIGGERED_BY="$GITLAB_USER_LOGIN"
      logtide_build_start
    - npm ci && npm run build
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - BUILD_DURATION=$SECONDS logtide_build_success

deploy_production:
  stage: deploy
  script:
    - kubectl set image deployment/myapp myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA -n production
    - source ./scripts/logtide-ci.sh
    - VERSION="$CI_COMMIT_SHA" ENVIRONMENT="production"
      DEPLOYED_BY="$GITLAB_USER_LOGIN" IMAGE_TAG="$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
      logtide_deploy
  when: manual
  only: [main]
```

### 4. Programmatic Deploy Script with Rollback

For more control, use the LogTide SDK directly in deploy scripts:

```typescript
// scripts/deploy.ts
import { LogTideClient } from '@logtide/sdk-node';
import { execSync } from 'child_process';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

async function deploy(service: string, env: string, imageTag: string) {
  const deployId = `deploy-${Date.now()}`;
  const startTime = Date.now();

  client.info('deploy-script', 'Deployment started', {
    event: 'deploy.started',
    deploy_id: deployId,
    service, environment: env, image_tag: imageTag,
    deployed_by: process.env.USER || 'automation',
    git_commit: execSync('git rev-parse HEAD').toString().trim(),
    git_message: execSync('git log -1 --pretty=%B').toString().trim(),
  });

  try {
    execSync(`kubectl set image deployment/${service} ${service}=${imageTag} -n ${env}`);
    execSync(`kubectl rollout status deployment/${service} -n ${env} --timeout=300s`);

    client.info('deploy-script', 'Deployment succeeded', {
      event: 'deploy.success',
      deploy_id: deployId,
      duration_seconds: Math.round((Date.now() - startTime) / 1000),
      service, environment: env,
    });
  } catch (error) {
    client.error('deploy-script', 'Deployment failed', {
      event: 'deploy.failure',
      deploy_id: deployId,
      error: error.message,
      service, environment: env,
    });

    // Automatic rollback
    client.warn('deploy-script', 'Initiating rollback', { event: 'deploy.rollback', deploy_id: deployId });
    execSync(`kubectl rollout undo deployment/${service} -n ${env}`);
    client.info('deploy-script', 'Rollback completed', { event: 'deploy.rollback.completed', deploy_id: deployId });

    throw error;
  } finally {
    await client.flush();
  }
}
```

### 5. Release Audit Trail

For compliance, capture comprehensive release metadata:

```typescript
// scripts/release-audit.ts
import { LogTideClient } from '@logtide/sdk-node';
import { execSync } from 'child_process';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

async function logRelease(version: string, environment: string) {
  const lastTag = execSync(
    'git describe --tags --abbrev=0 HEAD~1 2>/dev/null || echo HEAD~10'
  ).toString().trim();

  client.info('release-audit', 'Release published', {
    event: 'release.published',
    version,
    environment,
    released_by: process.env.USER || 'automation',
    commit_sha: execSync('git rev-parse HEAD').toString().trim(),
    commits: execSync(`git log --oneline ${lastTag}..HEAD`).toString().trim(),
    changed_files: execSync(`git diff --name-only ${lastTag}..HEAD`).toString().trim().split('\n'),
    has_dependency_changes: execSync(
      `git diff ${lastTag}..HEAD -- package-lock.json | head -1`
    ).toString().trim().length > 0,
  });

  await client.flush();
}
```

## Real-World Example: Platform Engineering Team

A platform team managing 30 microservices across 15 repositories needs visibility into their CI/CD health.

**Before LogTide:**
- Build failure notifications scattered across Slack channels
- No way to correlate a production incident to a specific deploy
- Compliance audits required manually gathering screenshots from GitHub
- Flaky tests went undetected for weeks

**After LogTide (daily volume: ~5,000 pipeline events):**

```
1. Production alert: "Order service error rate > 5%"

2. Query: What was the last deploy?
   event:deploy.completed AND environment:production AND time:>2h
   → order-service v2.4.1 deployed 45 min ago

3. Query: What changed in that release?
   event:release.published AND version:v2.4.1
   → 3 commits, changed: src/services/payment-client.ts

4. Query: Did tests pass?
   event:tests.completed AND pipeline_id:12345
   → 142 passed, 0 failed -- regression not covered by tests

5. Decision: Rollback to v2.4.0
   → event:deploy.rollback logged automatically
```

**Result:** Incident correlation time dropped from 30+ minutes to 5 minutes. Compliance audits are now automated exports from LogTide.

## Dashboard Query Patterns

```
# All deployments to production this week
event:deploy.completed AND environment:production AND time:>7d | sort desc

# Failed deployments by repository
event:deploy.failure AND time:>30d | group by repository

# Average deploy duration by environment
event:deploy.success AND time:>30d | group by environment | avg(duration_seconds)

# Deployment frequency (DORA metric)
event:deploy.completed AND environment:production AND time:>30d
  | group by date(timestamp) | count

# Change failure rate (DORA metric)
# rollbacks / total deploys
event:deploy.rollback AND time:>30d | count

# Build success rate by repository
(event:build.success OR event:build.failure) AND time:>30d
  | group by repository | ratio(event:build.success)

# Flaky test detection
event:tests.completed AND failed:>0 AND time:>14d
  | group by pipeline_id
```

## CI/CD Logging Checklist

### Pipeline Integration
- [ ] Build start and completion events logged
- [ ] Test results (passed, failed, skipped, coverage) logged
- [ ] Build failure step identified in log metadata
- [ ] Pipeline ID and triggering user captured

### Deployment Tracking
- [ ] Deploy events include version, environment, deployer
- [ ] Image tags and commit SHAs recorded
- [ ] Rollback events captured with reason
- [ ] Deploy duration tracked

### Release Audit Trail
- [ ] Commit list included in release events
- [ ] Changed files recorded
- [ ] Dependency changes flagged
- [ ] Approval records captured for gated environments

### Alerting
- [ ] Build failure alerts (immediate notification)
- [ ] Deploy failure alerts with auto-rollback
- [ ] Build duration regression alerts
- [ ] Flaky test detection (tests that fail intermittently)

## Common Pitfalls

### 1. "We only log failures"

If you only capture failures, you cannot calculate success rates, track deployment frequency, or establish baselines for build duration.

**Solution:** Log every pipeline event. The storage cost is negligible (a few thousand events per day), and the analytical value is enormous.

### 2. "Our CI platform already has logs"

GitHub Actions keeps logs for 90 days. GitLab CI keeps them for 30. When you need to investigate what happened 6 months ago during a compliance audit, those logs are gone.

**Solution:** Ship to LogTide in real-time. Configure retention based on your compliance requirements.

### 3. "We'll add deployment tracking later"

Without deployment markers in your logs, you cannot correlate production incidents with releases. The most common root cause of production issues is "something changed."

**Solution:** Add deploy logging on day one. It takes 10 minutes with the helper script above.

## Next Steps

- [Docker Integration](/integrations/docker/) - Container build logging
- [Kubernetes Integration](/integrations/kubernetes/) - Deployment orchestration
- [Incident Response](/use-cases/incident-response/) - Correlate deploys with incidents
- [Compliance Audit Trail](/use-cases/compliance-audit-trail/) - Meet regulatory requirements

---

**Ready to centralize your pipeline logs?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Share your CI/CD logging setup


---

# GDPR-Compliant Logging

> Implement privacy-first logging that meets GDPR requirements. Data minimization, retention policies, and right to erasure.

*Source: https://logtide.dev/use-cases/gdpr-compliance/*

---

GDPR compliance isn't just about consent forms—it extends to how you collect, store, and process log data. This guide shows you how to implement privacy-first logging with LogTide.

## The Problem with Traditional Logging

Most logging solutions treat logs as technical data without privacy considerations:

```typescript
// ❌ BAD: Logging personal data without consideration
logger.info('User logged in', {
  email: 'john.doe@example.com',
  ip: '192.168.1.100',
  name: 'John Doe',
  sessionId: 'sess_abc123',
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)...',
  location: { lat: 51.5074, lng: -0.1278 },
});
```

```d2
direction: down
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

app: Your Application { class: src }
pseudo: "Log with pseudonymization" { class: src }
lt: "LogTide\nEU infrastructure" { class: hub }
retention: "Retention policies applied" { class: src }
deletion: "Automatic deletion" { class: src }

app -> pseudo -> lt -> retention -> deletion { class: flow }
```

## Implementation: Step by Step

### 1. Data Minimization Strategy

First, audit what you're currently logging and categorize:

| Data Type | GDPR Category | Action |
|-----------|---------------|--------|
| User ID | Personal Data | Pseudonymize |
| Email | Personal Data | Hash or omit |
| IP Address | Personal Data | Truncate or hash |
| Session ID | Potentially Personal | Short retention |
| Error messages | Technical | Keep (no PII) |
| Request paths | Technical | Keep (no PII) |

### 2. Pseudonymization Helpers

Create utility functions for consistent data handling:

```typescript
import crypto from 'crypto';

// Hash personal identifiers (one-way)
export function hashPII(value: string, salt: string): string {
  return crypto
    .createHmac('sha256', salt)
    .update(value)
    .digest('hex')
    .substring(0, 16);
}

// Truncate IP addresses (GDPR-friendly)
export function truncateIP(ip: string): string {
  if (ip.includes(':')) {
    // IPv6: keep first 4 segments
    return ip.split(':').slice(0, 4).join(':') + '::';
  }
  // IPv4: zero last octet
  return ip.split('.').slice(0, 3).join('.') + '.0';
}

// Mask email (for debugging without full exposure)
export function maskEmail(email: string): string {
  const [local, domain] = email.split('@');
  if (!domain) return '***';
  return `${local.charAt(0)}***@${domain}`;
}

// Generate pseudonymous user ID
export function pseudonymizeUserId(userId: string, salt: string): string {
  return `user_${hashPII(userId, salt)}`;
}
```

### 3. Privacy-First Logger Wrapper

Create a logging wrapper that enforces privacy rules:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const PII_SALT = process.env.PII_HASH_SALT!;

class PrivacyLogger {
  private client: LogTideClient;

  constructor() {
    this.client = new LogTideClient({
      apiUrl: process.env.LOGTIDE_API_URL!,
      apiKey: process.env.LOGTIDE_API_KEY!,
      globalMetadata: {
        environment: process.env.NODE_ENV,
        dataClassification: 'gdpr-compliant',
      },
    });
  }

  // Safe logging method that sanitizes data
  log(level: string, message: string, data: Record<string, unknown> = {}) {
    const sanitized = this.sanitize(data);
    this.client[level]('app', message, sanitized);
  }

  private sanitize(data: Record<string, unknown>): Record<string, unknown> {
    const result: Record<string, unknown> = {};

    for (const [key, value] of Object.entries(data)) {
      // Handle known PII fields
      if (key === 'email' && typeof value === 'string') {
        result.email_hash = hashPII(value, PII_SALT);
        continue;
      }

      if (key === 'ip' && typeof value === 'string') {
        result.ip_truncated = truncateIP(value);
        continue;
      }

      if (key === 'userId' && typeof value === 'string') {
        result.user_pseudo = pseudonymizeUserId(value, PII_SALT);
        continue;
      }

      // Block known sensitive fields
      if (['password', 'ssn', 'creditCard', 'token'].includes(key)) {
        continue; // Don't log at all
      }

      result[key] = value;
    }

    return result;
  }

  info(message: string, data?: Record<string, unknown>) {
    this.log('info', message, data);
  }

  warn(message: string, data?: Record<string, unknown>) {
    this.log('warn', message, data);
  }

  error(message: string, data?: Record<string, unknown>) {
    this.log('error', message, data);
  }
}

export const logger = new PrivacyLogger();
```

### 4. Usage in Your Application

```typescript
import { logger } from './privacy-logger';

// ✅ GOOD: Privacy-compliant logging
app.post('/login', async (req, res) => {
  const { email, password } = req.body;

  try {
    const user = await authenticate(email, password);

    // Personal data is automatically sanitized
    logger.info('User login successful', {
      email,                    // → email_hash: "a1b2c3..."
      ip: req.ip,              // → ip_truncated: "192.168.1.0"
      userId: user.id,         // → user_pseudo: "user_d4e5f6..."
      userAgent: req.headers['user-agent'],
      loginMethod: 'password',
    });

    res.json({ success: true });
  } catch (error) {
    logger.warn('Login failed', {
      email,                    // → email_hash (can correlate attempts)
      ip: req.ip,
      reason: error.code,      // Technical, not personal
    });

    res.status(401).json({ error: 'Invalid credentials' });
  }
});
```

### 5. Retention Policies

Configure LogTide retention policies in your organization settings:

| Log Type | Retention | Justification |
|----------|-----------|---------------|
| Access logs | 90 days | Security monitoring |
| Error logs | 30 days | Debugging |
| Audit logs | 7 years | Legal requirement |
| Debug logs | 7 days | Development only |

In LogTide dashboard:
1. Go to **Organization Settings** → **Data Retention**
2. Set default retention period
3. Create rules for specific services or log levels

### 6. Right to Erasure (Article 17)

When a user requests deletion, you need to remove their log data:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

async function handleDeletionRequest(userId: string) {
  // 1. Delete from your database
  await db.users.delete({ where: { id: userId } });

  // 2. The pseudonymized logs don't directly identify the user
  // But if you need complete erasure, use LogTide's deletion API

  // 3. Log the deletion (audit trail)
  logger.info('GDPR deletion request processed', {
    user_pseudo: pseudonymizeUserId(userId, PII_SALT),
    requestedAt: new Date().toISOString(),
    processedBy: 'system',
  });

  // 4. Return confirmation
  return {
    deleted: true,
    retentionNote: 'Pseudonymized logs retained for 90 days per security policy',
  };
}
```

## Real-World Example: E-commerce Platform

Let's say you're building an e-commerce platform in the EU.

**Requirements:**
- Log user activity for fraud detection
- Meet GDPR data minimization
- Support right to erasure
- Retain order logs for 7 years (tax compliance)

**Implementation:**

```typescript
// Order logging (long retention for tax)
logger.info('Order placed', {
  orderId: order.id,           // Business identifier
  amount: order.total,         // Financial data
  currency: order.currency,
  items: order.items.length,   // Aggregate, not details
  // User data pseudonymized automatically
  userId: order.userId,
  email: order.email,
  ip: req.ip,
});

// Fraud detection (short retention)
logger.info('Payment attempt', {
  orderId: order.id,
  paymentMethod: 'card',       // Type only, no card numbers
  succeeded: true,
  // Pseudonymized for correlation
  userId: order.userId,
  ip: req.ip,
  fingerprint: deviceFingerprint,
});

// Marketing events (requires consent)
if (user.marketingConsent) {
  logger.info('Product viewed', {
    productId: product.id,
    category: product.category,
    userId: user.id,  // Pseudonymized, consent recorded
  });
}
```

## GDPR Compliance Checklist

Use this for your audit:

- [ ] **Data Minimization**
    - [ ] Audit all log statements for personal data
    - [ ] Remove unnecessary PII collection
    - [ ] Implement pseudonymization for required PII

- [ ] **Storage Location**
    - [ ] Confirm logs stored in EU (or adequate country)
    - [ ] Document data transfer mechanisms if using cloud

- [ ] **Retention**
    - [ ] Define retention periods for each log type
    - [ ] Configure automatic deletion in LogTide
    - [ ] Document justification for retention periods

- [ ] **Access Control**
    - [ ] Implement role-based access to logs
    - [ ] Enable audit logging for log access
    - [ ] Regular access review process

- [ ] **Right to Erasure**
    - [ ] Process for handling deletion requests
    - [ ] Mechanism to delete or anonymize user logs
    - [ ] Document what can/cannot be deleted (legal holds)

- [ ] **Documentation**
    - [ ] Update privacy policy with logging practices
    - [ ] Document lawful basis for log processing
    - [ ] Maintain records of processing activities

## Cost Comparison

**Traditional SaaS Logging (Datadog, Splunk):**
- Data leaves your infrastructure
- Limited control over data location
- Retention often unlimited or expensive to manage
- Per-GB pricing encourages data reduction

**LogTide self-hosted:**
- Data stays in your infrastructure
- Full control over data location
- Configurable retention with automatic deletion
- Fixed infrastructure cost

**Compliance benefit:** Easier GDPR compliance with self-hosted solution.

## Common Pitfalls

### 1. "We anonymize everything"

True anonymization is irreversible. If you can link logs back to users (even with internal keys), it's pseudonymization, not anonymization. GDPR still applies.

**Correct approach:** Use pseudonymization (one-way hashing) and document it as such.

### 2. "Logs are just technical data"

IP addresses, user agents, and session IDs can all be personal data under GDPR if they can identify a person.

**Correct approach:** Treat all user-related data as potentially personal.

### 3. "We'll delete logs when requested"

If you can't efficiently find and delete a user's logs, you can't fulfill deletion requests.

**Correct approach:** Use consistent user identifiers (pseudonymized) that can be searched and deleted.

## Legal Disclaimer

This guide provides technical implementation, not legal advice. GDPR interpretation varies by country and case. Consult with a data protection officer or legal counsel for your specific situation.

## Resources

- [GDPR Official Text](https://gdpr.eu/)
- [Article 29 Working Party Guidelines on Anonymisation](https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/index_en.htm)
- [ICO Guide to Data Protection](https://ico.org.uk/for-organisations/)
- [CNIL Logging Guidelines (French DPA)](https://www.cnil.fr/en)

---

**Need help with GDPR-compliant logging?**

- [Open a GitHub issue](https://github.com/logtide-dev/logtide/issues) for technical questions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) for architecture advice


---

# High-Volume Log Management

> Handle millions of log events per second with LogTide using batching, partitioning, and retention strategies at scale.

*Source: https://logtide.dev/use-cases/high-volume/*

---

When your systems generate millions of log events per second, naive logging approaches fail. Buffers overflow, disks fill up, and your observability pipeline becomes a liability. This guide covers the architecture and configuration patterns for handling high-volume log workloads with LogTide.

## The Problem with High-Volume Logging

### What Happens at Scale

At 10,000 events/sec, most logging setups work fine. At 100,000+, problems start:

```
❌ Common failure modes at high volume:

1. Network saturation     → SDK buffers fill, logs dropped
2. Disk I/O bottleneck    → Write latency spikes, queries slow
3. Memory pressure        → OOM kills on log processors
4. Ingestion lag          → Minutes of delay between event and visibility
5. Storage explosion      → Terabytes per day, costs spiral
```

### Scale Reference Points

| Events/sec | GB/day (avg 500 bytes/event) | Use Case |
|------------|------------------------------|----------|
| 1,000 | ~43 GB | Small SaaS |
| 10,000 | ~430 GB | Mid-size application |
| 100,000 | ~4.3 TB | Large platform |
| 1,000,000 | ~43 TB | High-traffic / IoT |

## The LogTide Approach

### Architecture for High Volume

```d2
direction: right
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

apps: "Your Apps\n(SDKs)" { class: src }
kafka: Kafka Cluster { shape: queue; class: hub }
ingesters: LogTide Ingesters { class: hub }
queries: "Queries & SIEM" { class: src }
s3: "S3 / GCS Archive" { shape: cylinder; class: src }
pg: "PostgreSQL\nTimescaleDB" { shape: cylinder; class: src }

apps -> kafka -> ingesters -> queries { class: flow }
kafka -> s3 { class: flow }
ingesters -> pg { class: flow }
```

**Key principles:**
1. **Buffer with Kafka** between your apps and LogTide — handle bursts without backpressure
2. **Horizontal scaling** of ingestion workers — add more consumers for more throughput
3. **Tiered storage** — hot data in PostgreSQL/TimescaleDB, cold data in S3
4. **Sampling and filtering** at the edge — not every debug log needs to be stored

## Implementation

### 1. SDK-Level Batching

Configure your application SDKs for high-throughput batching:

```typescript
// Node.js SDK - tuned for high volume
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,

  // Batching tuning
  batchSize: 500,           // Larger batches (default: 100)
  flushInterval: 2000,      // Flush every 2s (default: 5s)
  maxBufferSize: 50000,     // Larger in-memory buffer

  // Reliability
  maxRetries: 3,
  retryDelayMs: 1000,
});
```

```python
# Python SDK - tuned for high volume
import os

from logtide_sdk import LogTideClient, ClientOptions

client = LogTideClient(ClientOptions(
    api_url=os.environ["LOGTIDE_API_URL"],
    api_key=os.environ["LOGTIDE_API_KEY"],

    batch_size=500,
    flush_interval=2.0,

    global_metadata={
        "environment": "production",
        "tier": "critical",
    },
))
```

### 2. Log Level Filtering

At high volume, not every log needs to reach LogTide. Filter at the SDK level:

```typescript
const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Only ship warn+ in production, debug+ in staging
const LEVELS = ['debug', 'info', 'warn', 'error', 'critical'];
const minLevel = process.env.NODE_ENV === 'production' ? 'warn' : 'debug';
const enabled = (level: string) => LEVELS.indexOf(level) >= LEVELS.indexOf(minLevel);

// In hot paths, gate logging before the call
if (enabled('debug')) {
  client.debug('api', 'Processing item', { itemId, step: 'validation' });
}
```

### 3. Sampling for High-Frequency Events

For events that fire thousands of times per second, sample instead of logging everything:

```typescript
// Sample 1% of successful requests, 100% of errors
function shouldLog(statusCode: number): boolean {
  if (statusCode >= 400) return true;            // Always log errors
  if (statusCode >= 300) return true;            // Always log redirects
  return Math.random() < 0.01;                    // 1% sample for 2xx
}

app.use((req, res, next) => {
  const start = performance.now();

  res.on('finish', () => {
    if (shouldLog(res.statusCode)) {
      client.info('api', `${req.method} ${req.path} ${res.statusCode}`, {
        durationMs: Math.round(performance.now() - start),
        sampled: res.statusCode < 300,
        sampleRate: res.statusCode < 300 ? 0.01 : 1.0,
      });
    }
  });

  next();
});
```

### 4. Kafka Buffer Layer

For sustained high volume, add Kafka between your apps and LogTide:

```typescript
// Producer: Your application sends to Kafka
import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  brokers: process.env.KAFKA_BROKERS!.split(','),
});

const producer = kafka.producer({
  allowAutoTopicCreation: false,
});

await producer.connect();

// High-throughput producer configuration
await producer.send({
  topic: 'logs.application',
  compression: 2, // GZIP
  messages: batch.map(event => ({
    key: event.service,
    value: JSON.stringify(event),
  })),
});
```

See the [Kafka Integration Guide](/integrations/kafka/) for full setup.

### 5. Tiered Retention

Not all logs need the same retention:

```yaml
# LogTide retention configuration
retention:
  # Hot tier: fast queries, expensive storage
  hot:
    duration: 7d
    storage: postgresql     # TimescaleDB hypertable

  # Warm tier: slower queries, cheaper storage
  warm:
    duration: 30d
    storage: compressed     # Compressed chunks

  # Cold tier: archival, cheapest storage
  cold:
    duration: 365d
    storage: s3             # Object storage
    bucket: logtide-archive
```

**Cost impact at 100 GB/day:**

| Tier | Duration | Storage Cost |
|------|----------|-------------|
| Hot (SSD) | 7 days = 700 GB | ~$70/month |
| Warm (compressed) | 30 days = 1.5 TB (compressed) | ~$30/month |
| Cold (S3) | 365 days = 10 TB (compressed) | ~$23/month |
| **Total** | | **~$123/month** |

vs. keeping everything in hot storage: ~$1,000/month.

### 6. Horizontal Scaling

Scale LogTide ingestion horizontally:

```yaml
# Kubernetes: Scale ingestion workers
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logtide-ingester
spec:
  replicas: 6        # Scale based on volume
  selector:
    matchLabels:
      app: logtide-ingester
  template:
    spec:
      containers:
        - name: ingester
          image: logtide/logtide:latest
          env:
            - name: LOGTIDE_MODE
              value: "ingester"
            - name: KAFKA_BROKERS
              value: "kafka:9092"
            - name: KAFKA_GROUP_ID
              value: "logtide-ingest"
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "2Gi"
              cpu: "2000m"
---
# HPA for auto-scaling
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: logtide-ingester-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: logtide-ingester
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
```

## Real-World Example: Gaming Platform

A real-time multiplayer platform generating 500,000 events/sec during peak hours.

**Requirements:**
- Game server events (player actions, match state)
- Anti-cheat detection (suspicious patterns)
- Infrastructure monitoring (server health)
- 7-day hot retention, 90-day archival

**Architecture:**

```d2
direction: down
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

games: "Game Servers (200+)" { class: src }
kafka: "Kafka\n24 partitions · 3 brokers" { shape: queue; class: hub }
ingesters: "LogTide Ingesters\n6 replicas" { class: hub }
tsdb: "TimescaleDB\nhot: 7 days" { shape: cylinder; class: src }
s3: "S3\ncold: 90 days" { shape: cylinder; class: src }

games -> kafka: "UDP" { class: flow }
kafka -> ingesters: "consumer group" { class: flow }
ingesters -> tsdb { class: flow }
tsdb -> s3: "scheduled job" { class: flow }
```

**Configuration:**

```typescript
// Game server SDK config
const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  batchSize: 1000,
  flushInterval: 1000,
  maxBufferSize: 100000,
});

// Only log actionable events in production
// Debug events sampled at 0.1%
client.info('game-server', 'match_started', {
  matchId,
  players: playerIds.length,
  map: mapName,
  mode: gameMode,
});
```

**Results:**
- Sustained 500k events/sec during peak
- P99 ingestion latency: 200ms
- Storage: 2.1 TB/day → 350 GB/day compressed
- Monthly infrastructure cost: ~$800

## Performance Tuning Checklist

### SDK Layer
- [ ] Batch size increased to 500+ for high volume
- [ ] Flush interval reduced to 1-2 seconds
- [ ] Compression enabled for network efficiency
- [ ] Log level filtering applied (no debug in production)
- [ ] Sampling configured for high-frequency events
- [ ] Graceful shutdown with flush on SIGTERM

### Transport Layer
- [ ] Kafka deployed with replication factor 3
- [ ] Partitions sized for parallelism (12+ for high throughput)
- [ ] LZ4 compression on Kafka topics
- [ ] Consumer group with enough consumers to match partitions
- [ ] Consumer lag monitoring configured

### Storage Layer
- [ ] TimescaleDB hypertables with appropriate chunk intervals
- [ ] Compression policies for chunks older than 1 day
- [ ] Retention policies automated (don't rely on manual cleanup)
- [ ] Disk provisioned with 2x expected peak capacity
- [ ] IOPS sufficient for write workload

### Monitoring
- [ ] Consumer lag alerts (>10,000 = warning, >100,000 = critical)
- [ ] Ingestion throughput dashboard
- [ ] Disk usage alerts at 70% and 85%
- [ ] Query latency monitoring for degradation

## Common Pitfalls

### 1. "We'll just log everything"

At 100,000 events/sec, "everything" means 4.3 TB/day. Storage costs dominate.

**Solution:** Define what's actionable. Sample routine events. Always log errors, alerts, and security events at full fidelity.

### 2. "Our SDK handles backpressure"

SDKs buffer in memory. If LogTide or Kafka is down for 5 minutes at 100k events/sec, that's 30 million events in memory — potentially GBs of RAM.

**Solution:** Set `maxBufferSize` limits. Accept that during extended outages, some logs may be dropped. Log the drop count itself.

### 3. "We'll tune performance later"

At high volume, defaults fail fast. A 5-second flush interval with batch size 100 means only 20 batches/sec — that's 2,000 events/sec max throughput per client.

**Solution:** Tune batch size and flush interval before going to production at scale.

### 4. "Same retention for everything"

Keeping debug logs for a year at $0.10/GB/month is wasteful.

**Solution:** Tiered retention. 7 days hot for debugging, 30 days warm for analysis, 365 days cold for compliance.

## Cost Comparison at Scale

**100 GB/day ingestion:**

| Solution | Monthly Cost |
|----------|-------------|
| Datadog | $3,000+ |
| Splunk Cloud | $4,500+ |
| Azure Monitor | $5,880+ |
| AWS CloudWatch | $1,500+ |
| **LogTide (self-hosted)** | **$400-800** |

**1 TB/day ingestion:**

| Solution | Monthly Cost |
|----------|-------------|
| Datadog | $30,000+ |
| Splunk Cloud | $45,000+ |
| Azure Monitor | $58,800+ |
| **LogTide (self-hosted)** | **$2,000-4,000** |

## Next Steps

- [Apache Kafka Integration](/integrations/kafka/) - Build the buffer layer
- [Kubernetes Integration](/integrations/kubernetes/) - Scale LogTide on K8s
- [Cost Optimization](/use-cases/cost-optimization/) - Detailed savings analysis
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Alert at high volume


---

# Homelab Logging & Monitoring

> Centralize logs from your Proxmox homelab with LogTide: LXC containers, Docker services, AdGuard DNS, firewall packets and exposed services — in ~3 GB of RAM.

*Source: https://logtide.dev/use-cases/homelab/*

---

A homelab generates more logs than most small companies: a Proxmox host, a stack of LXC containers, Docker services like Forgejo, Vikunja and AdGuard Home, a reverse proxy, and — the part that actually matters — one or two services exposed to the public internet. Each one logs into its own corner, and when something breaks (or something probes you), you're SSH-ing into five boxes running `journalctl` and `docker logs` in parallel.

LogTide gives a homelab the thing enterprises pay five figures for: every log searchable in one place, with alerting and security detection — at homelab scale. The whole stack runs in a single LXC or VM and stays around **3 GB of RAM under heavy load**.

## What a homelab deployment looks like

```d2
direction: right
style.fill: transparent

classes: {
  src: {
    style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true }
  }
  hub: {
    style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true }
  }
  flow: {
    style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" }
  }
}

prox: Proxmox host { class: src }
forgejo: LXC · Forgejo { class: src }
vikunja: LXC · Vikunja { class: src }
adguard: LXC · AdGuard { class: src }
docker: Docker services { class: src }
fw: Firewall { class: src }

fb: Fluent Bit { shape: hexagon; class: hub }
lt: "LogTide · TimescaleDB\n~3 GB RAM" { shape: cylinder; class: hub }

prox -> fb: journald / rsyslog { class: flow }
forgejo -> fb: journald { class: flow }
vikunja -> fb: journald { class: flow }
adguard -> fb: query log { class: flow }
docker -> fb: log driver { class: flow }
fw -> fb: nftables / OPNsense { class: flow }
fb -> lt: { class: flow }
```

One central Fluent Bit (or Vector) instance receives everything and ships it to LogTide's HTTP ingest API. No agents to license, no per-GB bill — your only constraint is disk, and homelab log volumes are tiny by enterprise standards (a busy homelab produces 100-500 MB/day; even a year of retention fits on a small SSD).

## Step 1: Deploy LogTide in an LXC or VM

A container or VM with **2 vCPU / 4 GB RAM / 50+ GB disk** is plenty. For homelab-sized hardware, use the **lightweight compose** (3 containers — PostgreSQL, backend, frontend — no Redis):

```bash
mkdir logtide && cd logtide
curl -O https://raw.githubusercontent.com/logtide-dev/logtide/main/docker/docker-compose.simple.yml
curl -O https://raw.githubusercontent.com/logtide-dev/logtide/main/docker/.env.example
mv .env.example .env

# Edit .env: set DB_PASSWORD and API_KEY_SECRET (openssl rand -base64 32)
nano .env

docker compose -f docker-compose.simple.yml up -d
```

Create an organization, a project (e.g. `homelab`), and an API key. Full steps in the [getting started guide](/docs/getting-started/) — including the low-resource Docker setup.

> Running Docker inside a Proxmox LXC works well with a privileged container or with `keyctl=1,nesting=1` features enabled on an unprivileged one. If you'd rather avoid Docker-in-LXC, use a small VM.

## Step 2: Collect the host and LXC containers

Every Debian/Ubuntu-based LXC already logs to journald, so the [systemd integration](/integrations/systemd/) pattern covers most of the fleet: forward journald (or rsyslog) from each container to your central collector.

The lazy-but-effective variant: instead of touching every container, log from the **Proxmox host**, which can see its containers' journals, and tag entries by container. Either way, Fluent Bit ships them on:

```ini
[INPUT]
    Name systemd
    Tag host.systemd
    Read_From_Tail On

[INPUT]
    Name syslog
    Mode udp
    Listen 0.0.0.0
    Port 5140
    Parser syslog-rfc5424
    Tag lxc.syslog

[OUTPUT]
    Name http
    Match *
    Host logtide.lan
    Port 8080
    URI /api/v1/ingest/single
    Format json_lines
    Header X-API-Key lp_your_api_key
    Header Content-Type application/json
```

Docker-based services (Forgejo, Vikunja, etc.) plug in through the [Docker integration](/integrations/docker/) — either the Fluent Bit forward driver or tailing the container log files.

## Step 3: Watch your packets (firewall logging)

This is the part homelabbers ask about most: *what is actually hitting my network?* You don't need a packet capture appliance — firewall logs give you the connection-level story (who, what port, allowed or dropped) at a fraction of the volume.

### Linux router / Proxmox firewall (nftables)

Add logging to the rules you care about — typically dropped inbound traffic and accepted connections to exposed ports:

```bash
# Log dropped inbound packets, rate-limited to protect the journal
nft add rule inet filter input limit rate 10/second log prefix \"fw-drop: \" counter drop

# Log accepted connections to your exposed web service
nft add rule inet filter input tcp dport { 80, 443 } limit rate 50/second log prefix \"fw-accept-web: \" counter accept
```

These entries land in the kernel log (journald), which your Fluent Bit pipeline from Step 2 already ships. The `fw-drop:` / `fw-accept-web:` prefixes become trivially searchable in LogTide, and the log line carries source IP, destination port, and protocol.

> Always rate-limit firewall logging (`limit rate`). A port scan against an unlimited `log` rule can flood the journal faster than you'd expect.

### OPNsense / pfSense

If a BSD firewall fronts your network, point its remote syslog at the Fluent Bit syslog input from Step 2 (`Settings → Logging → Remote`) and select the firewall log category. Same pipeline, zero extra moving parts.

### What you get

- A searchable history of every dropped probe against your WAN — by source IP, port, and time
- Alert rules like *"more than 50 drops from one IP in 5 minutes"* (port scan) or *"any accepted connection to a port that isn't 80/443"*
- Correlation between a firewall accept and what your reverse proxy and application logged seconds later — one timeline instead of three

## Step 4: The exposed service gets the SIEM treatment

Anything public-facing attracts scanners within hours. Ship your reverse proxy logs ([nginx](/integrations/nginx/) or Traefik) and auth logs, then let LogTide's built-in detection work:

- **SSH brute force**: Sigma rules flag repeated `Failed password` bursts — useful even if you only expose SSH through a VPN, as a canary
- **Web scanning**: bursts of 404s and probes for `/wp-admin`, `/.env`, `/phpmyadmin` against a site that has none of them
- **AdGuard DNS anomalies**: ship AdGuard Home's query log and alert on lookups to known-bad domains from your own LAN — the cheapest malware tripwire there is

Set up [real-time alerts](/use-cases/real-time-alerting/) to Telegram, Discord or email, and the homelab tells *you* when something's wrong — instead of you finding out three weeks later in a forgotten log file.

## Why not ELK / Loki / a SaaS free tier?

| | ELK Stack | Grafana Loki | SaaS free tiers | LogTide |
|---|---|---|---|---|
| RAM footprint | 8-16 GB | 2-4 GB (+ Grafana) | n/a | **~3 GB under load** |
| Full-text + field search | Yes | Labels only | Limited | Yes |
| Retention | Disk-bound | Disk-bound | Days, capped volume | Disk-bound |
| SIEM / detections | Paid tier | No | No | **Included** |
| Data leaves your network | No | No | **Yes** | No |

SaaS free tiers cap volume and retention exactly where a homelab gets interesting, and shipping your home network's logs (every DNS query, every failed login) to a third party defeats half the point of self-hosting. ELK works but eats a homelab-sized server by itself. LogTide is built for exactly this footprint: one service, one database, everything included.

## Realistic resource budget

| Component | Sizing |
|-----------|--------|
| LogTide LXC/VM | 2 vCPU, 4 GB RAM (≈3 GB used under load), 50 GB disk |
| Fluent Bit collector | ~50 MB RAM (can live in the same container) |
| Typical homelab volume | 100-500 MB/day → years of retention on 50 GB |

Start with everything on one box; split the collector out only if you grow into multiple physical hosts.

---

**Get started:**

- [Getting started guide](/docs/getting-started/) — Docker Compose deployment, low-resource setup included
- [systemd integration](/integrations/systemd/) — journald forwarding for host and LXCs
- [Docker integration](/integrations/docker/) — for your containerized services
- [No-SDK setup](/docs/no-sdk-setup/) — Fluent Bit patterns for everything else


---

# Incident Response with Structured Logs

> Reduce MTTR with structured logging, correlation IDs, and runbook-driven incident management using LogTide.

*Source: https://logtide.dev/use-cases/incident-response/*

---

When production breaks at 3 AM, the difference between a 5-minute fix and a 3-hour outage comes down to one thing: can you find the relevant logs fast enough? This guide shows how to structure your logging for rapid incident response with LogTide.

## The Incident Response Problem

### What Goes Wrong During Incidents

Most teams waste time during incidents because:

```typescript
// ❌ BAD: Unstructured log that's useless during incidents
console.log('Error processing request');

// ❌ BAD: No correlation between services
logger.error('Payment failed');
// Which user? Which order? Which service triggered this?

// ❌ BAD: Missing context
logger.error('Database timeout');
// Which query? How long did it wait? Is it the primary or replica?
```

**Common time sinks during incidents:**
- 30+ minutes searching for the right logs
- No way to trace a request across services
- Key context missing from error messages
- No timeline of events leading to the failure
- Multiple team members searching independently

### The Cost of Slow Incident Response

| Metric | Impact |
|--------|--------|
| MTTR > 30 min | Lost revenue, SLA violations |
| MTTR > 1 hour | Customer churn, brand damage |
| MTTR > 4 hours | Regulatory reporting (for financial services) |
| Repeat incidents | Team burnout, on-call fatigue |

## The LogTide Approach

Build your logging for incident response from day one:

1. **Structured events** with consistent fields
2. **Correlation IDs** that trace across services
3. **Context-rich errors** with debugging data
4. **Alert-driven detection** for fast notification
5. **Real-time streaming** for live investigation
6. **Built-in incidents** for tracking and post-mortems

## Implementation

### 1. Request Correlation

Every request gets a trace ID that follows it across all services:

```typescript
// middleware/correlation.ts
import { AsyncLocalStorage } from 'async_hooks';
import crypto from 'crypto';

interface RequestContext {
  requestId: string;
  traceId: string;
  spanId: string;
  userId?: string;
  service: string;
  startTime: number;
}

export const requestStorage = new AsyncLocalStorage<RequestContext>();

export function correlationMiddleware(serviceName: string) {
  return (req: Request, res: Response, next: NextFunction) => {
    const context: RequestContext = {
      requestId: req.headers['x-request-id'] as string || crypto.randomUUID(),
      traceId: req.headers['x-trace-id'] as string || crypto.randomUUID(),
      spanId: crypto.randomUUID().split('-')[0],
      userId: req.user?.id,
      service: serviceName,
      startTime: Date.now(),
    };

    // Pass correlation headers to downstream services
    res.setHeader('x-request-id', context.requestId);
    res.setHeader('x-trace-id', context.traceId);

    requestStorage.run(context, () => next());
  };
}

export function getContext(): RequestContext | undefined {
  return requestStorage.getStore();
}
```

### 2. Context-Rich Logger

```typescript
// lib/logger.ts
import { LogTideClient } from '@logtide/sdk-node';
import { getContext } from './middleware/correlation';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

function enrichMetadata(metadata: Record<string, unknown> = {}) {
  const ctx = getContext();
  if (!ctx) return metadata;

  return {
    ...metadata,
    request_id: ctx.requestId,
    trace_id: ctx.traceId,
    span_id: ctx.spanId,
    user_id: ctx.userId,
    service: ctx.service,
    duration_ms: Date.now() - ctx.startTime,
  };
}

export const logger = {
  info(message: string, metadata?: Record<string, unknown>) {
    const ctx = getContext();
    client.info(ctx?.service || 'app', message, enrichMetadata(metadata));
  },

  warn(message: string, metadata?: Record<string, unknown>) {
    const ctx = getContext();
    client.warn(ctx?.service || 'app', message, enrichMetadata(metadata));
  },

  error(message: string, metadata?: Record<string, unknown>) {
    const ctx = getContext();
    client.error(ctx?.service || 'app', message, enrichMetadata(metadata));
  },

  // Special method for incident-critical events
  critical(message: string, metadata?: Record<string, unknown>) {
    const ctx = getContext();
    client.critical(ctx?.service || 'app', message, enrichMetadata({
      ...metadata,
      incident_relevant: true,
    }));
  },
};
```

### 3. Error Logging with Context

```typescript
// ✅ GOOD: Rich context for debugging
app.post('/api/orders', async (req, res) => {
  const { items, paymentMethod } = req.body;

  logger.info('Order creation started', {
    itemCount: items.length,
    paymentMethod,
    totalAmount: calculateTotal(items),
  });

  try {
    // Step 1: Validate inventory
    const inventory = await checkInventory(items);
    if (!inventory.available) {
      logger.warn('Order failed: insufficient inventory', {
        unavailableItems: inventory.unavailable,
        requestedItems: items.map(i => i.sku),
      });
      return res.status(400).json({ error: 'Items unavailable' });
    }

    // Step 2: Process payment
    const payment = await processPayment({
      amount: calculateTotal(items),
      method: paymentMethod,
      userId: req.user.id,
    });

    logger.info('Payment processed', {
      paymentId: payment.id,
      amount: payment.amount,
      processingTimeMs: payment.duration,
    });

    // Step 3: Create order
    const order = await createOrder({
      userId: req.user.id,
      items,
      paymentId: payment.id,
    });

    logger.info('Order created successfully', {
      orderId: order.id,
      itemCount: items.length,
      totalAmount: order.total,
    });

    res.json(order);
  } catch (error) {
    // ✅ Rich error context for incident response
    logger.error('Order creation failed', {
      error: error.message,
      errorCode: error.code,
      stack: error.stack,
      step: error.step || 'unknown',
      itemCount: items.length,
      paymentMethod,
      // Include enough to reproduce without exposing PII
    });

    res.status(500).json({ error: 'Order failed' });
  }
});
```

### 4. Service-to-Service Correlation

When calling downstream services, forward correlation headers:

```typescript
// lib/http-client.ts
import { getContext } from './middleware/correlation';

export async function serviceCall(url: string, options: RequestInit = {}) {
  const ctx = getContext();
  const headers = {
    ...options.headers,
    'Content-Type': 'application/json',
  };

  if (ctx) {
    headers['x-request-id'] = ctx.requestId;
    headers['x-trace-id'] = ctx.traceId;
    headers['x-caller-service'] = ctx.service;
  }

  const startTime = Date.now();

  try {
    const response = await fetch(url, { ...options, headers });
    const duration = Date.now() - startTime;

    logger.info('Service call completed', {
      target_url: url,
      status: response.status,
      duration_ms: duration,
    });

    return response;
  } catch (error) {
    const duration = Date.now() - startTime;

    logger.error('Service call failed', {
      target_url: url,
      error: error.message,
      duration_ms: duration,
    });

    throw error;
  }
}
```

### 5. Health Check Logging

```typescript
// health/checks.ts
import { logger } from '@/lib/logger';

interface HealthCheck {
  name: string;
  check: () => Promise<boolean>;
  critical: boolean;
}

const checks: HealthCheck[] = [
  {
    name: 'database',
    check: async () => {
      const start = Date.now();
      await db.$queryRaw`SELECT 1`;
      return Date.now() - start < 5000;
    },
    critical: true,
  },
  {
    name: 'redis',
    check: async () => {
      await redis.ping();
      return true;
    },
    critical: false,
  },
  {
    name: 'external_api',
    check: async () => {
      const res = await fetch('https://api.payment.com/health');
      return res.ok;
    },
    critical: true,
  },
];

export async function runHealthChecks() {
  const results = await Promise.allSettled(
    checks.map(async (check) => {
      try {
        const healthy = await check.check();
        return { name: check.name, healthy, critical: check.critical };
      } catch (error) {
        return { name: check.name, healthy: false, critical: check.critical, error: error.message };
      }
    })
  );

  const unhealthy = results
    .map(r => r.status === 'fulfilled' ? r.value : null)
    .filter(r => r && !r.healthy);

  if (unhealthy.length > 0) {
    const criticalDown = unhealthy.filter(r => r.critical);

    if (criticalDown.length > 0) {
      logger.critical('Critical health check failure', {
        failed: criticalDown.map(r => r.name),
        errors: criticalDown.map(r => ({ name: r.name, error: r.error })),
      });
    } else {
      logger.warn('Non-critical health check failure', {
        failed: unhealthy.map(r => r.name),
      });
    }
  }

  return results;
}
```

## Incident Response Workflow

### Phase 1: Detection (0-5 minutes)

LogTide detects the issue through:
- Alert rules trigger on error thresholds
- Sigma detection rules fire on patterns
- Health check failures are logged

### Phase 2: Triage (5-10 minutes)

The on-call engineer investigates in LogTide:

**"What's happening right now?"**
```
level:error AND time:>5m
```

**"Which services are affected?"**
```
level:error AND time:>15m | group by service
```

**"What changed recently?"**
```
service:audit AND action:deployment.* AND time:>1h
```

### Phase 3: Investigation (10-20 minutes)

**Trace a specific request:**
```
trace_id:abc-123-def
```

**Find the root cause:**
```
service:payment-api AND level:error AND time:>30m
```

**Check if it's a known pattern:**
```
error_code:ECONNREFUSED AND service:payment-api
```

### Phase 4: Resolution

Fix the issue, verify with real-time streaming, then document.

### Phase 5: Post-Mortem

Use LogTide data for the post-mortem timeline:

```typescript
// Generate incident timeline from logs
async function generateTimeline(
  traceId: string,
  startTime: string,
  endTime: string
) {
  const { logs: events } = await logtide.query({
    q: traceId,
    from: startTime,
    to: endTime,
    limit: 1000,
  });

  return events
    .sort((a, b) => a.time.localeCompare(b.time))
    .map(event => ({
      time: event.time,
      service: event.service,
      level: event.level,
      message: event.message,
      metadata: event.metadata,
    }));
}
```

## Incident Response Checklist

- [ ] **Logging Infrastructure**
    - [ ] Correlation IDs (request_id, trace_id) in all logs
    - [ ] Context-rich error messages
    - [ ] Service-to-service header forwarding
    - [ ] Health check logging

- [ ] **Alert Configuration**
    - [ ] Error rate alerts per service
    - [ ] Latency alerts (p99 > threshold)
    - [ ] Health check failure alerts
    - [ ] Critical business event alerts

- [ ] **On-Call Preparedness**
    - [ ] Common LogTide queries documented
    - [ ] Runbook for each service
    - [ ] Escalation paths defined
    - [ ] Access to LogTide verified for all on-call

- [ ] **Post-Incident**
    - [ ] Timeline generation from logs
    - [ ] Root cause analysis process
    - [ ] Action items tracked
    - [ ] Detection improvements implemented

## Common Pitfalls

### 1. "We'll add correlation IDs later"

By the time you need them during an incident, it's too late.

**Solution:** Add correlation middleware on day one. It's a one-time setup.

### 2. "Our error messages are descriptive enough"

`"Database error"` tells you nothing. `"PostgreSQL connection timeout after 5000ms to primary db at db.internal:5432"` tells you everything.

**Solution:** Include the what, where, and why in every error log.

### 3. "We don't need structured logging, grep works fine"

Grep through 500 GB of unstructured logs during a 3 AM incident. Good luck.

**Solution:** Use structured JSON logs with consistent fields from the start.

## Performance Considerations

| Aspect | Impact | Mitigation |
|--------|--------|------------|
| Correlation middleware | <1ms per request | AsyncLocalStorage is fast |
| Extra metadata fields | ~100 bytes per log | Negligible at any volume |
| Service call logging | <1ms per call | Async, non-blocking |
| Health checks | 1-5 seconds | Run in background, cache results |

## Next Steps

- [Node.js SDK](/integrations/nodejs/) - Full logging setup
- [nginx Integration](/integrations/nginx/) - Web server logging
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Alert configuration
- [Security Monitoring](/use-cases/security-monitoring/) - Threat detection

---

**Ready to improve your incident response?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Share your incident response setup


---

# Logging Cost Optimization

> Cut log management costs by 80-90% by switching from SaaS platforms to self-hosted LogTide. Real TCO comparison with numbers.

*Source: https://logtide.dev/use-cases/cost-optimization/*

---

Log management costs are the fastest-growing line item in many engineering budgets. If you're spending $5K-50K+/month on Datadog, Splunk, or Elastic Cloud, this guide shows you exactly how much you can save with LogTide.

## The SaaS Logging Cost Problem

### How SaaS Pricing Works

Most SaaS logging platforms charge based on data volume:

| Platform | Pricing Model | Typical Cost (100 GB/day) |
|----------|---------------|--------------------------|
| Datadog | $0.10-$1.70/GB ingested + per-user | $3,000-$15,000/month |
| Splunk Cloud | $150-$1,800/GB/day indexed | $15,000-$180,000/year |
| Elastic Cloud | $0.274/GB storage + compute | $2,000-$8,000/month |
| New Relic | $0.30/GB ingested (beyond free) | $900-$3,000/month |
| Grafana Cloud | $0.50/GB logs | $1,500-$5,000/month |

### The Hidden Costs

Volume-based pricing creates perverse incentives:

```typescript
// ❌ BAD: Engineers stop logging to save money
if (process.env.NODE_ENV === 'production') {
  // "We can't afford to log debug data"
  logger.setLevel('error');
}

// ❌ BAD: Sampling reduces visibility
const SAMPLE_RATE = 0.01; // Only log 1% of requests
if (Math.random() < SAMPLE_RATE) {
  logger.info('Request processed', metadata);
}
```

**Consequences of under-logging:**
- Missed bugs that only appear in specific conditions
- Security incidents detected days late
- Longer MTTR (Mean Time to Resolution) during outages
- Compliance gaps (audit trails with holes)

### Cost Growth Is Exponential

As your application grows, log volume grows faster:

| Stage | Daily Log Volume | Datadog Cost/Month | Splunk Cost/Year |
|-------|-----------------|-------------------|-----------------|
| Seed (2 services) | 5 GB/day | $150-$500 | $9,000-$18,000 |
| Growth (10 services) | 50 GB/day | $1,500-$5,000 | $90,000-$180,000 |
| Scale (50 services) | 500 GB/day | $15,000-$50,000 | $900,000+ |
| Enterprise (200+ services) | 2 TB/day | $60,000-$200,000 | $3,600,000+ |

## The LogTide Approach: Infrastructure-Only Costs

LogTide is self-hosted. You pay for compute and storage, not per-GB.

### TCO Comparison: 100 GB/Day

**Datadog (SaaS):**
- Log ingestion: 100 GB × $0.10 × 30 = $300/month (cheapest tier)
- Log retention (15 days): included in ingestion
- 10 users × $23/user = $230/month
- Cloud SIEM: 100 GB × $0.20 × 30 = $600/month
- **Total: ~$1,130/month minimum** (often 3-5x higher with retention and features)

**Splunk Cloud (SaaS):**
- 100 GB/day license: ~$15,000-$50,000/year
- **Total: ~$1,250-$4,166/month**

**LogTide (Self-hosted on typical cloud provider):**
- VPS (8 vCPU, 32 GB RAM): $150-$200/month
- SSD storage (2 TB): $50-$100/month
- Backup storage: $20-$40/month
- **Total: ~$220-$340/month**

**Savings: 70-90% compared to Datadog, 80-95% compared to Splunk.**

### TCO Comparison: 500 GB/Day

| Cost Item | Datadog | Splunk Cloud | LogTide |
|-----------|---------|-------------|---------|
| Ingestion/License | $5,000-$15,000/mo | $75,000-$150,000/yr | $0 |
| Retention (30 days) | $1,500/mo | Included | $0 |
| Users (20) | $460/mo | Included | $0 |
| SIEM | $3,000/mo | $25,000/yr extra | $0 (built-in) |
| Infrastructure | N/A | N/A | $500-$800/mo |
| **Monthly Total** | **$10,000-$20,000** | **$8,300-$14,500** | **$500-$800** |
| **Annual Total** | **$120,000-$240,000** | **$100,000-$175,000** | **$6,000-$9,600** |

### What You Get With LogTide

For that infrastructure cost, you get:

- **Unlimited log ingestion** (no per-GB fees)
- **Unlimited users** (no per-seat licensing)
- **Built-in SIEM** with Sigma detection rules
- **Incident management** included
- **MITRE ATT&CK mapping** included
- **Full-text search** across all logs
- **Real-time streaming** via SSE
- **OpenTelemetry** native support
- **REST API** with SDKs for Node.js, Python, PHP, Go, Kotlin, C#

## Implementation: Migrating to Save Costs

### 1. Calculate Your Current Spend

Before migrating, understand your baseline:

```bash
# If using Datadog, check your billing
# Settings → Organization → Billing

# If using Splunk, check license usage
# Settings → Licensing → Usage Report

# Estimate daily volume from your current platform
# This is the number you'll use to size LogTide
```

### 2. Size Your LogTide Deployment

| Daily Volume | CPU | RAM | Storage (30d retention) | Monthly Cost |
|-------------|-----|-----|------------------------|-------------|
| 10 GB/day | 4 cores | 16 GB | 500 GB SSD | $80-$120 |
| 50 GB/day | 8 cores | 32 GB | 2 TB SSD | $200-$350 |
| 100 GB/day | 8 cores | 32 GB | 4 TB SSD | $300-$500 |
| 500 GB/day | 16 cores | 64 GB | 20 TB SSD | $800-$1,200 |
| 1 TB/day | 32 cores | 128 GB | 40 TB SSD | $1,500-$2,500 |

### 3. Deploy LogTide

```bash
# Clone and configure
git clone https://github.com/logtide-dev/logtide.git
cd logtide/docker

cp .env.example .env
# Edit .env with your settings

# Start
docker compose up -d

# Verify
curl http://localhost:8080/health
```

### 4. Migrate Gradually

Run both platforms in parallel during migration:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Dual-write during migration
function log(level: string, message: string, meta?: object) {
  // Keep sending to existing platform
  existingLogger[level](message, meta);

  // Also send to LogTide
  logtide[level]('my-service', message, meta);
}
```

### 5. Validate and Cut Over

After 1-2 weeks of parallel operation:

1. Compare log counts between platforms
2. Verify search results match
3. Test alert rules in LogTide
4. Confirm team is comfortable with LogTide UI
5. Switch off the old platform
6. Cancel SaaS subscription

## Real-World Example: B2B SaaS Startup

**Company profile:**
- 15 microservices
- 80 GB/day log volume
- 12 engineers
- Previously on Datadog

**Before (Datadog):**
- Log ingestion: $2,400/month
- APM: $1,200/month
- Infrastructure monitoring: $600/month
- 12 users × $23 = $276/month
- **Total: $4,476/month ($53,712/year)**

**After (LogTide on Hetzner):**
- Dedicated server (AMD Ryzen 9, 64 GB RAM, 2×2 TB NVMe): €65/month
- Backup storage: €10/month
- **Total: €75/month (~$82/month, ~$984/year)**

**Annual savings: $52,728 (98%)**

The team also stopped sampling logs and enabled debug-level logging in production, improving their MTTR by 40%.

## When LogTide Isn't the Right Choice

Be honest with yourself about trade-offs:

**LogTide might not fit if:**
- You need advanced APM with deep metrics correlation (LogTide supports basic metrics, but focuses on logs + SIEM)
- You have no DevOps capacity to manage infrastructure
- You need 100+ pre-built integrations with cloud services
- Compliance requires a SOC2-certified SaaS vendor

**LogTide is ideal if:**
- Logging costs are a significant budget concern
- You value data ownership and privacy (GDPR)
- You need SIEM capabilities without extra cost
- You have basic infrastructure management skills
- You want to log everything without worrying about per-GB costs

## Cost Optimization Checklist

- [ ] **Audit Current Costs**
    - [ ] Document monthly logging spend
    - [ ] Calculate cost per GB
    - [ ] Identify hidden costs (users, retention, SIEM add-ons)

- [ ] **Size LogTide Deployment**
    - [ ] Measure daily log volume
    - [ ] Choose infrastructure provider
    - [ ] Estimate monthly infrastructure cost
    - [ ] Calculate projected savings

- [ ] **Migration Plan**
    - [ ] Set up LogTide instance
    - [ ] Configure SDK/log forwarding
    - [ ] Run parallel for 1-2 weeks
    - [ ] Validate data consistency
    - [ ] Cut over and cancel SaaS

- [ ] **Ongoing Optimization**
    - [ ] Monitor storage usage
    - [ ] Configure retention policies
    - [ ] Right-size infrastructure quarterly

## Common Pitfalls

### 1. "Self-hosted is too much work"

LogTide runs as a single Docker Compose stack. If you can run PostgreSQL, you can run LogTide.

**Reality:** Initial setup takes 1-2 hours. Ongoing maintenance is minimal (backups, updates).

### 2. "We'll lose features"

LogTide includes SIEM, alerting, and incident management that cost extra on SaaS platforms.

**Reality:** You gain features (Sigma rules, MITRE ATT&CK) that would cost $10K+/year on Datadog or Splunk.

### 3. "The savings aren't worth the risk"

Run both platforms in parallel during migration. Zero risk.

**Reality:** Most teams complete migration in 1-2 weeks with no incidents.

## Next Steps

- [Docker Deployment](/integrations/docker/) - Production deployment guide
- [Kubernetes Integration](/integrations/kubernetes/) - K8s deployment
- [Security Monitoring](/use-cases/security-monitoring/) - Built-in SIEM features
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Data privacy benefits

---

**Ready to cut your logging costs?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Ask migration questions


---

# Microservices Observability

> Achieve full observability across distributed microservices with centralized logging, correlation IDs, and service dependency mapping in LogTide.

*Source: https://logtide.dev/use-cases/microservices-observability/*

---

In a microservices architecture, every user action fans out across dozens of services. When something breaks, the error you see in one service is often just a symptom -- the root cause lives three hops upstream, buried in a different service's logs. This guide shows how to build full observability across your distributed system with LogTide.

## The Problem with Distributed Logs

In a monolith, debugging is straightforward: one process, one log stream, one place to look. Microservices shatter that simplicity:

```d2
direction: right
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

req: User Request { class: src }
gw: API Gateway { class: src }
auth: Auth Service { class: src }
order: Order Service { class: src }
inv: Inventory API { class: src }
pay: Payment Service { class: src }
notif: Notification Svc { class: src }
ana: Analytics Service { class: src }

stdout1: stdout { class: dest }
filed: file { class: dest }
stdout2: stdout { class: dest }
cloudwatch: CloudWatch { class: dest }
datadog: Datadog { class: dest }
syslogd: syslog { class: dest }
nowhere: nowhere { class: dest }

req -> gw -> stdout1 { class: flow }
req -> auth -> filed { class: flow }
req -> order -> stdout2 { class: flow }
order -> inv -> cloudwatch { class: flow }
order -> pay -> datadog { class: flow }
order -> notif -> syslogd { class: flow }
req -> ana -> nowhere { class: flow }
```

```
❌ Distributed logging problems:

1. Fragmented logs       → Each service logs independently, no unified view
2. Missing correlation   → No way to trace a request across service boundaries
3. Inconsistent formats  → JSON here, plain text there, different field names
4. Blind spots           → Async workers and message consumers often unlogged
5. Blast radius unknown  → One failure cascades, but you can't see where
```

| Problem | Impact |
|---------|--------|
| No correlation IDs | 40+ minutes per incident tracing request paths manually |
| Fragmented log stores | Context-switching between 3-5 tools during debugging |
| Inconsistent schemas | Queries break across services, dashboards unreliable |
| No dependency mapping | Cascading failures go undetected until customers report them |

## The LogTide Approach

LogTide solves distributed observability with three principles:

1. **One destination** -- All services ship logs to a single LogTide instance
2. **Correlation by design** -- Trace context propagates automatically across service boundaries
3. **Structure everything** -- Consistent schemas make cross-service queries possible

### Architecture Overview

```d2
direction: down
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

gw: "API GW\n(Express)" { class: src }
auth: "Auth Svc\n(Fastify)" { class: src }
order: "Order Svc\n(Python)" { class: src }
payment: "Payment\n(Go)" { class: src }

sdk: "LogTide SDK Layer\n@logtide/express · @logtide/fastify · …" { class: hub }

server: LogTide Server {
  class: group
  grid-rows: 2
  correlation: Correlation Engine { class: hub }
  siem: "SIEM / Sigma Detection" { class: hub }
}

gw -> sdk: x-trace-id { class: flow }
auth -> sdk: x-trace-id { class: flow }
order -> sdk: x-trace-id { class: flow }
payment -> sdk: x-trace-id { class: flow }
sdk -> server: "batched, compressed" { class: flow }
```

## Implementation

### 1. Trace Context Propagation

The foundation of microservices observability is a trace ID that follows every request from ingress to the last downstream call:

```typescript
// shared/correlation.ts
import { AsyncLocalStorage } from 'async_hooks';
import crypto from 'crypto';

export interface TraceContext {
  traceId: string;       // Unique per user request, propagated across services
  spanId: string;        // Unique per service hop
  parentSpanId?: string; // The span that called this one
  service: string;
  startTime: number;
}

export const traceStorage = new AsyncLocalStorage<TraceContext>();

export function createTraceMiddleware(serviceName: string) {
  return (req: Request, res: Response, next: NextFunction) => {
    const context: TraceContext = {
      traceId: req.headers['x-trace-id'] as string || crypto.randomUUID(),
      spanId: crypto.randomUUID().slice(0, 16),
      parentSpanId: req.headers['x-span-id'] as string,
      service: serviceName,
      startTime: Date.now(),
    };

    res.setHeader('x-trace-id', context.traceId);
    res.setHeader('x-span-id', context.spanId);

    traceStorage.run(context, () => next());
  };
}

export function getTrace(): TraceContext | undefined {
  return traceStorage.getStore();
}
```

### 2. Service Logger with Auto-Enrichment

Wrap the LogTide SDK so every log event automatically includes trace context:

```typescript
// shared/logger.ts
import { LogTideClient } from '@logtide/sdk-node';
import { getTrace } from './correlation';

const SERVICE_NAME = process.env.SERVICE_NAME || 'unknown';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  batchSize: 200,
  flushInterval: 3000,
});

function enrichWithTrace(metadata: Record<string, unknown> = {}) {
  const trace = getTrace();
  if (!trace) return metadata;

  return {
    ...metadata,
    trace_id: trace.traceId,
    span_id: trace.spanId,
    parent_span_id: trace.parentSpanId,
    service: trace.service,
    elapsed_ms: Date.now() - trace.startTime,
  };
}

export const logger = {
  info(message: string, meta?: Record<string, unknown>) {
    client.info(SERVICE_NAME, message, enrichWithTrace(meta));
  },
  warn(message: string, meta?: Record<string, unknown>) {
    client.warn(SERVICE_NAME, message, enrichWithTrace(meta));
  },
  error(message: string, meta?: Record<string, unknown>) {
    client.error(SERVICE_NAME, message, enrichWithTrace(meta));
  },
};

process.on('SIGTERM', async () => {
  await client.flush();
  process.exit(0);
});
```

### 3. HTTP Client with Trace Propagation

When services call each other, trace headers must be forwarded automatically:

```typescript
// shared/http-client.ts
import { getTrace } from './correlation';
import { logger } from './logger';

export async function serviceCall(url: string, options: RequestInit = {}): Promise<Response> {
  const trace = getTrace();
  const headers: Record<string, string> = {
    'Content-Type': 'application/json',
    ...(options.headers as Record<string, string>),
  };

  if (trace) {
    headers['x-trace-id'] = trace.traceId;
    headers['x-span-id'] = trace.spanId;
  }

  const startTime = Date.now();
  const targetService = new URL(url).hostname;

  try {
    const response = await fetch(url, { ...options, headers });
    const duration = Date.now() - startTime;

    logger.info('Downstream call completed', {
      target_service: targetService,
      status: response.status,
      duration_ms: duration,
    });

    if (duration > 2000) {
      logger.warn('Slow downstream call detected', {
        target_service: targetService,
        duration_ms: duration,
      });
    }

    return response;
  } catch (error) {
    const err = error as Error;
    logger.error('Downstream call failed', {
      target_service: targetService,
      error: err.message,
      error_code: (err as any).code,
      duration_ms: Date.now() - startTime,
    });
    throw error;
  }
}
```

### 4. Python (FastAPI) Service Example

Not all services are in Node.js. Here is a Python downstream service using LogTide:

```python
# services/inventory-service/main.py
import os, time, uuid
from contextvars import ContextVar
from fastapi import FastAPI, Request, Response
from logtide_sdk import LogTideClient, ClientOptions

trace_context: ContextVar[dict] = ContextVar("trace_context", default={})

client = LogTideClient(ClientOptions(
    api_url=os.environ["LOGTIDE_API_URL"],
    api_key=os.environ["LOGTIDE_API_KEY"],
    batch_size=200,
    flush_interval=3.0,
    global_metadata={"service": "inventory-service"},
))

app = FastAPI()

@app.middleware("http")
async def trace_middleware(request: Request, call_next):
    ctx = {
        "trace_id": request.headers.get("x-trace-id", str(uuid.uuid4())),
        "span_id": str(uuid.uuid4())[:16],
        "parent_span_id": request.headers.get("x-span-id"),
    }
    trace_context.set(ctx)

    start = time.time()
    response: Response = await call_next(request)
    duration_ms = round((time.time() - start) * 1000)

    client.info(
        "inventory-service",
        f"{request.method} {request.url.path} {response.status_code}",
        {**ctx, "duration_ms": duration_ms},
    )
    return response

@app.post("/check")
async def check_inventory(items: list[dict]):
    ctx = trace_context.get({})
    client.info("inventory-service", "Inventory check started", {**ctx, "item_count": len(items)})

    unavailable = []
    for item in items:
        stock = await get_stock(item["sku"])
        if stock < item["quantity"]:
            unavailable.append({"sku": item["sku"], "requested": item["quantity"], "available": stock})

    return {"available": len(unavailable) == 0, "unavailable": unavailable}
```

```d2
direction: right
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

map: "Service Dependency Map" {
  class: group
  apigw: api-gw { class: hub }
  authsvc: "auth-service\n45ms avg" { class: src }
  notif: "notification\n120ms avg" { class: src }
  ordersvc: "order-service\n85ms avg" { class: src }
  inv: "inventory\n30ms avg" { class: src }
  pay: "payment\n200ms avg" { class: src }

  apigw -> authsvc { class: flow }
  apigw -> notif { class: flow }
  apigw -> ordersvc { class: flow }
  ordersvc -> inv { class: flow }
  ordersvc -> pay { class: flow }
}
```

### 6. Kubernetes Deployment

Deploy your microservices with LogTide credentials as a shared Secret:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: logtide-credentials
  namespace: ecommerce
type: Opaque
stringData:
  LOGTIDE_API_URL: "https://logtide.internal"
  LOGTIDE_API_KEY: "lp_your_api_key"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway
  namespace: ecommerce
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:
        - name: api-gateway
          image: myregistry/api-gateway:latest
          ports:
            - containerPort: 3000
          env:
            - name: SERVICE_NAME
              value: "api-gateway"
            - name: LOGTIDE_API_URL
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: LOGTIDE_API_URL
            - name: LOGTIDE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: logtide-credentials
                  key: LOGTIDE_API_KEY
```

Repeat this pattern for each service, changing the `SERVICE_NAME` and image. See the [Kubernetes Integration](/integrations/kubernetes/) guide for full setup.

### 7. Cross-Service Alerting

Set up alerts that detect problems spanning multiple services:

```yaml
# Cascade failure: multiple services erroring simultaneously
- name: cascade-failure
  query: 'level:error AND time:>5m | group by service | count > 10'
  threshold: 3
  window: 5m
  severity: critical

# Downstream latency spike
- name: downstream-latency-spike
  query: 'message:"Downstream call completed" AND duration_ms:>3000'
  threshold: 10
  window: 5m
  severity: warning

# Service unreachable
- name: service-unreachable
  query: 'message:"Downstream call failed" AND error_code:ECONNREFUSED'
  threshold: 3
  window: 2m
  severity: critical
```

## Real-World Example: E-Commerce Platform

An e-commerce company with 8 microservices (Express, Fastify, FastAPI, Go) handles 50,000 orders per day. Before LogTide, debugging a failed order meant checking logs in 5 different systems.

**Before LogTide:**
- MTTR for cross-service issues: 90 minutes
- Time to find relevant logs: 30+ minutes
- Recurring "mystery errors" with no root cause

**After LogTide:**
- MTTR for cross-service issues: 15 minutes
- Time to find relevant logs: < 2 minutes (search by trace_id)
- Zero unresolved incidents in 6 months

**Debugging a failed order:**

```
1. Alert fires: "order-service error rate > 5%"

2. Query recent errors:
   service:order-service AND level:error AND time:>15m

3. Find common error: "Downstream call failed: inventory-service"
   → trace_id: 8f3a-b2c1-d4e5

4. Search by trace to see the full picture:
   trace_id:8f3a-b2c1-d4e5

5. Trace timeline reveals:
   10:42:01.123  api-gateway       → POST /api/orders (started)
   10:42:01.145  auth-service      → Token verified (22ms)
   10:42:01.167  order-service     → Order creation started
   10:42:01.189  inventory-service → Connection refused ← ROOT CAUSE
   10:42:01.190  order-service     → Downstream call failed
   10:42:01.191  api-gateway       → 500 Internal Server Error

6. Root cause: inventory-service pod OOMKilled
   → Fix: increase memory limits, add HPA
```

## Cross-Service Query Patterns

```
# Trace a single request across all services
trace_id:8f3a-b2c1-d4e5

# Error rate by service (last hour)
level:error AND time:>1h | group by service | sort by count desc

# Services with elevated p99 latency
message:"Request completed" AND time:>30m
  | group by service | percentile(duration_ms, 99)

# Detect retry storms
target_service:inventory-service AND time:>10m
  | group by trace_id | count > 3

# Find all callers of a service
target_service:payment-service | group by service
```

## Observability Checklist

### Trace Propagation
- [ ] Trace middleware installed on all services
- [ ] `x-trace-id` header forwarded in all service-to-service calls
- [ ] `x-span-id` set per service for parent-child relationships
- [ ] Async workers and message consumers inherit trace context

### Consistent Logging
- [ ] All services use the LogTide SDK
- [ ] Standard fields: `trace_id`, `span_id`, `service`, `duration_ms`
- [ ] Downstream calls logged with target service and duration
- [ ] Errors include stack traces and contextual metadata

### Alerting and Infrastructure
- [ ] Cascade failure detection (multiple services erroring)
- [ ] Per-service error rate and latency alerts
- [ ] Service unreachable alerts (ECONNREFUSED)
- [ ] LogTide credentials stored as Kubernetes Secrets
- [ ] Graceful shutdown with `client.flush()` on SIGTERM

## Common Pitfalls

### 1. "We only need to trace errors"

If you only log error paths, you have no baseline for normal behavior. When latency doubles but nothing errors, you are blind.

**Solution:** Log request start and completion for every request. Use sampling for debug logs, but always capture lifecycle events.

### 2. "Each team picks their own logging library"

Service A uses Winston, Service B uses Pino, Service C uses Python logging. Field names differ, log levels differ. Cross-service queries become impossible.

**Solution:** Standardize on LogTide SDKs across all services. The SDKs handle field naming, batching, and delivery consistently.

### 3. "Async jobs don't need tracing"

Background workers, cron jobs, and message consumers are often the source of subtle bugs. Without trace context, their failures are orphaned.

**Solution:** Include the trace_id in message payloads and restore it when consuming:

```typescript
// Producer: include trace in message
await queue.publish('order.process', {
  traceId: getTrace()?.traceId,
  orderId: order.id,
});

// Consumer: restore trace context
queue.subscribe('order.process', (msg) => {
  const ctx = { traceId: msg.traceId, spanId: crypto.randomUUID().slice(0, 16) };
  traceStorage.run(ctx, () => processOrder(msg));
});
```

## Next Steps

- [Express Integration](/integrations/express/) - Detailed Express SDK setup
- [Fastify Integration](/integrations/fastify/) - Fastify SDK and middleware
- [Kubernetes Integration](/integrations/kubernetes/) - Deploy LogTide on K8s
- [Incident Response](/use-cases/incident-response/) - Use traces for faster debugging
- [Real-Time Alerting](/use-cases/real-time-alerting/) - Alert on cross-service failures

---

**Ready to unify your microservices logs?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Share your observability setup


---

# Multi-Tenant SaaS Logging

> Implement per-tenant log isolation and analytics for B2B SaaS applications with LogTide's organization-based architecture.

*Source: https://logtide.dev/use-cases/multi-tenant-saas/*

---

Multi-tenant SaaS applications need logging that respects tenant boundaries while enabling platform-wide observability. This guide shows you how to implement tenant-aware logging with LogTide.

## The Multi-Tenant Logging Challenge

### Common Mistakes

```typescript
// ❌ BAD: No tenant context
logger.info('User created account');

// ❌ BAD: Tenant ID buried in unstructured message
logger.info(`User created account in tenant ${tenantId}`);

// ❌ BAD: Inconsistent tenant identification
logger.info('User created', { org: tenantId });      // sometimes 'org'
logger.info('Order placed', { tenant: tenantId });   // sometimes 'tenant'
logger.info('Payment failed', { customer: tenantId }); // sometimes 'customer'
```

**Issues:**
- Can't filter logs by tenant for debugging
- Can't provide tenant-specific log access
- Can't calculate per-tenant costs
- Can't ensure data isolation for compliance
- Support team sees all tenants mixed together

## The LogTide Approach

LogTide's architecture naturally supports multi-tenancy:

**Data Flow:**
1. **Tenants** (Acme, TechCorp, StartupX) generate logs from their activity
2. **Tenant-Aware Middleware** intercepts all requests and adds `tenant_id` to every log
3. **LogTide** stores logs with tenant context, enabling per-tenant filtering and isolation

## Implementation

### 1. Tenant Context Middleware

Create middleware that extracts and propagates tenant context:

```typescript
// middleware/tenant-context.ts
import { AsyncLocalStorage } from 'async_hooks';

interface TenantContext {
  tenantId: string;
  tenantName: string;
  tenantPlan: string;
  userId?: string;
  requestId: string;
}

export const tenantStorage = new AsyncLocalStorage<TenantContext>();

export function getTenantContext(): TenantContext | undefined {
  return tenantStorage.getStore();
}

// Express middleware
export function tenantContextMiddleware(req: Request, res: Response, next: NextFunction) {
  // Extract tenant from JWT, subdomain, header, etc.
  const tenantId = extractTenantId(req);
  const tenant = await getTenantDetails(tenantId);

  const context: TenantContext = {
    tenantId: tenant.id,
    tenantName: tenant.name,
    tenantPlan: tenant.plan,
    userId: req.user?.id,
    requestId: req.headers['x-request-id'] || crypto.randomUUID(),
  };

  // Run the rest of the request in this context
  tenantStorage.run(context, () => {
    next();
  });
}

function extractTenantId(req: Request): string {
  // Option 1: From subdomain (acme.yourapp.com)
  const subdomain = req.hostname.split('.')[0];
  if (subdomain && subdomain !== 'www' && subdomain !== 'app') {
    return subdomain;
  }

  // Option 2: From JWT claims
  if (req.user?.tenantId) {
    return req.user.tenantId;
  }

  // Option 3: From header (for internal services)
  if (req.headers['x-tenant-id']) {
    return req.headers['x-tenant-id'] as string;
  }

  throw new Error('Tenant not identified');
}
```

### 2. Tenant-Aware Logger

```typescript
// lib/logger.ts
import { LogTideClient } from '@logtide/sdk-node';
import { getTenantContext } from './middleware/tenant-context';

const baseClient = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  globalMetadata: {
    service: 'saas-platform',
    environment: process.env.NODE_ENV,
  },
});

class TenantAwareLogger {
  private client: LogTideClient;

  constructor(client: LogTideClient) {
    this.client = client;
  }

  private enrichWithTenant(metadata: Record<string, unknown> = {}): Record<string, unknown> {
    const context = getTenantContext();

    if (context) {
      return {
        ...metadata,
        tenant_id: context.tenantId,
        tenant_name: context.tenantName,
        tenant_plan: context.tenantPlan,
        user_id: context.userId,
        request_id: context.requestId,
      };
    }

    return {
      ...metadata,
      tenant_id: 'system', // For background jobs without tenant context
    };
  }

  info(message: string, metadata?: Record<string, unknown>) {
    this.client.info('saas-platform', message, this.enrichWithTenant(metadata));
  }

  warn(message: string, metadata?: Record<string, unknown>) {
    this.client.warn('saas-platform', message, this.enrichWithTenant(metadata));
  }

  error(message: string, metadata?: Record<string, unknown>) {
    this.client.error('saas-platform', message, this.enrichWithTenant(metadata));
  }

  debug(message: string, metadata?: Record<string, unknown>) {
    this.client.debug('saas-platform', message, this.enrichWithTenant(metadata));
  }
}

export const logger = new TenantAwareLogger(baseClient);
```

### 3. Usage in Application Code

```typescript
import { logger } from '@/lib/logger';

// ✅ GOOD: Tenant context automatically included
app.post('/api/projects', async (req, res) => {
  const { name, description } = req.body;

  logger.info('Creating project', {
    projectName: name,
  });

  try {
    const project = await db.project.create({
      data: {
        name,
        description,
        tenantId: req.tenant.id, // Still store in DB
      },
    });

    logger.info('Project created successfully', {
      projectId: project.id,
      projectName: name,
    });

    res.json(project);
  } catch (error) {
    logger.error('Failed to create project', {
      projectName: name,
      error: error.message,
    });

    res.status(500).json({ error: 'Failed to create project' });
  }
});
```

Output in LogTide:

```json
{
  "level": "info",
  "message": "Project created successfully",
  "service": "saas-platform",
  "tenant_id": "acme-corp",
  "tenant_name": "Acme Corporation",
  "tenant_plan": "enterprise",
  "user_id": "user_123",
  "request_id": "req_abc456",
  "metadata": {
    "projectId": "proj_789",
    "projectName": "Q4 Campaign"
  }
}
```

### 4. Background Job Logging

For background jobs without HTTP context:

```typescript
// jobs/process-invoice.ts
import { tenantStorage, TenantContext } from '@/middleware/tenant-context';
import { logger } from '@/lib/logger';

interface InvoiceJob {
  tenantId: string;
  invoiceId: string;
}

export async function processInvoice(job: InvoiceJob) {
  // Manually set tenant context for background jobs
  const tenant = await getTenantDetails(job.tenantId);

  const context: TenantContext = {
    tenantId: tenant.id,
    tenantName: tenant.name,
    tenantPlan: tenant.plan,
    requestId: `job_${crypto.randomUUID()}`,
  };

  await tenantStorage.run(context, async () => {
    logger.info('Processing invoice', {
      invoiceId: job.invoiceId,
    });

    try {
      await generateInvoicePDF(job.invoiceId);
      await sendInvoiceEmail(job.invoiceId);

      logger.info('Invoice processed successfully', {
        invoiceId: job.invoiceId,
      });
    } catch (error) {
      logger.error('Invoice processing failed', {
        invoiceId: job.invoiceId,
        error: error.message,
      });
      throw error;
    }
  });
}
```

## Tenant-Specific Log Access

### Providing Tenants Access to Their Logs

Create an API endpoint for tenants to view their own logs:

```typescript
// api/tenant-logs.ts
import { LogTideClient } from '@logtide/sdk-node';

app.get('/api/admin/logs', requireTenantAdmin, async (req, res) => {
  const { tenantId } = req.tenant;
  const { startDate, endDate, level, search } = req.query;

  // Query LogTide, then enforce the tenant boundary on metadata
  const { logs } = await logtide.query({
    level,
    from: startDate,
    to: endDate,
    q: search,
    limit: 500,
  });

  const tenantLogs = logs.filter(log => log.metadata?.tenant_id === tenantId);

  // Sanitize logs before returning (remove internal fields)
  const sanitizedLogs = tenantLogs.slice(0, 100).map(log => ({
    timestamp: log.time,
    level: log.level,
    message: log.message,
    metadata: sanitizeMetadata(log.metadata ?? {}),
  }));

  res.json(sanitizedLogs);
});

function sanitizeMetadata(metadata: Record<string, unknown>) {
  // Remove internal fields
  const { tenant_id, tenant_name, ...safe } = metadata;
  return safe;
}
```

## Cross-Tenant Analytics

### Platform-Wide Metrics

For your internal dashboards:

```typescript
// Analytics for platform operators
async function getPlatformMetrics() {
  const last24h = new Date(Date.now() - 24 * 60 * 60 * 1000);

  // Hourly volume + top services and errors, straight from the stats API
  const stats = await logtide.getAggregatedStats({
    from: last24h,
    to: new Date(),
    interval: '1h',
  });

  // Errors by tenant (find problematic tenants): query, then group on metadata
  const { logs: recentErrors } = await logtide.query({
    level: 'error',
    from: last24h,
    limit: 1000,
  });

  const errorsByTenant = new Map<string, number>();
  for (const log of recentErrors) {
    const tenant = String(log.metadata?.tenant_id ?? 'unknown');
    errorsByTenant.set(tenant, (errorsByTenant.get(tenant) ?? 0) + 1);
  }

  // Slowest tenants: average duration_ms per tenant from the same window
  const { logs: requests } = await logtide.query({
    q: 'HTTP',
    from: last24h,
    limit: 1000,
  });

  const durations = new Map<string, number[]>();
  for (const log of requests) {
    const tenant = String(log.metadata?.tenant_id ?? 'unknown');
    const ms = Number(log.metadata?.duration_ms);
    if (!Number.isNaN(ms)) {
      durations.set(tenant, [...(durations.get(tenant) ?? []), ms]);
    }
  }
  const slowestTenants = [...durations.entries()]
    .map(([tenant, ms]) => ({ tenant, avgMs: ms.reduce((a, b) => a + b, 0) / ms.length }))
    .sort((a, b) => b.avgMs - a.avgMs)
    .slice(0, 10);

  return { stats, errorsByTenant, slowestTenants };
}
```

For heavier analytics (billing-grade counts over a month, percentiles across millions of rows), query the storage engine directly — LogTide's data lives in TimescaleDB or ClickHouse you own, so plain SQL with `GROUP BY` on the tenant metadata is always available without API pagination.

### Per-Tenant Dashboards

```typescript
// Dashboard for tenant admins
async function getTenantDashboard(tenantId: string) {
  const last24h = new Date(Date.now() - 24 * 60 * 60 * 1000);

  // Pull the tenant's recent logs once, then derive the dashboard numbers
  const { logs } = await logtide.query({ q: tenantId, from: last24h, limit: 1000 });
  const tenantLogs = logs.filter(log => log.metadata?.tenant_id === tenantId);

  const errorCount = tenantLogs.filter(log => log.level === 'error').length;
  const requestLogs = tenantLogs.filter(log => log.metadata?.duration_ms != null);

  const byEndpoint = new Map<string, number[]>();
  for (const log of requestLogs) {
    const path = String(log.metadata?.path ?? 'unknown');
    byEndpoint.set(path, [...(byEndpoint.get(path) ?? []), Number(log.metadata?.duration_ms)]);
  }
  const slowestEndpoints = [...byEndpoint.entries()]
    .map(([path, ms]) => ({ path, avgMs: ms.reduce((a, b) => a + b, 0) / ms.length }))
    .sort((a, b) => b.avgMs - a.avgMs)
    .slice(0, 5);

  return {
    errorRate: requestLogs.length ? errorCount / requestLogs.length : 0,
    requestCount: requestLogs.length,
    slowestEndpoints,
  };
}
```

## Cost Allocation

### Track Log Volume Per Tenant

A month of per-tenant volume is an aggregation job, not an API loop — run it as SQL directly on the storage engine you already own (TimescaleDB here; the ClickHouse equivalent is nearly identical):

```sql
-- Monthly log volume per tenant, straight from TimescaleDB
SELECT
  metadata->>'tenant_id'   AS tenant_id,
  metadata->>'tenant_plan' AS plan,
  count(*)                 AS log_count
FROM logs
WHERE time >= date_trunc('month', now())
GROUP BY 1, 2
ORDER BY log_count DESC;
```

Then price it in application code:

```typescript
// Monthly cost allocation report
async function generateCostReport() {
  const logCounts = await db.query(MONTHLY_VOLUME_SQL); // the SQL above

  return logCounts.rows.map(tenant => ({
    tenantId: tenant.tenant_id,
    plan: tenant.plan,
    logCount: Number(tenant.log_count),
    cost: calculateCost(tenant.plan, Number(tenant.log_count)),
  }));
}

function calculateCost(plan: string, logCount: number): number {
  const rates = {
    free: { includedLogs: 10000, perExtraLog: 0.001 },
    pro: { includedLogs: 100000, perExtraLog: 0.0005 },
    enterprise: { includedLogs: 1000000, perExtraLog: 0.0001 },
  };

  const rate = rates[plan] || rates.free;
  const extraLogs = Math.max(0, logCount - rate.includedLogs);
  return extraLogs * rate.perExtraLog;
}
```

## Real-World Example: Project Management SaaS

Let's say you're building a project management tool like Linear or Asana.

### Tenant Structure

```typescript
interface Tenant {
  id: string;           // 'acme-corp'
  name: string;         // 'Acme Corporation'
  plan: 'free' | 'pro' | 'enterprise';
  features: string[];
  maxUsers: number;
}
```

### Key Events to Log

```typescript
// User actions
logger.info('User logged in', { method: 'sso' });
logger.info('User invited teammate', { inviteeEmail: maskEmail(email) });

// Project operations
logger.info('Project created', { projectId, projectName });
logger.info('Task created', { taskId, projectId, assigneeId });
logger.info('Task status changed', { taskId, from: 'todo', to: 'in_progress' });
logger.info('Comment added', { taskId, commentLength: comment.length });

// Integrations
logger.info('GitHub integration connected', { repos: repos.length });
logger.info('Slack notification sent', { channel, messageType });

// Billing events
logger.info('Subscription upgraded', { from: 'pro', to: 'enterprise' });
logger.info('Payment processed', { amount, currency });

// Feature usage (for analytics)
logger.info('Feature used', { feature: 'gantt_chart', duration_ms: 5000 });
```

### Queries for Common Scenarios

**"Acme Corp is reporting slow performance"**
```
tenant_id:acme-corp AND duration_ms:>1000
```

**"How many projects did each enterprise tenant create this month?"**
```
tenant_plan:enterprise AND message:"Project created"
| group by tenant_id
| count
```

**"Find all errors for a specific user's session"**
```
tenant_id:acme-corp AND user_id:user_456 AND level:error
```

## Multi-Tenant Checklist

- [ ] **Tenant Identification**
    - [ ] Extract tenant from JWT/session consistently
    - [ ] Handle missing tenant gracefully (reject or log as 'system')
    - [ ] Support service-to-service calls with tenant header

- [ ] **Logging Standards**
    - [ ] Tenant ID in every log entry
    - [ ] Consistent field naming (`tenant_id`, not `org`/`customer`)
    - [ ] User ID alongside tenant ID
    - [ ] Request ID for tracing

- [ ] **Data Isolation**
    - [ ] Tenants can only query their own logs
    - [ ] Internal fields stripped from tenant-facing APIs
    - [ ] Audit logging for log access

- [ ] **Analytics**
    - [ ] Platform-wide metrics for operators
    - [ ] Per-tenant dashboards for admins
    - [ ] Cost allocation tracking

- [ ] **Compliance**
    - [ ] Retention policies per tenant plan
    - [ ] Data deletion for churned tenants
    - [ ] Export capability for tenant data

## Common Pitfalls

### 1. "We'll add tenant context later"

By the time you need it, you have months of useless logs.

**Solution:** Add tenant context from day one. It's a one-time setup cost.

### 2. "Background jobs don't need tenant context"

Then you can't debug why a specific tenant's invoice failed.

**Solution:** Always propagate tenant context to background jobs.

### 3. "All our tenants are small, we don't need isolation"

Until one enterprise customer asks for their log export.

**Solution:** Design for isolation from the start.

## Performance Considerations

| Aspect | Impact | Mitigation |
|--------|--------|------------|
| Context propagation | <1ms per request | AsyncLocalStorage is fast |
| Extra fields in logs | ~50 bytes per log | Negligible |
| Tenant lookup | 1-5ms | Cache tenant details |
| Cross-tenant queries | Depends on volume | Add tenant_id to indexes |

## Next Steps

- [Node.js SDK](/integrations/nodejs/) - Full logging setup
- [PostgreSQL Integration](/integrations/postgresql/) - Database logging
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy requirements for EU tenants


---

# Real-Time Log Alerting

> Configure intelligent log-based alerts with LogTide. Threshold alerts, anomaly detection, and notification routing to Slack and email.

*Source: https://logtide.dev/use-cases/real-time-alerting/*

---

Good alerting tells you about problems before your users do. Bad alerting wakes you up at 3 AM for nothing. This guide shows how to build effective log-based alerting with LogTide that catches real issues without alert fatigue.

## The Alerting Problem

### Too Many Alerts

```
🔔 3:01 AM - Error count > 5 in service: worker (5 errors)
🔔 3:01 AM - Error count > 5 in service: worker (6 errors)
🔔 3:02 AM - Error count > 5 in service: api (7 errors)
🔔 3:02 AM - Error count > 5 in service: worker (8 errors)
🔔 3:03 AM - Error count > 5 in service: api (12 errors)
... 47 more alerts ...
```

**Result:** Engineer mutes notifications, misses the real incident next week.

### Too Few Alerts

```
(silence)
```

**User report at 9 AM:** "We haven't been able to log in since midnight."

**Result:** 9-hour outage, SLA violation, angry customers.

### The Right Balance

Effective alerting has three properties:
1. **Actionable** - every alert requires human action
2. **Timely** - fires within minutes of the issue
3. **Contextual** - tells you what to investigate

## LogTide Alerting Features

LogTide provides two alerting mechanisms:

1. **Alert Rules** - Threshold-based alerts on log volume, error rates, or patterns
2. **Sigma Detection Rules** - Pattern-based security detection (brute force, anomalies)

Both support:
- Email notifications
- Webhook notifications (Slack, PagerDuty, Teams, etc.)
- Configurable time windows and thresholds

## Implementation

### 1. Alert Rules via API

Create alert rules programmatically:

```bash
# Create an alert rule
curl -X POST "http://logtide.internal:8080/api/v1/alerts" \
  -H "Authorization: Bearer YOUR_SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "organizationId": "your-org-id",
    "projectId": "your-project-id",
    "name": "High Error Rate - API",
    "enabled": true,
    "service": "api",
    "level": ["error", "critical"],
    "threshold": 50,
    "timeWindow": 5,
    "emailRecipients": ["oncall@company.com"],
    "webhookUrl": "https://hooks.slack.com/services/xxx/yyy/zzz"
  }'
```

### 2. Essential Alert Rules

Here are the alert rules every production system should have:

**Error Rate Alert (per service)**
```json
{
  "name": "High Error Rate - API",
  "enabled": true,
  "service": "api",
  "level": ["error", "critical"],
  "threshold": 50,
  "timeWindow": 5,
  "emailRecipients": ["oncall@company.com"],
  "webhookUrl": "https://hooks.slack.com/services/xxx"
}
```

**Critical Error Alert (any service)**
```json
{
  "name": "Critical Errors",
  "enabled": true,
  "level": ["critical"],
  "threshold": 1,
  "timeWindow": 1,
  "emailRecipients": ["oncall@company.com", "eng-lead@company.com"],
  "webhookUrl": "https://hooks.slack.com/services/xxx"
}
```

**Service Down Alert**
```json
{
  "name": "Health Check Failures",
  "enabled": true,
  "service": "health",
  "searchQuery": "critical health check failure",
  "threshold": 3,
  "timeWindow": 5,
  "emailRecipients": ["oncall@company.com"],
  "webhookUrl": "https://hooks.slack.com/services/xxx"
}
```

### 3. Slack Webhook Integration

LogTide sends webhook payloads you can route to Slack:

```json
{
  "alert": {
    "name": "High Error Rate - API",
    "threshold": 50,
    "timeWindow": 5,
    "currentCount": 73
  },
  "triggeredAt": "2025-02-01T03:15:00Z",
  "service": "api",
  "level": ["error", "critical"],
  "sampleLogs": [
    {
      "timestamp": "2025-02-01T03:14:58Z",
      "level": "error",
      "message": "Database connection timeout",
      "metadata": { "host": "db-primary", "timeout_ms": 5000 }
    }
  ]
}
```

To integrate with Slack, use an incoming webhook URL from your Slack workspace settings.

### 4. PagerDuty Integration via Webhook

Route critical alerts to PagerDuty by using a webhook middleware:

```typescript
// webhook-router/index.ts
import express from 'express';

const app = express();
app.use(express.json());

app.post('/logtide-alert', async (req, res) => {
  const alert = req.body;

  // Route based on severity
  if (alert.alert.name.includes('Critical')) {
    // PagerDuty for critical alerts
    await fetch('https://events.pagerduty.com/v2/enqueue', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        routing_key: process.env.PAGERDUTY_ROUTING_KEY,
        event_action: 'trigger',
        dedup_key: `logtide-${alert.alert.name}-${alert.service}`,
        payload: {
          summary: `${alert.alert.name}: ${alert.alert.currentCount} events in ${alert.alert.timeWindow}m`,
          severity: 'critical',
          source: 'LogTide',
          custom_details: alert,
        },
      }),
    });
  }

  // Slack for all alerts
  await fetch(process.env.SLACK_WEBHOOK_URL!, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      text: `🚨 *${alert.alert.name}*\nService: ${alert.service}\nCount: ${alert.alert.currentCount} (threshold: ${alert.alert.threshold})\nTime: ${alert.triggeredAt}`,
    }),
  });

  res.json({ ok: true });
});

app.listen(3001);
```

### 5. Sigma Rules for Security Alerts

LogTide's built-in Sigma support detects security patterns:

```yaml
# Brute force detection
title: Brute Force Login Attempt
status: stable
level: high
logsource:
  category: authentication
detection:
  selection:
    message|contains: "login failed"
  timeframe: 5m
  condition: selection | count() > 10
tags:
  - attack.credential_access
  - attack.t1110
```

```yaml
# Privilege escalation
title: Suspicious Admin Role Assignment
status: stable
level: critical
logsource:
  category: audit
detection:
  selection:
    action: "role.update"
    new_state|contains: "admin"
  condition: selection
tags:
  - attack.privilege_escalation
  - attack.t1078
```

Import Sigma rules via the LogTide UI at `/dashboard/security/sigma`.

## Alert Design Patterns

### Pattern 1: Tiered Severity

Don't treat all errors the same:

| Tier | Condition | Notification | Response |
|------|-----------|-------------|----------|
| P1 (Critical) | Service down, data loss, security breach | PagerDuty + Slack + Email | Wake up on-call immediately |
| P2 (High) | Error rate > 5x normal, degraded performance | Slack + Email | Investigate within 30 minutes |
| P3 (Medium) | Elevated errors, non-critical failures | Slack channel | Investigate during business hours |
| P4 (Low) | Warnings, unusual patterns | Weekly digest | Review in next sprint |

### Pattern 2: Alert on Symptoms, Not Causes

```json
// ❌ BAD: Alert on cause (too specific, many false positives)
{
  "name": "Database Error",
  "searchQuery": "ECONNREFUSED",
  "threshold": 1,
  "timeWindow": 1
}

// ✅ GOOD: Alert on symptom (catches the real impact)
{
  "name": "API Error Rate High",
  "service": "api",
  "level": ["error"],
  "threshold": 50,
  "timeWindow": 5
}
```

### Pattern 3: Percentage-Based Thresholds

Absolute thresholds break when traffic changes. Use percentages when possible:

```typescript
// Alert setup: Monitor error percentage
// Check every 5 minutes
async function checkErrorRate(service: string) {
  const total = await logtide.count({
    service,
    from: '-5m',
  });

  const errors = await logtide.count({
    service,
    level: 'error',
    from: '-5m',
  });

  const errorRate = total > 0 ? (errors / total) * 100 : 0;

  if (errorRate > 5) { // > 5% error rate
    logger.critical('Error rate threshold exceeded', {
      service,
      errorRate: errorRate.toFixed(2),
      totalRequests: total,
      errorCount: errors,
    });
  }
}
```

### Pattern 4: Alert Deduplication

Avoid alert storms by deduplicating:

```typescript
// Use consistent dedup keys
// LogTide will only fire once per unique combination
// within the time window
{
  "name": "Service Error Rate",
  "service": "api",        // Scoped to service
  "timeWindow": 5,         // 5-minute window
  // Only one alert per service per 5 minutes
}
```

## Reducing Alert Fatigue

### Step 1: Audit Current Alerts

List all your alerts and classify:

| Alert | Last Triggered | Actionable? | Keep/Modify/Delete |
|-------|---------------|-------------|-------------------|
| Error > 5 | Daily | No (too sensitive) | Modify: threshold to 50 |
| CPU > 90% | Never | N/A | Delete |
| 5xx rate > 1% | Weekly | Yes | Keep |
| Disk > 80% | Monthly | Yes | Keep |

### Step 2: Apply the SRE Alert Framework

For each alert, ask:
1. Does this alert require human action? If no, delete it.
2. Can this wait until business hours? If yes, make it P3/P4.
3. Is the threshold set correctly? Tune based on historical data.
4. Is the alert well-documented? Add a runbook link.

### Step 3: Implement Alert Routing

Not every alert needs to page someone:

```d2
direction: right
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

p1: "P1 · Critical" { class: hub }
p2: "P2 · High" { class: src }
p3: "P3 · Medium" { class: src }
p4: "P4 · Low" { class: src }

pagerduty: PagerDuty { class: hub }
slack_inc: "Slack #incidents" { class: hub }
slack_alerts: "Slack #alerts" { class: hub }
email: Email digest { class: hub }

phone: Phone call { class: src }
within30: Investigate within 30m { class: src }
business: Business hours { class: src }
weekly: Weekly review { class: src }

p1 -> pagerduty -> phone { class: flow }
p2 -> slack_inc -> within30 { class: flow }
p3 -> slack_alerts -> business { class: flow }
p4 -> email -> weekly { class: flow }
```

## Alert Configuration Checklist

- [ ] **Essential Alerts**
    - [ ] Error rate per critical service
    - [ ] Critical/fatal errors (any service)
    - [ ] Health check failures
    - [ ] Authentication failures (brute force)
    - [ ] Deployment events (verify after deploy)

- [ ] **Notification Routing**
    - [ ] P1 alerts → PagerDuty (phone)
    - [ ] P2 alerts → Slack #incidents
    - [ ] P3 alerts → Slack #alerts
    - [ ] P4 alerts → Email digest

- [ ] **Alert Quality**
    - [ ] Every alert has a runbook
    - [ ] Thresholds tuned to avoid false positives
    - [ ] Alert fatigue audit quarterly
    - [ ] On-call handoff process documented

- [ ] **Security Alerts (Sigma)**
    - [ ] Brute force detection enabled
    - [ ] Privilege escalation monitoring
    - [ ] Suspicious IP detection
    - [ ] After-hours admin activity

## Common Pitfalls

### 1. "Alert on every error"

You'll get 500 alerts per day and ignore them all.

**Solution:** Alert on error *rates*, not individual errors.

### 2. "Set thresholds once and forget"

As traffic grows, static thresholds become useless.

**Solution:** Review thresholds quarterly. Consider percentage-based thresholds.

### 3. "No runbook for alerts"

An alert without a runbook just says "something is wrong, figure it out."

**Solution:** Every alert should link to a runbook with investigation steps.

### 4. "Same notification for everything"

If everything pages, nothing is important.

**Solution:** Tier your alerts. Only P1 should wake someone up.

## Next Steps

- [Incident Response](/use-cases/incident-response/) - What to do when alerts fire
- [Security Monitoring](/use-cases/security-monitoring/) - Sigma rules for threat detection
- [Node.js SDK](/integrations/nodejs/) - Application logging setup
- [Docker Integration](/integrations/docker/) - Container log collection

---

**Ready to set up intelligent alerting?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Share your alerting setup


---

# Security Monitoring and Threat Detection

> Build SIEM-like security monitoring with LogTide. Detect suspicious logins, brute force attacks, and security anomalies.

*Source: https://logtide.dev/use-cases/security-monitoring/*

---

Traditional SIEM solutions are expensive and complex. LogTide provides the foundation for security monitoring at a fraction of the cost. This guide shows you how to detect threats, monitor for anomalies, and maintain audit trails.

## Why Security Monitoring with LogTide?

- **Cost-effective**: No per-GB pricing that punishes security logging
- **Self-hosted option**: Keep security data in your infrastructure
- **Real-time alerting**: Immediate notification of security events
- **Correlation**: Link security events across your entire stack
- **Compliance**: Audit trails for SOC2, ISO27001, HIPAA

## Security Logging Architecture

**Data Sources:**
- **nginx/WAF**: Access logs, blocked requests, rate limiting
- **API Servers**: Authentication events, authorization failures
- **Database**: Query audit logs, connection events
- **Auth Service**: Login attempts, password resets, MFA events

**Processing:**
- All security events flow into LogTide with severity levels
- Real-time detection rules analyze incoming logs
- Alerts trigger via Slack, PagerDuty, or webhooks
- Long-term archive to S3 for compliance

## Implementation

### 1. Security Event Logging

Create a dedicated security logger:

```typescript
// lib/security-logger.ts
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  globalMetadata: {
    service: 'security',
    environment: process.env.NODE_ENV,
  },
});

interface SecurityEvent {
  event: string;
  severity: 'low' | 'medium' | 'high' | 'critical';
  userId?: string;
  ip?: string;
  userAgent?: string;
  metadata?: Record<string, unknown>;
}

export const securityLogger = {
  log(event: SecurityEvent) {
    const level = event.severity === 'critical' || event.severity === 'high'
      ? 'error'
      : event.severity === 'medium'
      ? 'warn'
      : 'info';

    client[level]('security', `SECURITY: ${event.event}`, {
      security_event: event.event,
      severity: event.severity,
      user_id: event.userId,
      source_ip: event.ip,
      user_agent: event.userAgent,
      ...event.metadata,
    });
  },

  // Convenience methods
  loginSuccess(userId: string, ip: string, method: string) {
    this.log({
      event: 'login_success',
      severity: 'low',
      userId,
      ip,
      metadata: { method },
    });
  },

  loginFailed(email: string, ip: string, reason: string) {
    this.log({
      event: 'login_failed',
      severity: 'medium',
      ip,
      metadata: { email_hash: hashEmail(email), reason },
    });
  },

  suspiciousActivity(event: string, details: Record<string, unknown>) {
    this.log({
      event,
      severity: 'high',
      ...details,
    });
  },

  criticalAlert(event: string, details: Record<string, unknown>) {
    this.log({
      event,
      severity: 'critical',
      ...details,
    });
  },
};
```

### 2. Authentication Monitoring

```typescript
// auth/login.ts
import { securityLogger } from '@/lib/security-logger';
import { RateLimiter } from '@/lib/rate-limiter';

const loginLimiter = new RateLimiter({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 5, // 5 attempts per window
});

app.post('/api/auth/login', async (req, res) => {
  const { email, password } = req.body;
  const ip = req.ip;
  const userAgent = req.headers['user-agent'];

  // Check rate limit
  const rateLimitResult = await loginLimiter.check(ip);
  if (!rateLimitResult.allowed) {
    securityLogger.suspiciousActivity('brute_force_detected', {
      ip,
      userAgent,
      attempts: rateLimitResult.attempts,
      metadata: { email_hash: hashEmail(email) },
    });

    return res.status(429).json({
      error: 'Too many login attempts. Please try again later.',
    });
  }

  try {
    const user = await authenticate(email, password);

    if (!user) {
      securityLogger.loginFailed(email, ip, 'invalid_credentials');
      return res.status(401).json({ error: 'Invalid credentials' });
    }

    // Check for suspicious patterns
    await checkLoginAnomalies(user, ip, userAgent);

    securityLogger.loginSuccess(user.id, ip, 'password');

    const token = generateToken(user);
    res.json({ token });
  } catch (error) {
    securityLogger.log({
      event: 'login_error',
      severity: 'high',
      ip,
      metadata: {
        email_hash: hashEmail(email),
        error: error.message,
      },
    });

    res.status(500).json({ error: 'Login failed' });
  }
});

async function checkLoginAnomalies(user: User, ip: string, userAgent: string) {
  // Check for new device
  const knownDevices = await getKnownDevices(user.id);
  const deviceFingerprint = generateFingerprint(userAgent);

  if (!knownDevices.includes(deviceFingerprint)) {
    securityLogger.log({
      event: 'new_device_login',
      severity: 'medium',
      userId: user.id,
      ip,
      userAgent,
      metadata: { fingerprint: deviceFingerprint },
    });

    // Optionally send notification to user
    await notifyNewDeviceLogin(user, ip, userAgent);
  }

  // Check for impossible travel
  const lastLogin = await getLastLogin(user.id);
  if (lastLogin && isImpossibleTravel(lastLogin.ip, ip, lastLogin.time)) {
    securityLogger.suspiciousActivity('impossible_travel', {
      userId: user.id,
      ip,
      metadata: {
        previous_ip: lastLogin.ip,
        previous_time: lastLogin.time,
        time_diff_minutes: (Date.now() - lastLogin.time) / 60000,
      },
    });
  }

  // Check for unusual time
  const userTimezone = user.timezone || 'UTC';
  const localHour = getLocalHour(userTimezone);
  if (localHour >= 1 && localHour <= 5) {
    securityLogger.log({
      event: 'unusual_hour_login',
      severity: 'low',
      userId: user.id,
      ip,
      metadata: { local_hour: localHour, timezone: userTimezone },
    });
  }
}
```

### 3. API Rate Limiting and Abuse Detection

```typescript
// middleware/rate-limit.ts
import { securityLogger } from '@/lib/security-logger';

interface RateLimitConfig {
  windowMs: number;
  max: number;
  keyGenerator?: (req: Request) => string;
}

export function createRateLimiter(config: RateLimitConfig) {
  const hits = new Map<string, { count: number; resetAt: number }>();

  return async (req: Request, res: Response, next: NextFunction) => {
    const key = config.keyGenerator?.(req) || req.ip;
    const now = Date.now();

    let record = hits.get(key);
    if (!record || record.resetAt < now) {
      record = { count: 0, resetAt: now + config.windowMs };
      hits.set(key, record);
    }

    record.count++;

    if (record.count > config.max) {
      // Log rate limit violation
      securityLogger.log({
        event: 'rate_limit_exceeded',
        severity: record.count > config.max * 2 ? 'high' : 'medium',
        ip: req.ip,
        metadata: {
          path: req.path,
          method: req.method,
          attempts: record.count,
          limit: config.max,
          user_id: req.user?.id,
        },
      });

      res.set('Retry-After', Math.ceil((record.resetAt - now) / 1000).toString());
      return res.status(429).json({
        error: 'Too many requests',
        retryAfter: record.resetAt,
      });
    }

    next();
  };
}

// Apply different limits to different endpoints
app.use('/api/auth/*', createRateLimiter({
  windowMs: 15 * 60 * 1000,
  max: 10,
}));

app.use('/api/password-reset', createRateLimiter({
  windowMs: 60 * 60 * 1000,
  max: 3,
}));

app.use('/api/*', createRateLimiter({
  windowMs: 60 * 1000,
  max: 100,
  keyGenerator: (req) => req.user?.id || req.ip,
}));
```

### 4. Privilege Escalation Monitoring

```typescript
// middleware/audit-sensitive.ts
import { securityLogger } from '@/lib/security-logger';

const SENSITIVE_OPERATIONS = [
  { method: 'POST', path: '/api/users', event: 'user_created' },
  { method: 'DELETE', path: '/api/users/:id', event: 'user_deleted' },
  { method: 'PUT', path: '/api/users/:id/role', event: 'role_changed' },
  { method: 'POST', path: '/api/api-keys', event: 'api_key_created' },
  { method: 'DELETE', path: '/api/api-keys/:id', event: 'api_key_revoked' },
  { method: 'PUT', path: '/api/settings/security', event: 'security_settings_changed' },
];

export function auditSensitiveOperations(req: Request, res: Response, next: NextFunction) {
  const operation = SENSITIVE_OPERATIONS.find(op =>
    op.method === req.method && matchPath(op.path, req.path)
  );

  if (operation) {
    // Log before operation
    securityLogger.log({
      event: `${operation.event}_attempted`,
      severity: 'medium',
      userId: req.user?.id,
      ip: req.ip,
      metadata: {
        path: req.path,
        method: req.method,
        target_id: req.params.id,
      },
    });

    // Capture response
    const originalJson = res.json.bind(res);
    res.json = function(body) {
      // Log after operation
      securityLogger.log({
        event: res.statusCode < 400 ? operation.event : `${operation.event}_failed`,
        severity: res.statusCode < 400 ? 'medium' : 'high',
        userId: req.user?.id,
        ip: req.ip,
        metadata: {
          path: req.path,
          method: req.method,
          target_id: req.params.id,
          status_code: res.statusCode,
        },
      });

      return originalJson(body);
    };
  }

  next();
}
```

### 5. SQL Injection and XSS Detection

```typescript
// middleware/input-validation.ts
import { securityLogger } from '@/lib/security-logger';

const SQL_INJECTION_PATTERNS = [
  /(\b(SELECT|INSERT|UPDATE|DELETE|DROP|UNION|ALTER)\b)/i,
  /(--|#|\/\*)/,
  /(\bOR\b\s+\d+\s*=\s*\d+)/i,
  /('\s*OR\s*')/i,
];

const XSS_PATTERNS = [
  /<script\b[^>]*>/i,
  /javascript:/i,
  /on\w+\s*=/i,
  /<iframe/i,
  /<object/i,
];

export function detectMaliciousInput(req: Request, res: Response, next: NextFunction) {
  const inputs = [
    ...Object.values(req.query),
    ...Object.values(req.params),
    ...(req.body ? Object.values(flattenObject(req.body)) : []),
  ].filter(v => typeof v === 'string');

  for (const input of inputs) {
    // Check for SQL injection
    for (const pattern of SQL_INJECTION_PATTERNS) {
      if (pattern.test(input)) {
        securityLogger.criticalAlert('sql_injection_attempt', {
          ip: req.ip,
          userId: req.user?.id,
          userAgent: req.headers['user-agent'],
          metadata: {
            path: req.path,
            method: req.method,
            pattern: pattern.source,
            // Don't log the actual input to avoid log injection
            input_length: input.length,
          },
        });

        return res.status(400).json({ error: 'Invalid input' });
      }
    }

    // Check for XSS
    for (const pattern of XSS_PATTERNS) {
      if (pattern.test(input)) {
        securityLogger.suspiciousActivity('xss_attempt', {
          ip: req.ip,
          userId: req.user?.id,
          userAgent: req.headers['user-agent'],
          metadata: {
            path: req.path,
            method: req.method,
            pattern: pattern.source,
          },
        });

        return res.status(400).json({ error: 'Invalid input' });
      }
    }
  }

  next();
}
```

## Detection Rules

### Brute Force Attack

```
service:security AND security_event:login_failed
| window 5m
| group by source_ip
| count > 10
```

Alert: "Brute force attack detected from {source_ip}"

### Account Takeover Attempt

```
service:security AND security_event:login_success AND new_device_login
| join (security_event:password_reset_requested) on user_id
| within 24h
```

Alert: "Possible account takeover for user {user_id}"

### Privilege Escalation

```
service:security AND security_event:role_changed
| where metadata.new_role = "admin"
```

Alert: "User {target_id} promoted to admin by {user_id}"

### API Abuse

```
service:security AND security_event:rate_limit_exceeded
| window 1h
| group by source_ip
| count > 50
```

Alert: "Sustained API abuse from {source_ip}"

### Impossible Travel

```
service:security AND security_event:impossible_travel
```

Alert: "Impossible travel detected for user {user_id}"

### Off-Hours Admin Activity

```
service:security AND severity:high
| where hour >= 22 OR hour <= 6
| where user_role = "admin"
```

Alert: "Admin activity outside business hours"

## Real-Time Alerting

### Slack Integration

```typescript
// lib/security-alerts.ts
import { IncomingWebhook } from '@slack/webhook';

const webhook = new IncomingWebhook(process.env.SLACK_SECURITY_WEBHOOK!);

export async function sendSecurityAlert(event: SecurityEvent) {
  if (event.severity !== 'high' && event.severity !== 'critical') {
    return;
  }

  const color = event.severity === 'critical' ? '#FF0000' : '#FFA500';

  await webhook.send({
    attachments: [{
      color,
      title: `🚨 Security Alert: ${event.event}`,
      fields: [
        { title: 'Severity', value: event.severity.toUpperCase(), short: true },
        { title: 'IP Address', value: event.ip || 'Unknown', short: true },
        { title: 'User ID', value: event.userId || 'N/A', short: true },
        { title: 'Time', value: new Date().toISOString(), short: true },
      ],
      footer: 'LogTide Security Monitoring',
    }],
  });
}
```

### PagerDuty Integration

```typescript
// lib/pagerduty.ts
import { event } from '@pagerduty/pdjs';

export async function triggerIncident(securityEvent: SecurityEvent) {
  if (securityEvent.severity !== 'critical') {
    return;
  }

  await event({
    data: {
      routing_key: process.env.PAGERDUTY_ROUTING_KEY!,
      event_action: 'trigger',
      dedup_key: `security-${securityEvent.event}-${securityEvent.ip}`,
      payload: {
        summary: `Critical Security Event: ${securityEvent.event}`,
        severity: 'critical',
        source: 'LogTide Security Monitoring',
        custom_details: {
          event: securityEvent.event,
          ip: securityEvent.ip,
          user_id: securityEvent.userId,
          ...securityEvent.metadata,
        },
      },
    },
  });
}
```

## Audit Trail for Compliance

### SOC2 / ISO27001 Requirements

```typescript
// All security events should include:
interface AuditEvent {
  // Who
  user_id: string;
  user_email: string;
  user_role: string;

  // What
  action: string;
  resource_type: string;
  resource_id: string;

  // When
  timestamp: string;

  // Where
  source_ip: string;
  user_agent: string;
  session_id: string;

  // Result
  success: boolean;
  error_message?: string;
}

export function logAuditEvent(event: AuditEvent) {
  client.info('audit', 'AUDIT', {
    ...event,
    audit: true, // Flag for compliance queries
  });
}
```

### Retention Policy

For compliance, configure retention:
- Security events: 1 year minimum
- Audit logs: 7 years (financial services)
- Access logs: 90 days

## Security Monitoring Checklist

- [ ] **Authentication Events**
    - [ ] Login success/failure with IP and device
    - [ ] Password reset requests
    - [ ] MFA enablement/disablement
    - [ ] Session creation/termination

- [ ] **Authorization Events**
    - [ ] Role changes
    - [ ] Permission grants/revokes
    - [ ] API key creation/revocation
    - [ ] Resource access attempts

- [ ] **Data Access**
    - [ ] Sensitive data queries
    - [ ] Bulk exports
    - [ ] PII access

- [ ] **Infrastructure**
    - [ ] Failed SSH attempts
    - [ ] Firewall blocks
    - [ ] Certificate errors

- [ ] **Anomaly Detection**
    - [ ] Impossible travel
    - [ ] Unusual access patterns
    - [ ] Off-hours activity

## Common Pitfalls

### 1. "We log everything, we're secure"

Logging without monitoring is useless.

**Solution:** Set up active alerts for critical events.

### 2. "Our logs contain too much noise"

Then you'll miss the real threats.

**Solution:** Severity levels and filtered dashboards.

### 3. "We'll investigate breaches after they happen"

By then it's too late.

**Solution:** Real-time alerting for suspicious patterns.

## Next Steps

- [nginx Integration](/integrations/nginx/) - Web server security logs
- [Node.js SDK](/integrations/nodejs/) - Application security logging
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Privacy requirements


---

# SOC2 & ISO27001 Audit Trail Logging

> Build compliant audit trails for SOC2 and ISO27001 certifications with LogTide. Evidence collection, retention, and access controls.

*Source: https://logtide.dev/use-cases/compliance-audit-trail/*

---

SOC2 and ISO27001 certifications require comprehensive audit trails. This guide shows how to implement compliant logging with LogTide that satisfies auditor requirements and makes evidence collection straightforward.

## Why Audit Trail Logging Matters

### SOC2 Trust Service Criteria

SOC2 Type II requires demonstrable controls across five categories. Logging is central to three of them:

| Trust Criteria | Logging Requirement | Evidence Needed |
|---------------|--------------------|-----------------|
| Security (CC6) | Access controls, authentication events | Login/logout, permission changes, MFA events |
| Availability (A1) | System monitoring, incident detection | Uptime logs, error rates, incident timelines |
| Processing Integrity (PI1) | Data processing accuracy | Transaction logs, data validation events |

### ISO27001 Annex A Controls

ISO27001 Annex A specifies controls that require audit logging:

- **A.8.15** - Logging: Event logs recording user activities, exceptions, faults
- **A.8.16** - Monitoring activities: Monitoring for anomalous behavior
- **A.5.33** - Protection of records: Ensuring log integrity and retention
- **A.5.23** - Information security for cloud services: Cloud access logging

### What Auditors Actually Ask

From real SOC2 and ISO27001 audits:

1. *"Show me who accessed the production database in the last 90 days"*
2. *"How do you detect unauthorized access attempts?"*
3. *"What's your log retention policy and how is it enforced?"*
4. *"Can logs be tampered with? How do you ensure integrity?"*
5. *"Show me the incident response timeline for your last security event"*

If you can't answer these with data, you fail the audit.

## The LogTide Approach

LogTide provides the infrastructure for compliant audit trails:

1. **Structured audit events** with consistent schema
2. **Immutable storage** with TimescaleDB
3. **Role-based access** to log data
4. **Configurable retention** with automatic enforcement
5. **Built-in SIEM** for anomaly detection (Sigma rules)
6. **Self-hosted** for full data sovereignty

## Implementation

### 1. Audit Event Schema

Define a consistent schema for all audit events:

```typescript
// lib/audit-logger.ts
import { LogTideClient } from '@logtide/sdk-node';

interface AuditEvent {
  // WHO performed the action
  actorId: string;
  actorEmail: string;
  actorRole: string;
  actorIp: string;
  actorUserAgent: string;

  // WHAT happened
  action: string;
  resourceType: string;
  resourceId: string;
  outcome: 'success' | 'failure' | 'denied';

  // DETAILS
  details?: Record<string, unknown>;
  previousState?: Record<string, unknown>;
  newState?: Record<string, unknown>;

  // CONTEXT
  sessionId: string;
  requestId: string;
}

const client = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
  globalMetadata: {
    service: 'audit',
    environment: process.env.NODE_ENV,
    auditVersion: '1.0',
  },
});

export function logAudit(event: AuditEvent) {
  const level = event.outcome === 'denied' ? 'warn' :
                event.outcome === 'failure' ? 'error' : 'info';

  client[level]('audit', `AUDIT: ${event.action}`, {
    audit: true,
    actor_id: event.actorId,
    actor_email: event.actorEmail,
    actor_role: event.actorRole,
    actor_ip: event.actorIp,
    actor_user_agent: event.actorUserAgent,
    action: event.action,
    resource_type: event.resourceType,
    resource_id: event.resourceId,
    outcome: event.outcome,
    session_id: event.sessionId,
    request_id: event.requestId,
    details: event.details,
    previous_state: event.previousState,
    new_state: event.newState,
  });
}
```

### 2. Authentication Events (SOC2 CC6.1)

```typescript
// auth/login.ts
import { logAudit } from '@/lib/audit-logger';

app.post('/api/auth/login', async (req, res) => {
  const { email, password } = req.body;

  try {
    const user = await authenticate(email, password);

    if (!user) {
      logAudit({
        actorId: 'unknown',
        actorEmail: email,
        actorRole: 'unknown',
        actorIp: req.ip,
        actorUserAgent: req.headers['user-agent'] || '',
        action: 'auth.login',
        resourceType: 'session',
        resourceId: 'N/A',
        outcome: 'failure',
        sessionId: 'N/A',
        requestId: req.id,
        details: { reason: 'invalid_credentials' },
      });

      return res.status(401).json({ error: 'Invalid credentials' });
    }

    const session = await createSession(user);

    logAudit({
      actorId: user.id,
      actorEmail: user.email,
      actorRole: user.role,
      actorIp: req.ip,
      actorUserAgent: req.headers['user-agent'] || '',
      action: 'auth.login',
      resourceType: 'session',
      resourceId: session.id,
      outcome: 'success',
      sessionId: session.id,
      requestId: req.id,
      details: { method: 'password', mfaUsed: user.mfaEnabled },
    });

    res.json({ token: session.token });
  } catch (error) {
    logAudit({
      actorId: 'unknown',
      actorEmail: email,
      actorRole: 'unknown',
      actorIp: req.ip,
      actorUserAgent: req.headers['user-agent'] || '',
      action: 'auth.login',
      resourceType: 'session',
      resourceId: 'N/A',
      outcome: 'failure',
      sessionId: 'N/A',
      requestId: req.id,
      details: { reason: 'system_error', error: error.message },
    });

    res.status(500).json({ error: 'Login failed' });
  }
});
```

### 3. Authorization & Access Control (SOC2 CC6.3)

```typescript
// middleware/audit-access.ts
import { logAudit } from '@/lib/audit-logger';

// Audit all sensitive data access
export function auditDataAccess(resourceType: string) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const startTime = Date.now();

    // Capture original json to audit response
    const originalJson = res.json.bind(res);
    res.json = function (body) {
      const duration = Date.now() - startTime;
      const outcome = res.statusCode < 400 ? 'success' :
                      res.statusCode === 403 ? 'denied' : 'failure';

      logAudit({
        actorId: req.user.id,
        actorEmail: req.user.email,
        actorRole: req.user.role,
        actorIp: req.ip,
        actorUserAgent: req.headers['user-agent'] || '',
        action: `${resourceType}.${req.method.toLowerCase()}`,
        resourceType,
        resourceId: req.params.id || 'list',
        outcome,
        sessionId: req.sessionId,
        requestId: req.id,
        details: {
          method: req.method,
          path: req.path,
          statusCode: res.statusCode,
          durationMs: duration,
        },
      });

      return originalJson(body);
    };

    next();
  };
}

// Apply to sensitive endpoints
app.use('/api/users', auditDataAccess('user'));
app.use('/api/billing', auditDataAccess('billing'));
app.use('/api/settings', auditDataAccess('settings'));
app.use('/api/api-keys', auditDataAccess('api_key'));
```

### 4. Change Tracking (SOC2 CC8.1)

```typescript
// middleware/audit-changes.ts
import { logAudit } from '@/lib/audit-logger';

export function auditChanges(resourceType: string) {
  return async (req: Request, res: Response, next: NextFunction) => {
    // Only audit mutating operations
    if (!['POST', 'PUT', 'PATCH', 'DELETE'].includes(req.method)) {
      return next();
    }

    // Capture previous state for updates
    let previousState: Record<string, unknown> | undefined;
    if (req.params.id && ['PUT', 'PATCH', 'DELETE'].includes(req.method)) {
      previousState = await getResourceState(resourceType, req.params.id);
    }

    const originalJson = res.json.bind(res);
    res.json = function (body) {
      const action = req.method === 'POST' ? 'create' :
                     req.method === 'DELETE' ? 'delete' : 'update';

      logAudit({
        actorId: req.user.id,
        actorEmail: req.user.email,
        actorRole: req.user.role,
        actorIp: req.ip,
        actorUserAgent: req.headers['user-agent'] || '',
        action: `${resourceType}.${action}`,
        resourceType,
        resourceId: req.params.id || body?.id || 'unknown',
        outcome: res.statusCode < 400 ? 'success' : 'failure',
        sessionId: req.sessionId,
        requestId: req.id,
        previousState,
        newState: req.method !== 'DELETE' ? sanitizeForAudit(req.body) : undefined,
        details: {
          changedFields: previousState
            ? getChangedFields(previousState, req.body)
            : undefined,
        },
      });

      return originalJson(body);
    };

    next();
  };
}

function getChangedFields(
  previous: Record<string, unknown>,
  current: Record<string, unknown>
): string[] {
  return Object.keys(current).filter(
    key => JSON.stringify(previous[key]) !== JSON.stringify(current[key])
  );
}

function sanitizeForAudit(data: Record<string, unknown>): Record<string, unknown> {
  const { password, token, secret, ...safe } = data;
  return safe;
}
```

### 5. Retention Policy Configuration

Configure retention to meet compliance requirements:

| Log Type | SOC2 Requirement | ISO27001 Requirement | Recommended Retention |
|----------|-----------------|---------------------|----------------------|
| Authentication events | 1 year | Per risk assessment | 1 year |
| Access control changes | 1 year | Per risk assessment | 2 years |
| Data modifications | 1 year | Per risk assessment | 1 year |
| System errors | 90 days | Per risk assessment | 90 days |
| Security incidents | 3 years | Per risk assessment | 3 years |
| Financial transactions | 7 years | Per risk assessment | 7 years |

In LogTide, configure retention per-service or per-level through your organization settings.

## Evidence Collection for Auditors

### Common Auditor Queries

**"Show authentication events for the last 90 days"**
```
service:audit AND action:auth.* AND time:>90d
```

**"Show all permission changes"**
```
service:audit AND action:*.update AND resource_type:role
```

**"Show failed access attempts"**
```
service:audit AND outcome:denied
```

**"Show all admin actions"**
```
service:audit AND actor_role:admin
```

**"Show incident timeline for specific date"**
```
service:audit AND time:2025-01-15 AND (outcome:failure OR outcome:denied)
```

### Generating Audit Reports

```typescript
// scripts/generate-audit-report.ts
import { LogTideClient } from '@logtide/sdk-node';

async function generateAuditReport(startDate: string, endDate: string) {
  const client = new LogTideClient({
    apiUrl: process.env.LOGTIDE_API_URL!,
    apiKey: process.env.LOGTIDE_API_KEY!,
  });

  // Authentication summary
  const { logs: authEvents } = await client.query({
    service: 'audit',
    q: 'auth.',
    from: startDate,
    to: endDate,
    limit: 1000,
  });

  // Access denied events
  const { logs: deniedEvents } = await client.query({
    service: 'audit',
    q: 'denied',
    from: startDate,
    to: endDate,
    limit: 1000,
  });

  // Configuration changes
  const { logs: configChanges } = await client.query({
    service: 'audit',
    q: 'settings.',
    from: startDate,
    to: endDate,
    limit: 1000,
  });

  return {
    period: { start: startDate, end: endDate },
    summary: {
      totalAuthEvents: authEvents.length,
      failedLogins: authEvents.filter(e => e.metadata.outcome === 'failure').length,
      accessDenied: deniedEvents.length,
      configChanges: configChanges.length,
    },
    // Detailed data for auditor review
    authEvents,
    deniedEvents,
    configChanges,
  };
}
```

## SOC2 Controls Mapping

| SOC2 Control | What to Log | LogTide Implementation |
|-------------|------------|----------------------|
| CC6.1 (Access Control) | Login/logout, MFA events | `auth.login`, `auth.logout`, `auth.mfa` |
| CC6.2 (User Registration) | Account creation, deactivation | `user.create`, `user.deactivate` |
| CC6.3 (Role-Based Access) | Permission grants/revokes | `role.update`, `permission.grant` |
| CC6.6 (External Access) | API key usage, webhook calls | `api_key.create`, `api_key.use` |
| CC6.8 (Malicious Software) | Security alerts, threat detection | Sigma rules + alert triggers |
| CC7.2 (Monitoring) | System health, error rates | Alert rules on error thresholds |
| CC7.3 (Change Detection) | Config changes, deployments | `settings.update`, `deployment.create` |
| CC8.1 (Change Management) | Code/config change tracking | `change.create`, `change.approve` |

## ISO27001 Controls Mapping

| ISO27001 Control | LogTide Feature |
|-----------------|----------------|
| A.5.23 (Cloud Security) | Self-hosted, full data sovereignty |
| A.5.33 (Record Protection) | Immutable TimescaleDB storage |
| A.8.15 (Logging) | Structured audit events with consistent schema |
| A.8.16 (Monitoring) | Sigma rules, alert rules, real-time streaming |
| A.8.24 (Cryptography) | TLS for data in transit, encryption at rest via PostgreSQL |

## Compliance Audit Checklist

Use this before your audit:

- [ ] **Audit Logging**
    - [ ] All authentication events logged (login, logout, MFA)
    - [ ] All authorization changes logged (roles, permissions)
    - [ ] All data access to sensitive resources logged
    - [ ] All configuration changes logged with before/after state
    - [ ] Consistent audit event schema across all services

- [ ] **Log Integrity**
    - [ ] Logs stored in append-only storage (TimescaleDB)
    - [ ] Access to log storage restricted to authorized personnel
    - [ ] Log deletion requires explicit admin action

- [ ] **Retention**
    - [ ] Retention policies documented and configured
    - [ ] Automatic enforcement verified
    - [ ] Retention periods meet regulatory requirements
    - [ ] Long-term archive for financial/legal records

- [ ] **Access Control**
    - [ ] Role-based access to LogTide dashboard
    - [ ] API key rotation policy in place
    - [ ] Access to logs audited (meta-audit)

- [ ] **Incident Detection**
    - [ ] Sigma rules enabled for common threats
    - [ ] Alert rules configured for critical events
    - [ ] Incident response playbook documented
    - [ ] Escalation paths defined

- [ ] **Evidence Readiness**
    - [ ] Audit report generation tested
    - [ ] Common auditor queries documented
    - [ ] Export capability verified
    - [ ] Sample reports prepared

## Common Pitfalls

### 1. "We log everything, that's enough"

Auditors want structured, queryable audit trails, not raw application logs.

**Solution:** Implement dedicated audit events with consistent schema.

### 2. "Our cloud provider handles compliance"

Using a SOC2-certified cloud provider doesn't make your application SOC2 compliant.

**Solution:** Implement application-level audit logging regardless of infrastructure.

### 3. "We'll set up logging before the audit"

Auditors for SOC2 Type II need 6-12 months of historical evidence.

**Solution:** Start logging now. You need at least 6 months of data.

### 4. "Audit logs and application logs are the same"

Application logs are for debugging. Audit logs are for compliance. Different purposes, different retention, different access controls.

**Solution:** Use separate service names and retention policies for audit vs application logs.

## Legal Disclaimer

This guide provides technical implementation guidance for audit trail logging. It does not constitute legal, compliance, or audit advice. Work with qualified auditors and compliance professionals to ensure your specific implementation meets applicable requirements.

## Resources

- [SOC2 Trust Service Criteria (AICPA)](https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services)
- [ISO/IEC 27001:2022 Standard](https://www.iso.org/standard/27001)
- [NIST Cybersecurity Framework](https://www.nist.gov/cyberframework)

---

**Need help with compliance logging?**

- [Open a GitHub issue](https://github.com/logtide-dev/logtide/issues) for technical questions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) for architecture advice



# Comparisons

---

# LogTide vs AWS CloudWatch Logs

> Compare LogTide and AWS CloudWatch Logs for log management. Self-hosted vs cloud-native, pricing, and migration path.

*Source: https://logtide.dev/vs/aws-cloudwatch/*

---

AWS CloudWatch Logs is the default log management service for AWS workloads. LogTide is a self-hosted, open-source log management platform with built-in SIEM. Here's an honest comparison.

## Cost Comparison

AWS CloudWatch pricing scales linearly with volume — costs can grow faster than your traffic.

### CloudWatch Logs Pricing

| Component | Cost |
|-----------|------|
| Log ingestion | $0.50/GB |
| Log storage | $0.03/GB/month |
| Log Insights queries | $0.005/GB scanned |
| CloudWatch Logs subscription filter | $0.00 (but Lambda costs) |
| Cross-account log delivery | $0.10/GB |
| Live Tail | $0.01/minute |

**Real-world example:** A mid-size SaaS running on AWS with 50 GB/day log ingestion:
- Ingestion: 50 GB × $0.50 × 30 = $750/month
- Storage (30-day retention): 1,500 GB × $0.03 = $45/month
- Log Insights (10 queries/day): ~$75/month
- **Total: ~$870/month ($10,440/year)**

### LogTide Pricing

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| EC2 instance (c6i.2xlarge) | ~$250/month |
| EBS storage (1 TB gp3) | ~$80/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario on AWS infrastructure:**
- EC2 + storage: ~$330/month
- **Total: ~$330/month ($3,960/year)**

**Savings: ~$6,480/year (62%)**

At higher volumes (200+ GB/day), savings exceed 80%.

## Feature Comparison

| Feature | CloudWatch Logs | LogTide |
|---------|----------------|---------|
| Log ingestion API | Yes | Yes (HTTP + SDKs) |
| Full-text search | Log Insights (query-based) | Yes (real-time) |
| Real-time streaming | Live Tail ($0.01/min) | SSE (included) |
| Alert rules | CloudWatch Alarms | Built-in |
| Dashboards | CloudWatch Dashboards | SIEM dashboard |
| Structured queries | Log Insights (SQL-like) | SQL + Sigma rules |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | No | Included |
| Cross-account logs | Yes ($0.10/GB) | Multi-org (included) |
| Retention control | 1 day to indefinite | Configurable |
| S3 export | Yes (async) | Roadmap |
| Self-hosted | No (AWS only) | Yes |
| Open source | No | AGPLv3 |

## Where CloudWatch Wins

**Native AWS integration.** CloudWatch is deeply integrated with every AWS service. EC2, Lambda, ECS, RDS, and API Gateway all ship logs to CloudWatch automatically with zero configuration.

**Serverless logging.** For Lambda functions, CloudWatch is the only option that captures logs without additional infrastructure. There's no log agent to install.

**Auto-scaling.** CloudWatch scales automatically with your workload. No capacity planning, no disks to resize, no instances to upgrade.

**Log Insights.** CloudWatch Log Insights provides a powerful query language with automatic field discovery. Complex aggregations work across terabytes of logs.

## Where LogTide Wins

**Cost at scale.** CloudWatch's $0.50/GB ingestion fee adds up fast. LogTide's self-hosted model means you pay only for infrastructure, which is dramatically cheaper for 50+ GB/day.

**Full-text search.** CloudWatch requires Log Insights queries to search logs. LogTide provides real-time full-text search without per-query costs.

**Built-in SIEM.** LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management. CloudWatch has no native SIEM — you'd need Amazon Security Lake ($$$) or a third-party tool.

**Data sovereignty.** CloudWatch stores data in AWS. With LogTide, your logs stay in your infrastructure. Run on any cloud or on-premises for full GDPR compliance.

**No query costs.** Every Log Insights query costs $0.005/GB scanned. With frequent debugging, this adds up. LogTide queries are unlimited.

**No vendor lock-in.** CloudWatch logs are in a proprietary format. LogTide stores logs in TimescaleDB or ClickHouse — standard SQL, easy to export, easy to migrate.

## When to Choose CloudWatch

- You're running entirely on AWS and want zero-config log collection
- You use Lambda heavily and need serverless log capture
- Log volume is under 10 GB/day (cost difference is small)
- Your team doesn't want to manage additional infrastructure
- You need native integration with AWS services (ECS, RDS, API Gateway)

## When to Choose LogTide

- Log volume exceeds 50 GB/day (significant cost savings)
- You need SIEM capabilities (Sigma rules, incident management)
- Data sovereignty or GDPR compliance requires self-hosting
- You want unlimited search without per-query costs
- You run multi-cloud or hybrid infrastructure
- You prefer open-source with no vendor lock-in

## Migration Path

### Step 1: Ship Logs to Both

Run LogTide in parallel with CloudWatch during evaluation:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Your existing CloudWatch logging continues
// LogTide receives the same events via SDK
logtide.info('my-service', 'Request processed', { path: '/api/users', status: 200 });
```

### Step 2: Forward CloudWatch to LogTide

Use a Lambda function to forward existing CloudWatch logs:

```typescript
// Lambda function subscribed to CloudWatch log groups
export const handler = async (event: CloudWatchLogsEvent) => {
  const payload = Buffer.from(event.awslogs.data, 'base64');
  const parsed = JSON.parse(zlib.gunzipSync(payload).toString());

  const events = parsed.logEvents.map((e: any) => ({
    level: 'info',
    message: e.message,
    service: parsed.logGroup,
    metadata: {
      logStream: parsed.logStream,
      cloudwatch_id: e.id,
    },
  }));

  await fetch(`${process.env.LOGTIDE_API_URL}/api/v1/ingest`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Key': process.env.LOGTIDE_API_KEY!,
    },
    body: JSON.stringify({ logs: events }),
  });
};
```

### Step 3: Validate and Cut Over

Compare LogTide and CloudWatch results for 1-2 weeks. Once validated, switch your applications to ship directly to LogTide and reduce CloudWatch retention.

## Concept Mapping

| CloudWatch | LogTide | Notes |
|------------|---------|-------|
| Log Group | Project | Logical grouping |
| Log Stream | Service | Per-source streams |
| Log Insights | Search + SIEM | Real-time search included |
| CloudWatch Alarm | Alert Rule | Similar functionality |
| Subscription Filter | N/A | Ship directly to LogTide |
| Metric Filter | Detection Rule | Pattern matching |
| Live Tail | SSE Streaming | Real-time, no per-minute cost |

---

**Ready to reduce your AWS logging bill?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Docker Integration](/integrations/docker/) - Quick start on any infrastructure
- [Cost Optimization Guide](/use-cases/cost-optimization/) - Detailed savings analysis


---

# LogTide vs Azure Monitor Logs

> LogTide vs Azure Monitor: self-hosted, open-source log management with built-in SIEM. Azure Log Analytics charges $2.76/GB — LogTide saves 85-95% at scale.

*Source: https://logtide.dev/vs/azure-monitor/*

---

Azure Monitor with Log Analytics is Microsoft's observability platform for Azure workloads. LogTide is a self-hosted, open-source alternative with built-in SIEM. Here's how they compare.

## Cost Comparison

Azure Monitor is one of the most expensive log management options in the cloud.

### Azure Monitor Pricing

| Component | Cost |
|-----------|------|
| Log Analytics ingestion (pay-as-you-go) | $2.76/GB |
| Log Analytics ingestion (commitment tier, 100 GB/day) | $1.96/GB |
| Data retention (first 31 days) | Included |
| Data retention (additional) | $0.10/GB/month |
| Basic Logs ingestion | $0.55/GB |
| Basic Logs queries | $0.006/GB scanned |
| Microsoft Sentinel (SIEM add-on) | $2.46/GB |

**Real-world example:** An Azure SaaS company ingesting 50 GB/day:
- Log Analytics (commitment tier): 1,500 GB × $1.96 = $2,940/month
- Extra retention (90 days): ~$300/month
- **Total: ~$3,240/month ($38,880/year)**

With Sentinel SIEM added:
- Sentinel: 1,500 GB × $2.46 = $3,690/month
- **Total with SIEM: ~$6,930/month ($83,160/year)**

### LogTide Pricing

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Azure VM (Standard_D8s_v5) | ~$280/month |
| Managed disk (1 TB Premium SSD) | ~$120/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario on Azure infrastructure:**
- VM + storage: ~$400/month
- **Total: ~$400/month ($4,800/year)**

**Savings: ~$34,080/year (88%)**

With SIEM comparison: **Savings: ~$78,360/year (94%)**

## Feature Comparison

| Feature | Azure Monitor | LogTide |
|---------|--------------|---------|
| Log ingestion | Yes (agents, APIs) | Yes (HTTP + SDKs) |
| Full-text search | KQL queries | Yes (real-time) |
| Query language | KQL (Kusto) | SQL + Sigma rules |
| Real-time streaming | Limited | SSE (included) |
| Alert rules | Yes (complex conditions) | Built-in |
| Dashboards | Azure Workbooks | SIEM dashboard |
| SIEM | Sentinel ($2.46/GB extra) | Built-in (included) |
| Sigma detection rules | Sentinel only | Built-in |
| MITRE ATT&CK mapping | Sentinel only | Included |
| Playbooks/automation | Logic Apps (extra cost) | Webhooks |
| Self-hosted | No (Azure only) | Yes |
| Open source | No | AGPLv3 |

## Where Azure Monitor Wins

**Native Azure integration.** Azure Monitor collects data from every Azure service automatically — VMs, App Service, AKS, SQL Database, Functions. No agent configuration for most services.

**KQL query language.** Kusto Query Language is powerful for log analytics, with joins, aggregations, time-series analysis, and visualization. It's one of the most capable query languages for log data.

**Azure Workbooks.** Rich, interactive dashboards with parameterized queries, conditional visibility, and embedded Azure resource context.

**Sentinel integration.** Microsoft Sentinel provides enterprise SIEM with hundreds of pre-built detection rules, Microsoft threat intelligence, and automated response playbooks — if you can afford it.

**Compliance certifications.** Azure Monitor carries SOC2, ISO 27001, HIPAA, and FedRAMP certifications for regulated industries.

## Where LogTide Wins

**Cost.** Azure Monitor's $2.76/GB ingestion is the highest among major cloud providers. LogTide saves 85-95% at typical volumes. Even with commitment tiers, the savings are dramatic.

**SIEM included.** Sentinel adds $2.46/GB on top of Log Analytics costs. LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management at zero additional cost.

**No vendor lock-in.** Azure Monitor uses KQL and proprietary data formats. LogTide stores data in PostgreSQL — standard SQL, exportable, portable.

**Multi-cloud.** LogTide runs on any cloud or on-premises. If you use Azure alongside AWS or GCP, LogTide provides unified log management.

**Unlimited queries.** Basic Logs in Azure Monitor charge per query ($0.006/GB). LogTide queries are unlimited. For frequent debugging and investigation, this matters.

**Data ownership.** With LogTide self-hosted, your logs never leave your infrastructure. Full control over data residency, retention, and access.

## When to Choose Azure Monitor

- You run exclusively on Azure and want zero-config collection
- You need Sentinel for enterprise SIEM with Microsoft threat intelligence
- Your organization has Microsoft Enterprise Agreement discounts
- You need FedRAMP or specific Azure compliance certifications
- Budget is not a primary concern and convenience is valued

## When to Choose LogTide

- Log volume exceeds 20 GB/day (significant cost difference even at small volumes)
- You need SIEM without Sentinel's per-GB pricing
- You run multi-cloud or hybrid infrastructure
- You want to avoid Azure's complex pricing tiers
- Data sovereignty requires self-hosted log management
- You prefer open-source with no vendor lock-in

## Migration Path

### Step 1: Deploy LogTide on Azure

```bash
# Create a VM
az vm create \
  --resource-group logtide-rg \
  --name logtide-server \
  --image Ubuntu2204 \
  --size Standard_D4s_v5 \
  --admin-username azureuser \
  --generate-ssh-keys

# Or deploy on AKS
kubectl apply -f logtide-deployment.yaml
```

### Step 2: Dual Ship During Evaluation

Configure your applications to send logs to both Azure Monitor and LogTide:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Existing Azure Application Insights continues
// LogTide also receives events
logtide.info('my-service', 'Request processed', { path: '/api/users', status: 200 });
```

### Step 3: Forward via Event Hub

For existing Azure Monitor data, export to Event Hub and consume into LogTide:

```bash
# Create diagnostic setting to export to Event Hub
az monitor diagnostic-settings create \
  --name export-to-logtide \
  --resource /subscriptions/.../resourceGroups/.../providers/... \
  --event-hub logtide-logs \
  --event-hub-rule /subscriptions/.../providers/Microsoft.EventHub/namespaces/.../authorizationRules/RootManageSharedAccessKey \
  --logs '[{"category":"AppServiceHTTPLogs","enabled":true}]'
```

### Step 4: Reduce Log Analytics Retention

Once validated, reduce retention and switch off unnecessary data collection:

```bash
az monitor log-analytics workspace update \
  --resource-group logtide-rg \
  --workspace-name your-workspace \
  --retention-time 30
```

## Concept Mapping

| Azure Monitor | LogTide | Notes |
|---------------|---------|-------|
| Log Analytics workspace | Organization | Top-level grouping |
| Table | Project + Service | Log destination |
| KQL query | SQL query | Different syntax, similar power |
| Alert rule | Alert Rule | 1:1 mapping |
| Workbook | SIEM Dashboard | Visualization |
| Sentinel | Built-in SIEM | Included at no extra cost |
| Diagnostic Setting | SDK integration | Ship directly |
| Data Collection Rule | N/A | Not needed |

---

**Ready to cut your Azure Monitor bill?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Kubernetes Integration](/integrations/kubernetes/) - Run on AKS
- [Cost Optimization Guide](/use-cases/cost-optimization/) - Full savings analysis
- [Security Monitoring](/use-cases/security-monitoring/) - SIEM without Sentinel costs


---

# LogTide vs Better Stack for Log Management

> Compare LogTide and Better Stack (Logtail) for log management. Self-hosted vs SaaS, pricing, SIEM, and migration path.

*Source: https://logtide.dev/vs/better-stack/*

---

Better Stack (formerly Logtail) is a modern SaaS log management platform with uptime monitoring and incident management. LogTide is a self-hosted, open-source log management platform with built-in SIEM. Both target developers who want clean, modern tooling -- here's how they compare.

## Cost Comparison

Better Stack offers competitive SaaS pricing with a developer-friendly free tier, but costs still scale linearly with log volume and team size.

### Better Stack Pricing

Better Stack charges based on data volume and team size:

| Component | Cost |
|-----------|------|
| Free tier | 1 GB/month, 1 user, 3 days retention |
| Team plan | From $0.25/GB/month |
| Business plan | Custom pricing |
| Per-user (Team) | $25/user/month |
| Retention (Team, 30 days) | Included |
| Extended retention (90 days) | Additional fees |
| Uptime monitoring | Separate product (bundled in higher plans) |
| Status pages | Included in paid plans |

**Real-world example:** A growing startup ingesting 50 GB/day:

- Monthly ingestion: 1,500 GB x $0.25 = $375/month
- 8 team members: 8 x $25 = $200/month
- Extended retention (90 days): ~$100/month
- **Total: ~$675/month ($8,100/year)**

At 200 GB/day with 15 team members:
- Monthly ingestion: 6,000 GB x $0.25 = $1,500/month
- 15 team members: $375/month
- Extended retention: ~$300/month
- **Total: ~$2,175/month ($26,100/year)**

### LogTide Pricing

LogTide is free, open-source software. You pay only for infrastructure:

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Cloud VM (4 CPU, 16 GB RAM) | ~$100-200/month |
| Storage (1 TB SSD) | ~$50-80/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario:**

- Infrastructure: ~$200-300/month
- **Total: ~$250/month ($3,000/year)**

**Savings: ~$5,100/year (63%)**

At 200 GB/day:
- Infrastructure (8 CPU, 32 GB RAM, 4 TB): ~$400/month
- **Total: ~$400/month ($4,800/year)**

**Savings at scale: ~$21,300/year (82%)**

The cost gap widens further as team size grows, since LogTide has zero per-user fees.

## Feature Comparison

| Feature | Better Stack | LogTide |
|---------|-------------|---------|
| Log ingestion (HTTP API) | Yes | Yes |
| SDKs (Node.js, Python, Ruby, etc.) | Yes | Yes |
| OpenTelemetry support | Yes | Native OTLP |
| Full-text search | Yes | Yes |
| Real-time streaming | Live Tail | SSE |
| Alert rules | Yes | Yes |
| Email/Slack/webhook notifications | Yes | Yes |
| SQL querying | Yes | PostgreSQL (native) |
| Uptime monitoring | Yes (bundled) | Roadmap |
| Incident management | Yes (StatusPage) | Yes (SIEM-focused) |
| Status pages | Yes | N/A |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | No | Included |
| Custom dashboards | Yes | SIEM dashboard |
| Self-hosted option | No | Yes |
| Open source | No | AGPLv3 |
| Per-user pricing | Yes ($25/user/month) | No (unlimited) |
| Retention limits | Plan-dependent (3-30 days) | Unlimited (storage-bound) |
| Multi-org support | Yes | Yes (organizations/projects) |
| Data sovereignty | No (SaaS-only) | Yes (self-hosted) |

## Where Better Stack Wins

**Developer experience.** Better Stack is designed for developer ergonomics. The UI is clean, modern, and thoughtfully designed. Onboarding is guided with interactive setup wizards, and the SDK integration experience is polished. For teams that value design and usability, Better Stack sets a high bar.

**Uptime monitoring bundled.** Better Stack includes uptime monitoring, heartbeat checks, and status pages alongside log management. If you need logging, uptime monitoring, and incident management from one vendor with a unified interface, Better Stack packages them together seamlessly.

**Zero infrastructure management.** As a fully managed SaaS, there's nothing to deploy, patch, back up, or maintain. Your team focuses on using the platform rather than running it. This is especially valuable for small teams without dedicated DevOps resources.

**SQL-based querying.** Better Stack uses SQL for log queries, which is familiar to virtually every developer. No proprietary query language to learn -- just write standard SQL against your log data.

**Fast onboarding.** Better Stack offers a guided setup experience with copy-paste code snippets for dozens of languages and frameworks. You can go from signup to seeing your first logs in under 5 minutes, with pre-built parsing for common log formats.

**Beautiful live tail.** Better Stack's Live Tail provides a polished, real-time log streaming experience with syntax highlighting, filtering, and a clean visual design that makes debugging enjoyable.

## Where LogTide Wins

**Self-hosted option.** Better Stack is SaaS-only with no self-hosted offering. LogTide gives you full control over where your logs are stored and processed. Run on-premises, in your own cloud VPC, at the edge, or in an air-gapped environment. This is essential for organizations with strict data residency requirements.

**Built-in SIEM.** LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and security incident management at no additional cost. Better Stack focuses on observability and uptime monitoring rather than security detection. If you need to detect threats, investigate incidents, and map attacks to the MITRE framework, LogTide provides these capabilities out of the box.

**No per-seat pricing.** Better Stack charges $25/user/month on the Team plan. For a team of 20, that's $500/month ($6,000/year) in user fees alone, before any data costs. LogTide has unlimited users at no extra cost, encouraging broad access across engineering, security, DevOps, and support teams.

**Unlimited retention.** Better Stack's retention is plan-dependent: 3 days on the free tier, 30 days on Team, and extended retention costs extra. LogTide retains logs for as long as your storage allows -- months or years with no additional fees. This is critical for compliance audits, forensic investigations, and long-term trend analysis.

**Data sovereignty.** Your logs never leave your infrastructure. No third-party data processing agreements needed. Full GDPR compliance with your choice of data residency. No concerns about sensitive data -- PII, credentials, financial records -- being processed outside your perimeter.

**No vendor lock-in.** LogTide is AGPLv3 open-source. Your data lives in TimescaleDB or ClickHouse that you own and control. Your detection rules use the portable Sigma standard. You can inspect, fork, and modify every line of source code.

**Cost advantage at scale.** Better Stack's per-GB pricing means costs grow linearly with data volume, and per-user fees add a second scaling dimension. At 200+ GB/day with a growing team, LogTide's infrastructure-only cost model saves 80%+ compared to Better Stack.

**Real-time SSE streaming.** LogTide's Server-Sent Events streaming provides real-time log tailing with efficient, standards-based technology. No proprietary protocols, no WebSocket complexity -- just HTTP streaming that works through any proxy or CDN.

## When to Choose Better Stack

- You want the fastest possible onboarding with zero infrastructure management
- Log volume is small (under 5 GB/day) where cost differences are minimal
- You need uptime monitoring and status pages bundled with log management
- Your team values polished developer experience and modern UI design
- You don't need SIEM or security detection capabilities
- Your team is small (under 5 users, minimizing per-seat costs)
- You prefer a fully managed SaaS with no operational overhead

## When to Choose LogTide

- You need self-hosted log management for data sovereignty or compliance
- Per-user pricing is limiting access across your growing team
- You need SIEM capabilities (Sigma rules, MITRE ATT&CK mapping)
- Log volume exceeds 20 GB/day (significant cost savings begin)
- You need unlimited log retention without extra fees
- You prefer open-source with no vendor lock-in
- You want to keep all your data on your own infrastructure
- Security detection and incident management are core requirements

## Migration Path

### Step 1: Ship Logs to Both

Run LogTide alongside Better Stack during evaluation. Use both SDKs in parallel:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Your existing Better Stack (Logtail) SDK continues running
// LogTide receives the same events via its SDK
logtide.info('my-service', 'User signed up', { plan: 'pro', source: 'website' });
logtide.warn('my-service', 'Disk usage high', { mount: '/data', usage_pct: 87 });
logtide.error('my-service', 'Webhook delivery failed', { url: 'https://...', status: 503 });
```

### Step 2: Forward via Fluent Bit

Use Fluent Bit to ship infrastructure and application logs to both platforms:

```ini
[SERVICE]
    Flush 5
    Log_Level info

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app.*
    Parser json

[INPUT]
    Name tail
    Path /var/log/nginx/access.log
    Tag nginx.*
    Parser nginx

# Send to LogTide
[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest
    Format json
    Header X-API-Key lp_your_api_key

# Keep Better Stack output during migration
[OUTPUT]
    Name http
    Match *
    Host in.logs.betterstack.com
    Port 443
    URI /
    Format json
    Header Authorization Bearer ${BETTERSTACK_SOURCE_TOKEN}
    tls On
```

### Step 3: Validate and Cut Over

Compare LogTide and Better Stack results for 1-2 weeks:

1. Verify that search results match for the same queries and time ranges
2. Confirm alerts fire correctly in LogTide with equivalent thresholds
3. Test the LogTide SSE streaming as a replacement for Better Stack's Live Tail
4. Validate that your team is comfortable with the LogTide UI and workflows
5. Set up SIEM detection rules if security monitoring is a goal
6. Once validated, remove the Better Stack SDK and Fluent Bit output, then cancel your subscription

## Query Migration (Better Stack SQL to LogTide)

| Better Stack SQL | LogTide API |
|-----------------|-------------|
| `SELECT * FROM logs WHERE service = 'api'` | `GET /api/v1/logs?service=api` |
| `SELECT * FROM logs WHERE level = 'error'` | `GET /api/v1/logs?level=error` |
| `SELECT * FROM logs WHERE message LIKE '%timeout%'` | `GET /api/v1/logs?q=timeout` |
| `SELECT * FROM logs WHERE dt > NOW() - INTERVAL '1 hour'` | `GET /api/v1/logs?from=2025-01-15T11:00:00Z` |
| `SELECT count(*) FROM logs GROUP BY service` | `GET /api/v1/logs/aggregated?interval=1h` |
| `SELECT * FROM logs WHERE service = 'api' AND level = 'error'` | `GET /api/v1/logs?service=api&level=error` |

## Concept Mapping

| Better Stack | LogTide | Notes |
|-------------|---------|-------|
| Team | Organization | Top-level container |
| Source | Project | Logical grouping of logs |
| Source token | API Key (per project) | Prefix: `lp_` |
| Service | Service | 1:1 mapping |
| Live Tail | SSE Streaming | Real-time log streaming |
| Alert | Alert Rule | Threshold-based alerts |
| Dashboard | SIEM Dashboard | Security-focused dashboards |
| SQL query | REST API + PostgreSQL | Query via API or direct SQL |
| Metadata | metadata JSON | Structured fields in metadata |
| N/A | Sigma Rules + SIEM | LogTide exclusive feature |
| StatusPage | N/A | Use dedicated status page tool |
| Uptime Monitor | N/A | Use dedicated uptime monitoring |
| Heartbeat | N/A | Use health check endpoint |

---

**Ready to own your log management?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Docker Integration](/integrations/docker/) - Quick start on any infrastructure
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Datadog for Log Management

> LogTide vs Datadog: open-source, self-hosted log management with built-in SIEM and unlimited users. Save up to 90% versus Datadog's per-GB + per-seat pricing.

*Source: https://logtide.dev/vs/datadog/*

---

Datadog is a popular SaaS observability platform. LogTide is a self-hosted, open-source log management platform with built-in SIEM. Here's an honest comparison to help you decide which is right for your team.

## Cost Comparison

Cost is the biggest difference between LogTide and Datadog.

### Datadog Pricing

Datadog charges based on log volume plus per-user fees:

| Component | Cost |
|-----------|------|
| Log ingestion | $0.10/GB (with commitment) to $1.70/GB (on-demand) |
| Log retention (15 days) | Included with ingestion |
| Log retention (30+ days) | $0.05-$0.15/GB/month |
| Per-user (Pro) | $23/user/month |
| Cloud SIEM | $0.20/GB additional |

**Real-world example:** A mid-size SaaS company ingesting 100 GB/day:
- Log ingestion: ~$3,000/month
- 5 users (Pro): $115/month
- Cloud SIEM: $600/month
- **Total: ~$3,715/month ($44,580/year)**

### LogTide Pricing

LogTide is free, open-source software. You pay only for infrastructure:

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Cloud VM (8 CPU, 32 GB) | ~$150-300/month |
| Storage (2 TB SSD) | ~$50-100/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 100 GB/day scenario:**
- Infrastructure: ~$200-400/month
- **Total: ~$300/month ($3,600/year)**

**Savings: ~$41,000/year (92%)**

## Feature Comparison

| Feature | Datadog | LogTide |
|---------|---------|---------|
| Log ingestion (HTTP API) | Yes | Yes |
| SDKs (Node.js, Python, Go, etc.) | Yes | Yes |
| OpenTelemetry support | Partial (logs only) | Native OTLP |
| Full-text search | Yes | Yes |
| Real-time streaming | Yes | Yes (SSE) |
| Alert rules | Yes | Yes |
| Email/webhook notifications | Yes | Yes |
| Trace correlation | Yes | Yes |
| Sigma detection rules | No | Built-in |
| Incident management | Cloud SIEM ($0.20/GB) | Included |
| MITRE ATT&CK mapping | Cloud SIEM | Included |
| Self-hosted option | No | Yes |
| Custom dashboards | Yes | SIEM dashboard |
| APM/traces | Yes | Logs + OTLP traces |
| Infrastructure metrics | Yes | Yes (basic) |

## Where Datadog Wins

**Full observability platform.** Datadog offers advanced metrics, traces, logs, and infrastructure monitoring in a single SaaS platform. While LogTide now supports basic metrics, Datadog's metrics and APM capabilities are significantly more mature.

**Zero infrastructure management.** As a SaaS product, there's nothing to deploy or maintain. Your team focuses entirely on using the platform, not running it.

**Custom dashboards.** Datadog's dashboard builder is mature with drag-and-drop widgets, formula-based queries, and visualization options that LogTide doesn't match yet.

**Ecosystem integrations.** Datadog has 700+ integrations for cloud providers, databases, and third-party services with auto-discovery and auto-instrumentation.

## Where LogTide Wins

**Cost at scale.** For log-heavy workloads, LogTide's self-hosted model saves 80-95% compared to Datadog. The more logs you generate, the bigger the savings.

**Data ownership.** Your logs never leave your infrastructure. No third-party data processing agreements needed. Full GDPR compliance with EU data sovereignty.

**Built-in SIEM.** LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management at no extra cost. Datadog's Cloud SIEM adds $0.20/GB on top of log costs.

**No vendor lock-in.** LogTide is AGPLv3 open-source. You can inspect, modify, and extend the code. Your data lives in TimescaleDB or ClickHouse (your choice) that you control.

**Unlimited users.** No per-seat pricing. Add your entire engineering, security, and support teams without cost concerns.

## When to Choose Datadog

- You need advanced metrics, APM, and infrastructure monitoring with deep correlation
- Your team doesn't want to manage infrastructure
- Log volume is small (under 10 GB/day) where cost difference is minimal
- You need 700+ out-of-box integrations
- Budget is not a primary concern

## When to Choose LogTide

- Log costs are a significant line item (50+ GB/day)
- Data sovereignty or GDPR compliance is required
- You need SIEM capabilities without additional licensing
- You want to avoid per-user pricing
- You prefer open-source with no vendor lock-in
- You're comfortable managing a Docker/Kubernetes deployment

## SDK Migration

Migrating from Datadog SDK to LogTide SDK is straightforward:

```typescript
// Before (Datadog)
import { datadogLogs } from '@datadog/browser-logs';

datadogLogs.init({
  clientToken: 'pub_xxx',
  site: 'datadoghq.com',
  service: 'my-app',
});

datadogLogs.logger.info('User logged in', { userId: 123 });

// After (LogTide)
import { LogTideClient } from '@logtide/sdk-node';

const client = new LogTideClient({
  apiUrl: 'http://logtide.internal:8080',
  apiKey: 'lp_xxx',
});

client.info('my-app', 'User logged in', { userId: 123 });
```

## Concept Mapping

| Datadog | LogTide | Notes |
|---------|---------|-------|
| Organization | Organization | 1:1 mapping |
| Index | Project | Logs are scoped to projects |
| Service | Service | 1:1 mapping |
| Log Pipeline | N/A (automatic) | LogTide auto-parses JSON |
| Monitor | Alert Rule | Similar functionality |
| Dashboard | SIEM Dashboard | Security-focused dashboards |
| API Key | API Key (per project) | Prefix: `lp_` |
| Cloud SIEM | Sigma Rules + Incidents | Included at no extra cost |

## Migration Path

We have a detailed step-by-step migration guide covering SDK replacement, alert migration, parallel ingestion for validation, and cutover.

[View the full Datadog migration guide](/docs/migration/datadog/)

---

**Ready to switch from Datadog?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Migration Guide](/docs/migration/datadog/) - Step-by-step instructions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs ELK Stack for Log Management

> Compare LogTide and ELK Stack (Elasticsearch, Logstash, Kibana). Simpler architecture, lower resources, and built-in SIEM.

*Source: https://logtide.dev/vs/elk-stack/*

---

The ELK Stack (Elasticsearch, Logstash, Kibana) is the most widely deployed open-source log management solution. LogTide offers a simpler alternative with fewer components, lower resource requirements, and built-in SIEM. Here's a detailed comparison.

## Architecture Comparison

### ELK Stack

The ELK Stack consists of three (or more) components that must work together:

- **Elasticsearch** - Search and storage engine (requires cluster management)
- **Logstash** - Log processing pipeline (or Beats for lightweight collection)
- **Kibana** - Visualization and UI

Additional components often needed:
- **Filebeat/Metricbeat** - Data shippers
- **Elastic Agent** - Unified data shipper
- **Elastic SIEM** - Security features (paid)
- **ElastAlert** - Alerting (third-party)

### LogTide

LogTide is a single application stack:
- **Backend** - API, ingestion, search, alerting, SIEM
- **Frontend** - Built-in web UI
- **TimescaleDB or ClickHouse** - Storage (via Reservoir abstraction)

**That's it.** One `docker compose up -d` and you're running.

## Resource Comparison

One of LogTide's biggest advantages is resource efficiency:

| Component | ELK Stack | LogTide |
|-----------|-----------|---------|
| Elasticsearch | 16-32 GB RAM (heap) | - |
| Logstash | 4-8 GB RAM | - |
| Kibana | 2-4 GB RAM | - |
| Backend | - | 2-4 GB RAM |
| TimescaleDB | - | 4-8 GB RAM |
| **Total RAM** | **22-44 GB** | **6-12 GB** |

LogTide uses **60-75% less memory** for equivalent workloads.

## Feature Comparison

| Feature | ELK Stack | LogTide |
|---------|-----------|---------|
| Components | 3+ (ES, Logstash, Kibana) | Single stack |
| Log ingestion | Beats, Logstash | HTTP API, SDKs, OTLP |
| Query language | Lucene / KQL | REST API + Full-text |
| Full-text search | Yes | Yes |
| Real-time streaming | Kibana Discover | SSE |
| Alerting | Watcher / ElastAlert | Built-in |
| Security detection | Elastic SIEM (paid) | Sigma (included) |
| OpenTelemetry | APM Server | Native OTLP |
| Cluster management | Complex (shards, replicas) | Simple (TimescaleDB or ClickHouse) |
| Version compatibility | Must match all components | Single versioned release |
| Custom dashboards | Kibana (extensive) | SIEM dashboard |

## Where ELK Stack Wins

**Advanced search capabilities.** Elasticsearch is one of the best search engines ever built. Lucene-based queries, aggregations, and the Query DSL are incredibly powerful for complex log analysis.

**Kibana dashboards.** Kibana's visualization capabilities are extensive: custom dashboards, lens, maps, canvas, and dozens of visualization types. LogTide's dashboard is security-focused and less customizable.

**Elastic ecosystem.** Elastic has Beats agents for every data source imaginable: Filebeat, Metricbeat, Auditbeat, Packetbeat, Heartbeat. The agent ecosystem is mature.

**Battle-tested at scale.** Elasticsearch powers some of the largest search installations in the world. Its distributed architecture handles petabytes of data across hundreds of nodes.

## Where LogTide Wins

**Dramatically simpler.** No more managing Elasticsearch clusters (shards, replicas, split-brain), Logstash pipelines (grok patterns, codec issues), and Kibana (index patterns, saved objects). LogTide is one deployment.

**Lower resource requirements.** ELK requires 22-44 GB RAM minimum for production. LogTide runs on 6-12 GB. That's a 60-75% reduction in infrastructure costs.

**No version headaches.** ELK components must be version-matched. Upgrading Elasticsearch without matching Kibana and Logstash causes compatibility issues. LogTide is a single versioned release.

**Built-in SIEM.** Elastic's security features (SIEM, endpoint security) require paid licenses. LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management for free.

**Better compression.** Both TimescaleDB's columnar compression and ClickHouse's native compression are highly efficient for time-series log data, often achieving 10-20x compression ratios.

## When to Choose ELK Stack

- You need Elasticsearch's advanced query DSL for complex analysis
- You require Kibana's extensive dashboard and visualization capabilities
- You have existing Beats agents deployed across your infrastructure
- Your team has deep Elasticsearch expertise
- You need to handle petabyte-scale data with cluster management

## When to Choose LogTide

- You want simpler operations (single stack vs 3+ components)
- Your infrastructure budget is limited (60-75% less RAM needed)
- You need SIEM capabilities without paid Elastic licenses
- You're tired of ELK version compatibility issues
- You want built-in alerting without ElastAlert or Watcher
- You're starting fresh and want the easiest path to production

## Query Migration

### Elasticsearch Query DSL to LogTide

| Elasticsearch | LogTide API |
|--------------|-------------|
| `{"match": {"service": "api"}}` | `GET /api/v1/logs?service=api` |
| `{"match": {"level": "error"}}` | `GET /api/v1/logs?level=error` |
| `{"query_string": {"query": "timeout"}}` | `GET /api/v1/logs?q=timeout` |
| `{"range": {"@timestamp": {"gte": "now-1h"}}}` | `GET /api/v1/logs?from=2025-01-15T11:00:00Z` |

### KQL (Kibana Query Language) to LogTide

| KQL | LogTide |
|-----|---------|
| `service: api` | `?service=api` |
| `level: error OR level: critical` | `?level=error&level=critical` |
| `"connection timeout"` | `?q=connection%20timeout` |
| `service: api AND level: error` | `?service=api&level=error` |

## Concept Mapping

| ELK | LogTide | Notes |
|-----|---------|-------|
| Index | Project | One index pattern = One project |
| Document | Log entry | 1:1 mapping |
| Field | metadata key | Custom fields stored in metadata JSON |
| @timestamp | time | ISO 8601 format |
| Filebeat | Fluent Bit / SDK | Use Fluent Bit for file tailing |
| Logstash | Fluent Bit / SDK | Use Fluent Bit filters or preprocess in app |
| Kibana | LogTide UI | Built-in web interface |
| Watcher | Alert Rules | Simpler configuration |
| Elastic SIEM | Sigma Rules + SIEM Dashboard | Included at no extra cost |

## Migration Path

Our migration guide covers replacing Beats/Logstash with Fluent Bit, translating Elasticsearch queries, migrating Watcher alerts, and handling Logstash pipeline transformations.

[View the full ELK migration guide](/docs/migration/elk/)

---

**Ready to simplify your log stack?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Migration Guide](/docs/migration/elk/) - Step-by-step instructions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Google Cloud Logging

> LogTide vs Google Cloud Logging: self-hosted, open-source log management with built-in SIEM. Cut GCP logging costs ~73% beyond the 50 GiB free tier.

*Source: https://logtide.dev/vs/gcp-cloud-logging/*

---

Google Cloud Logging (formerly Stackdriver) is GCP's native log management service. LogTide is a self-hosted, open-source alternative with built-in SIEM. Here's how they compare.

## Cost Comparison

Google Cloud Logging offers a generous 50 GiB/month free tier, but costs scale quickly beyond that.

### Cloud Logging Pricing

| Component | Cost |
|-----------|------|
| First 50 GiB/month | Free |
| Additional ingestion | $0.50/GiB |
| Log storage (30 days default) | Included with ingestion |
| Log storage (>30 days, Log Buckets) | $0.01/GiB/month |
| Log Analytics (BigQuery linked) | BigQuery pricing applies |
| Log Router sinks | Free (but destination costs apply) |

**Real-world example:** A GCP-native SaaS ingesting 100 GiB/day:
- First 50 GiB free: -$0
- Remaining 2,950 GiB/month × $0.50 = $1,475/month
- Log Analytics: ~$200/month (BigQuery query costs)
- **Total: ~$1,675/month ($20,100/year)**

### LogTide Pricing

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| GCE instance (n2-standard-8) | ~$280/month |
| Persistent disk (2 TB SSD) | ~$170/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 100 GiB/day scenario on GCP infrastructure:**
- Compute + storage: ~$450/month
- **Total: ~$450/month ($5,400/year)**

**Savings: ~$14,700/year (73%)**

### When Cloud Logging Is Cheaper

If you're under 50 GiB/month, Cloud Logging's free tier wins. At exactly 50 GiB/month, you pay $0. LogTide would still require infrastructure costs.

**Break-even point:** ~80-100 GiB/month. Below that, Cloud Logging is cheaper or equivalent.

## Feature Comparison

| Feature | Cloud Logging | LogTide |
|---------|--------------|---------|
| Log ingestion | Yes (auto from GCP services) | Yes (HTTP + SDKs) |
| Full-text search | Logs Explorer | Yes (real-time) |
| Structured queries | Logging query language | SQL + Sigma rules |
| Real-time streaming | Log tailing | SSE (included) |
| Alert rules | Alerting policies | Built-in |
| Dashboards | Cloud Monitoring | SIEM dashboard |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | No | Included |
| Log Analytics | BigQuery-linked (extra cost) | Included |
| Log Router | Yes (sinks to GCS, BigQuery, Pub/Sub) | N/A |
| Error Reporting | Integrated | Roadmap |
| Self-hosted | No (GCP only) | Yes |
| Open source | No | AGPLv3 |

## Where Cloud Logging Wins

**Native GCP integration.** Every GCP service — GKE, Cloud Run, Cloud Functions, App Engine — ships logs automatically. No agent, no configuration. It just works.

**Free tier.** 50 GiB/month free is generous for small projects. If you're under this limit, Cloud Logging is effectively free.

**Log Router.** Cloud Logging's Log Router lets you route logs to BigQuery, Cloud Storage, Pub/Sub, or other projects. This is powerful for compliance archival and analytics.

**Error Reporting.** Integrated error grouping and alerting for application exceptions, directly from logs. No additional setup needed.

**Audit logs.** Cloud Audit Logs capture all GCP API calls automatically, providing a complete audit trail for compliance.

## Where LogTide Wins

**Cost at scale.** Beyond the free tier, Cloud Logging charges $0.50/GiB. At 100+ GiB/day, LogTide saves 70-85% on log management costs.

**Built-in SIEM.** LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management at no extra cost. Cloud Logging has no native SIEM — you'd need Chronicle SIEM or a third-party solution.

**Multi-cloud support.** LogTide runs anywhere — AWS, GCP, Azure, on-premises. Cloud Logging is GCP-only. If you run multi-cloud, LogTide provides a single pane of glass.

**No query costs.** Cloud Logging queries are free in the Logs Explorer, but Log Analytics (BigQuery-linked) charges per query. LogTide queries are always unlimited.

**Data sovereignty.** Cloud Logging stores data in Google's infrastructure. With LogTide, your logs stay in infrastructure you control, which matters for GDPR and data residency requirements.

**No vendor lock-in.** Your logs live in TimescaleDB or ClickHouse. Standard SQL, easy export, easy migration. Cloud Logging uses Google's proprietary format.

## When to Choose Cloud Logging

- You run exclusively on GCP
- Log volume is under 50 GiB/month (free tier)
- You need automatic GCP service log collection (Cloud Run, GKE)
- You want Log Router for complex routing (BigQuery analytics, GCS archival)
- Your team doesn't want to manage log infrastructure

## When to Choose LogTide

- Log volume exceeds 80 GiB/month (break-even point)
- You need SIEM capabilities without Chronicle costs
- You run multi-cloud or hybrid infrastructure
- Data residency requirements mandate self-hosting
- You want unlimited queries with no BigQuery charges
- You prefer open-source and no vendor lock-in

## Migration Path

### Step 1: Deploy LogTide on GCP

```bash
# Deploy LogTide on a GCE instance
gcloud compute instances create logtide \
  --machine-type=n2-standard-4 \
  --image-family=ubuntu-2204-lts \
  --image-project=ubuntu-os-cloud \
  --boot-disk-size=500GB

# Or use GKE
kubectl apply -f logtide-deployment.yaml
```

### Step 2: Forward Logs via Pub/Sub

Use Cloud Logging's Log Router to forward logs to LogTide via Pub/Sub:

```bash
# Create a Pub/Sub topic
gcloud pubsub topics create logtide-logs

# Create a Log Router sink
gcloud logging sinks create logtide-sink \
  pubsub.googleapis.com/projects/YOUR_PROJECT/topics/logtide-logs \
  --log-filter='resource.type="gce_instance" OR resource.type="gke_container"'
```

Then consume the Pub/Sub subscription and forward to LogTide's API.

### Step 3: Reduce Cloud Logging Retention

Once validated, reduce Cloud Logging retention to minimize costs:

```bash
# Set 1-day retention on specific log buckets
gcloud logging buckets update _Default \
  --location=global \
  --retention-days=1
```

## Concept Mapping

| Cloud Logging | LogTide | Notes |
|---------------|---------|-------|
| Project | Organization | Top-level grouping |
| Log name | Project + Service | Logical grouping |
| Logs Explorer | Search | Real-time search |
| Alerting Policy | Alert Rule | Similar functionality |
| Log Router Sink | N/A | Ship directly to LogTide |
| Log Analytics | SIEM Dashboard | No BigQuery costs |
| Error Reporting | Log-based alerts | Pattern matching |
| Audit Logs | Security Events | Manual capture required |

---

**Ready to reduce your GCP logging costs?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Kubernetes Integration](/integrations/kubernetes/) - Run on GKE
- [GDPR Compliance](/use-cases/gdpr-compliance/) - Data sovereignty guide


---

# LogTide vs Grafana Loki for Logs

> Compare LogTide and Grafana Loki for log management. Built-in alerting, full-text search, and SIEM vs label-based indexing.

*Source: https://logtide.dev/vs/grafana-loki/*

---

Grafana Loki and LogTide are both open-source log management tools. Loki excels at lightweight label-based indexing within the Grafana ecosystem. LogTide provides an all-in-one solution with full-text search, built-in alerting, and SIEM capabilities. Here's how they compare.

## Architecture Comparison

### Grafana Loki

Loki takes a unique approach: it indexes only labels (metadata), not log content. This keeps storage costs low but limits search capabilities. Loki is designed as one piece of the Grafana observability stack.

**Required components:** Loki + Promtail + Grafana + AlertManager (for alerts)

### LogTide

LogTide indexes both labels and content, providing fast full-text search. It's a self-contained platform with built-in UI, alerting, and SIEM.

**Required components:** LogTide (single Docker Compose stack)

## Feature Comparison

| Feature | Grafana Loki | LogTide |
|---------|-------------|---------|
| Log ingestion | Promtail, Fluent Bit | HTTP API, SDKs, OTLP |
| Query language | LogQL | REST API + Full-text |
| Full-text search | Limited (regex only) | Indexed full-text |
| Indexing strategy | Labels only | Labels + Content |
| Built-in alerting | No (requires Grafana) | Native |
| Built-in UI | No (requires Grafana) | Included |
| Sigma detection rules | No | Built-in |
| Incident management | No | Built-in |
| MITRE ATT&CK mapping | No | Built-in |
| OpenTelemetry | Yes | Native OTLP |
| Real-time streaming | Tail queries | SSE |
| Multi-tenancy | Yes (tenant header) | Yes (organizations/projects) |

## Where Grafana Loki Wins

**Lightweight and cost-efficient.** Loki's label-only indexing means significantly less storage and compute for pure aggregation use cases. If you primarily filter by labels and rarely search log content, Loki is extremely efficient.

**Grafana ecosystem.** If you already use Grafana for metrics dashboards (Prometheus, Mimir), adding Loki gives you logs in the same familiar interface alongside your metrics.

**Kubernetes-native.** Loki with Promtail is specifically designed for Kubernetes log collection. The integration is seamless with auto-discovery of pod labels.

**Mature scaling model.** Loki's microservices mode (distributor, ingester, querier, compactor) allows independent scaling of each component for very large deployments.

## Where LogTide Wins

**True full-text search.** Loki only indexes labels, so searching log content requires regex scanning across chunks. LogTide indexes everything, making content search fast and efficient.

**All-in-one solution.** Loki requires Grafana for UI, AlertManager for alerts, and Promtail for collection. LogTide includes UI, alerting, SDKs, and SIEM in a single deployment.

**Security detection.** Loki is purely a log aggregation tool. LogTide includes Sigma rules, MITRE ATT&CK mapping, and incident management for security teams.

**Simpler operations.** Instead of managing 3-4 separate services (Loki, Promtail, Grafana, AlertManager), LogTide runs as one Docker Compose stack.

**No label cardinality concerns.** Loki has strict limits on label cardinality. High-cardinality labels (user IDs, request IDs) can cause performance issues. LogTide stores these in metadata JSON without cardinality constraints.

## When to Choose Grafana Loki

- You already use Grafana extensively and want logs in the same UI
- You primarily filter logs by labels, not by content search
- You need extremely cost-efficient storage for very high volumes
- You're running Kubernetes and want native pod log collection
- You don't need SIEM or security detection capabilities

## When to Choose LogTide

- You need fast full-text search across log content
- You want a single tool instead of managing Loki + Grafana + AlertManager
- Security detection (Sigma rules, SIEM) is important to you
- You want built-in alerting without external dependencies
- You need native SDKs for application-level logging (not just file tailing)

## Query Migration (LogQL to LogTide)

| LogQL | LogTide API |
|-------|-------------|
| `{job="api"}` | `GET /api/v1/logs?service=api` |
| `{job="api"} \|= "error"` | `GET /api/v1/logs?service=api&level=error` |
| `{job="api"} \|~ "timeout\|connection"` | `GET /api/v1/logs?service=api&q=timeout` |
| `{job="api"} \| json` | Auto (metadata is JSON) |
| `count_over_time({job="api"}[5m])` | `GET /api/v1/logs/aggregated?interval=5m` |

**Key difference:** Loki requires regex patterns (`|~`) to search log content because it doesn't index log bodies. LogTide indexes everything, so you can search any text with the `q` parameter.

## Concept Mapping

| Loki | LogTide | Notes |
|------|---------|-------|
| Tenant | Organization / Project | Multi-tenancy via projects |
| Labels | service + metadata | service is indexed, extra fields in metadata |
| Stream | Service | One label set = one service |
| Promtail | Fluent Bit / SDK | Use Fluent Bit or application SDK |
| LogQL | REST API params | Simpler query syntax |
| Grafana | LogTide UI | Built-in web interface |
| Grafana Alerting | Alert Rules | Native alerting, no external tools |
| N/A | Sigma Rules + SIEM | LogTide exclusive |

## Migration Path

Migrating from Loki is straightforward since both support Fluent Bit and OpenTelemetry. Our migration guide covers replacing Promtail, translating LogQL queries, and setting up alerts.

[View the full Loki migration guide](/docs/migration/loki/)

---

**Ready to switch from Grafana Loki?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Migration Guide](/docs/migration/loki/) - Step-by-step instructions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Graylog for Log Management

> Compare LogTide and Graylog for log management. Docker Compose vs Java/MongoDB/Elasticsearch, SIEM, and migration path.

*Source: https://logtide.dev/vs/graylog/*

---

Graylog is an established open-source log management platform built on Java, MongoDB, and Elasticsearch (or OpenSearch). LogTide is a modern alternative with simpler architecture, lower resource needs, and built-in SIEM. Both are self-hosted and open-source, so this comparison focuses on architecture, operational complexity, and capabilities.

## Architecture Comparison

### Graylog

Graylog requires three separate components to function:

- **Graylog Server** - Java-based log processing and UI (requires JVM tuning)
- **MongoDB** - Configuration and metadata storage
- **Elasticsearch/OpenSearch** - Log storage and full-text search engine

Each component must be installed, configured, version-matched, and maintained independently. Production deployments typically require clustering Elasticsearch nodes for redundancy and performance.

**Minimum production setup:** 3 Elasticsearch nodes + 1 MongoDB replica set + 1-2 Graylog servers = 5-6 processes across multiple machines.

### LogTide

LogTide is a single application stack:

- **Backend** - API, ingestion, search, alerting, SIEM
- **Frontend** - Built-in web UI
- **TimescaleDB or ClickHouse** - Storage (via Reservoir abstraction)

**Production setup:** `docker compose up -d` = 3 containers, one command. No Elasticsearch, no MongoDB, no JVM.

## Resource Comparison

One of LogTide's biggest advantages over Graylog is resource efficiency:

| Component | Graylog | LogTide |
|-----------|---------|---------|
| Elasticsearch (3 nodes) | 24-48 GB RAM | - |
| MongoDB | 2-4 GB RAM | - |
| Graylog Server (JVM) | 4-8 GB RAM | - |
| Backend | - | 2-4 GB RAM |
| TimescaleDB | - | 4-8 GB RAM |
| **Total RAM** | **30-60 GB** | **6-12 GB** |
| **CPU cores** | **12-24** | **4-8** |
| **Disk IOPS** | High (ES indexing) | Moderate |

LogTide uses **60-80% less memory** for equivalent workloads, primarily because it avoids the Elasticsearch heap overhead and JVM memory requirements.

## Cost Comparison

Both offer free self-hosted options, but operational costs differ significantly due to infrastructure requirements and feature gating.

### Graylog Pricing

| Tier | Cost |
|------|------|
| Graylog Open (self-hosted) | Free (limited features) |
| Graylog Operations | From $1,250/month |
| Graylog Security | From $1,550/month |
| Graylog Cloud | From $1,250/month |

**Real-world infrastructure cost for 50 GB/day (self-hosted open source):**

- 3x Elasticsearch nodes (16 GB RAM each): ~$450/month
- MongoDB instance (4 GB RAM): ~$50/month
- Graylog server (8 GB JVM heap): ~$100/month
- Storage (2 TB SSD across ES nodes): ~$100/month
- **Infrastructure total: ~$700/month ($8,400/year)**

To get SIEM features, you need Graylog Security at $1,550+/month, bringing the total to **$2,250/month ($27,000/year)**.

### LogTide Pricing

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Cloud VM (4 CPU, 16 GB RAM) | ~$100-200/month |
| Storage (1 TB SSD) | ~$50-80/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario:**

- Infrastructure: ~$200-300/month
- **Total: ~$250/month ($3,000/year)**

**Infrastructure savings vs Graylog Open: ~$5,400/year (64%)**

**Savings vs Graylog Security: ~$24,000/year (89%)**

## Feature Comparison

| Feature | Graylog Open | LogTide |
|---------|-------------|---------|
| Log ingestion (Syslog, GELF) | Yes | Syslog (via Fluent Bit) + HTTP API |
| HTTP API ingestion | Yes (GELF HTTP) | Yes (native) |
| Native SDKs | No (GELF libraries) | Yes (Node.js, Python, Go) |
| OpenTelemetry support | Plugin | Native OTLP |
| Full-text search | Yes (Elasticsearch) | Yes (indexed) |
| Real-time streaming | Yes | Yes (SSE) |
| Alert rules | Yes | Yes |
| Pipelines / extractors | Yes (processing rules) | Auto JSON parsing |
| Lookup tables | Yes | metadata JSON |
| Sigma detection rules | No (Graylog Security only) | Built-in |
| MITRE ATT&CK mapping | No (Graylog Security only) | Included |
| Incident management | No (Graylog Security only) | Included |
| Multi-tenancy | Limited (streams) | Yes (organizations/projects) |
| Self-hosted | Yes | Yes |
| Open source | Yes (SSPL for some features) | AGPLv3 |
| Architecture complexity | High (Java + MongoDB + ES) | Low (Docker Compose) |
| Index management | Manual (rotation, retention) | Automatic |

## Where Graylog Wins

**Mature log processing.** Graylog's processing pipelines, extractors, and lookup tables provide sophisticated log transformation capabilities. If you need to parse complex log formats (syslog, CEF, LEEF, Windows Event Log) with rule-based extraction and enrichment, Graylog's pipeline system is well-established and battle-tested.

**Native Syslog and GELF support.** Graylog has first-class Syslog (TCP/UDP) and GELF input support, making it a natural fit for network devices, firewalls, switches, and legacy infrastructure that speaks Syslog natively. No intermediary agent is needed.

**Established community.** Graylog has been around since 2010 and has a large community with extensive documentation, content packs, and community plugins. There are answers to most common questions on forums and Stack Overflow.

**Stream-based routing.** Graylog's stream concept allows flexible log routing based on field-matching rules. Different log types can be sent to different indexes with independent retention policies, making it easy to manage data lifecycle.

**Elasticsearch query power.** Because Graylog uses Elasticsearch under the hood, you get the full power of Lucene queries and Elasticsearch aggregations for complex log analysis.

## Where LogTide Wins

**Dramatically simpler operations.** Graylog requires managing Elasticsearch clusters (shard allocation, replica management, index rotation, JVM garbage collection tuning), MongoDB (replica sets for HA, WiredTiger cache sizing), and the Graylog JVM (heap sizing, thread pool tuning). LogTide is a single Docker Compose deployment with your choice of TimescaleDB or ClickHouse.

**Lower resource footprint.** Graylog's Elasticsearch dependency alone needs 24-48 GB of RAM in production. Add MongoDB and the Graylog JVM, and you're looking at 30-60 GB minimum. LogTide runs on 6-12 GB total, saving 60-80% on infrastructure.

**Built-in SIEM at no extra cost.** Graylog locks SIEM features (Sigma rules, MITRE ATT&CK mapping, incident management) behind the paid Graylog Security tier at $1,550+/month. LogTide includes all SIEM capabilities in the free open-source version.

**Native SDKs.** Graylog relies on GELF (Graylog Extended Log Format) libraries or Syslog for application logging. GELF is non-standard and Graylog-specific. LogTide provides native SDKs for Node.js, Python, and Go with structured logging, automatic batching, and retry support.

**Flexible storage backends.** LogTide supports TimescaleDB (PostgreSQL-based, default) and ClickHouse via its Reservoir abstraction. Both are easier to manage than Elasticsearch. With TimescaleDB, standard SQL tooling, pg_dump, and pgAdmin all work out of the box. With ClickHouse, you get columnar storage optimized for high-volume analytical queries.

**True open source.** Some Graylog features are now under the Server Side Public License (SSPL), which restricts usage in certain scenarios. LogTide is fully AGPLv3 with no feature gating between free and paid tiers.

**No version compatibility headaches.** Graylog requires version-matching between Graylog Server, Elasticsearch/OpenSearch, and MongoDB. Upgrading one component without the others can cause breakage. LogTide is a single versioned release.

## When to Choose Graylog

- You receive logs primarily via Syslog or GELF from network infrastructure
- You need Graylog's mature log processing pipelines and extractors
- Your team has existing Graylog expertise, content packs, and dashboards
- You need stream-based routing for complex multi-tenant log management
- You're already running Elasticsearch and want to leverage that investment
- You need Elasticsearch's advanced aggregation capabilities

## When to Choose LogTide

- You want simpler operations without managing Elasticsearch and MongoDB
- Infrastructure budget is limited (60-80% less RAM needed)
- You need SIEM capabilities without paying for Graylog Security
- You prefer native application SDKs over GELF format
- You want a modern stack with flexible storage (TimescaleDB or ClickHouse)
- You're starting fresh and want the fastest path to production
- You want all features included with no commercial tier gating

## Migration Path

### Step 1: Ship Logs to Both

Run LogTide alongside Graylog during evaluation. Use Fluent Bit to fan out logs to both systems:

```ini
[SERVICE]
    Flush 5
    Log_Level info

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app.*
    Parser json

[INPUT]
    Name systemd
    Tag system.*
    Systemd_Filter _SYSTEMD_UNIT=nginx.service

# Send to LogTide
[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest
    Format json
    Header X-API-Key lp_your_api_key

# Keep Graylog output during migration
[OUTPUT]
    Name gelf
    Match *
    Host graylog.internal
    Port 12201
    Mode tcp
```

### Step 2: Forward Syslog Sources to LogTide

For network devices and infrastructure sending Syslog or GELF, use Fluent Bit as a protocol relay:

```ini
# Receive Syslog from network devices
[INPUT]
    Name syslog
    Listen 0.0.0.0
    Port 5140
    Mode tcp
    Parser syslog-rfc5424

# Receive GELF from legacy applications
[INPUT]
    Name tcp
    Listen 0.0.0.0
    Port 12201
    Format json

# Forward everything to LogTide
[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest
    Format json
    Header X-API-Key lp_your_api_key
```

### Step 3: Validate and Cut Over

Compare LogTide and Graylog results for 1-2 weeks:

1. Verify search results match for the same queries and time ranges
2. Confirm alerts fire correctly with equivalent thresholds
3. Test SIEM detection rules against your real log data
4. Validate multi-tenancy and access controls
5. Once validated, redirect all log sources to LogTide and decommission the Graylog stack (Elasticsearch, MongoDB, Graylog Server)

## Concept Mapping

| Graylog | LogTide | Notes |
|---------|---------|-------|
| Stream | Project | Logical grouping of logs |
| Index Set | Project | One index set = One project |
| Source | Service | Per-source identification |
| Extractor | N/A (auto JSON) | LogTide auto-parses JSON fields |
| Processing Pipeline | N/A | Preprocess in Fluent Bit or app |
| GELF input | HTTP API / SDK | Direct HTTP ingestion |
| Syslog input | Fluent Bit relay | Forward via Fluent Bit |
| Alert Condition | Alert Rule | Similar functionality |
| Notification | Webhook / Email | Alert delivery channels |
| Content Pack | N/A | Use Sigma rules from SigmaHQ |
| Lookup Table | metadata JSON | Store enrichment data in metadata |
| Graylog Security | Sigma Rules + SIEM | Built-in, no extra license |
| Dashboard | SIEM Dashboard | Security-focused dashboards |

---

**Ready to simplify your log management?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Docker Compose Integration](/integrations/docker-compose/) - One-command deployment
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Loggly for Log Management

> Compare LogTide and SolarWinds Loggly: self-hosted vs SaaS, daily-volume pricing, retention limits, SIEM capabilities, and a practical migration path.

*Source: https://logtide.dev/vs/loggly/*

---

Loggly (a SolarWinds product, like [Papertrail](/vs/papertrail/)) is a veteran SaaS log aggregator focused on JSON-aware ingestion and search. LogTide is a self-hosted, open-source log management platform with a built-in SIEM. The trade is the classic one — managed convenience against volume-capped pricing — but the details decide which side you should be on.

## Cost Comparison

### Loggly Pricing

Loggly bills by **daily ingestion volume** with retention tied to the plan tier (figures indicative — SolarWinds adjusts plans periodically):

| Plan | Daily volume | Retention | Approx. price |
|------|-------------|-----------|---------------|
| Lite (free) | 200 MB/day | 7 days | $0 |
| Standard | ~1 GB/day | ~2 weeks | from ~$79/month |
| Pro | several GB/day | ~2-4 weeks | from ~$159/month |
| Enterprise | custom | ~30+ days | from ~$279/month |

The structural issues mirror its sibling Papertrail:

1. **Daily caps, not monthly averages.** One verbose deploy or incident burst can hit the daily cap and get your overage throttled or dropped — exactly when you need logs most.
2. **Retention measured in weeks.** Investigating anything older means hoping you exported in time.
3. **Volume tiers compound with growth.** Every new service pushes you toward the next tier.

**Real-world example:** A team ingesting 5 GB/day needs Pro/Enterprise territory — realistically **$300-600+/month** with retention still capped around a month.

### LogTide Pricing

LogTide is free, open-source software. You pay only for infrastructure:

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| VPS (4 CPU, 8-16 GB RAM) | ~$40-100/month |
| Storage (1 TB SSD) | ~$50-80/month |
| SIEM features | Included |
| Users | Unlimited |
| Retention | Unlimited (storage-bound) |

**Same 5 GB/day scenario:** ~$100-180/month of infrastructure, with **months of searchable retention** and no daily cap to throttle you mid-incident.

## Feature Comparison

| Feature | Loggly | LogTide |
|---------|--------|---------|
| HTTP/S ingestion | Yes | Yes |
| Syslog ingestion | Yes | Yes (via Fluent Bit/Vector) |
| Agentless setup | Yes (core strength) | Yes (HTTP/syslog) |
| SDKs (Node.js, Python, JVM, Go, PHP, .NET) | Token-based, generic | Yes (official SDKs) |
| Automated parsing (apache, nginx, JSON…) | Yes | JSON-native + shipper parsing |
| Full-text search | Yes | Yes |
| Field/JSON queries | Yes | Yes |
| Live tail | Yes | Yes (SSE) |
| Dashboards | Yes | SIEM dashboard |
| Alerting | Yes | Yes |
| Anomaly detection | Basic | Sigma-rule based |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | No | Included |
| Incident management | No | Yes |
| Daily volume caps | Yes (plan-based) | No |
| Retention | ~7-30 days (plan-based) | Unlimited (storage-bound) |
| Self-hosted option | No | Yes |
| Open source | No | AGPLv3 |
| Data sovereignty | No (US SaaS) | Yes |

## Where Loggly Wins

**Zero infrastructure.** Fully managed: nothing to deploy, patch, scale, or back up. For teams without ops capacity, that's the entire decision.

**Agentless ingestion breadth.** Loggly's standard-syslog and HTTP endpoints plus scripted setup for common platforms make onboarding heterogeneous legacy systems genuinely easy.

**Automated parsing.** Out-of-the-box field extraction for dozens of formats (apache, nginx, Java stack traces, JSON) with derived fields for custom patterns — solid for teams who can't normalize logs at the source.

**Free tier for small workloads.** 200 MB/day free with 7-day retention covers side projects and small staging environments at literally zero cost.

## Where LogTide Wins

**No daily caps.** Incident bursts are precisely when logs matter; a platform that throttles overage during a traffic spike fails at its one job. LogTide ingests whatever your hardware handles.

**Retention.** Weeks vs months-to-years. Compliance lookbacks, post-mortems on month-old deploys, and security forensics all need history that Loggly's tiers simply don't keep.

**Built-in SIEM.** Sigma rules, MITRE ATT&CK mapping, and incident management included — Loggly offers log search and basic anomaly alerts, not security detection.

**Self-hosting and data sovereignty.** Loggly is US-hosted SaaS only. LogTide keeps logs — and the PII that inevitably leaks into them — on infrastructure you control, making GDPR a configuration detail.

**Cost trajectory.** Tiered SaaS pricing rises in steps with every growth phase; self-hosted infrastructure cost rises slowly and predictably with storage.

**Open source.** AGPLv3, standard SQL storage (TimescaleDB/ClickHouse), portable Sigma rules — no lock-in at any layer.

## When to Choose Loggly

- You have no capacity or appetite to operate any infrastructure
- Volume is small and stable (within or near the 200 MB/day free tier)
- You need automated parsing for legacy formats you can't change
- Two weeks of retention genuinely covers your investigation needs
- Data residency is not a requirement

## When to Choose LogTide

- Daily caps or retention limits have bitten you during an incident
- Volume is multiple GB/day and SaaS tiers are escalating
- You need months of searchable history for compliance or forensics
- Logs must stay on your own infrastructure (GDPR, data sovereignty)
- You want security detection (Sigma, MITRE ATT&CK) without a separate SIEM
- You prefer open source with no vendor lock-in

## Migration Path

### Step 1: Dual-ship from your existing pipeline

Loggly ingestion is syslog/HTTP based, so add a parallel output in whatever shipper you already run. With rsyslog:

```bash
# Existing Loggly forward (keep during migration)
*.* @@logs-01.loggly.com:514

# Add: forward to local Fluent Bit for LogTide
*.* @@127.0.0.1:5140
```

And Fluent Bit ships to LogTide:

```ini
[INPUT]
    Name syslog
    Mode tcp
    Listen 127.0.0.1
    Port 5140
    Parser syslog-rfc5424

[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest/single
    Format json_lines
    Header X-API-Key lp_your_api_key
    Header Content-Type application/json
```

Applications using Loggly's HTTP token endpoint can adopt the [LogTide SDKs](/docs/sdks/) directly and gain structured metadata in the process.

### Step 2: Concept mapping

| Loggly | LogTide |
|--------|---------|
| Customer token | API key (per project, `lp_` prefix) |
| Source group | Project / service filter |
| Derived fields | Structured metadata fields |
| Saved search | Saved query |
| Alert (search-based) | Alert rule + notification channel |
| Live tail | SSE streaming |
| Dashboards | SIEM dashboard |
| Archiving to S3 | Native retention (still searchable) |

### Step 3: Validate and cut over

1. Run both for 1-2 weeks; compare the same queries and time windows
2. Recreate alerts and confirm notification parity
3. Verify parsing: fields Loggly auto-extracted should arrive as JSON metadata from your shipper
4. Remove the Loggly output, downgrade to Lite or cancel

---

**Ready to own your log management?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [LogTide vs Papertrail](/vs/papertrail/) - The other SolarWinds logging product
- [Self-hosted log management](/self-hosted-log-management/) - Why teams move off metered SaaS


---

# LogTide vs New Relic for Log Management

> LogTide vs New Relic: self-hosted, open-source log management with built-in SIEM and no per-user pricing. Save 90%+ beyond New Relic's 100 GB free tier.

*Source: https://logtide.dev/vs/new-relic/*

---

New Relic is a popular SaaS observability platform offering metrics, traces, and logs in a unified experience. LogTide is a self-hosted, open-source log management platform with built-in SIEM capabilities. Here's an honest comparison to help you decide which fits your needs.

## Cost Comparison

New Relic's pricing model is simpler than most SaaS competitors, but costs still grow linearly with data volume. The per-user fees also add up quickly as teams grow.

### New Relic Pricing

New Relic charges based on data ingest volume plus per-user fees:

| Component | Cost |
|-----------|------|
| Free tier | 100 GB/month included |
| Additional data ingestion | $0.35/GB |
| Full Platform user | $49/user/month (Standard) to $99/user/month (Pro) |
| Core users | $0.35/user/hour |
| Data retention (8 days) | Included |
| Extended retention (30 days) | $0.05/GB/month |
| Data Plus (90 days retention) | $0.50/GB |

**Real-world example:** A mid-size team ingesting 50 GB/day of logs:

- Monthly ingestion: 1,500 GB - 100 GB free = 1,400 GB
- Data cost: 1,400 GB x $0.35 = $490/month
- 10 Full Platform users (Pro): $990/month
- Extended retention (30 days): ~$75/month
- **Total: ~$1,555/month ($18,660/year)**

At 200 GB/day with 15 users:
- Data cost: (6,000 - 100) x $0.35 = $2,065/month
- 15 Pro users: $1,485/month
- **Total: ~$3,700/month ($44,400/year)**

### LogTide Pricing

LogTide is free, open-source software. You pay only for infrastructure:

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Cloud VM (4 CPU, 16 GB RAM) | ~$100-200/month |
| Storage (1 TB SSD) | ~$50-80/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario:**

- Infrastructure: ~$200-300/month
- **Total: ~$250/month ($3,000/year)**

**Savings: ~$15,660/year (84%)**

At 200 GB/day:
- Infrastructure (8 CPU, 32 GB RAM, 4 TB): ~$400/month
- **Total: ~$400/month ($4,800/year)**

**Savings at scale: ~$39,600/year (89%)**

## Feature Comparison

| Feature | New Relic | LogTide |
|---------|-----------|---------|
| Log ingestion (HTTP API) | Yes | Yes |
| SDKs (Node.js, Python, Go, etc.) | Yes | Yes |
| OpenTelemetry support | Yes (OTLP) | Native OTLP |
| Full-text search | Yes (NRQL) | Yes |
| Real-time streaming | Live Tail | SSE |
| Alert rules | Yes | Yes |
| Email/webhook notifications | Yes | Yes |
| APM / traces | Yes | Logs + OTLP traces |
| Infrastructure metrics | Yes | Roadmap |
| Browser monitoring | Yes | No |
| Mobile monitoring | Yes | No |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | No | Included |
| Incident management | Yes | Included |
| Self-hosted option | No | Yes |
| Open source | No (proprietary) | AGPLv3 |
| Custom dashboards | Yes (extensive) | SIEM dashboard |
| Per-user pricing | Yes ($49-99/user/month) | No (unlimited) |
| Data retention (default) | 8 days | Unlimited (storage-bound) |
| Multi-org support | Yes (sub-accounts) | Yes (organizations/projects) |

## Where New Relic Wins

**Full observability platform.** New Relic provides metrics, traces, logs, browser monitoring, mobile monitoring, synthetic monitoring, and infrastructure monitoring in a single SaaS platform. If you need comprehensive observability across all telemetry types, New Relic is more complete.

**NRQL query language.** NRQL is a powerful SQL-like language for querying all telemetry data. Complex aggregations, facets, time-series comparisons, and cross-signal correlations are straightforward for analysts familiar with SQL. You can join logs with traces and metrics in a single query.

**Zero infrastructure management.** As a fully managed SaaS, there's nothing to deploy, scale, or maintain. Your team focuses on analyzing data rather than running the platform. No patching, no backups, no capacity planning.

**Generous free tier.** 100 GB/month of free data ingest is enough for small projects, staging environments, or proof-of-concept work. This makes it easy to evaluate without any upfront commitment or credit card.

**Extensive integrations.** New Relic has 700+ integrations and quickstarts for cloud providers, frameworks, databases, and third-party services. Guided installation and pre-built dashboards reduce time to value.

**AI-powered insights.** New Relic AI provides natural-language querying, automated anomaly detection, and root cause analysis across all telemetry data. These ML-powered features can accelerate troubleshooting.

## Where LogTide Wins

**Cost at scale.** Once you exceed the 100 GB/month free tier, New Relic's per-GB pricing adds up quickly. LogTide's self-hosted model saves 80-90%+ for log-heavy workloads. The more logs you generate, the bigger the savings.

**No per-user fees.** New Relic's per-user pricing ($49-99/user/month) discourages broad access. Organizations often limit who can access the platform, creating knowledge silos. LogTide has unlimited users, so your entire engineering, security, DevOps, and support teams can access logs without cost concerns.

**Data sovereignty.** Your logs never leave your infrastructure. No third-party data processing agreements needed. Full GDPR compliance with EU data residency. No concerns about sensitive data -- PII, credentials, healthcare records -- being processed by a third party.

**Built-in SIEM.** LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management at no extra cost. New Relic offers vulnerability management and some security features, but not a full SIEM with standardized, community-backed detection rules.

**No vendor lock-in.** LogTide is AGPLv3 open-source. Your data lives in TimescaleDB or ClickHouse (your choice) that you control. Your detection rules use the portable Sigma standard. You can inspect, modify, and extend every line of code.

**Unlimited retention.** New Relic's default retention is 8 days. Extended retention to 30 days costs $0.05/GB/month; 90 days requires the expensive Data Plus plan at $0.50/GB. LogTide retains logs as long as your storage allows, with no additional fees. Keep a year of logs for forensic analysis at just the cost of disk space.

**Real-time SSE streaming.** LogTide's Server-Sent Events streaming provides real-time log tailing with no per-minute charges. New Relic's Live Tail is available but with limitations on the free tier.

## When to Choose New Relic

- You need a full observability platform (metrics + traces + logs + browser + infrastructure)
- Your team doesn't want to manage infrastructure
- Log volume stays under 100 GB/month (free tier covers your needs)
- You need 700+ out-of-box integrations and pre-built dashboards
- You want NRQL's SQL-like query language for cross-signal analysis
- AI-powered anomaly detection and root cause analysis are important

## When to Choose LogTide

- Log volume exceeds 100 GB/month and costs are a concern
- Per-user pricing is limiting access across your team
- Data sovereignty or GDPR compliance requires self-hosting
- You need SIEM capabilities (Sigma rules, MITRE ATT&CK mapping)
- You want unlimited log retention without extra fees
- You prefer open-source with no vendor lock-in
- You're comfortable managing a Docker Compose deployment

## Migration Path

### Step 1: Ship Logs to Both

Run LogTide in parallel with New Relic during evaluation. Use the LogTide SDK alongside the New Relic agent:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Your existing New Relic agent continues capturing logs
// LogTide receives the same events via its SDK
logtide.info('my-service', 'Order processed', { orderId: 'abc-123', total: 49.99 });
logtide.warn('my-service', 'Slow query detected', { query: 'SELECT ...', duration_ms: 3200 });
logtide.error('my-service', 'Payment failed', { provider: 'stripe', error: 'card_declined' });
```

### Step 2: Forward with Fluent Bit

Use Fluent Bit to ship infrastructure logs to LogTide while keeping New Relic active:

```ini
[SERVICE]
    Flush 5
    Log_Level info

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app.*
    Parser json

[INPUT]
    Name systemd
    Tag system.*
    Systemd_Filter _SYSTEMD_UNIT=nginx.service

# Send to LogTide
[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest
    Format json
    Header X-API-Key lp_your_api_key

# Keep New Relic output during migration
[OUTPUT]
    Name newrelic
    Match *
    licenseKey ${NEW_RELIC_LICENSE_KEY}
```

### Step 3: Validate and Cut Over

Compare LogTide and New Relic log results for 1-2 weeks:

1. Verify search results match for the same time ranges and queries
2. Confirm alerts fire correctly in LogTide with equivalent thresholds
3. Validate that your team is comfortable with the LogTide UI
4. Test SIEM detection rules on your real log data
5. Once validated, remove the New Relic agent and Fluent Bit output, then cancel your subscription

## Query Migration (NRQL to LogTide)

| NRQL Query | LogTide API |
|-----------|-------------|
| `SELECT * FROM Log WHERE service = 'api'` | `GET /api/v1/logs?service=api` |
| `SELECT * FROM Log WHERE level = 'ERROR'` | `GET /api/v1/logs?level=error` |
| `SELECT * FROM Log WHERE message LIKE '%timeout%'` | `GET /api/v1/logs?q=timeout` |
| `SELECT count(*) FROM Log SINCE 1 hour ago TIMESERIES` | `GET /api/v1/logs/aggregated?interval=1h` |
| `SELECT * FROM Log WHERE service = 'api' SINCE 1 hour ago` | `GET /api/v1/logs?service=api&from=2025-01-15T11:00:00Z` |

## Concept Mapping

| New Relic | LogTide | Notes |
|-----------|---------|-------|
| Account | Organization | Top-level container |
| Sub-account | Project | Logs are scoped to projects |
| Entity (service) | Service | 1:1 mapping |
| Log forwarding | HTTP API / SDK | Direct ingestion |
| NRQL query | REST API params | Query via API parameters |
| Alert condition | Alert Rule | Threshold-based alerts |
| Alert policy | Alert Rule group | Group related alerts |
| Dashboard | SIEM Dashboard | Security-focused dashboards |
| License Key | API Key (per project) | Prefix: `lp_` |
| Vulnerability Mgmt | Sigma Rules + SIEM | Built-in detection engine |
| Logs in Context | metadata JSON | Structured context in metadata |
| Log patterns | N/A | Manual log analysis |

---

**Ready to switch from New Relic?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Docker Integration](/integrations/docker/) - Quick start on any infrastructure
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Papertrail for Log Management

> Compare LogTide and SolarWinds Papertrail: self-hosted vs SaaS, pricing per GB, retention limits, SIEM features, and a step-by-step migration path.

*Source: https://logtide.dev/vs/papertrail/*

---

Papertrail (acquired by SolarWinds) is one of the oldest and simplest hosted log management services: pipe syslog at it, get a live tail and searchable logs in seconds. LogTide is a self-hosted, open-source log management platform with structured queries and a built-in SIEM. One optimizes for zero-setup simplicity, the other for ownership, cost at scale, and security tooling — here's the honest comparison.

## Cost Comparison

### Papertrail Pricing

Papertrail charges by monthly log volume, with searchable retention measured in days and a 1-year archive on paid plans:

| Plan tier | Monthly volume | Searchable retention | Approx. price |
|-----------|---------------|----------------------|---------------|
| Free | 50 MB/month | 48 hours | $0 |
| Entry | 1 GB/month | 1 week | ~$7/month |
| Mid | 8 GB/month | 1 week | ~$35/month |
| Growth | 25 GB/month | 2 weeks | ~$75/month |
| Upper tiers | 100+ GB/month | 2-4 weeks | ~$230+/month |

(Prices are indicative — SolarWinds adjusts tiers periodically; check their pricing page for current numbers.)

Two structural limits matter more than the sticker price:

1. **Searchable retention is days, not months.** Even top tiers cap live search at a few weeks. Older logs go to an S3 archive you have to download and grep manually.
2. **Volume tiers punish growth.** A noisy deploy or a traffic spike pushes you into the next tier or triggers per-GB overage.

**Real-world example:** A team ingesting a modest 5 GB/day:

- Monthly volume: ~150 GB — beyond standard published tiers, requiring custom/enterprise pricing, realistically several hundred dollars per month
- Searchable window: still capped at ~2-4 weeks
- **Total: ~$300-500+/month with search limited to recent weeks**

### LogTide Pricing

LogTide is free, open-source software. You pay only for infrastructure:

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| VPS (4 CPU, 8-16 GB RAM) | ~$40-100/month |
| Storage (1 TB SSD) | ~$50-80/month |
| SIEM features | Included |
| Users | Unlimited |
| Searchable retention | Unlimited (storage-bound) |

**Same 5 GB/day scenario:**

- Infrastructure: ~$100-150/month
- Searchable retention: **months, not days** — query everything live
- **Savings: typically 60-75%, plus the retention difference**

The gap widens with volume: at 20+ GB/day Papertrail-class SaaS pricing becomes untenable while LogTide's infrastructure cost grows roughly with storage alone.

## Feature Comparison

| Feature | Papertrail | LogTide |
|---------|-----------|---------|
| Syslog ingestion (rsyslog, syslog-ng) | Yes (core strength) | Yes (via Fluent Bit/Vector) |
| HTTP API ingestion | Limited | Yes (native) |
| SDKs (Node.js, Python, JVM, Go, PHP, .NET) | No (syslog-oriented) | Yes |
| Live tail | Yes (excellent) | Yes (SSE streaming) |
| Full-text search | Yes (grep-style) | Yes |
| Structured JSON field queries | No (text search only) | Yes (native metadata fields) |
| Searchable retention | Days to weeks (plan-capped) | Unlimited (storage-bound) |
| Long-term archive | S3 archive (1 year, not searchable) | All data stays searchable |
| Alerting | Yes (search-based) | Yes (threshold + real-time) |
| Email/Slack/webhook notifications | Yes | Yes |
| Dashboards | Basic (velocity graphs) | SIEM dashboard |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | No | Included |
| Incident management | No | Yes (SIEM-focused) |
| Self-hosted option | No | Yes |
| Open source | No | AGPLv3 |
| Per-user pricing | No | No |
| Data sovereignty | No (US SaaS) | Yes (your infrastructure) |

## Where Papertrail Wins

**Setup speed.** Papertrail's claim to fame is going from signup to live logs in under a minute: point `remote_syslog2` or your rsyslog config at the provided endpoint and you're done. No containers, no database, nothing to operate.

**Zero operational overhead.** Fully managed SaaS — no patching, backups, or capacity planning. For a solo developer or a team with no ops capacity, that convenience is real.

**Simplicity as a feature.** The product is deliberately minimal: a tail, a search box, saved searches, and alerts. There is almost no learning curve, which makes it great for teams that only ever grep recent logs.

**Tiny workloads ride free.** 50 MB/month free with 48-hour search is enough for hobby projects and low-traffic side services.

## Where LogTide Wins

**Self-hosting and data sovereignty.** Papertrail is US-hosted SaaS with no self-hosted option — your logs (and whatever PII leaks into them) live on SolarWinds infrastructure. LogTide runs on your own servers, VPC, or air-gapped network, which makes GDPR and data-residency compliance a non-issue.

**Searchable retention.** This is the biggest practical difference. Papertrail caps live search at days or weeks; investigating an incident from last month means downloading S3 archives and grepping them offline. LogTide keeps everything searchable for as long as you give it disk.

**Structured queries.** Papertrail treats logs as text lines. LogTide ingests structured JSON with arbitrary metadata fields, so you can filter by `user_id`, `status_code`, or `deployment` instead of crafting grep patterns and hoping the format never changes.

**Built-in SIEM.** Sigma detection rules, MITRE ATT&CK mapping, and incident management ship in the box. Papertrail has no security detection capabilities at all — pairing it with a SIEM means buying and integrating a second product.

**Cost at scale.** Volume-tiered SaaS pricing grows linearly (or worse, in tier jumps) with traffic. LogTide's cost is your infrastructure bill, which grows far slower and has no overage anxiety.

**No vendor lock-in.** AGPLv3 source, data in standard SQL stores (TimescaleDB or ClickHouse) you can query and export directly.

## When to Choose Papertrail

- You want logs flowing in literally minutes with zero infrastructure
- Volume is tiny (within or near the free 50 MB/month tier)
- You only need recent-history grep and a live tail, never long lookbacks
- Nobody on the team can operate even a Docker Compose stack
- Data residency and security detection are not requirements

## When to Choose LogTide

- You need searchable retention measured in months, not days
- Log volume exceeds a few GB/day and SaaS tiers are getting expensive
- You need structured, field-based queries on JSON logs
- Data sovereignty or GDPR compliance requires keeping logs on your infrastructure
- You want SIEM capabilities (Sigma rules, MITRE ATT&CK) without a second product
- You prefer open source with no lock-in

## Migration Path

### Step 1: Ship logs to both

Papertrail ingestion is syslog-based, so the cleanest bridge is a log shipper that fans out to both destinations. With Fluent Bit:

```ini
[SERVICE]
    Flush 5

[INPUT]
    Name syslog
    Mode udp
    Listen 0.0.0.0
    Port 5140
    Parser syslog-rfc5424

# Send to LogTide
[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest/single
    Format json_lines
    Header X-API-Key lp_your_api_key
    Header Content-Type application/json

# Keep Papertrail during migration
[OUTPUT]
    Name syslog
    Match *
    Host logsN.papertrailapp.com
    Port 12345
    Mode tls
    Syslog_Format rfc5424
```

Point your existing rsyslog/`remote_syslog2` at the Fluent Bit listener (or add a second `*.* @@` forward rule in rsyslog) and both platforms receive identical streams.

For applications, you can also adopt the [LogTide SDKs](/docs/sdks/) directly and get structured metadata that syslog never carried.

### Step 2: Recreate saved searches and alerts

| Papertrail concept | LogTide equivalent |
|--------------------|--------------------|
| Saved search | Saved query / filter |
| Search alert (email/Slack/webhook) | Alert rule with notification channel |
| Live tail (`papertrail -f`) | SSE streaming / live tail |
| Groups (host grouping) | Services + projects |
| S3 archive | Native retention (still searchable) |
| Log velocity graph | SIEM dashboard |

Papertrail searches are plain text expressions, so translation is mostly mechanical: a saved search like `"connection refused" -staging` becomes a LogTide query on message text plus a service/environment filter on metadata.

### Step 3: Validate and cut over

1. Compare both platforms on the same queries for 1-2 weeks
2. Confirm alert parity (thresholds, notification channels, latency)
3. Verify the LogTide live tail covers your `tail -f` workflows
4. Export or let expire the Papertrail S3 archive, then remove the Papertrail output and cancel

---

**Ready to own your log management?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Self-hosted log management](/self-hosted-log-management/) - Why teams move off metered SaaS
- [Docker Integration](/integrations/docker/) - Quick start on any infrastructure


---

# LogTide vs SigNoz for Log Management

> Compare LogTide and SigNoz for log management. Both open-source, both support OpenTelemetry. See where they differ.

*Source: https://logtide.dev/vs/signoz/*

---

SigNoz and LogTide are both open-source, self-hosted platforms that support OpenTelemetry. SigNoz focuses on full observability (metrics, traces, logs). LogTide focuses on log management with built-in SIEM capabilities, plus basic metrics and traces support. Here's how they compare.

## Philosophy Comparison

### SigNoz

SigNoz positions itself as an open-source alternative to Datadog and New Relic, covering all three pillars of observability: metrics, traces, and logs. It's built on ClickHouse for high-performance analytics.

### LogTide

LogTide focuses on log management with security built in, plus basic support for metrics and traces. Rather than trying to replace your entire observability stack, LogTide excels at logs + security detection. It supports both TimescaleDB and ClickHouse as storage backends via its Reservoir abstraction layer.

## Feature Comparison

| Feature | SigNoz | LogTide |
|---------|--------|---------|
| OpenTelemetry | Native OTLP | Native OTLP |
| Logs | Yes | Yes |
| Traces | Yes | Yes (via OTLP) |
| Metrics | Yes | Yes (basic) |
| Custom SDKs | OTel only | OTel + Custom (Node.js, Python, Go, etc.) |
| Alerting | Yes | Yes |
| Sigma detection rules | No | Built-in |
| Incident management | No | Built-in |
| MITRE ATT&CK mapping | No | Built-in |
| Database | ClickHouse | TimescaleDB or ClickHouse (via Reservoir) |
| Full-text search | Yes | Yes |
| Real-time streaming | Yes | Yes (SSE) |
| Custom dashboards | Yes | SIEM dashboard |
| Multi-tenancy | Limited | Organizations + Projects |

## Where SigNoz Wins

**Full observability.** SigNoz covers metrics, traces, and logs with deep correlation across all three pillars. While LogTide now supports basic metrics, SigNoz's metrics capabilities are more mature.

**Deeper ClickHouse integration.** While LogTide also supports ClickHouse via its Reservoir storage abstraction, SigNoz is built natively on ClickHouse with deeper integration for cross-signal correlation and analytical queries.

**Custom dashboards.** SigNoz has a query builder and dashboard creator for building custom visualizations across metrics, traces, and logs with advanced correlation.

**Service maps.** SigNoz auto-generates service dependency maps from trace data, helping you understand your microservice architecture.

## Where LogTide Wins

**Security detection.** LogTide includes Sigma rules, MITRE ATT&CK mapping, and incident management. SigNoz is purely an observability tool with no security capabilities.

**Flexible storage.** LogTide supports both TimescaleDB (default, PostgreSQL-based) and ClickHouse via its Reservoir abstraction. Start with TimescaleDB for simplicity, or use ClickHouse for high-volume workloads — without changing application code.

**Native SDKs.** SigNoz relies exclusively on OpenTelemetry SDKs. LogTide provides lightweight, purpose-built SDKs for Node.js, Python, Go, PHP, Kotlin, and C# in addition to OTLP support.

**Multi-tenancy.** LogTide has built-in multi-tenancy with organizations and projects, each with separate API keys and access controls. SigNoz's multi-tenancy is more limited.

**SIEM dashboard.** LogTide provides a security-focused dashboard for threat monitoring, detection rule management, and incident tracking that SigNoz doesn't offer.

## When to Choose SigNoz

- You need advanced metrics with custom dashboards and deep correlation
- You want custom dashboards across all telemetry types
- Service dependency mapping is important
- You want deeper native ClickHouse integration for cross-signal analytics
- You don't need security detection or SIEM capabilities

## When to Choose LogTide

- Security detection (Sigma rules, SIEM) is a requirement
- You want flexible storage (TimescaleDB for simplicity or ClickHouse for scale)
- You need native SDKs beyond OpenTelemetry
- Incident management and MITRE ATT&CK mapping are important
- You need focused log management with basic metrics and built-in security

## Migration: Seamless via OpenTelemetry

Since both platforms support OTLP natively, migration is straightforward - just update the endpoint:

```typescript
// Before (SigNoz)
const logExporter = new OTLPLogExporter({
  url: 'http://signoz:4318/v1/logs',
});

// After (LogTide)
const logExporter = new OTLPLogExporter({
  url: 'http://logtide:8080/v1/otlp/logs',
  headers: { 'X-API-Key': 'lp_your_api_key' },
});
```

For OpenTelemetry Collector, update the exporter config:

```yaml
# Before (SigNoz)
exporters:
  otlp:
    endpoint: signoz-otel-collector:4317

# After (LogTide)
exporters:
  otlphttp/logtide:
    endpoint: http://logtide:8080
    headers:
      X-API-Key: lp_your_api_key
```

## Concept Mapping

| SigNoz | LogTide | Notes |
|--------|---------|-------|
| Service | Service | 1:1 mapping (from OTel resource) |
| Trace | trace_id | Indexed for correlation |
| Span | span_id | Indexed for correlation |
| Log attributes | metadata | Stored as JSON |
| Alert | Alert Rule | Similar configuration |
| Dashboard | SIEM Dashboard | Security-focused |
| N/A | Sigma Rules | LogTide exclusive |
| N/A | Incidents | LogTide exclusive |

## Migration Path

Our migration guide covers updating OTLP endpoints, migrating alerts, and enabling LogTide's security features that aren't available in SigNoz.

[View the full SigNoz migration guide](/docs/migration/signoz/)

---

**Ready to add security to your log management?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Migration Guide](/docs/migration/signoz/) - Step-by-step instructions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Splunk for Log Management

> Compare LogTide and Splunk for log management. Open-source vs enterprise licensing, features, and migration path.

*Source: https://logtide.dev/vs/splunk/*

---

Splunk is a legacy enterprise log management and SIEM platform. LogTide is a modern, open-source alternative with native Sigma rules support. Here's how they compare.

## Cost Comparison

Splunk's licensing model is one of the most expensive in the industry.

### Splunk Pricing

Splunk charges based on daily ingestion volume:

| Tier | Cost |
|------|------|
| Splunk Enterprise (1-10 GB/day) | ~$150/GB/day/year |
| Splunk Enterprise (100+ GB/day) | Negotiated (typically $500-1,800/GB/day/year) |
| Splunk Cloud | Higher than Enterprise |
| Splunk Enterprise Security | Additional license |
| Splunk SOAR | Additional license |

**Real-world example:** Enterprise with 50 GB/day:
- Splunk Enterprise license: ~$75,000-250,000/year
- Enterprise Security add-on: ~$25,000-50,000/year
- Infrastructure + admin staff: ~$50,000/year
- **Total: ~$150,000-350,000/year**

### LogTide Pricing

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Infrastructure (50 GB/day) | ~$300-600/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario:**
- Infrastructure: ~$400/month
- **Total: ~$4,800/year**

**Savings: $145,000-345,000/year (97%+)**

## Feature Comparison

| Feature | Splunk | LogTide |
|---------|--------|---------|
| Log ingestion | HEC, Forwarders | HTTP API, SDKs, OTLP |
| Query language | SPL (proprietary) | REST API + Full-text |
| Full-text search | Yes | Yes |
| Real-time streaming | Yes | Yes (SSE) |
| Alerts | Yes | Yes |
| Detection rules | Splunk ES (extra license) | Sigma (included) |
| MITRE ATT&CK | Splunk ES | Included |
| Incident management | Splunk ES / SOAR | Included |
| OpenTelemetry | Partial | Native OTLP |
| Self-hosted | Yes (licensed) | Yes (free) |
| Architecture | Complex (indexer clusters) | Simple (Docker Compose) |

## Where Splunk Wins

**Mature SPL query language.** SPL is powerful for complex data analysis with pipes, stats, evals, and lookups. If your team has years of SPL knowledge, that's significant institutional expertise.

**Enterprise ecosystem.** Splunk has thousands of apps and add-ons in Splunkbase. Enterprise customers get professional support, training, and consulting.

**Advanced analytics.** Splunk's ML toolkit and advanced correlations (especially in Splunk ES) are more sophisticated than LogTide's current detection capabilities.

**Proven at massive scale.** Splunk handles petabytes of data across global deployments. It's battle-tested in the largest enterprises.

## Where LogTide Wins

**Cost savings of 90-97%.** Splunk licensing is extremely expensive, especially for growing companies. LogTide eliminates per-GB licensing entirely.

**Industry-standard detection rules.** Splunk locks you into proprietary SPL. LogTide uses Sigma, the open standard for detection rules with 2,000+ community rules from SigmaHQ.

**Simpler architecture.** Splunk requires indexer clusters, search heads, deployment servers, and forwarders. LogTide runs as a single Docker Compose stack.

**No data limits.** Splunk penalizes you for exceeding your license. LogTide ingests as much as your infrastructure can handle.

**No vendor lock-in.** Your data lives in TimescaleDB or ClickHouse (your choice). Your detection rules are portable Sigma YAML. Nothing is proprietary.

## When to Choose Splunk

- You have existing SPL expertise and large investments in Splunk apps
- You need the most advanced correlation and ML-based analytics
- You require enterprise-grade support with SLAs
- Budget is not a constraint
- You're already deeply integrated with the Splunk ecosystem

## When to Choose LogTide

- Splunk licensing costs are unsustainable as data grows
- You want industry-standard Sigma rules instead of proprietary SPL
- You prefer simpler architecture without cluster management
- Data sovereignty or self-hosting is a requirement
- You need SIEM capabilities without additional Splunk ES licensing

## Query Migration (SPL to LogTide)

SPL queries translate to LogTide REST API parameters:

| SPL Query | LogTide API |
|-----------|-------------|
| `index=main sourcetype=app_logs` | `GET /api/v1/logs?service=app` |
| `index=main level=ERROR` | `GET /api/v1/logs?level=error` |
| `index=main "connection failed"` | `GET /api/v1/logs?q=connection%20failed` |
| `index=main earliest=-1h` | `GET /api/v1/logs?from=2025-01-15T11:00:00Z` |
| `index=main \| stats count by host` | `GET /api/v1/logs/aggregated?interval=1h` |

## Concept Mapping

| Splunk | LogTide | Notes |
|--------|---------|-------|
| Index | Project | One Splunk index = One LogTide project |
| Sourcetype | Service | Use service field to differentiate log sources |
| Host | metadata.host | Store in metadata JSON field |
| Universal Forwarder | Fluent Bit / SDK | Use Fluent Bit or application SDK |
| HEC | POST /api/v1/ingest | HTTP API endpoint |
| Saved Search | Alert Rule | Threshold-based alerts |
| Enterprise Security | Sigma Rules + SIEM | Built-in, no extra license |
| props.conf / transforms.conf | N/A (auto JSON parsing) | Send structured JSON logs |

## Migration Path

We provide a detailed migration guide covering forwarder replacement, SPL query translation, alert migration, and security detection migration.

[View the full Splunk migration guide](/docs/migration/splunk/)

---

**Ready to switch from Splunk?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Migration Guide](/docs/migration/splunk/) - Step-by-step instructions
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community


---

# LogTide vs Sumo Logic for Log Management

> Compare LogTide and Sumo Logic for log management. Self-hosted vs cloud SaaS, pricing, data caps, and migration path.

*Source: https://logtide.dev/vs/sumo-logic/*

---

Sumo Logic is a cloud-native SaaS platform for log management, analytics, and security. LogTide is a self-hosted, open-source log management platform with built-in SIEM. Here's an honest comparison to help you evaluate both options for your log management needs.

## Cost Comparison

Sumo Logic is one of the more expensive SaaS log management platforms, with pricing based on daily ingestion volume and committed tiers.

### Sumo Logic Pricing

Sumo Logic charges based on daily data ingest tiers:

| Tier | Cost |
|------|------|
| Free | 500 MB/day, 7 days retention, limited features |
| Essentials | From $3.00/GB/day (annual commitment) |
| Enterprise Operations | Custom pricing (negotiated) |
| Enterprise Security | Custom pricing (includes Cloud SIEM) |
| Enterprise Suite | Custom pricing (full platform) |

Additional costs:

| Add-on | Cost |
|--------|------|
| Cloud SIEM | Additional licensing on top of base plan |
| Extended retention (90+ days) | Additional per-GB fees |
| Continuous Intelligence credits | Usage-based |

**Real-world example:** A mid-size company ingesting 50 GB/day:

- Essentials tier: 50 GB/day x $3.00 = $150/day
- Monthly cost: ~$4,500/month
- Extended retention (90 days): additional ~$500/month
- Cloud SIEM add-on: additional ~$1,000-2,000/month
- **Total: ~$5,000-7,000/month ($60,000-84,000/year)**

At 200 GB/day (Enterprise tier, negotiated):
- Estimated: ~$15,000-25,000/month
- **Total: ~$180,000-300,000/year**

### LogTide Pricing

LogTide is free, open-source software. You pay only for infrastructure:

| Component | Cost |
|-----------|------|
| Software license | Free (AGPLv3) |
| Cloud VM (4 CPU, 16 GB RAM) | ~$100-200/month |
| Storage (1 TB SSD) | ~$50-80/month |
| SIEM features | Included |
| Users | Unlimited |

**Same 50 GB/day scenario:**

- Infrastructure: ~$200-300/month
- **Total: ~$250/month ($3,000/year)**

**Savings: ~$57,000-81,000/year (95%+)**

At 200 GB/day:
- Infrastructure (8 CPU, 32 GB RAM, 4 TB): ~$400/month
- **Total: ~$400/month ($4,800/year)**

**Savings at scale: ~$175,000-295,000/year (97%+)**

## Feature Comparison

| Feature | Sumo Logic | LogTide |
|---------|-----------|---------|
| Log ingestion (HTTP) | Yes | Yes |
| Installed collectors | Yes | Fluent Bit |
| SDKs | Limited (API-based) | Yes (Node.js, Python, Go) |
| OpenTelemetry support | Yes | Native OTLP |
| Full-text search | Yes | Yes |
| Real-time streaming | Live Tail | SSE |
| Query language | Sumo Logic QL (pipe-based) | REST API + Full-text |
| Alert rules | Yes | Yes |
| Custom dashboards | Yes (extensive) | SIEM dashboard |
| LogReduce (clustering) | Yes | N/A |
| Anomaly detection | Yes | Roadmap |
| Cloud SIEM | Yes (add-on) | Built-in |
| Sigma detection rules | No | Built-in |
| MITRE ATT&CK mapping | Yes (SIEM add-on) | Included |
| Compliance certifications | SOC 2, FedRAMP, HIPAA, PCI | Self-managed |
| Self-hosted option | No | Yes |
| Open source | No | AGPLv3 |
| Data caps | Yes (daily limits, overage fees) | No |
| Multi-tenancy | Yes (partitions) | Yes (organizations/projects) |
| Data retention | Plan-dependent | Unlimited (storage-bound) |

## Where Sumo Logic Wins

**Enterprise compliance certifications.** Sumo Logic holds SOC 2 Type II, FedRAMP Moderate, HIPAA, PCI DSS, and ISO 27001 certifications. If your organization requires a vendor with pre-existing compliance certifications for audit purposes, Sumo Logic provides them. With LogTide you manage compliance yourself, which gives you full control but requires your own audit effort.

**Advanced analytics and ML.** Sumo Logic's query language supports powerful analytics operators: outlier detection, anomaly detection, time-series comparison, prediction, and machine learning operators. Their LogReduce feature automatically clusters similar log messages to surface patterns, which is valuable for investigating large volumes of unfamiliar logs.

**Zero infrastructure management.** As a fully managed SaaS platform, Sumo Logic handles deployment, scaling, high availability, disaster recovery, and security patching. Your team focuses entirely on analysis rather than operations.

**Pre-built applications.** Sumo Logic offers 300+ pre-built apps for AWS, Azure, GCP, Kubernetes, and popular SaaS tools. These apps provide instant dashboards, pre-configured alerts, and parsing rules without manual configuration.

**Unified platform.** Sumo Logic combines log management, infrastructure monitoring, application observability, and Cloud SIEM in a single platform. Cross-correlation between data types (logs, metrics, traces) enables faster root cause analysis.

**Global availability.** Sumo Logic operates data centers in North America, Europe, and Asia-Pacific, providing low-latency access for geographically distributed teams.

## Where LogTide Wins

**Massive cost savings.** Sumo Logic's per-GB pricing is among the highest in the industry. At 50 GB/day, LogTide saves $57,000-81,000 per year. At 200 GB/day, savings can exceed $200,000 per year. The more data you ingest, the more dramatic the cost advantage.

**No data caps or throttling.** Sumo Logic enforces daily ingestion limits tied to your plan. Exceeding your committed tier triggers overage charges or log throttling, which can cause you to lose critical data during peak traffic or incidents -- exactly when you need logs most. LogTide ingests as much as your infrastructure can handle with no artificial caps.

**Data sovereignty.** Sumo Logic stores your data in their cloud infrastructure. With LogTide, logs never leave your infrastructure. Run on-premises, in your own VPC, or in any cloud region for full control over data residency. This is essential for organizations subject to GDPR, CCPA, HIPAA, or industry-specific data regulations.

**Built-in SIEM at no extra cost.** Sumo Logic's Cloud SIEM is a separate add-on with additional licensing that can double your costs. LogTide includes Sigma detection rules, MITRE ATT&CK mapping, and incident management in the free open-source version. You get 2,000+ community Sigma rules from SigmaHQ out of the box.

**No vendor lock-in.** LogTide is AGPLv3 open-source. Your data lives in TimescaleDB or ClickHouse (your choice) using standard SQL. Your detection rules are portable Sigma YAML files. Nothing is proprietary -- you can fork, modify, or migrate at any time.

**Unlimited retention.** Sumo Logic's free tier retains logs for only 7 days. Even paid tiers have retention limits, and extended retention incurs additional fees. LogTide retains logs for as long as your storage allows -- months or years at just the cost of disk space. This is critical for compliance requirements that mandate long-term log retention.

**Transparent operations.** With LogTide, you have full visibility into how your logs are stored, indexed, and queried. No black boxes, no proprietary formats, no surprises on your invoice.

## When to Choose Sumo Logic

- You need enterprise compliance certifications out of the box (FedRAMP, SOC 2, HIPAA)
- Your team doesn't want to manage infrastructure at all
- You need advanced analytics with LogReduce, anomaly detection, and ML operators
- Log volume is very small (under 500 MB/day, covered by free tier)
- You require 300+ pre-built integrations with instant dashboards
- Budget is not a primary concern and you value time-to-value over cost

## When to Choose LogTide

- Log costs are a significant line item (10+ GB/day makes a meaningful difference)
- Data sovereignty or regulatory compliance requires self-hosting your logs
- You need SIEM capabilities without additional licensing costs
- You want to avoid daily data caps and ingestion throttling
- You need unlimited log retention for compliance or forensic analysis
- You prefer open-source with no vendor lock-in
- You're comfortable managing a Docker Compose deployment

## Migration Path

### Step 1: Ship Logs to Both

Run LogTide in parallel with Sumo Logic during evaluation. Add the LogTide SDK alongside existing Sumo Logic collectors:

```typescript
import { LogTideClient } from '@logtide/sdk-node';

const logtide = new LogTideClient({
  apiUrl: process.env.LOGTIDE_API_URL!,
  apiKey: process.env.LOGTIDE_API_KEY!,
});

// Your existing Sumo Logic installed collectors continue running
// LogTide receives the same events via its SDK
logtide.info('my-service', 'Payment processed', { amount: 99.00, currency: 'USD' });
logtide.warn('my-service', 'Rate limit approaching', { endpoint: '/api/v1/users', usage: '85%' });
logtide.error('my-service', 'Database connection failed', { host: 'db-primary', retries: 3 });
```

### Step 2: Forward via Fluent Bit

Use Fluent Bit to ship infrastructure logs to both LogTide and Sumo Logic:

```ini
[SERVICE]
    Flush 5
    Log_Level info

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag app.*
    Parser json

[INPUT]
    Name systemd
    Tag system.*
    Systemd_Filter _SYSTEMD_UNIT=myapp.service

# Send to LogTide
[OUTPUT]
    Name http
    Match *
    Host logtide.internal
    Port 8080
    URI /api/v1/ingest
    Format json
    Header X-API-Key lp_your_api_key

# Keep Sumo Logic output during migration
[OUTPUT]
    Name http
    Match *
    Host collectors.sumologic.com
    Port 443
    URI /receiver/v1/http/${SUMO_HTTP_SOURCE}
    Format json
    tls On
```

### Step 3: Validate and Cut Over

Compare LogTide and Sumo Logic results for 1-2 weeks:

1. Verify that search queries return matching results across both systems
2. Confirm alerts fire correctly in LogTide with equivalent thresholds
3. Test SIEM detection rules (Sigma) against your real production log data
4. Validate that critical dashboards can be replicated or replaced
5. Ensure your team is comfortable navigating the LogTide UI
6. Once validated, remove Sumo Logic collectors and cancel your subscription

## Query Migration (Sumo Logic QL to LogTide)

| Sumo Logic Query | LogTide API |
|-----------------|-------------|
| `_sourceCategory=app/api` | `GET /api/v1/logs?service=api` |
| `_sourceCategory=app/api error` | `GET /api/v1/logs?service=api&level=error` |
| `_sourceCategory=app/api "connection timeout"` | `GET /api/v1/logs?service=api&q=connection%20timeout` |
| `* \| count by _sourceHost` | `GET /api/v1/logs/aggregated?interval=1h` |
| `* \| where status_code >= 500` | `GET /api/v1/logs?level=error` |
| `_sourceCategory=app/api \| timeslice 5m` | `GET /api/v1/logs/aggregated?service=api&interval=5m` |

## Concept Mapping

| Sumo Logic | LogTide | Notes |
|------------|---------|-------|
| Organization | Organization | Top-level container |
| Installed Collector | Fluent Bit / SDK | Log collection agent or SDK |
| Hosted Collector | HTTP API | Cloud-hosted HTTP endpoint |
| Source | Service | Per-source identification |
| Source Category | Project | Logical grouping for data |
| Partition | Project | Data isolation and routing |
| Search query | REST API params | Query via API parameters |
| Scheduled Search | Alert Rule | Threshold-based alerts |
| Dashboard | SIEM Dashboard | Security-focused dashboards |
| Cloud SIEM | Sigma Rules + SIEM | Built-in, no extra license |
| Access Key | API Key (per project) | Prefix: `lp_` |
| LogReduce | N/A | Manual log pattern analysis |
| Field Extraction Rule | N/A (auto JSON) | Send structured JSON logs |
| Ingest Budget | N/A | No ingestion limits |

---

**Ready to cut your log management costs?**

- [Deploy LogTide](https://github.com/logtide-dev/logtide) - Free, open-source
- [Docker Integration](/integrations/docker/) - Quick start on any infrastructure
- [Join GitHub Discussions](https://github.com/logtide-dev/logtide/discussions) - Get help from the community



# Blog

---

# AWS Lambda Logging: How It Works and the Practices That Matter

> How AWS Lambda logging really works — CloudWatch log groups, JSON structured logs, retention, costs, and how to ship Lambda logs somewhere searchable.

*Source: https://logtide.dev/blog/aws-lambda-logging-best-practices/*

---

Lambda logging looks trivial — `console.log` and the output shows up in CloudWatch. Then the bill arrives, an incident happens, and you discover the logs you need are unstructured text spread across forty log streams, the debug lines that would have helped were never written, and the stack trace is split across twelve CloudWatch events.

This guide covers how Lambda logging actually works under the hood, the platform features most teams don't turn on, and the practices that make serverless logs useful during an incident instead of merely present.

## How Lambda logging actually works

There is no agent to install and no file to tail. The pipeline is built into the service:

```d2
direction: right
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

fn: "Your handler\nstdout / stderr" { class: src }
rt: "Lambda runtime\ncaptures & batches" { class: hub }
cw: "CloudWatch Logs\n/aws/lambda/<fn>" { class: src }
out1: "Subscription filter\n→ forwarder / Firehose" { class: dest }
out2: "Logs Insights\n(billed per GB scanned)" { class: dest }

fn -> rt { class: flow }
rt -> cw { class: flow }
cw -> out1 { class: flow }
cw -> out2 { class: flow }
```

The details that matter:

- **Everything on stdout/stderr is captured.** `console.log`, `print`, a stray `System.out.println` from a dependency — all of it becomes log events. You pay ingestion for every byte.
- **One log group per function** (`/aws/lambda/<function-name>`), **one log stream per execution environment**. A function running at concurrency 50 writes to 50 interleaved streams — which is why grepping a single stream during an incident misses most of the picture.
- **Log group retention defaults to "never expire."** Nobody notices until the storage line item grows for two years. Set a retention policy on day one.
- **The execution environment freezes between invocations.** Anything your code buffered in memory but hadn't sent yet stays frozen with it. This is the single most common cause of "missing" logs when teams ship logs over HTTP from inside the handler.

## Turn on JSON log format — it's free and most teams don't

Since late 2023, Lambda has native **structured JSON logging** (part of what AWS calls advanced logging controls). You enable it per function — no code change:

```bash
aws lambda update-function-configuration \
  --function-name order-processor \
  --logging-config LogFormat=JSON,ApplicationLogLevel=INFO,SystemLogLevel=WARN
```

Three things you get:

1. **Automatic JSON wrapping.** On Node.js and Python runtimes, the built-in `console.*` / `logging` calls are emitted as JSON objects with `timestamp`, `level`, and `requestId` attached. Multiline messages — including stack traces — arrive as a *single* log event instead of one event per line.
2. **Platform-level log filtering.** `ApplicationLogLevel` drops logs below the threshold *before* they reach CloudWatch, so `DEBUG` noise in production stops costing $0.50/GB. Change it in configuration, not in code.
3. **Quieter system logs.** `SystemLogLevel=WARN` suppresses the `START`/`END`/`REPORT` chatter for routine invocations.

If you take one thing from this article, take this: **JSON format plus a retention policy** fixes the two most common Lambda logging mistakes with zero code.

## Best practices that survive contact with production

### Log structured fields, not sentences

`"Order 4521 failed for customer 812 after retry"` can only be grepped. `{"msg": "order_failed", "orderId": 4521, "customerId": 812, "retries": 1}` can be filtered, aggregated, and alerted on. Every runtime has a near-zero-cost way to do this — `console.log` with an object on Node.js (with JSON format enabled), `logger.info(..., extra={...})` on Python, a JSON layout on Java.

### Correlate everything with the request ID

`context.awsRequestId` is the thread that ties an invocation's logs together across retries and services. Attach it to every log line (JSON format does this automatically) and propagate it to downstream calls so one ID follows the request through API Gateway, your function, and whatever the function calls.

### Don't log the event payload

`console.log(JSON.stringify(event))` on every invocation is the classic serverless anti-pattern: it duplicates your ingestion volume, and event payloads are exactly where PII lives (user emails in API Gateway bodies, records in Kinesis batches). Log the fields you need — IDs, types, sizes — not the object.

### Track cold starts explicitly

A module-scope boolean is all it takes, and it turns "the API is sometimes slow" from a mystery into a query:

```typescript
let coldStart = true;

export const handler = async (event: any, context: any) => {
  console.log({ msg: "invocation_start", coldStart, requestId: context.awsRequestId });
  coldStart = false;
  // ...
};
```

### Flush before the freeze

If you ship logs from inside the handler with any batching SDK or agent, the handler returning is a cliff edge: the environment freezes mid-buffer. Await a flush as the last step before returning — it's one HTTP round-trip, typically 10-50 ms in-region. (Runtime-specific patterns, including the Java/Log4j2 case, are covered in the guides linked below.)

### Mind the runtime-specific traps

- **Node.js**: with JSON format enabled, plain `console.log` is fine — you don't need a logging library for structure. Without it, use one (pino is the usual choice) or stack traces will fragment.
- **Python**: the stdlib `logging` module integrates with JSON format directly; resist the urge to `print()`.
- **Java**: Log4j2 inside Lambda has real sharp edges — the plugin cache gets destroyed by JAR shading and the Lambda appender silently vanishes. We wrote a dedicated guide: [AWS Lambda Java logging with Log4j2](/integrations/aws-lambda-java/).

## The part nobody budgets for: reading the logs

CloudWatch ingestion is $0.50/GB, storage $0.03/GB-month — and **every CloudWatch Logs Insights query costs $0.005 per GB scanned**. Debugging an incident across a month of logs for a busy function can cost more than the ingestion did, and the interactive experience (per-log-group selection, KQL-ish syntax, no live tail worth the name) is not where you want to be at 3 a.m.

That's why most teams eventually forward Lambda logs somewhere built for search. The standard options:

| Route | How | Good for |
|---|---|---|
| Subscription filter → forwarder | CloudWatch pushes matching events to a small forwarder function | No code changes, works for the whole fleet |
| Lambda extension | Telemetry API sidecar ships logs directly, can bypass CloudWatch | Cutting CloudWatch ingestion costs entirely |
| SDK from the handler | Structured logs with your own fields, sent directly | Richest data, new services |

All three patterns are covered step-by-step in our [AWS Lambda integration guide](/integrations/aws-lambda/). The SDK route, for reference, is a few lines with the [LogTide JavaScript SDK](/docs/sdks/nodejs/):

```typescript
import { hub } from '@logtide/core';

// Outside the handler — reused across warm invocations
hub.init({ dsn: process.env.LOGTIDE_DSN, service: 'order-processor' });

export const handler = async (event: any, context: any) => {
  hub.captureLog('info', 'invocation_start', { requestId: context.awsRequestId });

  const result = await processOrder(event);

  await hub.flush(); // before the environment freezes
  return result;
};
```

Self-hosted LogTide gives you field-based queries, live tail, and alerting on those logs with no per-GB ingestion fee and no per-query scan charge — the full comparison is in [LogTide vs CloudWatch](/vs/aws-cloudwatch/).

## Checklist

- [ ] JSON log format enabled (`LogFormat=JSON`)
- [ ] `ApplicationLogLevel` set per environment — `DEBUG` never ships to production
- [ ] Retention policy on every log group
- [ ] Request ID on every log line, propagated downstream
- [ ] Event payloads not logged wholesale
- [ ] Cold starts tracked
- [ ] Buffers flushed before the handler returns
- [ ] A search story that doesn't bill you per query

---

**Related guides:**

- [AWS Lambda integration](/integrations/aws-lambda/) — SDK, forwarding, and extension setup for LogTide
- [AWS Lambda Java logging with Log4j2](/integrations/aws-lambda-java/) — shading, appenders, and flush patterns for the JVM
- [LogTide vs AWS CloudWatch](/vs/aws-cloudwatch/) — feature and cost comparison


---

# Azure Monitor vs Azure Sentinel: What's the Difference?

> Azure Monitor and Microsoft Sentinel overlap more than their names suggest. What each one does, how the pricing stacks, and when you actually need both.

*Source: https://logtide.dev/blog/azure-monitor-vs-azure-sentinel/*

---

Searches for "Azure Monitor vs Azure Sentinel" usually come from a reasonable confusion: both products ingest logs, both live in the Azure portal, both query with KQL, and both can fire alerts. Microsoft's renaming of Azure Sentinel to **Microsoft Sentinel** didn't help.

The short answer: **they're not competitors — Sentinel runs on top of Azure Monitor's data layer.** The real question is whether you need the security layer, and what that layer costs. Let's break it down.

## What Azure Monitor actually is

Azure Monitor is Microsoft's **observability platform** — the umbrella for everything operational:

- **Metrics**: near-real-time numeric telemetry from Azure resources (CPU, memory, request rates)
- **Logs**: resource logs, activity logs, and application logs collected into **Log Analytics workspaces**
- **Application Insights**: APM — distributed traces, dependency maps, performance profiling
- **Alerts & dashboards**: threshold and KQL-based alert rules, workbooks, Grafana integration

Its job is answering operational questions: *Is the service healthy? Why is p95 latency up? What did this pod log before it crashed?*

The data backbone is the **Log Analytics workspace**. This matters because everything else in this article — including Sentinel — sits on that same backbone.

## What Microsoft Sentinel actually is

Sentinel is a **cloud-native SIEM and SOAR** platform. You enable it *on* a Log Analytics workspace, and it adds a security layer over data that's already there (plus security-specific connectors for Microsoft 365, Entra ID, firewalls, and third-party sources):

- **Analytics rules**: scheduled KQL detections and Microsoft-curated threat rules
- **Incidents**: correlation of alerts into cases with owners, severity, and investigation graphs
- **SOAR playbooks**: automated response via Logic Apps (disable a user, isolate a VM, open a ticket)
- **UEBA & threat intelligence**: behavioral baselines and IOC matching
- **Content hub**: prebuilt detections and workbooks per data source

Its job is answering security questions: *Is someone brute-forcing our VPN? Did this service principal just do something it never does? Which machines talked to that C2 domain?*

## The relationship in one picture

```d2
direction: down
style.fill: transparent

classes: {
  src:   { style: { fill: "#f5f3ff"; stroke: "#7c3aed"; stroke-width: 2; font-color: "#3b0764"; border-radius: 8; shadow: true } }
  hub:   { style: { fill: "#7c3aed"; stroke: "#6d28d9"; stroke-width: 2; font-color: "#ffffff"; border-radius: 8; shadow: true } }
  dest:  { style: { fill: "#f1f5f9"; stroke: "#94a3b8"; stroke-width: 1; font-color: "#475569"; border-radius: 6 } }
  group: { style: { fill: "#ede9fe"; stroke: "#c4b5fd"; stroke-width: 2; font-color: "#3b0764"; border-radius: 10 } }
  flow:  { style: { stroke: "#94a3b8"; stroke-width: 2; font-color: "#64748b" } }
}

azure: "Azure resources\napps, agents, App Insights" { class: src }
connectors: "M365 / Entra ID / firewalls\n(connectors)" { class: src }
law: "Log Analytics workspace\nshared data layer" { class: hub }
monitor: "Azure Monitor\nops alerts · dashboards · APM · autoscale" { class: src }
sentinel: "Microsoft Sentinel\ndetections · incidents · hunting · SOAR" { class: src }

azure -> law { class: flow }
connectors -> law { class: flow }
law -> monitor { class: flow }
law -> sentinel { class: flow }
```

Same data, two lenses. Which is exactly why the pricing model stings.

## Pricing: where the overlap gets expensive

Both products bill per GB on the **same ingested data**:

| Component | Cost (pay-as-you-go) |
|-----------|---------------------|
| Log Analytics ingestion | ~$2.76/GB |
| Microsoft Sentinel surcharge | ~$2.46/GB analyzed |
| **Security-relevant data, combined** | **~$5.22/GB** |
| Extra retention (beyond included) | ~$0.10/GB/month |

Commitment tiers discount both meters at volume (e.g. ~$1.96/GB for Log Analytics at 100 GB/day), but the structure stays the same: **security data is billed twice** — once to land in the workspace, once for Sentinel to analyze it.

A concrete example at 50 GB/day of security-relevant logs:

- Log Analytics: 1,500 GB × $1.96 (commitment) ≈ **$2,940/month**
- Sentinel: 1,500 GB × $2.46 ≈ **$3,690/month**
- **Total: ~$6,930/month (~$83,000/year)** before retention

For the full breakdown including the self-hosted comparison, see [LogTide vs Azure Monitor](/vs/azure-monitor/).

## So which one do you need?

**You need Azure Monitor** (or some observability stack) if you run anything on Azure, full stop. Operational telemetry isn't optional.

**You need Sentinel** if:

- You have a SOC team (or an MSSP) that will actually triage incidents and tune detections
- You're deep in the Microsoft security ecosystem (Defender, Entra ID, M365) — the native connectors and curated detections are genuinely good there
- Compliance requires a formal SIEM with incident management and audit trails

**You probably don't need Sentinel** if:

- Nobody would own the incident queue — an unwatched SIEM is an expensive log archive
- Your detection needs are log-based and rule-driven rather than UEBA/ML-driven
- The $2.46/GB surcharge forces you to *exclude* data sources from security analysis to control cost — which defeats the purpose

That last failure mode is common: teams route only a fraction of their logs into the Sentinel-enabled workspace to cap the bill, and attackers live in the logs that didn't make the cut.

## The third option: take log-based detection off the meter

If what you need is **detection on your logs** — failed logins, suspicious process activity, anomalous API usage, firewall events — rather than the full Microsoft security suite, a self-hosted platform changes the economics entirely.

[LogTide](/) includes SIEM capabilities in the box: [Sigma detection rules](https://sigmahq.io/) (the portable, community-driven detection standard), MITRE ATT&CK mapping, real-time alerting, and incident management — with **no per-GB surcharge**, because you run it on your own infrastructure. The same 50 GB/day scenario above runs on roughly **$400/month** of VM and storage, versus ~$6,930 for the Log Analytics + Sentinel pair — and you can afford to analyze *all* your logs, not just the ones that fit the budget.

It's not a full Sentinel replacement — there's no UEBA, no Logic Apps SOAR, no M365-native connectors. But for log-based detection with data sovereignty (your security telemetry never leaves your infrastructure), it covers what most teams outside a dedicated SOC actually use.

## Bottom line

| | Azure Monitor | Microsoft Sentinel | LogTide (self-hosted) |
|---|---|---|---|
| Category | Observability | SIEM / SOAR | Log management + SIEM |
| Primary question | Is it healthy? | Is it under attack? | Both, log-based |
| Data layer | Log Analytics | Log Analytics (same) | Your TimescaleDB/ClickHouse |
| Detection rules | KQL alerts (ops) | Analytics rules, UEBA, TI | Sigma rules, MITRE ATT&CK |
| Pricing | ~$2.76/GB | +$2.46/GB on top | Infrastructure only |
| Best for | Every Azure workload | Microsoft-centric SOCs | Cost-conscious, sovereignty-first teams |

Azure Monitor and Sentinel aren't an either/or — one is the data and ops layer, the other a security product on top of it. The decision that actually affects your budget is whether per-GB security pricing fits how much log data you want eyes on. If it doesn't, [self-hosted log management with built-in SIEM](/self-hosted-log-management/) is the lever that changes the equation.

---

**Related reading:**

- [LogTide vs Azure Monitor](/vs/azure-monitor/) — full feature and cost comparison
- [Cloud logging pricing breakdown](/cloud-logging-pricing/) — CloudWatch, GCP and Azure compared
- [Security monitoring use case](/use-cases/security-monitoring/) — Sigma rules and ATT&CK mapping in practice


---

# Complete Node.js Observability: Logs, Errors, and Traces with LogTide

> Instrument a Node.js service end to end with the LogTide SDK: structured logs, error capture, per-request scopes, breadcrumbs, and complete traces.

*Source: https://logtide.dev/blog/complete-nodejs-observability-logtide/*

---

One span per request tells you *that* a request was slow. It doesn't tell you *where* the time went, which downstream call failed, or what the payload looked like. Full observability means logs, errors, and traces that are all stitched to the same request — and the **LogTide JavaScript SDK** gives you that without changing how you write Node.

This post walks through instrumenting a Node.js service end to end: structured logs, error capture, per-request context, breadcrumbs, *complete* distributed traces, and request/response body capture — shipped to a LogTide backend you host yourself.

## What you get

The SDK is built around `@logtide/core`, with thin adapters per framework:

- **Leveled logging** — `debug`, `info`, `warn`, `error`, plus structured error capture
- **Hub / Scope model** — per-request isolation for tags, user, extras, and breadcrumbs
- **Distributed tracing** — root spans, child spans, OTLP export, and W3C `traceparent` in and out
- **Framework plugins** — Fastify, Express, Hono, Elysia, Next.js, Nuxt, SvelteKit, Angular: automatic request spans, error capture, and per-request scope with no wiring
- **Console + global-error integrations** — capture `console.*` and uncaught exceptions automatically
- **Reliable transport** — batching, retry with backoff, a circuit breaker, and a bounded buffer so logging never blocks or destabilises your app

## Install

```bash
# Core SDK
npm install @logtide/core

# Plus your framework adapter (includes @logtide/core as a dependency)
npm install @logtide/fastify    # or @logtide/express, @logtide/hono, …
```

## Initialize

For a web app, registering the framework plugin is all the setup you need — it calls `hub.init` for you:

```typescript
import Fastify from 'fastify';
import { logtide } from '@logtide/fastify';

const app = Fastify();

await app.register(logtide, {
  dsn: process.env.LOGTIDE_DSN,
  service: 'api',
  environment: process.env.NODE_ENV ?? 'development',
  release: process.env.GIT_SHA,
  tracesSampleRate: 1.0,
});
```

The plugin already installs `ConsoleIntegration` and `GlobalErrorIntegration` — **don't pass them again** in `integrations`. (Since v0.9.0 a duplicate is ignored anyway, but adding them was previously a common way to double-capture every `console.*` call.)

For non-HTTP code (workers, scripts, queues), initialize the hub once at startup:

```typescript
import { hub, ConsoleIntegration, GlobalErrorIntegration } from '@logtide/core';

hub.init({
  dsn: process.env.LOGTIDE_DSN,
  service: 'worker',
  integrations: [new ConsoleIntegration(), new GlobalErrorIntegration()],
});
```

## Logs and errors

```typescript
import { hub } from '@logtide/core';

hub.captureLog('info', 'User signed up', { userId: '123', plan: 'pro' });
hub.captureLog('warn', 'Rate limit approaching', { current: 90, max: 100 });

try {
  await chargeCard(order);
} catch (error) {
  hub.captureError(error, { orderId: order.id, amount: order.total });
}
```

`captureError` serializes the error with its message, stack frames, and cause chain.

## Per-request context

Framework adapters create an isolated scope per request and hang it off the request object. Tags, user, and extras you set on it are attached to every event captured during that request:

```typescript
app.addHook('preHandler', async (request) => {
  request.logtideScope.setTag('tenant', request.headers['x-tenant-id']);
  request.logtideScope.setExtra('userId', request.user?.id);
});
```

## Breadcrumbs

Breadcrumbs are the trail of events leading up to a log or error. Console output is captured automatically; you can add your own:

```typescript
hub.addBreadcrumb({
  type: 'navigation',
  category: 'checkout',
  message: 'Entered payment step',
  timestamp: Date.now(),
});
```

Since **v0.9.0**, when an event is captured with a scope, it carries *that scope's* breadcrumbs (request-local) rather than the global, process-wide buffer — so a single request's metadata no longer accumulates unrelated history from the rest of the app.

## Complete traces with child spans

This is the part most people miss. The plugin opens **one root span per request**. To see *where the time actually went*, nest the work under that span with child spans. `startChildSpan(name, scope, attributes)` parents a new span on the scope's `traceId` and `spanId`:

```typescript
import { startChildSpan, finishChildSpan } from '@logtide/core';

app.post('/checkout', async (request) => {
  const scope = request.logtideScope;

  // DB transaction as a child span
  const dbSpan = startChildSpan('checkout.db_tx', scope);
  const sale = await db.transaction().execute(/* … */);
  finishChildSpan(dbSpan.spanId, 'ok');

  // Upstream fiscal call as a sibling child span
  const fiscalSpan = startChildSpan('fiscal.create_receipt', scope, { saleId: sale.id });
  try {
    await fiscal.createReceipt(sale);
    finishChildSpan(fiscalSpan.spanId, 'ok');
  } catch (error) {
    finishChildSpan(fiscalSpan.spanId, 'error');
    throw error;
  }

  return { ok: true };
});
```

Both children share the request's `traceId` and nest under the HTTP span — one connected trace instead of disconnected events. For deeper nesting, clone the scope and set `childScope.spanId = span.spanId` before passing it to grandchild spans.

## Background jobs get their own trace

Code that runs outside a request (cron, queue workers) has no request scope, so it would otherwise collapse into one shared trace. Give each run its own:

```typescript
import { hub } from '@logtide/core';

async function processItem(item) {
  const client = hub.getClient();
  const scope = client.createScope();              // fresh traceId per run
  const root = client.startSpan({ name: 'queue.process_item', traceId: scope.traceId });
  scope.spanId = root.spanId;

  try {
    await doWork(item, scope);                      // pass scope for child spans
    client.finishSpan(root.spanId, 'ok');
  } catch (error) {
    client.finishSpan(root.spanId, 'error');
    client.captureError(error, {}, scope);
  }
}
```

## Capturing request and response bodies

On Fastify, opt in to body capture. Bodies are attached to the request span as `http.request_body` and `http.response_body`, truncated to 4096 characters:

```typescript
await app.register(logtide, {
  dsn: process.env.LOGTIDE_DSN,
  service: 'api',
  includeRequestBody: true,    // → span attribute  http.request_body
  includeResponseBody: true,   // → span attribute  http.response_body  (v0.9.0+)
  includeRequestHeaders: true, // sensitive headers are always stripped
});
```

Capture is **raw, with no redaction**, so enable it only on non-sensitive routes or behind your own gating logic.

## Distributed tracing across services

Framework adapters extract incoming `traceparent` headers and inject them into responses automatically. For outgoing calls you make yourself, build the header:

```typescript
import { createTraceparent, generateTraceId, generateSpanId } from '@logtide/core';

const header = createTraceparent(generateTraceId(), generateSpanId(), true);
await fetch('https://api.example.com/data', { headers: { traceparent: header } });
```

## Reliability and graceful shutdown

The transport batches entries, retries with backoff, trips a circuit breaker on repeated failures, and drops from a bounded buffer rather than growing unbounded — logging never blocks your request path. Flush on shutdown so nothing is lost:

```typescript
process.on('SIGTERM', async () => {
  await hub.flush();   // send buffered logs/spans
  await hub.close();   // tear down integrations + transport
  process.exit(0);
});
```

## The 100% checklist

- [x] Register the framework plugin with `dsn` + `service` (don't re-add the default integrations)
- [x] Set per-request context with `request.logtideScope` tags/extras
- [x] Wrap meaningful operations in `startChildSpan` / `finishChildSpan` for complete traces
- [x] Give background jobs their own scope + root span per run
- [x] Opt into `includeRequestBody` / `includeResponseBody` where useful (non-sensitive routes)
- [x] Propagate `traceparent` on outgoing requests
- [x] `flush()` + `close()` on `SIGTERM`

That's the full picture: every log, error, and span tied to the same request, on a backend you own. Point your `LOGTIDE_DSN` at your instance and you're done.


---

# Structured Logging and Error Tracking for Ruby and Rails

> Ship structured logs, exceptions, and traces from Ruby and Rails with the new LogTide Ruby SDK — a stdlib-only gem with Rack and Rails middleware.

*Source: https://logtide.dev/blog/structured-logging-ruby-rails/*

---

Ruby's built-in `Logger` is fine until you need to *find* something. Plain text lines don't carry the structure you want when an incident hits at 2am: which user, which request, which trace, what was the exception's cause chain. The **LogTide Ruby SDK** — the new `logtide` gem — adds that structure without changing how you write Ruby, and ships it to a log backend you host yourself.

This post walks through getting structured logs, exception capture, and distributed tracing out of a Ruby or Rails app, end to end.

## What you get

The `logtide` gem implements the LogTide SDK spec v1.0:

- **Leveled logging** — `debug`, `info`, `warn`, `error`, `critical`, plus `capture_exception`
- **Structured exception capture** with cause chains and stack frames
- **Hub / Scope model** — per-request isolation for tags, user, and breadcrumbs
- **Distributed tracing** — spans, OTLP export, and W3C `traceparent` in and out
- **Rack middleware + Rails Railtie** — request instrumentation with no wiring
- **stdlib `Logger` bridge** — a drop-in replacement for `Logger`
- **Reliability** — batching, exponential-backoff retry, a circuit breaker, and a bounded buffer

And it uses the **standard library only** — no runtime dependencies, and it requires Ruby 3.1+.

## Install

```ruby
# Gemfile
gem "logtide"
```

```bash
bundle install
```

## Plain Ruby in two minutes

Initialise once with your DSN, then log:

```ruby
require "logtide"

Logtide.init(
  dsn: "https://lp_your_key@logs.example.com",
  service: "checkout",
  environment: "production"
)

Logtide.info("order placed", { order_id: 42, amount: 19.99 })

begin
  charge_card!
rescue => e
  Logtide.capture_exception(e)
end

Logtide.flush
```

The second argument to any log method is a metadata hash — that's the "structured" part. Instead of interpolating values into a string, you attach them as fields you can later filter and aggregate on.

`capture_exception` is the one to reach for on errors: it serialises the exception, its cause chain, and (when `attach_stacktrace` is on, which it is by default) the stack frames — far more useful than `Logtide.error(e.message)`.

For short-lived scripts and rake tasks, call `Logtide.flush` before exit so buffered entries are delivered. A best-effort `at_exit` hook exists, but an explicit flush is the safe choice.

## Rails: the Railtie does the wiring

In Rails you don't touch Rack at all. Add an initializer and the Railtie installs the middleware for you:

```ruby
# config/initializers/logtide.rb
Logtide.init(
  dsn: ENV["LOGTIDE_DSN"],
  service: "my-app",
  environment: Rails.env,
  release: ENV["GIT_SHA"]
)
```

From here, every request gets an isolated scope, HTTP tags, inbound `traceparent` parsing, and request/response breadcrumbs — automatically.

### Enrich the request scope

Attach the current user and any request-specific tags once, and every log entry within that request carries them:

```ruby
class ApplicationController < ActionController::Base
  before_action do
    Logtide.configure_scope do |scope|
      scope.set_user(id: current_user&.id, email: current_user&.email)
      scope.set_tag("tenant", current_tenant.slug)
    end
  end
end
```

### Capture exceptions app-wide

```ruby
class ApplicationController < ActionController::Base
  rescue_from StandardError do |e|
    Logtide.capture_exception(e)
    raise
  end
end
```

### Keep your existing logger

If you'd rather not change call sites, swap the Rails logger for the bridge — a drop-in replacement for the stdlib `Logger`:

```ruby
Rails.logger = Logtide::LoggerBridge.new
```

`fatal` calls arrive as `critical` entries.

## Background jobs

The web middleware only covers HTTP requests, so wrap job execution in `with_scope` to keep each job isolated:

```ruby
class ProcessOrderJob < ApplicationJob
  def perform(order_id)
    Logtide.with_scope do |scope|
      scope.set_tag("job", self.class.name)
      Logtide.info("processing order", { order_id: order_id })
      # ... work ...
    end
  end
end
```

## Distributed tracing

Logs inside a span automatically carry its `trace_id` and `span_id`, so you can jump from a log line to the full trace. Create spans explicitly where it helps:

```ruby
Logtide.start_span("checkout", kind: :server) do |span|
  span.set_attribute("cart.size", 3)
  Logtide.info("processing") # carries the span's trace_id / span_id
end
```

To continue a trace across services, attach the propagation headers to your outbound call:

```ruby
headers = Logtide.trace_propagation_headers
# => { "traceparent" => "00-..." }

Net::HTTP.post(uri, body, headers)
```

The Rack middleware reads inbound `traceparent` headers automatically, so the other side of that call picks the trace back up with no extra code.

## Why self-hosted

Hosted log platforms price by the gigabyte, and Ruby apps under load generate a lot of gigabytes. LogTide is self-hosted: you run it on your own infrastructure and pay for that infrastructure, not for ingestion or retention. At meaningful volume that's a large saving, and your log data — which often contains user and request detail — never leaves systems you control. That matters for data residency and compliance as much as for cost.

## Next steps

- [Ruby SDK reference](/docs/sdks/ruby/) — every method and configuration option
- [Rails integration guide](/integrations/rails/) — the full Rails walkthrough
- [Self-hosted log management](/self-hosted-log-management/) — how LogTide is deployed