Loki is het log aggregation systeem van ASD-PMA, geoptimaliseerd voor efficiënte opslag en querying van logs.
| Eigenschap |
Waarde |
| Poort |
9214 |
| Container |
asd-loki |
| Collector |
Promtail |
| Health endpoint |
/ready |
| Categorie |
Logging |
| Classificatie |
Core |
flowchart TB
subgraph Sources["Log Sources"]
DOCKER[Docker Containers]
SYSLOG[Syslog]
FILES[Log Files]
end
subgraph Collection["Collection Layer"]
PROMTAIL[Promtail<br/>Log Collector]
end
subgraph Storage["Loki Stack"]
LOKI[asd-loki<br/>:9214]
CHUNKS[(Chunks Storage)]
INDEX[(Index)]
end
subgraph Query["Query Layer"]
GRAFANA[Grafana<br/>:9200]
API[Loki API]
end
DOCKER --> PROMTAIL
SYSLOG --> PROMTAIL
FILES --> PROMTAIL
PROMTAIL --> LOKI
LOKI --> CHUNKS
LOKI --> INDEX
GRAFANA --> LOKI
API --> LOKI
| Bron |
Labels |
Beschrijving |
| Docker |
container, compose_service |
Alle container stdout/stderr |
| Syslog |
host, facility |
System logs |
| App Logs |
app, level |
Application log files |
# Standaard labels
container: "asd-redmine"
compose_service: "redmine"
compose_project: "asd"
job: "docker"
LogQL is de query taal van Loki, gebaseerd op PromQL.
# Alle logs van een container
{container="asd-n8n"}
# Meerdere containers
{container=~"asd-n8n|asd-redmine"}
# Alle ASD containers
{container=~"asd-.*"}
# Exclusie
{container!="asd-postgres"}
# Bevat tekst (case sensitive)
{container="asd-n8n"} |= "error"
# Bevat niet
{container="asd-n8n"} != "healthcheck"
# Regex match
{container="asd-n8n"} |~ "error|warning"
# Case insensitive
{container="asd-n8n"} |~ "(?i)error"
# Parse JSON logs
{container="asd-n8n"} | json
# Filter op JSON veld
{container="asd-n8n"} | json | level="error"
# Specifieke velden extracten
{container="asd-n8n"} | json | line_format "{{.level}}: {{.message}}"
# Count logs per tijdseenheid
count_over_time({container="asd-n8n"}[5m])
# Rate van errors
rate({container="asd-n8n"} |= "error" [1m])
# Bytes per service
bytes_over_time({compose_service=~".+"}[1h])
# Top services by log volume
topk(5, sum by (compose_service) (bytes_over_time({compose_service=~".+"}[1h])))
# Sum per label
sum by (container) (count_over_time({container=~"asd-.*"} |= "error" [1h]))
# Average
avg(rate({container="asd-n8n"} [5m]))
# Top errors
topk(10, sum by (msg) (count_over_time({container="asd-n8n"} | json | level="error" [1h])))
¶ Command Reference
¶ Lifecycle Commands
# Start Loki stack
just start loki
# Stop Loki
just stop loki
# Status bekijken
just status loki
# Logs bekijken
just logs loki -f
# Restart
just restart loki
¶ Management Commands
# Service info
just info loki
# Health check
just health loki
# Backup logs
just backup loki
# Compact data
just loki-compact
¶ Query Commands
# Direct LogQL query
just loki-query '{container="asd-n8n"}' --limit=100
# Tail logs
just loki-tail '{container="asd-n8n"}'
# Export logs
just loki-export '{container="asd-n8n"}' --start="1h" --output=logs.json
Loki biedt een HTTP API voor queries en ingestion.
# Query logs
curl -s "http://localhost:9214/loki/api/v1/query_range" \
--data-urlencode 'query={container="asd-n8n"}' \
--data-urlencode 'start=1h' \
--data-urlencode 'limit=100' | jq .
# Labels ophalen
curl -s "http://localhost:9214/loki/api/v1/labels" | jq .
# Label values
curl -s "http://localhost:9214/loki/api/v1/label/container/values" | jq .
# Tail endpoint (WebSocket)
websocat "ws://localhost:9214/loki/api/v1/tail?query={container=~'.+'}"
# Push log entry
curl -X POST "http://localhost:9214/loki/api/v1/push" \
-H "Content-Type: application/json" \
-d '{
"streams": [{
"stream": {"app": "custom", "level": "info"},
"values": [["'$(date +%s)000000000'", "Custom log message"]]
}]
}'
# /etc/loki/config.yaml
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
shared_store: filesystem
filesystem:
directory: /loki/chunks
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h # 7 days
# /etc/promtail/config.yaml
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://asd-loki:3100/loki/api/v1/push
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
target_label: container
regex: '/(.*)'
- source_labels: ['__meta_docker_container_label_com_docker_compose_service']
target_label: compose_service
| Variable |
Beschrijving |
Default |
LOKI_RETENTION_PERIOD |
Log retention |
168h (7 days) |
LOKI_INGESTION_RATE_MB |
Max ingest rate |
4 |
LOKI_QUERY_TIMEOUT |
Query timeout |
5m |
| Volume |
Pad |
Inhoud |
loki_data |
/loki |
Chunks, indexes |
promtail_positions |
/tmp/positions.yaml |
Read positions |
# In loki config
limits_config:
retention_period: 168h # 7 days
compactor:
retention_enabled: true
retention_delete_delay: 2h
compaction_interval: 10m
¶ Handmatige Cleanup
# Compact en cleanup
just loki-compact
# Delete oude data
just loki-cleanup --older-than=7d
- Beperkt aantal labels - Max 10-15 labels
- Lage cardinality - Vermijd unieke values
- Statische labels - Geen timestamps of IDs
- Stream selectors first - Filter vroeg
- Time range beperken - Kleinere ranges zijn sneller
- Limit gebruiken - Niet meer ophalen dan nodig
- Caching - Herhaalde queries worden gecached
- SSD gebruiken - NVMe aanbevolen
- Retention instellen - Voorkom disk full
- Compaction - Regelmatig laten draaien