Operations

Use this guide for day-to-day operation after Kiln is running: register a system service, back up data, upgrade safely, and configure logs, health checks, metrics, tracing, and resource adaptation.

Running as a service

systemd

Passing --service to the install script registers a systemd unit and enables it at boot. This step needs root and only works on Linux with systemd available.

curl -fsSL https://raw.githubusercontent.com/babywbx/Kiln/main/install.sh -o /tmp/kiln-install.sh
sudo sh /tmp/kiln-install.sh --yes --service

The script provisions a dedicated system account first. If the kiln user does not exist, it creates one with useradd -r -U using a non-login shell (nologin or /bin/false) and /var/lib/kiln as its home, then creates /etc/kiln and /var/lib/kiln and chowns the latter to that account. The generated unit:

[Unit]
Description=Kiln
After=network-online.target
Wants=network-online.target

[Service]
User=kiln
Group=kiln
ExecStart=/usr/local/bin/kiln -config /etc/kiln/kiln.toml
WorkingDirectory=/var/lib/kiln
Restart=on-failure
RestartSec=3
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/kiln

[Install]
WantedBy=multi-user.target

Four directives carry the hardening. NoNewPrivileges=true closes off privilege escalation. ProtectSystem=strict makes the entire filesystem read-only to the process, with ReadWritePaths=/var/lib/kiln carving out the single writable directory. ProtectHome=true hides every home directory. PrivateTmp=true gives the process its own temp namespace. Because of ProtectSystem=strict, data_dir has to live under /var/lib/kiln. Pointing it anywhere else fails at startup.

ExecStart hardcodes the absolute path to the config file. As its last step the script enables and starts the unit if /etc/kiln/kiln.toml already exists; otherwise it installs the unit, leaves it disabled, and tells you to create the config first. If you configured Kiln after installing, one command closes the gap:

sudo systemctl enable --now kiln
sudo systemctl status kiln
sudo journalctl -u kiln -f

To remove the service:

sudo systemctl disable --now kiln
sudo rm /etc/systemd/system/kiln.service
sudo systemctl daemon-reload

The install script’s --uninstall also stops and removes the service, but it deliberately keeps the config under /etc/kiln and the data under /var/lib/kiln.

Windows service

On Windows the built-in kiln.exe service subcommands register the service, so no external supervisor is needed. Installation, removal, and log locations are covered in Binary deployment.

Data directory and backups

server.data_dir defaults to ./data and relative paths resolve against the process working directory. The systemd unit sets WorkingDirectory=/var/lib/kiln, and the Windows service switches its working directory to the config file’s directory, so ./data lands somewhere predictable in both cases. The directory itself is created with mode 0750.

The simplest reliable backup is to stop Kiln and copy the entire data directory. For an online backup, use the SQLite Backup API or a snapshot tool with equivalent consistency guarantees. Copying kiln.db and kiln.db-wal separately does not guarantee a point-in-time snapshot. You can skip sessions/; Kiln recreates it when a new session starts.

The media decryption keys in packager.keys_file live outside data_dir. Relative paths there resolve against the directory containing kiln.toml, so back that file up separately.

Upgrading

curl -fsSL https://raw.githubusercontent.com/babywbx/Kiln/main/install.sh | sh

sudo systemctl restart kiln

kiln -version

docker pull ghcr.io/babywbx/kiln:latest
docker compose up -d

Database migrations run automatically at startup, with no manual step. Kiln keeps a schema_version table in the database, reads the current version on boot, and applies every missing migration in order until it catches up with the version the binary supports. The whole sequence runs inside one transaction, so a failure rolls back completely and the process exits with sqlite open failed.

Upgrading across several releases uses the same migration chain. You do not need to step through intermediate versions; drop in the newest binary and go.

Logging

Logging is driven by three fields under [logging]. The environment variables take precedence, which makes them convenient for one-off changes in containers.

Level parsing accepts a few aliases: dbg and trace map to debug, warning and wrn map to warn, and err, erro, and fatal map to error. Anything unrecognized falls back to info. Format only has two outcomes: structured is an alias for json, and everything else is treated as text.

Coloring applies to text only. auto emits ANSI sequences solely when the output is a character device, so redirecting to a file or pipe disables it automatically. On top of that, a non-empty NO_COLOR environment variable turns auto off unconditionally. To disable color while still attached to a terminal, set KILN_LOG_COLOR=never.

Under json, every record carries a service=kiln field, which makes filtering straightforward in a central log system.

Access-log severity is derived from the response: 5xx logs at error, 4xx at warn, everything else at info. High-frequency paths (/healthz, /readyz, /, and anything containing /live/ or /u/) are demoted to debug so they do not flood the default level. Raise the level to debug when you need the full segment-request trail.

Play tokens never reach the log verbatim. Paths shaped like /p/<token>/... are rewritten to /p/<prefix>…/<suffix> before being logged or written to the access audit table, keeping just enough of the token to correlate requests.

Where the logs land depends on the deployment:

• systemd: stdout, read with journalctl -u kiln.
• Docker: docker logs kiln.
• Windows service: the SCM discards stdout, so the process writes kiln.log in the config file’s directory. Once it exceeds 16 MB it is renamed to kiln.log.1 on the next start, keeping one generation.
• Foreground: straight to the terminal.

Health checks

Neither endpoint requires credentials, and they mean different things. Do not use them interchangeably.

When security.public_hosts is set, requests whose Host header is not on the list are rejected with 403 host not allowed. So that probes are not caught by this rule, /healthz and /readyz requests originating from a loopback address are explicitly exempt. If you probe from another machine, add that hostname or IP to public_hosts.

The binary ships a health-check subcommand with a 3-second timeout. It exits with code 0 for a 2xx response and code 1 otherwise:

kiln -healthcheck http://127.0.0.1:8080/healthz

The images already declare HEALTHCHECK. core and full are Alpine-based and probe /healthz with wget; lite is built FROM scratch and has neither a shell nor wget, so it uses the subcommand above.

Metrics

GET /metrics emits Prometheus text format (text/plain; version=0.0.4). Process-level series are kiln_uptime_seconds, kiln_bytes_in_total, kiln_bytes_out_total, kiln_http_requests_total, kiln_errors_total, kiln_goroutines, and kiln_sessions.

Each active session adds a kiln_session_info sample labeled with channel, engine, and state. Packager statistics are labeled by channel and cover counters such as kiln_packager_segments_published_total, kiln_packager_segment_fetch_errors_total, kiln_packager_manifest_errors_total, and kiln_packager_key_mismatches_total, plus gauges like kiln_packager_cache_bytes and kiln_packager_clock_offset_seconds. When chasing upstream flakiness, look at the rate of segment_fetch_errors and manifest_errors first.

The endpoint is gated on [observe].enabled. Omitting the key means on, so core and full serve /metrics by default; an explicit false makes the route return 404. lite does not register the route at all.

OTLP tracing

The exporter is only initialized when [observe].otlp_endpoint is set and [observe].enabled has not been explicitly turned off. Leave either out of the picture and tracing costs nothing.

[observe]
otlp_endpoint = "https://collector.example.com/v1/traces"
otlp_insecure = false
trace_sample_ratio = 0.1
service_name = "kiln"

Export is OTLP over HTTP with batching. Set otlp_insecure = true for a plaintext collector on a trusted network. The sampler is ParentBased(TraceIDRatioBased): an upstream sampling decision is honored when present, otherwise trace_sample_ratio applies. A ratio at or below 0, or above 1, is treated as 1, meaning sample everything. An empty service_name becomes kiln, and the resource attributes also carry the build version.

Context propagation uses W3C traceparent plus baggage, and inbound headers are extracted so traces continue across the hop.

A failed exporter setup does not take the process down: it logs a single OpenTelemetry setup failed warning and continues without tracing. The lite variant refuses to start when otlp_endpoint appears in its config rather than ignoring it silently.

pprof diagnostics

pprof is off by default. Turn it on only while investigating a memory or CPU problem, and turn it back off when you are done.

[debug.pprof]
enabled = true
listen = "127.0.0.1:6060"

go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30
go tool pprof http://127.0.0.1:6060/debug/pprof/heap
go tool pprof http://127.0.0.1:6060/debug/pprof/block
go tool pprof http://127.0.0.1:6060/debug/pprof/mutex

lite does not include pprof; [debug.pprof].enabled = true makes it refuse to start.

Resource adaptation

At startup Kiln detects the memory and CPU available to it and scales the memory-related budgets down accordingly, so one config file works on a 256 MB box and a many-core server alike.

Three modes

server.resource_mode accepts exactly three values:

Memory profiles

Under auto, effective memory selects one of four profiles. The resource_profile field in the startup log is the profile name:

For balanced and standard the EPG per-source cap is derived as one 128th of effective memory, clamped between 4 MiB and 64 MiB. A 768 MB container works out to 6 MiB, which is exactly the epg_max_source_mb=6 you see in the startup log.

The first three profiles also flip one extra switch: after writing and reading media files, Kiln advises the kernel to drop the corresponding page cache (drop_file_cache=true in the startup log), so the container’s memory accounting is not inflated by cache. large leaves this off.

CPU caps

CPU is evaluated separately from memory and only affects pipeline depth and EPG refresh concurrency:

• Below 4000 effective milli-CPU, the pipeline cap is ceil(milli / 1000). At 4 cores and above no CPU cap is applied.
• EPG refresh concurrency is the smaller of ceil(milli / 2000) and the memory size in GiB rounded to nearest, with a floor of 1.
• The memory profile and the CPU cap each produce an upper bound; the final value is the minimum of the configured value, the profile bound, and the CPU bound.

Detection handles cgroup v1 and v2, nested cgroups, limits inherited from a parent cgroup, and fractional CPU quotas. A container given --cpus=1.5 reports effective_cpus=2 and effective_cpu_milli=1500.

The Lite fixed budget

The lite variant does not participate in profile selection. Under both auto and constrained it always uses a 24 MiB Go soft target, 24 MiB inflight, a 20 MiB max segment, a 1/1 pipeline, and GOGC=50, which keeps its memory footprint consistent across hosts. Only performance opts out.

Overrides and precedence

Verifying from the startup log

The kiln starting record prints both the detection results and every budget that took effect, which is the fastest way to confirm a profile:

resource_mode=auto resource_profile=compact resource_constrained=true
effective_cpus=1 effective_cpu_milli=1000 effective_memory_mb=192
memory_limit_mb=48 effective_go_memory_limit_mb=48
inflight_mb=32 max_segment_mb=20 gc_percent=75 drop_file_cache=true
start_segments=1 prefetch_segments=1
epg_refresh_concurrency=1 epg_max_source_mb=4

effective_memory_mb is the detected limit, memory_limit_mb is the Go soft target Kiln set, and effective_go_memory_limit_mb is the value the runtime applies. If they differ, check GOMEMLIMIT first. A resource_profile of configured means the configured values remained in effect, either because resource_mode = "performance" disabled adaptation or because automatic memory detection found no usable limit. In the latter case, set KILN_RESOURCE_MEMORY_MB explicitly.

Reproducing a profile locally

deploy/docker/resource-smoke.toml is a minimal config; combined with Docker resource limits it reproduces any profile:

docker run --rm --cpus=1 --memory=192m --memory-swap=192m \
  -v "$PWD/deploy/docker/resource-smoke.toml:/etc/kiln/kiln.toml:ro" \
  kiln:core

--cpus=2 --memory=384m yields balanced, --cpus=2 --memory=768m yields standard, and --cpus=4 --memory=1g yields large. Adding -e KILN_RESOURCE_MODE=constrained exercises the forced low-resource path on a large machine.

Next steps