Troubleshooting

Find the section that matches what you see, then follow Symptom, Diagnosis, and Fix in order. Each section includes exact log keywords and API messages you can search for. Related configuration is covered in Operations.

Startup failures

Config validation rejected

Symptom: the process exits immediately and the only log line is config load failed, with err and config fields.

How to tell: the config is fully validated during load, and any failing check aborts startup. The err field is the reason. Common ones:

Fix: follow the message. One worth explaining is auth requires token_private_key, token_private_key_file, or server.data_dir for auto-managed Ed25519 keys, which means you need at least one of the three: inject an Ed25519 private key explicitly, or leave a data directory where Kiln can generate one.

Media key file rejected

Symptom: config load failed with an err starting with packager.keys_file:.

How to tell: the key file is validated in full at startup, line by line, and the first problem aborts the load. Messages carry the line number:

• line 3: expected kid:key: the line has no colon.
• line 3: empty kid or key: one side of the colon is blank.
• line 3: key must be 32 hex characters, got 30: both the KID and the key must be 32 hex characters.
• line 3: key must not contain dashes: dashes are allowed in the KID for grouping, but not in the key.
• line 3: kid is not hexadecimal: a non-hex character crept in.
• line 3: KID has a conflicting key: the same KID appears twice with different keys.
• no keys found: the file contains only blank lines and comments.

Fix: correct the file and restart. Keys are read once at startup, so edits do not take effect live. Relative paths resolve against the directory containing kiln.toml, not the working directory.

Port already in use

Symptom: listening is immediately followed by http server stopped with err set to listen tcp :8080: bind: address already in use, and the process exits.

How to tell and fix:

ss -ltnp | grep 8080

Either stop whatever holds the port or change server.listen, or override it temporarily with KILN_LISTEN. When running several instances on one host, remember that data_dir must be distinct as well, not just the port.

Data directory not writable

Symptom: create data dir failed or sqlite open failed.

How to tell: the systemd unit sets ProtectSystem=strict, which makes the whole filesystem read-only except for ReadWritePaths=/var/lib/kiln. Pointing data_dir anywhere else always fails.

Fix: move data_dir back under /var/lib/kiln and make sure it is owned by the kiln user. In a read-only container, mount a writable volume at the data directory.

Lite refused the config

Symptom: the lite variant exits at startup with a message naming an unavailable capability.

How to tell: lite rejects out-of-scope configuration outright rather than ignoring it:

• packager.engine must be native in lite
• channel "x" requires native packager engine
• epg is not available in lite
• OpenTelemetry export is not available in lite
• pprof is not available in lite

Fix: remove the offending section, or switch to the core image.

Playback returns 401 or 403

First confirm which credential you are using. Four credential types cover four surfaces, and mixing them up is the most common cause: public endpoints need none, session JWTs serve the admin UI, admin API tokens serve scripts, and path-style play keys serve player distribution. See Authentication.

Wrong credential type

Symptom: using an admin API token against /v1/play/..., or a session JWT against /p/<token>/....

Fix: playback accepts either a session JWT (on /v1/play/...) or a path-style play key (on /p/<token>/...). API tokens are for management endpoints, and some of those routes are marked session-only, which returns 403 API token permission denied for a token.

play_require_auth

Symptom: you disabled playback auth but the play endpoints still demand a credential.

How to tell: security.play_require_auth defaults to on when omitted. Confirm the config really carries play_require_auth = false, then confirm no stray KILN_PLAY_OPEN=0 survives in the environment, since the variable outranks the file.

Fix: in a debugging environment, set play_require_auth = false under [security], or set KILN_PLAY_OPEN=1.

Host not on the allow list

Symptom: after moving to a new domain or adding a reverse proxy, every endpoint returns 403 host not allowed.

How to tell: when security.public_hosts is set, middleware compares the request Host (port and brackets stripped) against each entry and rejects anything that does not match. A literal * allows everything. Loopback requests to /healthz and /readyz are exempt, so the telltale sign is a local probe returning 200 while every external request returns 403.

Fix: add the public hostname to public_hosts. The reverse proxy must forward Host unchanged; rewriting it to the backend address produces the same 403.

A channel will not start

Upstream unreachable

Symptom: playback returns 502 with upstream request failed or upstream status 403 Forbidden: .... The logs show session restarting with attempt, delay, and err; after 8 attempts it becomes session restart budget exceeded and the session goes to failed.

How to tell: restart backoff starts at 2 seconds and caps at 30, and the counter resets after 90 seconds of stability. Repeated restarts mean the upstream really is not serving content, not that you hit a blip.

Fix: verify reachability from the server itself, including the channel’s configured headers and user_agent. If the upstream needs a proxy, see Proxies do not apply below.

Upstream host not allowlisted

Symptom: 403 with upstream host not allowed, and an err like private host not allowlisted: 192.168.1.10.

How to tell: outbound requests go through an SSRF check. Public hosts pass by default, but loopback and private-range addresses pass only when explicitly listed in security.allowed_hosts; cloud metadata addresses (169.254.169.254, for example) and link-local addresses are always rejected. Declaring a host under upstreams or channels does not grant a private-network exemption.

Fix: add the LAN origin’s hostname or IP to security.allowed_hosts and restart. Upstreams and channels do not enter this list automatically, whether they come from the config file or the admin UI.

The engine cannot handle this source

Symptom: 502 with engine=native but the source cannot be served natively.

How to tell: engine = "native" forbids fallback. When the native engine decides it cannot handle a source it fails outright instead of reaching for ffmpeg. The reason surfaces as fallback_reason: multi_period (multi-period MPD), addressing_unsupported, manifest_codec_unsupported, no_video_representation, no_audio_representation, missing_key_for_kid, or manifest_kid_conflicts_with_tenc.

Fix: set that channel’s packager to auto so it can fall back to the compatibility engine, which requires running the full variant with ffmpeg. Otherwise switch to a source the native engine handles.

A related message is native cannot handle this source and ffmpeg would decode it incorrectly. In that case even auto will not fall back, because the compatibility engine’s output would be wrong. Those sources have to be replaced.

ffmpeg unavailable

Symptom: 503 with ffmpeg compatibility engine is not available, or, when a fallback was needed, source requires ffmpeg compatibility, but the ffmpeg engine is not available. /readyz returns 503 with code not_ready at the same time.

How to tell: two startup-log fields answer this directly:

packager_engine=auto ffmpeg_available=false

The dependency is probed once at startup. When it is missing, the log carries ffmpeg compatibility engine unavailable with dependency and err fields, where err reads like find ffmpeg dependency "ffmpeg": exec: "ffmpeg": executable file not found in $PATH.

Fix:

• The core and lite images do not ship ffmpeg. Use full if you need compatibility fallback.
• Windows builds do not ship ffmpeg either. Install it and add it to PATH.
• If [ffmpeg].binary points at a path that does not exist, use an absolute path or make sure the binary is on PATH.
• With [ffmpeg].mode = "docker", the probe targets the docker command itself. A host without a Docker client reports the engine as unavailable.

Missing decryption keys

Symptom: 400 with no global keys configured, or a startup failure with channel "x" dash ingress requires global packager.keys_file.

Fix: configure [packager].keys_file with one kid:key pair per line. The per-channel keys field has been removed; keys are global only.

Playback stalls mid-stream

Symptom: the channel starts fine, then freezes after a few minutes. The logs show a session restart with err set to no segment published for 3m0s while the manifest kept updating.

How to tell: this is the packager.stall_timeout_sec detector, 180 seconds by default. Its trigger is specific: the upstream manifest is still refreshing normally, and no track has produced a segment within the timeout window. Both conditions must hold.

• A manifest that cannot be fetched at all (unreachable upstream) is not a stall. That path produces the upstream errors above.
• As long as any single track is still advancing, it is not a stall.

Once it fires, the session is treated as fatally failed and restarts, which re-resolves the manifest and re-selects tracks.

Fix: an isolated occurrence usually means the upstream encoder hiccuped and the restart already recovered it, so no tuning is needed. Frequent hits mean the upstream keeps updating its manifest without publishing media, which has to be chased upstream. Raising stall_timeout_sec only delays recovery; it does not fix the cause.

Memory higher than expected

Symptom: container RSS above what you budgeted, or the OOM killer stepping in.

How to tell: confirm the profile and budgets that actually took effect from the startup log:

resource_profile=standard resource_constrained=true
effective_memory_mb=768 memory_limit_mb=192 effective_go_memory_limit_mb=192
inflight_mb=64 max_segment_mb=32 gc_percent=100 drop_file_cache=true

Compare that against the profile table in Operations. Three common misreadings:

• These numbers bound the Go heap and the media working set. They are not a guarantee about total container RSS. SQLite, goroutine stacks, and kernel page cache all sit outside.
• When the full image falls back to the compatibility engine it spawns an FFmpeg subprocess whose memory is entirely outside the Go soft limit. The FFmpeg memory is outside the Go soft limit warning exists to say so, with an ffmpeg_scope field distinguishing a subprocess from an external container. It is advisory only (advisory_only=true) and does not affect operation.
• If GOMEMLIMIT is set, server.memory_limit_mb is no longer written to the Go soft target.

Fix:

• Lower packager.inflight_bytes. It is the shared cross-channel segment memory budget, so shrinking it lowers peak memory at the cost of slower cold starts on high-bitrate sources.
• Run fewer concurrent channels, or reduce start_segments and prefetch_segments.
• When you need a predictable per-process memory boundary, run the native engine in core or lite so no FFmpeg subprocess is involved.
• To validate low-resource behavior, force the tightest profile with KILN_RESOURCE_MODE=constrained.

EPG will not refresh

Symptom: the guide is empty or never updates.

How to tell: check three things in order.

Fix: the disk cache defaults to data_dir/epg. It is fully rebuildable, so if you suspect corruption, delete the directory and restart. The refresh interval comes from epg.refresh_interval_min, 360 minutes by default; lowering it means watching out for upstream rate limits. See EPG.

Proxies do not apply

Symptom: a proxy is configured but traffic still goes direct, or a proxy-related error appears.

The rule never matched

How to tell: routing walks the rules in ascending priority order, takes the first match, and falls back to egress.default when nothing matches. A lower priority number is evaluated first. Five rule kinds exist: host_exact, host_suffix (the default), host_regex, channel_id, and url_regex.

host_suffix matches the hostname itself or any hostname ending in .pattern, not a plain substring. A pattern of example.com matches edge.example.com but not notexample.com.

Fix: lower the priority of the more specific rule. Remember that host rules match the hostname, not the full URL; use url_regex for the latter.

The profile is missing or disabled

How to tell: an active rule that names a missing or disabled profile now fails both configuration loading and hot reload instead of silently becoming direct. The error identifies an unknown or disabled proxy; an unknown egress.default also prevents startup.

Fix: check that the id under [[proxies]] matches exactly and that the profile is not disabled.

FFmpeg container cannot reach Kiln

Symptom: under [ffmpeg].mode = "docker", the FFmpeg container cannot reach the short-lived guarded proxy started by Kiln.

How to tell: Kiln advertises the guarded proxy to the FFmpeg container through egress.docker_proxy_host (default host.docker.internal) and automatically adds the default host-gateway mapping. Kiln itself connects to any configured upstream proxy, so those credentials are never handed directly to FFmpeg.

Fix: make sure docker_proxy_host reaches the Kiln process from the FFmpeg container. On a non-standard Docker network, set it to a reachable host or Kiln network address.

The full routing model is documented in Egress proxies.

Container quirks

Detection picks the wrong profile

Symptom: resource_profile in the startup log is not what you expected, or reads configured.

Diagnosis: configured means the configured values remained in effect. This is expected when resource_mode = "performance" disables adaptation. In auto mode, it means memory detection found no usable limit. Detection reads cgroup v2 memory.max, cgroup v1 memory.limit_in_bytes, and MemTotal from /proc/meminfo, then uses the smallest positive value. Nested cgroups, unusual mount layouts, or an unset memory limit can produce an unexpected result.

Fix: override detection directly and skip the guesswork:

docker run --rm --cpus=1 --memory=192m --memory-swap=192m \
  -e KILN_RESOURCE_MEMORY_MB=192 -e KILN_RESOURCE_CPUS=1 \
  -v "$PWD/kiln.toml:/etc/kiln/kiln.toml:ro" \
  ghcr.io/babywbx/kiln:core

After that, effective_memory_mb and effective_cpu_milli reflect exactly what you supplied. The verification procedure is in Operations.

Read-only mounts

Symptom: running with --read-only fails at startup with create data dir failed.

How to tell: even lite writes an auto-generated login key and temporary media files. A read-only root filesystem needs a writable volume mounted at the data directory.

Fix:

docker run --rm -p 8080:8080 --read-only \
  --cap-drop=ALL --security-opt=no-new-privileges \
  -v "$PWD/kiln.toml:/etc/kiln/kiln.toml:ro" \
  -v kiln-lite-data:/var/lib/kiln \
  ghcr.io/babywbx/kiln:lite

Unknown runtime variant

Symptom: an unknown runtime variant; using standalone warning with an environment field.

How to tell: the core and full images set KILN_RUNTIME_VARIANT themselves, so this warning only appears when it was set manually to an unrecognized value. The runtime_variant field in the startup log is the value that took effect.

Fix: do not set the variable by hand unless you are building your own image.

Health check never passes

How to tell: core and full probe /healthz with wget. lite is built FROM scratch with no shell and no wget, so it uses the binary’s own subcommand. Verify manually:

docker exec kiln wget -q -O /dev/null http://127.0.0.1:8080/healthz
docker exec kiln kiln -healthcheck http://127.0.0.1:8080/healthz

If /healthz passes but /readyz does not, the compatibility engine is missing. Go back to ffmpeg unavailable.

Gathering diagnostics

Where the logs are

Raising verbosity

The default level is info, which demotes /healthz, /readyz, /, and any path containing /live/ or /u/ to debug so they never appear. To see the full segment-request trail, raise the level temporarily:

KILN_LOG_LEVEL=debug KILN_LOG_FORMAT=json kiln -config /etc/kiln/kiln.toml

Under systemd use systemctl edit kiln and add Environment=KILN_LOG_LEVEL=debug; under Docker just pass -e KILN_LOG_LEVEL=debug. Turn it back down when you are done, because debug is very loud under load.

Color is disabled automatically when output is redirected to a file. To strip ANSI sequences while still attached to a terminal, use KILN_LOG_COLOR=never or set NO_COLOR.

Collecting a pprof profile

For memory or CPU anomalies, enable pprof briefly. The listener must be a loopback IP, and you should turn it off as soon as you have your profiles.

[debug.pprof]
enabled = true
listen = "127.0.0.1:6060"

go tool pprof http://127.0.0.1:6060/debug/pprof/heap
go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30
go tool pprof http://127.0.0.1:6060/debug/pprof/block
go tool pprof http://127.0.0.1:6060/debug/pprof/mutex

Which metrics to read first

In /metrics, the rates of kiln_packager_segment_fetch_errors_total and kiln_packager_manifest_errors_total say the most about upstream health. kiln_packager_cache_bytes shows current segment cache usage, and the state label on kiln_session_info shows at a glance which sessions have gone to failed.

What to include in a bug report

• The full output of kiln -version.
• The entire kiln starting line from the startup logs.
• Reproduction steps and the log excerpt covering that window. Strip play tokens and upstream credentials yourself. Kiln already truncates tokens in /p/ paths to a prefix in its own logs, but config snippets and URLs you paste are not sanitized for you.

Issues are tracked on GitHub.