TLDR

kl-scan is a simple tool for searching for secrets in Kubernetes pod logs. This helps identify sensitive keys,tokens, or secrets that may have been inadvertently logged.

Based on my experience with Kubernetes security assessments, I have found that application logs frequently contain sensitive information that was never intended to be exposed. Debug output, verbose error messages, and misconfigured applications can inadvertently log API keys, authentication tokens, cloud credentials, and other secrets. While source code and container images are routinely scanned for secrets, runtime logs are often overlooked despite representing another common source of credential exposure.

To address this, I initially developed kl-scan as a subcommand for Bloodhound-Kube. However, as the tool evolved, it became clear that it would be better suited as a standalone Kubernetes log scanning utility. kl-scan retrieves Pod logs directly from the Kubernetes API and analyzes them using common secret detection techniques. Designed to scale across large clusters, it supports configurable batch processing and a watch mode for continuous monitoring.

The Problem

Reviewing logs for leaked secrets is manageable in small clusters but becomes increasingly difficult as environments scale. Production clusters often contain hundreds or thousands of Pods across multiple namespaces, with workloads constantly starting, stopping, and generating new logs.

Although centralized logging platforms provide search capabilities, they are not usually available or configured to support offensive security testing or comprehensive secret discovery during engagements. kl-scan is designed as a penetration testing utility that operates directly against the Kubernetes API, enabling testers to quickly enumerate and analyze Pod logs for exposed credentials without relying on additional infrastructure.

How It Works

The tool authenticates to the Kubernetes API, enumerates Pods, and retrieves container logs for analysis. To support large environments, log collection is performed in configurable batches, allowing concurrency to be tuned for the size of the cluster. Retrieved logs are processed locally using configurable detection engines, including regular expression matching, entropy-based analysis, and integrations with existing secret scanning tools. Matches include the affected namespace, Pod, container, and relevant log context to simplify investigation.

kl-scan -l run=kl-scan-test --since 5m --log info --detectors betterleaks,trufflehog

In addition to one-time scans, kl-scan includes a watch mode that periodically scans the cluster for newly generated logs. The scan interval and batch size are configurable, allowing organizations to balance detection of non periodic leaks with API utilization. This enables continuous monitoring without repeatedly processing the entire cluster.

 kl-scan --watch --watch-since 1m --watch-window 1m --log INFO

Detection Considerations

The tool uses the standard Kubernetes Logs API and performs authenticated, read-only requests. Organizations with Kubernetes Audit Logging enabled can monitor access to the pods/log endpoint using existing audit data, including the authenticated identity, source IP, and requested resources. As with any Kubernetes client, visibility is limited by the RBAC permissions granted to the authenticated user.

Future Work

Potential enhancements include:

• Additional detection engines
• Ignore and suppression rules
• Expanded reporting capabilities

Conclusion

Runtime logs are an often-overlooked source of credential exposure, particularly in large Kubernetes environments where manual review is impractical. kl-scan provides a simple, scalable approach to detecting secrets in Kubernetes Pod logs by combining Kubernetes-native log collection with common detection engines.