Observability in EKS with Fluent Bit

In this article we explore why it matters to treat logs as data streams rather than static files in the context of modern applications. Logs are essential for gaining visibility into application behavior, detecting and resolving problems, and operating production platforms with confidence.

We’ll see how STDOUT and STDERR work in Linux — especially in container environments — and how Fluent Bit is configured to collect, transform, and ship records to destinations like Elasticsearch or AWS CloudWatch.

Logs as streams

In modern applications, the recommendation is to treat logs as a data stream, not as static files. Logs are generated continuously and transmitted to a central location where they can be collected, analyzed, and stored. In containers, applications send logs to STDOUT and STDERR.

To understand the pattern, it helps to recall that in Linux every process has three standard file descriptors:

STDIN — standard input, from which a process reads its input.
STDOUT — standard output, where a process writes its output.
STDERR — standard error output, where a process writes its error messages.

In a container environment, each container has its own STDOUT and STDERR. The application is coded so that the logs it generates are sent to STDOUT or STDERR. Often this is achieved by configuring the log driver of the development framework. If the application can’t be modified, an alternative is to create a symbolic link between the log files and /dev/stdout or /dev/stderr. This technique appears, for example, in the official nginx Dockerfile.

By managing logs as streams, we can take advantage of various strategies to collect, transform, and route them. This approach frees applications from the responsibility of determining the final storage destination, encouraging modular applications in line with microservices architecture.

Fluent Bit and EKS

Fluent Bit is a lightweight log processor and router. It lets you take the logs generated by containers, transform them, and deliver them to their final destination.

In the context of Amazon EKS, the default location for container logs is the /var/log/containers directory. The logs for each container are stored in a separate file with a name that includes the container ID and a timestamp.

Fluent Bit can be configured to read logs from this directory, enrich them with Kubernetes metadata (namespace, pod, labels), and deliver them to different destinations.

Installing Fluent Bit on EKS

We’re going to install and configure Fluent Bit on Amazon EKS using Helm.

First, we add the Fluent Bit Helm repository:

helm repo add fluent https://fluent.github.io/helm-charts

Then we install Fluent Bit:

helm upgrade --install fluent-bit fluent/fluent-bit

This command starts the Fluent Bit installation process on the EKS cluster. Helm lets you customize the configuration through a custom values file, which is useful for overriding the chart’s default values and adapting them to specific needs.

Routing logs to OpenSearch

A very common task is configuring where we want to route the logs. For example, we can route them to an Amazon OpenSearch cluster. To do that we create a values.yaml file that overrides the chart’s default values.yaml:

config:
  outputs: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${ES_ENDPOINT}
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ${AWS_REGION}
        Retry_Limit     6

Once the file is created, we run:

helm upgrade --install fluent-bit fluent/fluent-bit -f /ruta/al/values.yaml

With this configuration, Fluent Bit routes the container logs to the Amazon OpenSearch cluster, enabling centralized log search and analysis.

Other output options

Fluent Bit supports many destinations common in AWS architectures:

Amazon CloudWatch Logs — the cloudwatch_logs output
Amazon Kinesis Data Firehose — for streaming to S3, Redshift, or custom destinations
Amazon S3 — for low-cost long-term archiving
OpenSearch / Elasticsearch — for search and analysis (shown above)
Datadog, New Relic, Splunk — native outputs for multi-cloud observability

The choice depends on storage cost and on the search SLO the business requires.

Best practices for production

Early filtering and parsing — use filters (kubernetes, parser, modify) to enrich and reduce log volume before sending it to the destination. This cuts the bill significantly.
Multi-output with buffering — write simultaneously to CloudWatch (operational) and to S3 (long-term archive). The on-disk buffer prevents loss when the destination goes down.
Sample verbose logs — for very high-volume DEBUG or INFO logs, consider sampling to reduce cost without losing visibility into patterns.
Configure memory limits for the Fluent Bit DaemonSet in line with the log volume per node.

How we apply this at Caleidos

For clients with medium and large EKS platforms, Fluent Bit is usually the most efficient piece for managing logs as streams toward OpenSearch or CloudWatch. Combined with Edge Delta for edge processing, we frequently reduce the billed log volume by 40% to 60% without losing operational visibility. Combining it with Feature Flags in AWS AppConfig gives you the ability to change behaviors at runtime and observe the impact immediately.

Caleidos designs, implements, and operates full-stack observability platforms as part of our end-to-end cloud Observability service, including executive dashboards, predictive alerting, and 24×7 operations with Caleidos Lens©.

Want better visibility into your EKS platform? Let’s talk →

Observability in EKS: managing logs as streams with Fluent Bit