Apache Kafka is an open platform for moving data at the moment it happens. Applications that generate information —a sale, a click, a sensor reading— publish it as an event, and the applications that need it read at their own pace. Instead of every system talking directly to every other system, they all talk to Kafka.
What problem does Kafka solve?
Most companies still move information in batches. The sales system closes the day, produces a file, and overnight a process loads it into the data warehouse. The next morning the dashboard shows what happened yesterday. That design worked for decades and remains reasonable for many reports.
It stops being reasonable when the decision cannot wait until tomorrow. A fraudulent transaction has to be stopped before it is approved, not in the following day’s report. A product that ran out has to leave the digital channel now, not in the next load. A machine that starts to fail signals it with data that loses value by the minute.
The other problem is architectural. When every system connects directly to every other system, the number of integrations grows far faster than the number of systems: ten applications talking to each other can mean dozens of connections to maintain, each with its own format and schedule. Adding application number eleven forces changes in several of the previous ones.
Kafka addresses both at once. It provides a common place events flow through: whoever produces publishes once, whoever consumes subscribes to what they need, and neither has to know the other. Adding a new consumer stops being an integration project.
How does Kafka work?
The model is simpler than the tool’s reputation suggests. There are four pieces.
An event is a fact that already happened, with its timestamp and its data: “customer 4821 paid S/ 129 at 10:42”. A topic is the category where events of the same kind are published, something like a channel: payments, clicks, sensor readings. A producer is the application that publishes to a topic. A consumer is the application that reads from it.
What makes Kafka distinctive is how all of that is stored. Kafka is not a mailbox that delivers a message and deletes it: it is an ordered, durable log. Events are appended at the end, retained for as long as you configure, and each consumer keeps its own marker of how far it has read. Three properties follow from that, and they explain why Kafka became the standard:
- Several consumers, no interference. The fraud team, the operations dashboard and the analytical warehouse read the same payment events without stepping on each other or duplicating ingestion.
- Recovery without loss. If a consumer goes down, it resumes from its last marker when it returns. The events were still there while it was away.
- Reprocessing. A new model or a logic fix can re-read the available history and recalculate, without asking anything of the system that originated the data.
Each topic is divided into partitions, and that is the lever for scale: more partitions allow more consumers working in parallel. The price is that ordering is only guaranteed within a partition. That makes the key deciding which partition each event lands in —the account number, the equipment identifier— a design decision with consequences, not a parameter filled in at the end.
Use cases that come up again and again
In the projects that come our way, streaming almost always enters through one of these doors:
- Risk and fraud in the moment. Assessing the transaction while it happens, not after approving it.
- Live inventory and pricing. Digital channels reflecting real state in seconds rather than in the overnight load.
- Equipment and fleet telemetry. Maintenance that anticipates failure instead of reacting to it.
- Synchronization between systems. Replacing file exchange between the core system and satellite applications with events.
- Continuous feeding of analytics. Dashboards and models that stop looking at yesterday.
The common pattern is that the value of the data drops quickly with time. If the data is worth the same tomorrow, a well-built batch ETL process is usually the correct and cheaper answer to operate.
Managed Kafka on AWS: Amazon MSK
Running Kafka on your own is where the budget goes: brokers to size, storage to watch, versions to upgrade, rebalances to coordinate. Amazon Managed Streaming for Apache Kafka (Amazon MSK) is the AWS managed service that takes over that layer. AWS handles the control-plane operations —creating, updating and deleting clusters— and runs open-source versions of Apache Kafka, so existing applications, tooling and plugins from partners and the Apache Kafka community are supported without requiring changes to application code. That last point is what turns the conversation from “migrate to another product” into “stop administering servers”.
On availability: MSK creates broker nodes in the Availability Zones you specify, with a minimum of one broker per zone and each zone in its own isolated subnet. When it detects an unhealthy broker, it mitigates the failure or replaces the broker with a new one and, where possible, reuses the storage from the older broker to reduce the data Kafka needs to replicate. After recovery, producer and consumer applications keep communicating with the same broker addresses they used before the failure.
Metadata management also changed. The Apache Kafka community developed KRaft to replace Apache ZooKeeper: cluster metadata is propagated within a group of controllers that are part of the Kafka cluster itself. In MSK, those KRaft controllers are included at no additional cost and require no additional setup or management.
There are two service modes —and one of them offers two node types—. The choice is about operating model:
| Mode | Who sizes capacity | When it fits |
|---|---|---|
| MSK Serverless | AWS, automatically | Demand that rises and falls unpredictably; teams that do not want to reason about capacity |
| MSK Provisioned — standard brokers | The team | Fine control over sizing and storage |
| MSK Provisioned — Express brokers | The team, with less work | High throughput and fast scaling, without managing storage |
MSK Serverless automatically provisions and scales capacity while managing the partitions in your topics, with a throughput-based pricing model. It is fully compatible with Apache Kafka, so any compatible client works to produce and consume, and it integrates with AWS PrivateLink for private connectivity, AWS Identity and Access Management (IAM) for authentication and authorization, AWS Glue Schema Registry for schema management, Amazon Managed Service for Apache Flink for stream processing, and AWS Lambda for event processing. One detail worth knowing beforehand rather than afterwards: MSK Serverless requires IAM access control for all clusters and does not support Apache Kafka access control lists.
The Express brokers in MSK Provisioned are the development that most changes the calculation when volume is high. They include pay-as-you-go storage that scales automatically and requires no sizing, provisioning or proactive monitoring. Depending on the instance size selected, each broker node can provide up to 3x more throughput, scale up to 20x faster, and recover 90% quicker compared to standard Apache Kafka brokers. They come pre-configured with Amazon MSK’s best practice defaults and have no maintenance windows: AWS updates the cluster hardware on an ongoing basis. In exchange there are limits worth reviewing at design time: they are only available in a three-Availability-Zone configuration, only on select instance sizes, on Apache Kafka versions 3.6, 3.8, 3.9 and 4.2, and they do not yet fully support the KStreams API.
Around the cluster there are two pieces that solve very frequent needs. MSK Connect streams data to and from the Apache Kafka cluster, which is what you need when the source is an existing database rather than a new application. MSK Replicator reliably replicates data across MSK Provisioned clusters, in the same AWS Region or in different ones, which is the piece that appears when the requirement is continuity in the face of a full Region outage.
A planning note: not every cluster type is available in every AWS Region. Before fixing the architecture it is worth confirming the availability of the chosen mode in the target Region, especially when data residency requirements are involved.
Kafka versus Kinesis: how to choose
This is the question that appears as soon as the conversation moves from “we need real time” to “what do we build it on”. Amazon Kinesis Data Streams is the AWS service for collecting and processing large streams of data records in real time. The delay between a record entering the stream and being retrievable is typically less than one second, and multiple applications can consume from the same stream concurrently and independently. In terms of capability, both options solve the same problem. The difference lies elsewhere.
| Axis | Apache Kafka (on Amazon MSK) | Amazon Kinesis Data Streams |
|---|---|---|
| Nature | Open platform, run as a managed service | Native AWS service |
| Ecosystem | Broad: connectors, tooling and plugins from the community | Natively integrated with the rest of AWS |
| Portability | High: the same platform runs in other environments | Tied to AWS |
| Learning curve | Requires understanding the Kafka model | Lower, if the team already works in AWS |
| Usual fit | Existing Kafka investment, or portability is a goal | The shortest path inside AWS is the goal |
The practical reading: if the team already has Kafka applications, connectors or experience, MSK avoids rewriting and removes server administration. If you are starting from scratch, the team lives inside AWS and nobody misses the Kafka ecosystem, Kinesis reaches the first result sooner. It is a decision about ecosystem and team, which is why it is worth making early: switching streaming platforms midway costs considerably more than choosing well at the start.
What usually goes wrong
Three patterns repeat often enough to be worth naming.
Treating Kafka as a database. Kafka is the path data travels along, not the place where it is queried. Business questions are answered in the operational database, the data warehouse or the lake that Kafka feeds.
Deciding partitions last. The number of partitions and the key that determines them define maximum scale and ordering guarantees. Correcting that with the system in production is uncomfortable and sometimes means reprocessing.
Adopting streaming without a case that justifies it. If no business decision changes by receiving data in seconds rather than tomorrow, event-driven architecture adds complexity without returning value. It is worth starting from the use case rather than the tool.
Kafka within a data architecture
Adopting Kafka is rarely an isolated project. It almost always arrives alongside a redesign of how data moves and is consumed: which events matter, who produces them, what is retained and for how long, and how all of that connects with microservices, with analytics and with the messaging patterns already in place —the map of when to use a queue, a notification or an event bus is in our comparison of SQS, SNS and EventBridge.
At Caleidos we design and implement these architectures within our Data Engineering on AWS and cloud-native applications practices, and we sustain them with Caleidos Lens©, our 24×7 service desk. You can see how we apply it in our success stories.
Frequently asked questions
What is Kafka in simple terms? An open platform where applications publish business facts at the moment they happen and other applications read them at their own pace, without either side having to know the other.
Does Kafka replace a database? No. It is the path data travels along; queries still live in the operational database, the analytical warehouse or the data lake that Kafka feeds.
Do you need to administer servers to use Kafka? Not on AWS. Amazon MSK takes over the control plane and runs open-source versions of Apache Kafka, and MSK Serverless additionally provisions and scales capacity automatically.
When does Kafka fit and when does Kinesis? Kafka when there is existing investment in its ecosystem or portability is a goal; Kinesis when the team lives inside AWS and wants the shortest path.
Evaluating real-time streaming for your operation?
Let’s talk about your case and we will give you a concrete reading on whether your need is better solved with managed Kafka on AWS, with Kinesis, or with a good batch process you almost already have.
