What is data ingestion?

• Integration and ingestion made up 47% of developer questions about data pipelines in Foidl et al.’s 2023 mining study of GitHub and Stack Overflow.
• Incorrect data types were the most common root cause of pipeline data issues, at 33%, in that same 2023 study.
• Azure Data Explorer’s queued ingestion batches by default until 5 minutes, 1,000 items or 1 GB is reached, per Microsoft’s documentation updated in January 2026.
• OpenAI’s vector stores chunk documents at 800 tokens with a 400-token overlap by default, per its retrieval guide as of October 2026.
• Google’s BigQuery guidance, updated October 13, 2026, recommends the ELT approach for most customers.

What is data ingestion? A plain-language definition

Data ingestion is the process of collecting data from wherever it lives and moving it into a system where it can be stored, queried and used. The IEEE Technology Navigator’s definition of data ingestion puts it almost exactly that way, and places ingestion at the very start of a data pipeline. Everything downstream (reports, dashboards, machine learning models, AI agents) sees only what ingestion let in.

That one-line definition hides a lot of work.

In practice, ingestion is a standing pipeline stage with its own design problems, rather than a single copy job. Microsoft’s Azure Data Explorer documentation (updated January 2026) describes what happens on the way into a table: the system validates the incoming data, converts formats, matches the data to a schema, then indexes, encodes and compresses it. Only after all of that is the data queryable. A copy command does none of this.

Two shapes of ingestion exist, and most teams run both:

• One-time ingestion: a historical backfill, loading years of records once so new data has context.
• Continuous ingestion: a pipeline that keeps running, either streaming records as they arrive or queuing them into batches.

There’s a third framing that I think deserves more attention than it gets. Zhao, Megdiche and Ravat’s 2021 paper on data lake ingestion management argues that ingestion is “a complicated and important phase” whose job includes keeping data findable, accessible, interoperable and reusable. Those are the FAIR principles. Their proposed metadata model records which sources fed the lake, which ingestion processes ran, how trustworthy the data is and what security applies to it.

Why does that matter? Because data that arrives without a record of where it came from is data nobody will trust in six months. Anyone who’s inherited a data lake full of unlabeled Parquet files knows the feeling.

So the plain-language version is this. Data ingestion collects data from sources, checks and reshapes it enough to land safely, records where it came from, and makes it available for use. The transport is the easy part.

How a data ingestion pipeline works from source to storage

A data ingestion pipeline works by pulling records from source systems, buffering them in a durable queue, landing them in a raw storage layer with metadata attached, and then making that layer queryable. The stages are the same whether the source is a Postgres database or a fleet of sensors. The tools change.

Here’s the path a record takes:

1. Source. Databases, application APIs, files dropped into object storage, and event sources such as sensors or clickstreams. Apache Kafka’s own introduction lists databases and sensors as typical event sources.
2. Collection. A connector reads from the source. For databases, log-based change data capture (CDC) is the method that catches everything: Debezium’s documentation says reading the database’s native log “ensures that all data changes are captured,” with very low delay, no polling load and no need to add a “last updated” column. The MySQL connector reads the binlog; the PostgreSQL connector reads a logical replication stream.
3. Buffering. Real-time sources need a message buffer between collection and storage. Microsoft’s big data architecture guide (updated August 2026) calls this stream buffering and names Event Hubs, IoT Hub and Kafka as examples. Google Cloud Pub/Sub plays the same role on its platform.
4. Landing. Records arrive in a raw layer, often called bronze. Microsoft’s medallion guidance (September 2026) recommends storing most fields as string, VARIANT or binary at this stage and adding provenance metadata such as the source file name.
5. Making it queryable. Indexing, compression and schema registration happen here, after which analysts and models can read the data.

Step 2 has a catch that the vendor docs tend to gloss over. A database log holds recent changes, never the table’s full history, so a fresh CDC connector has no way to build an initial snapshot from the log alone. Netflix’s 2020 DBLog paper solves this by interleaving log events with chunked table selects, using watermarks to keep the two in order and taking no locks on the source. Netflix reported dozens of microservices running it at the time.

Delivery guarantees shape everything after the buffer. Google’s Pub/Sub architecture documentation states the default is at-least-once delivery with no ordering guarantee. Translation: your pipeline will see the same record twice, sometimes out of sequence, and the fix lives on your side. Google advises idempotent subscribers or deduplication in Dataflow. Microsoft’s ETL guide gives the same list for any streaming pipeline:

• Checkpointing, so a restart resumes instead of replaying from the beginning.
• Idempotent transforms, so a duplicate record produces the same result as a single one.
• Watermarking, so late events are handled rather than silently dropped or double-counted.
• Dead-letter queues, so a malformed record is parked for inspection instead of halting the stream.

Build those four in from day one. Retrofitting idempotency into a pipeline that’s already producing duplicates is a miserable job.

Batch, streaming and micro-batch: the three data ingestion modes compared

Batch ingestion processes all available data on a schedule with latency measured in hours to minutes, streaming ingests each event as it arrives with sub-second latency, and micro-batch sits between them at seconds to minutes. The right mode depends on how fresh the data needs to be and how much complexity and cost you’ll accept to get there.

The clearest comparison comes from Microsoft’s Azure Databricks guide to batch versus streaming (updated October 13, 2026) and its Azure Data Explorer ingestion overview (updated January 2026):

A few details behind those cells deserve spelling out.

Batch’s simplicity comes from amnesia. The Databricks guide says a batch engine doesn’t track what it has already processed. It reads everything the source has on each run. That makes the logic easy to reason about and the results complete, at the price of efficiency.

Micro-batch is how most “streaming” platforms actually operate. Azure Data Explorer’s streaming ingestion works in micro-batches and gives near-real-time latency for small data sets. Its queued ingestion, which Microsoft recommends for most scenarios, batches by default until one of three limits is hit: 5 minutes, 1,000 items or 1 GB. Those defaults tell you where the vendor thinks the throughput sweet spot lies. The guide reserves true real-time mode for sub-second operational cases such as fraud detection.

Cost tracks latency. Microsoft’s medallion architecture guidance (September 2026) lists continuous incremental ingestion as the highest-cost, lowest-latency option, with triggered incremental and batch ingestion cheaper and slower.

Streaming “results cannot always be accurate” given out-of-order and late data, according to Microsoft’s October 2026 Azure Databricks guide to batch versus streaming.

That line is the one I’d pin to the wall. Teams reach for streaming because it sounds modern, then discover that a count computed at 10:00:00 differs from the same count recomputed at 10:00:05 once the stragglers arrive.

Lambda or Kappa? The two classic architectures answer the batch-versus-stream question differently. Lambda runs a batch path and a speed path side by side, which Microsoft’s big data guide (August 2026) faults for duplicating processing logic across two frameworks. Kappa runs everything through a single stream and replays it when a recompute is needed. Databricks’ own recommendation for its platform is a split: stream the raw bronze layer, since that’s mostly appends, and batch the silver and gold transformation layers. That’s guidance for one vendor’s platform. It is a sensible default all the same.

Why data ingestion matters for analytics, machine learning and AI agents

Data ingestion matters because every analytics query, every trained model and every AI agent’s answer is bounded by what the ingestion stage let through and how cleanly it arrived. Reports can only count records that landed. Models can only learn from inputs that passed validation. A retrieval-augmented agent can only cite documents that were parsed, chunked and embedded. Three consumers, one upstream dependency.

For analytics the stake is completeness. A dashboard built on a load that skipped a partition is confidently wrong, and nobody reading it can tell.

For machine learning the stake is input quality. Breck et al.’s 2019 paper on data validation for machine learning at Google argues that bad input data can erase gains made anywhere else in an ML system, and describes the schema-based validation built into the TFX platform. The paper reported hundreds of product teams running it on several petabytes of production data a day. That is a 2019 figure with no stated measurement method, so read it as an order of magnitude.

Training data adds a second problem: size. Hugging Face’s datasets library (docs v4.8.4) streams a dataset when you set streaming=True, returning an IterableDataset that reads records on the fly, and its own example is the 45 TB English split of FineWeb. You give up random access. You keep Parquet column and filter selection, and state_dict() checkpoints let a training run resume mid-stream after a crash.

For AI agents and RAG, ingestion means turning documents into something searchable. Gao et al.’s RAG survey (first posted December 2023, v5 March 2024) describes the indexing path as PDF, HTML and Word files converted to text, split into chunks, embedded and stored in a vector database. OpenAI’s retrieval guide shows the managed version of that path, with these defaults as of October 2026:

• Chunking, embedding and indexing run asynchronously after a file is added to a vector store.
• Default chunk size is 800 tokens with a 400-token overlap.
• Chunk size can be set anywhere from 100 to 4,096 tokens.
• A single file can be at most 512 MB and 5,000,000 tokens.

Those numbers are OpenAI-specific and will change. What they imply is more durable: with 800-token chunks and 400 tokens of overlap, nearly every token in your corpus gets embedded twice. Teams tend to discover this when the embedding bill arrives. Building a RAG pipeline that holds up in production starts with ingestion decisions like these, long before anyone tunes a prompt.

ETL vs ELT: how data ingestion fits with transformation

Data ingestion covers transport and landing, transformation reshapes the landed data for use, and ETL versus ELT is simply the question of whether that reshaping happens before or after the landing. Microsoft’s Azure Architecture Center ETL guide (October 2025) states that ELT “differs from ETL solely in where the transformation takes place.” In ELT the target data store does the transforming. No standards body draws a formal line between ingestion and transformation. The split is industry convention, and a useful one.

ELT is why the medallion layering exists. Microsoft’s medallion guidance (September 2026) describes the bronze layer as one that keeps the raw state of the source, is appended incrementally, serves as the single source of truth, enables reprocessing and auditing, and can combine streaming and batch sources. Silver and gold are where transformation lives. Ingestion stops at bronze.

The same guidance says why. Writing from ingestion straight into silver tables means a schema change or a corrupt record breaks the load, whereas a raw bronze copy absorbs the surprise and lets you fix the transform later. Microsoft labels this a best practice rather than a requirement.

I’d go further than Microsoft does. If you work in healthcare or financial services, the compliance case for ETL-style staging audits is real, but it argues for an audited bronze layer, since the raw copy is exactly the evidence an auditor wants. Few teams need to throw away the raw data to satisfy a regulator.

Which data ingestion tools and platforms are worth considering

The data ingestion tools worth considering fall into five categories: message buses, change data capture connectors, orchestrators, warehouse-native loaders, and AI-specific ingestion libraries. Most production pipelines combine two or three of them. The descriptions below come from each vendor’s own documentation, which naturally presents its product in the best light, so weigh them accordingly.

Two things stand out from that table.

The bus is rarely optional. Pub/Sub describes itself as an “asynchronous and scalable messaging service” for streaming analytics and data integration pipelines, and Kafka pitches itself the same way. Any source that produces events faster than your store can absorb them needs one, whichever vendor’s logo it carries.

CDC tooling has matured into a category of its own. Debezium reads the MySQL binlog and the PostgreSQL logical replication stream natively, and Google now sells managed CDC as Datastream alongside its batch and streaming loaders. Five years ago this was custom code at most companies. It no longer needs to be.

What the table can’t tell you is which of these your team will keep running at 3 a.m. That depends on who you already employ and what you already pay for, which is a different question from which tool has the best documentation page.

What does data ingestion actually cost?

Data ingestion costs scale with how fresh you need the data to be, how much state the pipeline has to hold, how much raw data you keep, and how many engineering hours the pipeline eats. No reliable market-size or average-spend figures exist for ingestion as a category, so anyone quoting one is guessing. What the vendor documentation and the research do support is a clear picture of the drivers.

Four cost drivers, roughly in order of how much they surprise people:

• Latency: Microsoft’s medallion architecture guidance (September 2026) ranks continuous incremental ingestion as the highest-cost option, with triggered incremental and scheduled batch cheaper in that order. Always-on compute is the price of always-fresh data.
• State: stateful streaming operations (joins, aggregations, deduplication) hold data in memory between events. Microsoft’s October 2026 Databricks guide flags them as the main source of streaming complexity, and complexity here is compute you pay for continuously.
• Raw storage: a bronze layer keeps the source’s raw state and only ever appends. That’s the point of it. It also means storage grows without an upper bound unless you set retention rules.
• Engineering time: this one dwarfs the others at most companies.

That last driver has a number behind it. Foidl et al.’s 2023 study of data pipeline quality mined GitHub and Stack Overflow and found that integration and ingestion made up 47% of developer questions about pipelines. Nearly half. The study is the most recent measurement available, and nothing since has suggested the share has dropped.

So the honest cost model is this. Compute and storage are predictable and show up on an invoice. The engineers debugging a connector that stopped emitting at midnight are the expensive part, and that line never appears on a cloud bill.

Why data ingestion pipelines fail: schema drift, duplicates, latency and silent data loss

Data ingestion pipelines fail most often because a source changes its data types or schema without warning, because at-least-once delivery produces duplicates the pipeline never deduplicated, because late events arrive after a result was already computed, and because records go missing with no error to show for it. Each failure has a known cause and a known fix. Most teams still get bitten by all four.

Type errors come first for a reason. Foidl et al.’s 2023 taxonomy of data pipeline quality, accepted to the Journal of Systems and Software, identified 41 factors affecting pipeline data quality across five themes, validated them with eight expert interviews, and traced root causes in mined developer issues. Incorrect data types were the single most common root cause, at 33% of data issues.

Incorrect data types caused 33% of data-related pipeline issues in Foidl et al.’s 2023 mining study of GitHub and Stack Overflow, the most common root cause found.

A third of your data bugs from one cause. That statistic is the whole argument for storing bronze-layer fields as string, VARIANT or binary, as Microsoft’s medallion guidance recommends: a column that arrives as text one day and as an integer the next can’t break a load if the load never asked for an integer.

Two rows in that table deserve a closer look.

Silent data loss is the one that keeps me up. Debezium’s documentation contrasts log-based capture with polling and dual writes and says only the log-based approach “ensures that all data changes are captured.” Polling misses a row that’s updated twice between polls. Dual writes miss whatever the application forgot to write. And even a log-based connector starts blind, because the log never holds a table’s full history, which is the gap Netflix’s 2020 DBLog design fills with chunked, watermarked selects and no locks. A 2026 theoretical follow-up on arXiv formalizes why that interleaving stays consistent.

Stale lineage is the failure nobody budgets for. A 2025 arXiv paper on schema lineage extraction argues that the metadata captured when data first lands goes out of date as it moves through multi-stage pipelines. Meta’s Unified Lineage System paper, published at ACM in 2025, describes tracking data flows across heterogeneous storage at Meta’s scale. Both papers exist because the problem is unsolved in general. If a company Meta’s size needed a dedicated system for it, your spreadsheet of column mappings won’t hold.

Then there’s the streaming accuracy caveat, which is less a failure than a property. Microsoft’s Databricks guide says streaming results “cannot always be accurate” when data arrives late or out of order. Plan for the recompute instead of hoping it won’t be needed.

How to choose a data ingestion approach for your team and workload

Choose a data ingestion approach by answering five questions in order: how fresh the data must be, what kind of source you’re reading, what your team already knows how to run, what you can spend, and what a regulator will ask to see. The answers narrow the field fast. Most teams that work through them land on batch or queued ingestion for the bulk of their sources and streaming for one or two.

The five criteria:

1. Latency need: if seconds don’t matter, batch wins on cost, simplicity and accuracy. Reserve sub-second streaming for operational cases such as fraud detection, which is where Microsoft’s Databricks guide places real-time mode.
2. Source type: databases call for log-based change data capture. Event producers (sensors, clickstreams, application events) call for a message bus and streaming or micro-batch. Files landing in object storage call for scheduled batch.
3. Team skill: stateful streaming is a specialist discipline. A team that has never debugged a watermark should start with queued ingestion and earn its way into streams.
4. Budget: continuous incremental ingestion is the costliest option in Microsoft’s medallion guidance (September 2026), and the compute runs whether or not anyone is querying.
5. Compliance: healthcare and financial services teams need an auditable raw copy. A bronze layer with provenance metadata gives an auditor exactly that.

One pattern covers most cases. Databricks recommends streaming into the raw bronze layer, since that layer only appends, and batching the silver and gold transformation layers where the stateful work lives. That’s one vendor’s guidance for one platform, and it happens to match what I’d build on any platform: cheap, append-only capture at the edge and scheduled, well-tested transforms behind it.

The question I’d ask before any of the five is whether you already have ingestion running that nobody trusts. Replacing a pipeline is harder than building one, and an audit of the AI and data automation you already run usually reveals that the “streaming” system is a cron job with a bus in front of it. Fix the trust problem first. Pick the mode second.

What changed in data ingestion in 2026

The biggest change in data ingestion in 2026 is that document ingestion for AI retrieval has become a mainstream pipeline workload alongside the traditional database and event sources, while the core batch, streaming and CDC mechanics have stayed put. The vendor guidance got refreshed. The standards did not arrive.

What’s new, as of October 14, 2026:

• RAG ingestion research is now specific enough to be useful. A 2026 arXiv factorial study of parsers, chunkers and embedding models, run on Indian government regulatory documents, reports that parser and chunker choices interact: the best chunker depends on which parser produced the text. One document type, one country’s regulatory corpus, so treat it as a lead to test on your own files rather than a rule.
• CDC got its theory. A 2026 arXiv paper formalizes the watermark-based interleaving that Netflix’s 2020 DBLog design introduced, giving a proof-level account of why chunked selects and log events stay consistent. That matters for anyone building their own snapshot logic instead of adopting a connector.
• Vendor guidance was rewritten this year. Microsoft updated its Databricks batch-versus-streaming guide on October 13, 2026 and its medallion architecture page on September 11, 2026. Google updated its BigQuery loading guidance on October 13, 2026 and still recommends ELT for most customers.

What hasn’t changed is just as telling. No standards body has issued a formal definition of ingestion or a canonical batch/stream/CDC taxonomy. The IEEE one-liner remains the closest thing to an authoritative definition, and the rest is platform documentation. Published research on data contracts and schema evolution is still thin in 2026, even though schema drift sits near the top of every practitioner’s pain list. The most recent measurement of where developers struggle with pipelines is still Foidl et al.’s 2023 study.

So the picture for 2026 is a field whose practice moves fast and whose formal ground moves slowly. Build on the practice. Don’t wait for the standard.

Frequently asked questions about data ingestion

Is data ingestion the same as ETL?

No. Data ingestion is the transport and landing of data from a source into a storage system, while ETL (extract, transform, load) adds a transformation step before loading. Microsoft’s Azure Architecture Center ETL guide (October 2025) says ELT differs from ETL solely in where the transformation happens, so ingestion is the “E” and “L” of either pattern. Industry convention treats ingestion as ending when raw data lands in a bronze layer.

What is the difference between data ingestion and data integration?

Data ingestion moves data from sources into a target system, and data integration is the broader work of combining data from multiple systems into a coherent, queryable whole. Ingestion is one stage inside integration. Foidl et al.’s 2023 mining study grouped the two together and found they accounted for 47% of developer questions about data pipelines, which suggests practitioners treat them as one problem area.

What is real-time data ingestion?

Real-time data ingestion captures each event as it happens and lands it with sub-second latency, typically through a message bus such as Kafka, Pub/Sub or Event Hubs and a stream processor. Microsoft’s Databricks guide (October 2026) reserves true real-time mode for operational cases like fraud detection and notes that results “cannot always be accurate” when events arrive late or out of order. Most pipelines described as real-time run micro-batches with latency measured in seconds.

What is change data capture?

Change data capture (CDC) reads row-level inserts, updates and deletes from a database’s transaction log and emits them as events, so downstream systems receive every change without polling the tables. Debezium’s feature documentation says log-based CDC captures all changes with very low delay, adds no polling load and needs no “last updated” column. Because the log never holds a table’s full history, a CDC connector also needs an initial snapshot, which Netflix’s DBLog design handles with watermarked chunk selects.

What is a data ingestion framework?

A data ingestion framework is the reusable set of connectors, buffers, landing conventions and reliability controls a team standardizes on so each new source follows the same path. In practice it means a message bus, a CDC or batch connector per source type, a raw bronze layer with provenance metadata, and built-in checkpointing, idempotent processing and dead-letter queues. No formal standard defines one. The term describes a house pattern rather than a product.

Is Kafka a data ingestion tool?

Kafka is the buffer and transport layer in many ingestion pipelines rather than a complete ingestion tool on its own. Apache Kafka’s introduction describes it as combining publish/subscribe messaging, durable storage and stream processing, with Kafka Connect handling continuous import and export. Paired with connectors such as Debezium for databases, it becomes the backbone of an ingestion pipeline, but something still has to read the sources and something still has to land the data.

Where to start with your first data ingestion pipeline

Start your first data ingestion pipeline with one source, a raw bronze layer, and batch or queued ingestion, then add streaming only when a real latency need forces it. That order keeps the first version boring, which is what you want from something every report and model will depend on.

The sequence that works:

1. Pick the single source that matters most, and read it with a log-based CDC connector if it’s a database or a scheduled batch load if it’s files.
2. Land every record raw, with fields typed as string or VARIANT and the source name and arrival time attached.
3. Run it as queued or scheduled ingestion first. Azure Data Explorer’s default of 5 minutes, 1,000 items or 1 GB per batch is a reasonable starting shape on any platform.
4. Make the consumer idempotent and route malformed records to a dead-letter queue before the second source arrives.
5. Move one source to streaming when someone can name the decision that needs sub-second data.

Trust in the raw layer is the asset. Everything else can be rebuilt from it.

If your team is standing up ingestion to feed an AI agent or a retrieval system and wants engineers who have shipped that path in production, talk to AlphaCorp AI about your ingestion pipeline.