{"id":60022,"date":"2026-11-03T10:01:49","date_gmt":"2026-11-03T10:01:49","guid":{"rendered":"https:\/\/www.bravent.net\/?p=60022"},"modified":"2026-11-03T10:01:49","modified_gmt":"2026-11-03T10:01:49","slug":"snowpipe-continuous-ingestion-in-snowflake-without-a-scheduler","status":"publish","type":"post","link":"https:\/\/www.bravent.net\/en\/news\/snowpipe-continuous-ingestion-in-snowflake-without-a-scheduler","title":{"rendered":"Snowpipe: Continuous Ingestion in Snowflake Without a Scheduler"},"content":{"rendered":"

There comes a point in any data pipeline when scheduled batch processing no longer makes sense<\/strong>. Not because it is poorly designed, but because the business starts demanding fresher data than a job running every hour, or every fifteen minutes, can deliver. And the obvious answer \u2014 reducing the scheduler interval \u2014 has its limits: at some point, you are paying to keep a warehouse running almost constantly to process fewer and fewer records per execution.<\/p>\n

Snowpipe<\/strong><\/a> solves this problem from a different angle. Instead of asking \u201cAre there any new files yet?\u201d every X minutes, it listens. When a file arrives in the stage, loading starts automatically. There is no warehouse to start or scheduler to maintain: Snowflake manages the compute behind the scenes through a serverless service designed specifically for this purpose.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

How It Fits into the Architecture<\/strong><\/p>\n

Snowpipe does not replace a scheduled COPY INTO; it complements it when latency matters. The central component is the pipe<\/strong> object: a definition that wraps a COPY INTO statement and associates it with a stage. That pipe can be triggered in two ways.<\/p>\n

The first is auto-ingest<\/strong>: you configure an event notification in your cloud storage (S3 + SQS<\/a>, Azure Blob Storage + Event Grid<\/a>, or Google Cloud Storage + Pub\/Sub<\/a>) to notify Snowflake whenever a new file appears. Snowflake picks up the event, executes the corresponding pipe’s COPY INTO statement, and that is it. There is no polling of any kind; it is purely event-driven.<\/p>\n

The second option is through the Snowpipe REST API<\/strong><\/a>: you call Snowpipe’s insertFiles endpoint when your own process knows that a file is ready. This is the option when you do not want to depend on the cloud provider’s native messaging system, or when a file is generated by a system you already orchestrate and you prefer to control the trigger yourself.<\/p>\n

In both cases, the pipe does not run on a regular virtual warehouse. It uses serverless compute<\/strong>, managed by Snowflake and billed according to actual usage. There is no need to size anything or keep a warehouse running while waiting: compute becomes available when needed and stops when the job finishes.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

What You Gain<\/strong><\/h2>\n

Low latency without managing infrastructure.<\/strong> The typical time between a file arriving in the stage and the data becoming available in the table is around one minute, sometimes less. Compared with an hourly batch, that is a significant improvement, and you do not need to change Airflow, ADF, or any other orchestrator to achieve it.<\/p>\n

Costs aligned with actual usage.<\/strong> You do not pay for a warehouse sitting idle while waiting for files. You pay for the compute actually consumed by each load, measured in seconds. For continuous but non-massive ingestion volumes, this is often considerably cheaper than running an XS warehouse every five minutes \u201cjust in case.\u201d See the official Snowpipe billing documentation<\/a> for more details.<\/p>\n

Complete decoupling from the data-producing process.<\/strong> The system that places files in the stage does not need to know whether Snowpipe is listening or not. It simply writes to S3, Blob Storage, or whichever storage service is being used, at its usual pace. Ingestion into Snowflake becomes a separate concern, without coupling the source pipeline to the loading logic.<\/p>\n

Built-in retries and tracking.<\/strong> Snowpipe keeps track of which files have already been loaded, so if the same file appears twice \u2014 because of reprocessing or retries in the source system \u2014 it does not load it again. And if a load fails, the failure is recorded in the history, without having to build error-tracking mechanisms from scratch.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

Snowpipe Streaming: When You Do Not Even Want to Use Files<\/strong><\/h2>\n

The more recent evolution is Snowpipe Streaming<\/strong><\/a>, which changes the approach quite significantly. Instead of writing files to a stage and waiting for Snowpipe to pick them up, the Snowpipe Streaming SDK allows you to insert rows directly<\/strong>, without an intermediate file. Think of a Kafka consumer that pushes rows into Snowflake one by one, or in micro-batches, as they arrive, rather than writing them to disk and loading them afterwards.<\/p>\n

This reduces latency even further \u2014 we are talking seconds rather than minutes \u2014 and eliminates an entire step in the process: there is no longer any need to generate, name, and manage files that exist only briefly before being loaded. It is a natural fit for use cases involving application events, IoT, or CDC from transactional databases.<\/p>\n

The trade-off is that Snowpipe Streaming operates at the code level: it integrates with a Java application or through Kafka Connect. It is not something you configure declaratively like a traditional pipe. It is a different tool for a related problem, not simply a \u201cfaster mode\u201d of the Snowpipe we have always known.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

Costs and Observability<\/strong><\/h2>\n

Snowflake exposes several views and functions that let you understand what is happening with your pipes without having to guess. PIPE_USAGE_HISTORY<\/strong><\/a> provides credit consumption by pipe and time interval, so when the bill arrives, you can see exactly which pipe is costing what. COPY_HISTORY<\/strong><\/a> shows the details of each load: how many rows were processed, how many errors occurred, and which file was involved. And SYSTEM$PIPE_STATUS<\/strong><\/a> tells you in real time whether a pipe is paused, whether there are files pending in the notification queue, or whether something has become stuck.<\/p>\n

It is worth setting up an alert for the latter. A pipe can silently stop receiving notifications \u2014 for example, if someone changes the bucket configuration or rotates the role’s credentials \u2014 and in that case, data simply stops arriving without any visible error in the normal flow. Without active monitoring, it is easy to miss the issue until someone checks the dashboard and notices that it has not been updated for days.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

What You Should Know Before Taking It to Production<\/strong><\/h2>\n

It is not magic without friction, and there are a few things worth understanding before relying on it for anything critical.<\/p>\n

File size matters, a lot.<\/strong> Snowpipe is designed for files of a reasonable size (Snowflake recommends between 100 and 250 MB compressed). If your source system generates thousands of small files \u2014 just a few KB each \u2014 you will pay file-management overhead instead of taking advantage of actual throughput. It is worth grouping files before writing them to the stage if the source pattern tends to produce many tiny files.<\/p>\n

Load order is not guaranteed.<\/strong> If several files arrive at almost the same time, Snowpipe does not guarantee that they will be processed in the same order in which they were written. For most fact tables, this does not matter, but if your downstream logic assumes a strict arrival order, you need to handle it with a timestamp column rather than relying on load order.<\/p>\n

It is not truly real-time.<\/strong> Typical latency of around one minute is excellent compared with an hourly batch, but if the actual requirement is milliseconds, neither classic Snowpipe nor Streaming will meet it without additional work. You need to be honest with the business about what SLA is reasonable to expect.<\/p>\n

Event configuration lives outside Snowflake.<\/strong> Setting up auto-ingest means working with IAM, SQS, Event Grid, or Pub\/Sub, depending on the cloud provider. Normally, that means requesting permissions from another team. It is not difficult, but it is an external dependency that needs to be planned for and can fail silently if someone changes a policy without notifying the team.<\/p>\n

Pipes are immutable in important ways.<\/strong> If you need to make a substantial change to a pipe’s COPY INTO definition, you normally have to recreate it rather than simply alter it. This is worth considering when versioning the definition in your repository, because a \u201csmall\u201d change may require pausing the pipe, recreating it, and activating it again, with the resulting downtime window.<\/p>\n

Snowpipe is not the solution for everything that goes into Snowflake, nor does it claim to be: for traditional batch loads, a scheduled COPY INTO remains simpler and cheaper. But for the specific case of \u201cI need this data to be available shortly after it exists, without building or maintaining streaming infrastructure,\u201d it is probably the option with the best effort-to-result ratio within the Snowflake ecosystem. And if, later on, minute-level latency is no longer enough, Snowpipe Streaming is there as the next step, without having to rebuild the architecture from scratch.<\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>

Looking to Optimize Your Data Pipelines?<\/strong><\/h2>\n

Are you looking to improve data ingestion in your organization and make the most of Snowflake? At Bravent, we help businesses design more efficient data architectures tailored to their needs and built to scale. Email us at <\/strong>info@bravent.net<\/strong><\/a> to discover how you can optimize your data pipelines and get the right information where you need it, when you need it.<\/strong><\/p>\n<\/div>[vc_empty_space]<\/div><\/div><\/div><\/section>[vc_section]

[vc_single_image image=”57776″ img_size=”128×128″ alignment=”center” style=”vc_box_circle_2″ css=”” el_class=”align-inherit”]<\/div><\/div><\/div>

Daniel Galvez Ramos<\/h3>Senior Consultant - Bravent<\/span>