A while back I wrote about making Snowflake query Iceberg data sitting in Google Cloud Storage without copying a byte. This is the mirror image: a team on Google Cloud that needs to read an Apache Iceberg lake living in AWS S3. Same open table format, opposite direction. The tool for it is BigQuery Omni, and I spent a few days actually running it — not reading the marketing page — to find out where it shines and where it quietly falls over.
Compute goes to the data, not the other way around
The clever bit of Omni is that Google runs BigQuery compute inside AWS, in the same region as your S3 bucket. When you query an Iceberg table there, the scan happens next to the data and only the result crosses back to Google Cloud. I ran a full table read and checked the execution details: 151 bytes shuffled across the cloud boundary. Not 151 megabytes — bytes. If BigQuery had pulled the data to a GCP region to scan it, that number would have been the size of the table, every single time.
The auth is the part I liked most. There is no AWS access key stored anywhere in Google Cloud. The IAM role trusts a Google identity and BigQuery assumes it with a short-lived web-identity token — the same keyless pattern I use in the other direction. Nothing to rotate, nothing to leak.
So far, so good. Then I tried to actually use it.
Where it stops being magic
The first wall: I wanted Dataflow to read the Omni table and it flatly refused. The reason is that the BigQuery Storage Read API — the high-throughput path that Dataflow and the Spark connector use — is not available in Omni regions. I confirmed it directly: the read session came back with "Read API can be used to read temporary tables only in this region." That is not a config you can fix. Anything that reads BigQuery at volume cannot read an Omni table at all.
The second wall was subtler. Cross-cloud joins — joining the S3 data with a native BigQuery table — only work if the native dataset sits in the region colocated with the Omni region. For aws-us-east-1 that is us-east4, not us-east1. Put your dataset one region over and you get "does not support exporting to us-east1," which tells you nothing about the actual cause. It cost me an afternoon.
Then the list of things that simply don't work in Omni: no DML, no BigQuery ML, no JavaScript UDFs, no streaming inserts, no customer-managed keys. And results over 20 GiB can't come back to Google Cloud at all — you EXPORT them to S3 (not GCS) from a statement that runs inside AWS. Omni is a real BigQuery, but a deliberately narrow one.
The pattern that actually feeds your systems
Here is the mistake I nearly made: treating Omni as the way to sync S3 data into Bigtable or AlloyDB. It is the wrong tool for that, and the cost model tells you why. Every run that materializes the whole dataset pays a per-gigabyte cross-cloud transfer charge, separate from the scan you already paid for. Do that on a schedule and you have built a very expensive copy.
So I built the other thing. Iceberg records every commit as a snapshot, and an append snapshot's metadata lists exactly the data files it added. That means you can read the diff — only the new rows — straight from S3, with no query engine in the loop, and stream it to Bigtable or Pub/Sub. It is change data capture that the table format hands you for free. I proved it end to end: appended a batch, read only the new snapshot's files, published them to Pub/Sub, and pulled them back on the other side. No Omni anywhere in that path.
The mental model
Omni is a query window into your S3 lake, not a data-movement layer. Use it when a human or a light application asks a question and wants a small answer back — ad-hoc analysis from BigQuery, Cloud Run, or a notebook, or a cross-cloud join to enrich S3 data with something native. Do not make it the integration layer that feeds your operational systems. When a GCP system needs the data rather than an answer, move the data — with snapshot-diff CDC from S3, or a deliberate materialization into a native table — and leave Omni to do the one thing it is genuinely good at.
The durable skill here isn't memorizing the Omni docs. It's knowing the difference between a query and a sync, and refusing to let a clever tool blur it.
0 Comments
Leave a Comment