Build and edit data pipelines with an agent
Describe a pipeline in plain language and an agent drafts the transformation steps in BigQuery for a data engineer to review.
The use case
Data engineers spend time writing similar ingest-clean-join pipelines. BigQuery's data engineering agent takes a description such as 'load these files, clean dates, join to customers and build a daily summary' and drafts the pipeline in BigQuery pipelines and Dataform, and can modify existing ones. The engineer sees the generated SQL and steps, edits them, and tests on sample data. The engineer still owns schema decisions, data quality checks and scheduling, and reviews changes to pipelines that feed production reports. Starting with a new, low-stakes pipeline is a sensible way to learn how much editing the drafts need.
Example workflow
- Describe sources, transforms and target tables
- Agent drafts pipeline steps and SQL
- Review and edit each step
- Test on sample data and add quality checks
- Schedule after review
Tools
- BigQuery
- Dataform
- Gemini in BigQuery
Risks and guardrails
- Drafts can silently drop or duplicate rows; add row-count checks
- Changes to production pipelines need review
- Watch for personal data copied into new tables
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- Google Cloud docs: data engineering agent pipelines: docs.cloud.google.com/bigquery/docs/data-engineering-agent-pipelines
- BigQuery skills (official page): github.com/google/skills (checked 2026-11-19)