Databricks CEO and co-founder @alighodsi joined @technology to discuss the launch of Genie Code, an autonomous AI agent built specifically for data teams.

Building Reliable Data Agents with Databricks Genie Code

Why this matters for builders

Genie Code lets you hand off entire data-engineering tasks to an autonomous AI agent that writes, iterates, and debugs code on your behalf inside your Databricks workspace. Six months ago the conversation was about inline autocompletion; today the conversation is about full agentic engineering for data teams. This shift moves analysts and engineers from writing every line of Spark or SQL to defining outcomes and letting the agent handle implementation, iteration, and quality control.

The change unlocks a new workflow: describe the business problem in natural language once, get a production-ready pipeline that respects your lakehouse, Unity Catalog, and governance rules. Builders who master this pattern ship 5–10× faster while keeping control over correctness and cost.

When to use it

• You need to productionize a recurring analytics request that currently lives in a notebook.
• You want to generate ETL pipelines for new source tables without writing boilerplate.
• You are maintaining a large lakehouse and want to automate schema evolution, quality checks, and incremental loads.
• You have domain experts who understand the data but are not fluent in PySpark or Delta Lake.
• You want to explore multiple implementation strategies quickly and pick the best one.

The full process

Define the goal

Start every Genie Code session with a crisp, measurable outcome.

Bad: “Make a dashboard for sales.” Good: “Create an incremental daily pipeline that ingests from the raw.sales_transactions Delta table, deduplicates on transaction_id, applies business rules from the reference.discount_rules table, writes to silver.sales_cleaned with Unity Catalog lineage, and publishes a quality report to the monitoring schema. Run on the small cluster with a 15-minute SLA.”

Write this goal in a Markdown cell at the top of a new notebook. This becomes your system prompt and acceptance criteria.

Shape the spec/prompt

Genie Code performs best when you give it structured context. Use this starter template (copy-paste ready):

**Task:** <one-sentence objective>

**Input tables:**
- raw.sales_transactions (append-only, partitioned by date)
- reference.discount_rules (static, < 10k rows)

**Output tables:**
- silver.sales_cleaned (idempotent, merge on transaction_id + date)
- monitoring.quality_daily (append)

**Requirements:**
- Use Delta Live Tables or Auto Loader where appropriate
- All tables must be Unity Catalog registered
- Add data quality expectations using @expect_or_fail
- Keep total cost under 0.5 DBU per run
- Include full column-level lineage comments

**Success criteria:**
- Zero duplicate transaction_ids in silver
- Quality report shows < 0.1% null rate on amount
- Pipeline completes in < 12 minutes on current data volume

Add any relevant sample data, business rules, or existing notebook snippets. The more precise you are, the fewer correction cycles the agent needs.

Scaffold

Create a fresh Databricks notebook in a repo-backed workspace (required for version control).

1. Add the goal markdown cell.
2. Run the first Genie Code prompt: “Scaffold the complete project structure using Databricks Asset Bundles and Delta Live Tables.”
3. Let the agent create the folder layout, databricks.yml, pipeline definitions, and test notebooks.

Review the generated files. You should see:

• A clean DLT pipeline definition
• Expected schema declarations
• Quality expectation decorators
• A separate monitoring notebook

Implement

Now iterate with the agent. Typical sequence:

1. “Implement the bronze ingestion layer using Auto Loader with schema evolution.”
2. “Add the silver transformation with merge and the exact business rules I described earlier.”
3. “Write the quality expectations and monitoring table writer.”
4. “Generate a unit test notebook that uses pytest and loads 100 synthetic rows.”

After each step, read the code. Genie Code is autonomous but not infallible. You remain the reviewer.

Validate

Run these checks every time:

• Syntax & catalog check — Execute the pipeline in a test catalog. Verify all tables are created in the right schema.
• Lineage — Open the Unity Catalog lineage tab. Confirm column-level provenance.
• Quality — Intentionally inject bad data and confirm expectations fail the job.
• Performance — Run with realistic volume on a small cluster. Measure DBU and wall-clock time.
• Idempotency — Re-run the pipeline on the same data. Row count and content must not change.

If any check fails, reply to Genie Code with the exact error and your desired behavior:

“The merge is creating duplicates on re-run. Fix the merge condition to use both transaction_id and ingestion_date.”

Ship safely

1. Commit the entire bundle to Git (the agent can generate the PR description).
2. Promote the bundle from dev → staging using Databricks Asset Bundles CLI or the UI.
3. Run one full end-to-end test in staging.
4. Schedule the pipeline and set up alerts on the monitoring table.
5. Document the prompt that created the pipeline in the repo’s README so future engineers can evolve it.

Copy-paste prompts or snippets

Starter system prompt (paste into Genie Code):

You are an expert Databricks data engineer. Always use Delta Lake, Unity Catalog, and current best practices (2025). Prefer Delta Live Tables for pipelines. Never invent columns. Ask for clarification if business logic is ambiguous. Output complete, ready-to-run code with comments.

Quality expectation example the agent should produce:

@dlt.expect_or_fail("valid_amount", "amount > 0")
@dlt.expect_or_fail("has_customer_id", "customer_id IS NOT NULL")
def silver_sales_cleaned():
    return dlt.read("bronze_sales").merge(...)

Pitfalls and guardrails

What if the agent hallucinates columns that don’t exist?

Always give it the exact schema first. Paste DESCRIBE TABLE output or Unity Catalog table comments into the prompt. If it still invents columns, reply: “Only use columns that appear in the provided schema.”

What if the generated pipeline is too expensive?

Add a cost guardrail in the initial spec: “Target < 0.5 DBU per run.” After the first run, ask Genie Code to optimize: “The job used 1.8 DBU. Refactor to use predictive optimization and smaller cluster.”

What if the agent ignores Unity Catalog?

Explicitly list it in every prompt: “All output tables must be written to catalog prod schema silver with proper grants.”

What if tests are flaky or missing?

After implementation, always add: “Generate a pytest notebook that runs with synthetic data and asserts the exact success criteria I listed.”

What to do next

• Pick one recurring manual notebook in your workspace and replicate it with Genie Code this week.
• Create a “Genie Code prompt library” notebook that stores your best performing prompts.
• Set up a weekly 30-minute “agent review” meeting with your team to share successful and failed prompts.
• Measure before/after: track how many hours it took to ship the same pipeline manually vs. with Genie Code.

The age of data agents is here. Builders who treat Genie Code as a senior data engineer—giving clear specs, reviewing output, and iterating—will ship faster and with higher quality than teams still writing every line by hand.

Sources

• Databricks press release: “Databricks Launches Genie Code: Bringing Agentic Engineering to Data Work” (https://www.databricks.com/company/newsroom/press-releases/databricks-launches-genie-code-bringing-agentic-engineering-data)
• Original announcement tweet by @databricks (https://x.com/databricks/status/2031830130373374234)
• Interview with Ali Ghodsi on the shift from code assistance to autonomous agents (PR Newswire, CXOToday, HPCwire coverage)