Menu
Back
Discuss a project
Book a call
Back
Back
Articles
8 MIN READ

An Agentic Data Migration Studio That Saves ~50 Hours Per Client

Helping a regulated software provider take an AI-assisted data conversion app from working MVP to a governed data migration studio on Databricks, with a live demo showing around 50 hours of manual work saved per client.

Every new client, a new data puzzle

‍

Our client builds software for a heavily regulated market, where accuracy and tight security matter as much as speed. Onboarding a new customer meant converting their data by hand in SQL and COBOL, and profiling a single customer's files took anywhere from 8 to more than 40 hours. With two disconnected legacy target platforms, only 10 to 15% of fields mapped easily. The rest needed one-off manual work.

‍

That process relied on scarce SQL and COBOL skills and left a limited audit trail, which made it hard to repeat, and harder still to scale as the business grew.

‍

Agents suggest, engineers approve

‍

So we built an agentic workflow that keeps people in control. AI agents suggest mappings, generate transformation code and validate outputs, while engineers review and approve at each gate. We delivered it iteratively, so the client could use and test it early.

‍

The aim was to replace two siloed conversion processes with one AI-assisted data migration studio on Databricks: delivery controls first, then automated mapping, then a platform the client can demo and scale with confidence.

‍

The project covered three core stages: target schema, ingest and profiling, and automated mapping. As requirements sharpened through end-to-end testing, we refined the work in focused increments.

‍

Built on Databricks, one layer at a time

‍

Foundation and control. Approved data sits under version control with end-to-end audit lineage in Unity Catalog and Delta Lake, so every conversion can be traced. Databricks Asset Bundles and Azure Pipelines make each release repeatable.

‍

Ingest and profiling. Ingestion and profiling run as independent workflows on Databricks Jobs, with Strands Agents on Databricks Model Serving reading source files from Amazon S3. That means a step can be rerun without redoing the whole conversion.

‍

Mapping and automation. Automated mapping generates source-to-target code in an app on Databricks Apps, using Databricks Foundation Model APIs, with MLflow tracing and model registration in Unity Catalog, and some UI data held in Lakebase. All of it runs within role-based access, sandboxing and governance controls, and one process now produces outputs for both target platforms.

‍

What the client has today

  • ~50 hours saved per client: a live demo measured roughly 50 hours of manual SQL and COBOL work saved across the conversion steps the platform covers.
  • Ready to demonstrate: the platform now supports live demonstrations of the end-to-end conversion workflow.
  • One process instead of two: a single AI-assisted platform maps and validates outputs for both legacy targets.
  • In production-ready shape: a production environment is set up and has been tested end to end, with auditable lineage and role-based access.

‍

Extending the workflow

‍

Next, we're extending the workflow to further stages of the conversion process, starting with a design workshop alongside the client's team.

‍

Not sure where to start?

Can’t find the answer you’re looking for? Please chat to our friendly team.

Discuss a project