Most of my projects have been some kind of AI workflow. Load documents or emails, read the attachments, classify them, and extract the key information. Decide whether that information is enough to judge or triage on, then push it further down the line. Save it in a database and log what happened. A job like that might run once a day or once an hour.
All of that is easy enough to understand. What I think is missing when we talk about workflows is the feedback loop, the part where we look at how the workflow went and make it better next time. That loop has always been there, but it was implicit. The whole process, as I see it, is about making implicit stuff explicit.
I picture the workflow and the feedback loop as two stones rolling together.
This extends the loop I wrote about before. It's still that same loop. I only pull it apart here to show that one part runs and operates while another part observes.
Data engineering workflows from before AI had this loop too. We had a workflow, it worked, and then someone noticed a bug, or a stakeholder wanted to add or change something. That's how we moved the software in the direction it needed to go. People were the feedback loop, observing the running system and suggesting changes. It just wasn't really noticeable, because it was implicit in the software development process.
With AI, this loop will just be obvious. Run a workflow once or a couple of times, and some AI extraction may not work the way you'd like. So from time to time, or on demand, you run the feedback loop. It will probably be AI powered too, and to some extent it already should be.
I think we should try to make everything implicit here explicit and visible, so that we can use it fully, not hoping for the best but actually doing the best.
Take a workflow that works and then fails for some input. That failure is already an input to the feedback loop. At a really high level, the feedback loop is an AI agent, or some other workflow, that runs when something like this happens. It takes the workflow, along with the logs and statistics from the run, and its goal is to make the workflow better next time.
By better I mean optimized towards some metrics. They might be set in stone or they might not, but everyone should know them. Those metrics are our hypothesis of what a better workflow is. The hypothesis can be wrong, so the metrics may change over time. This is just like the scientific method. We test hypotheses and they change, and we can't really prove any one of them. We can only show that a hypothesis doesn't match the results of a particular experiment.
A lot of this work can already be delegated to agents. If something is failing, we can wake up a coding agent to pick it up, gather evidence, and propose a PR. The evidence that the bug was actually fixed is a unit test that fails, and then passes after the change. That's hard to dispute, and it tells us this is the change to merge.
Fixing things that don't work, like a bug, an infrastructure problem, or missing configuration, is the fairly simple case. The feedback loop can do much more, because it also reaches things we haven't observed or thought about yet. That can be just a question to the agent: find me something. Find me some characteristic of this data that's worth paying attention to. Imagine the agent figures out that 90% of the cases are really simple. Straightforward matching could handle those with no thorough AI analysis, and the pipeline's performance could improve a lot.
The workflow doesn't need AI for any of this. A plain data processing pipeline can have a feedback loop that uses AI. The job there is to surface what's unintuitive to us, or what we currently consider not worth our time, more than to change the process itself.
The really good thing is that, if you approach this correctly, everything is versioned. The pipeline lives in a repository, and the data is versioned through the ingestion process. So if you propose a change to a metric, or to how you measure something, you can run it back on the historical data and see whether you regress anywhere.
We'll probably find stuff that never really worked, and we just weren't aware of it. That's a very human thing, I guess. Using AI here won't necessarily make the workflow bulletproof. It does let us explore wider domains in a more robust way and become aware of much more intricate things. A regression doesn't necessarily mean we made a mistake. It can mean we weren't aware of something, and that in itself is valid input for the discussion and for figuring things out.