We Thought Our AI Workflow Compiler Was Ready. Then We Doubled the Eval Set.

A little while ago, I wrote about why we added an intermediate representation and compiler between the LLM planner and the workflows Fetch Hive actually creates. The reason was fairly straightforward: as Fetch Hive became more capable, we were asking the model to remember an increasing amount of implementation detail that it really shouldn’t have been responsible for in the first place.

LLMs are very good at understanding what somebody wants. They are considerably less reliable when you also expect them to remember every workflow node, parameter, interpolation path, model slug, connection rule and graph dependency needed to turn that request into a valid production workflow. Every new capability made the direct-generation approach more fragile because every new capability also gave the model another way to generate something that looked sensible but was subtly wrong.

The compiler was our attempt to put a proper boundary around that. Instead of asking the model to create the final Fetch Hive workflow directly, the planner describes the intended plan using a smaller Plan IR, and deterministic code takes care of the rest. Graph wiring, interpolation paths, model selection and prompt envelopes are compiler output rather than things we expect the planner to construct itself.

Architecturally, this made much more sense to us. The problem was that a cleaner architecture is not necessarily a reliable one, so after building the compiler we needed a way to measure whether the planner could actually use it consistently across the sorts of requests people give Fetch Hive.

That turned into a much bigger eval project than I initially expected.

What we actually needed to test

The obvious way to test a compiler is to give it some input, make sure compilation succeeds and mark the test as passed. That works perfectly well when the thing you care about is whether some deterministic source code can be lowered into another deterministic representation, but it is nowhere near enough when the input is being authored by an LLM.

A workflow can compile successfully and still be completely wrong.

If somebody asks Fetch Hive to take a CSV of companies, research each one, find the CEO, structure the results and write them into Google Sheets, then a workflow which searches for the companies and stops there is not a successful result. The graph might validate and every individual step might be perfectly legal, but the workflow still hasn’t done what the user asked for.

The same problem exists with integrations, scheduling and execution mode. If somebody asks for a report to be sent to Slack and the resulting workflow writes it to Google Sheets, or the planner invents a daily schedule for a workflow that was supposed to run manually, the fact that the result technically compiles doesn’t make it correct.

Because of that, our eval harness scores the resulting plan rather than just the compiler response. Depending on the request, we can assert which resources should exist, which workflow node types should be present, whether a particular integration is required, whether execution should be persistent or one-shot, whether a schedule should exist, and how many compile attempts or clarification questions we consider acceptable.

The two main measurements we ended up focusing on were first-try success and final success. First-try success means the planner produced the expected result without needing another compile attempt or asking an unnecessary question. Final success means the system reached the expected result within the allowed retry and repair budget.

For the current gate, we require the mean across three runs to reach at least 80% first-try success and 95% final success. We also require zero invariant violations, zero internal compiler errors, zero applicable no-op calls, and every applicable request has to complete on every run.

The three-run requirement turned out to matter a lot more than I expected.

The first eval set

Our original production gate contained 24 applicable requests, and the first result was pretty rough. We started at 33.3% first-try success and 45.8% final success, which at least removed any temptation to pretend the compiler was finished.

From there we started working through the failures and running the gate again. The numbers moved from 33.3% / 45.8% to 58.3% / 62.5%, then 66.7% / 83.3%, then 79.2% / 83.3%. Another pass got us to 91.7% first-try and 91.7% final.

The useful part of this process was that every failed request left behind the generated plan, compiler diagnostics and the conversation that produced it. We could see what the planner had tried to do, why the compiler had rejected it and whether the next attempt actually fixed the underlying problem.

That started exposing a pattern which became important later. Some failures were genuinely planner mistakes, but others were cases where the planner’s intent was obvious and the compiler was being unnecessarily strict. In those cases there was very little value in spending another model call asking the planner to rephrase the same idea in a slightly different way when deterministic code could fix it safely.

We gradually moved more of those repairs into the compiler, and the original gate started looking quite healthy.

By the end of Gate v1, the three-run mean was 86.1% first-try success and 97.2% final success.

At that point I was pretty happy with the result. We had gone from failing more than half the requests to comfortably clearing the final-success threshold, and the architecture seemed to be doing what we had hoped it would do.

There was one thing which stopped me from feeling completely comfortable with it, though. The three identical runs were not especially identical in terms of outcome.

The first run hit 91.7% first-try and 100% final success. The next run, using the same compiler build, same workspace and same request set, dropped to 79.2% first-try and 95.8% final. The third landed at 87.5% and 95.8%.

That wasn’t necessarily a problem by itself because the planner is stochastic, but it was a useful reminder that one good run does not prove much. If we had stopped after the first run, we would have had a lovely 100% final-success number and a fairly misleading idea of how stable the system actually was.

The bigger concern was that by this point we knew the original 24 requests extremely well. Every recurring failure had been investigated, every awkward edge case had been discussed and the compiler had been changed repeatedly based on what those same requests were exposing.

Eventually it became difficult to tell whether the system was becoming more generally reliable or whether we were just getting very good at passing our own test set.

So rather than declaring the compiler finished, we made the test harder.

Doubling the eval set

Gate v2 doubled the number of applicable requests from 24 to 48. There are 55 rows in the complete set, with known capability gaps kept outside the scored success rate. The original Gate v1 requests remained unchanged and we added another 24 applicable cases around areas which the first gate didn’t exercise enough.

The additional requests covered more combinations of APIs, existing workflows, knowledge bases, one-shot execution, scheduling, transforms, integrations and looping behaviour. We also checked recent production thumbs-downs to see whether there were newer real-world failures we could add, but the six production cases already represented in the original gate were still the only ones available, so the additional requests were synthetic.

That doesn’t mean we were generating random nonsense in an attempt to make the model fail. The idea was to construct reasonable requests that exercised combinations of capabilities the original set didn’t cover properly.

The result was fairly immediate.

The first Gate v2 baseline averaged 83.3% first-try success and 88.9% final success across the three runs. All three runs were NO-GO.

I don’t think it would be accurate to call that a regression from the 97.2% Gate v1 result because the request set had changed. The compiler hadn’t suddenly become worse. We had simply made the definition of reliability broader, and the system no longer met it.

That was probably the most important point in the whole eval process. The original test set had been good enough to improve the compiler substantially, but it wasn’t broad enough to justify the level of confidence we were starting to put in the score.

Once the larger gate failed, the next job was working out why.

What the larger gate actually exposed

The Gate v2 failures were useful because they weren’t all variations of the same bug. Some came from the planner, some came from the compiler, some came from missing context, and one of them turned out to be functionality that Fetch Hive didn’t actually support yet.

That meant the solution couldn’t simply be to keep adding instructions to the planner prompt. We had to look at each failure and decide which part of the system ought to own it.

One good example was scheduling. The planner occasionally attached a schedule to a workflow even though the user hadn’t asked for one. We already validated schedules, so malformed triggers were caught, but the larger eval exposed a more subtle case where the invented trigger was completely valid. Because there was nothing structurally wrong with it, the compiler quite reasonably accepted it.

The actual problem was that the schedule should never have existed in the first place.

The fix was to make the compiler drop a trigger when the original request, plan intent and resource name contain no scheduling intent. There is no useful reason to send that back to the model and ask it to reconsider. The compiler already has enough information to know the schedule was not requested.

Another failure involved image generation. The planner produced size: "16:9", which is completely reasonable if you’re describing an image, but the backend expects one of its supported concrete dimensions. Rather than teaching the planner another implementation-specific enum, the compiler now normalises common aspect-ratio hints. 16:9 becomes 1536x1024, 9:16 becomes 1024x1536, and 1:1 becomes 1024x1024.

The planner had already understood what the user wanted, so making it memorise the backend representation would have been solving the wrong problem.

Existing workflows exposed a different kind of issue. One request needed the system to use a saved workflow, but the planner didn’t have enough useful information about which workflows existed in the workspace. At the same time, the compiler expected a real workflow reference.

The result was references such as MOST_RECENTLY_UPDATED_SAVED_WORKFLOW, which made perfect sense conceptually and were completely useless technically.

That wasn’t really a reasoning failure. We had asked the planner to select something without giving it the available options. We added recent saved workflows to the planner context and improved compiler-side workflow resolution so workflow references can be matched by name. If the request is genuinely ambiguous, the planner can still ask the user which workflow they mean.

One of the most useful failures was R55, because it turned out not to be a planner or compiler problem at all. The request needed to take RSS results, filter them and remove duplicates. The planner kept failing to produce the expected workflow shape, and once we investigated it properly the reason was obvious: Fetch Hive didn’t actually have generic filter or dedupe transforms.

Our transform step could flatten arrays, concatenate arrays, build rows and convert objects into grids, but it could not perform those two operations. We had effectively written an eval asking the planner to use functionality that didn’t exist.

The solution was to add the functionality. filter and dedupe became real runtime operations, were added to the IR and got corresponding settings in the workflow editor.

That was probably the point where the eval work stopped feeling like a separate testing exercise and started feeling like part of product development. The eval wasn’t only telling us that the compiler was wrong. It was exposing missing pieces in the system around it.

The first fix pass

After working through the Gate v2 baseline, the first fix pass ended up touching several layers of the stack.

The compiler gained things like image-hint normalisation, unrequested-trigger handling, automatic declaration of referenced inputs, safer outside-loop reference handling, Google Sheets argument aliases, workflow resolution and knowledge-base defaults. The planner got better examples around APIs, scheduling and transforms, as well as better workspace context. The runtime itself gained the new filter and dedupe operations.

When we reran the complete 48-request gate three times, first-try success improved from 83.3% to 88.9% and final success moved from 88.9% to 93.1%. That was a good improvement, but it was still below the 95% final-success threshold, so all three runs remained NO-GO.

This is one reason I like defining the gate before looking at the result. If you decide what counts as “good enough” after the number appears, 93.1% starts looking extremely tempting.

We had already decided that wasn’t the threshold, so we kept going.

The second fix pass

By the time we reached the second pass, most of the obvious problems had been removed, so the remaining failures tended to be awkward interactions between otherwise reasonable planner decisions.

One request created a nested research loop. The workflow was already iterating through competitors, then the planner added another for_each around a scrape operation inside that loop. We deliberately don’t support arbitrary nested iteration because the behaviour becomes messy very quickly once you start mixing in delivery steps, human input and references between loop scopes.

In this particular case, however, the planner’s intention wasn’t ambiguous. It was just another way of expressing research over the current item. Instead of rejecting the whole graph, the compiler now recognises that safe pattern, flattens the nested research operation and rewires the relevant references. We still record the repair as nested_for_each_flattened so it remains visible in diagnostics.

Another failure involved runtime step identifiers. Plan IR deliberately doesn’t expose values such as step_5 because those are supposed to be assigned by the compiler later. The planner should refer to its own IR node ids and let the compiler handle runtime wiring.

Occasionally the model generated something like {{step_5.output}} anyway. Technically that breaks the IR contract, but there is no real ambiguity about what it means. Spending another inference call asking the planner to express the same reference differently doesn’t improve the result, so the compiler now resolves those accidental runtime identifiers back to their IR nodes before doing its normal reference rewriting.

Scheduling produced a similar edge case around one-shot execution. A request effectively asked for something to run tomorrow at a specific time and not repeat. The planner interpreted that as one_shot, which is a reasonable interpretation if you only think about how many times the workflow should execute.

The problem is that a workflow scheduled for tomorrow still has to exist until tomorrow. We had conflated execution count with persistence.

The compiler now treats a workflow that combines one-shot execution with a scheduled trigger as persistent unless the user explicitly says not to save it, for example with language such as “don’t save this” or “just run this now”.

That kind of distinction is exactly what the broader eval set was good at exposing. None of these were huge architectural failures, but together they were enough to make the system less reliable than the original gate suggested.

We also found a bug in the eval itself.

R53 expected a Slack delivery. On two of the runs, the generated workflow actually contained both Google Sheets and Slack delivery steps, but the scorer only looked at the first deliver node. If Google Sheets appeared first, the whole plan was marked wrong even though Slack was also present.

The fix was to change the scorer so the requested integration passes when any matching delivery node exists. The fixture expectation itself did not change.

I think this is worth mentioning because eval suites can become dangerous once people start treating their output as unquestionable truth. Sometimes the product is wrong, but sometimes the test is.

The final result

After the second fix pass, we ran the full Gate v2 set another three times.

Across 144 applicable executions, first-try success reached 96.5% (139/144) and final success reached 99.3% (143/144). More importantly, every individual run passed. The first run scored 97.9% first-try and 100% final, the second scored 95.8% and 100%, and the third scored 95.8% and 97.9%. We also had zero invariant violations, zero internal errors and zero applicable no-op calls on every run.

At that point I was genuinely very happy with the result, but mostly because of how we’d reached it rather than because 99.3% happens to look nice on a chart.

We had started with a 24-request gate, improved it until the numbers looked strong, then questioned whether we had simply become too good at our own benchmark. We doubled the coverage and immediately dropped final success back below 90%, which was a fairly good indication that the broader test had found something real. We then worked through those new failures across the planner, compiler, runtime and eval framework itself until the expanded gate passed on all three runs.

That made the final number considerably more convincing than the earlier 100% single run.

There was still one final miss across the 144 scored executions. R23 successfully compiled a workflow, but after the allowed attempts the resulting plan contained search without the structure node that particular request required. The workflow itself was valid, but it wasn’t the workflow the eval expected, so it failed.

I’m fine with that.

The goal isn’t to force the benchmark to say 100% because 100% makes a nicer screenshot. A 99.3% result with one remaining failure we understand is much more useful than a suspiciously perfect benchmark we’ve accidentally optimised around.

What we actually learned from the evals

When we started this work, I thought the eval suite would mainly act as a release gate. We’d build the compiler, test it, make sure the score was high enough and then turn the feature on.

It still serves that purpose, but the more useful part has been how much it changed the way we think about the planner/compiler boundary.

A failed request can mean several very different things. The planner might genuinely have misunderstood the request. It might have understood the request perfectly but expressed part of the IR slightly incorrectly. The compiler might be rejecting something it could safely repair. The planner might not have been given enough workspace context. The underlying runtime might not support the requested operation at all. Occasionally, the eval itself might be wrong.

Those failures need different solutions.

If we’d responded to all of them by making the system prompt longer, we would probably have ended up with a much bigger prompt and many of the same problems.

Instead, the compiler now performs more structural repair, the planner has better context, scheduling behaviour is stricter, existing resources are easier to resolve and the runtime gained capabilities that didn’t exist before. A surprising amount of that came directly from running the evals and investigating why individual requests failed.

That, more than the final percentage, is probably what made the work worthwhile.

Where we go from here

Doubling the gate from 24 to 48 applicable requests immediately found problems that the original set couldn’t see, so I don’t think there is any reason to assume 48 is somehow enough.

The next step is to keep expanding the gate towards 100 applicable requests and eventually beyond that as Fetch Hive gains more capabilities. More requests is useful, but the quality and diversity of those requests matters just as much. We want more combinations of workflow features, more awkward interactions between otherwise normal capabilities and, increasingly, more cases taken directly from real production failures.

Failed and repaired compiles already feed into compiler telemetry with enough information for production failures to become future fixture candidates. Over time, I want the eval set to look less like a benchmark we invented internally and more like a growing collection of the ways real users have managed to break the system.

That will probably make the score worse again at some point, which is fine. In fact, based on what happened when we moved from 24 to 48 requests, I would be more suspicious if it didn’t.

The original reason we built the compiler was that I don’t think an LLM should be responsible for every implementation detail involved in constructing a production workflow. The eval work has only reinforced that view, but it has also helped us work out where the boundary should actually sit.

The model should handle the parts that genuinely require understanding and reasoning. The compiler should handle the things we can determine safely and consistently, including repairing harmless mistakes where the intended meaning is obvious. The user should only need to step in when there is real ambiguity.

The final Gate v2 result of 99.3% was obviously something I was very happy with, but the more useful outcome was understanding why the larger gate initially dropped us back to 88.9% and then using those failures to improve the system.

The score was nice. The failures were probably more valuable.