The biggest cost lever in a generative AI system is whether you call a model at all. That is the core argument AWS engineers make: teams routinely route every task through a large language model—including input validation, formatting, and routing—when deterministic code could do the same work. While individual model calls are inexpensive, their cumulative effect at production scale can make them a significant architectural cost. Employing deterministic rules instead of models for suitable tasks can lead to cost savings, reduced latency, and increased predictability [1].

The Core Dilemma: Code for Rules, Models for Judgment

The AWS guidance starts one step earlier than most cost advice. Conventional cost optimization, including the AWS Well-Architected Generative AI Lens, begins at model selection—which model to use for a step. AWS argues the more important question goes unasked: does the step need a model at all?

The distinction comes down to the nature of the task. Some tasks are deterministic: the same input always yields the same correct output, and a rule can capture that mapping. Others require judgment, creativity, or handling of ambiguity that is hard to encode as rules. AWS frames the decision as a flowchart with three labeled lanes: "Runs as code," "Questions," and "Runs on a model." Tasks fall into the code lane when rules fully specify them; they fall into the model lane when rules are difficult to define or the task is inherently ambiguous.

The cost of getting this wrong compounds. AWS notes that when every step goes through a model, testing becomes harder and audits run slower, and each new model launch restarts the debate about which model to use—while the earlier question, whether the step needs a model, never gets asked.

Generative AI Task Decision Framework

Use the matrix below to classify any task in your workflow. Evaluate the task against the criteria in order: if a deterministic rule can fully specify the input-to-output mapping, implement it as code; if the task requires judgment under ambiguity, use a model.

Decision matrix for classifying workflow tasks as code or model implementations
TaskImplementationCost/Performance ImpactKey Consideration
Input validationImplement as CodeEliminates per-call model cost and latencyValidation rules are fully specifiable; each LLM call is cheap in isolation but costs accumulate at production scale
Data formattingImplement as CodeReduces cost, latency, and unpredictabilityDeterministic transformations need no judgment
Workflow routingImplement as Code (if rule-based)Avoiding unnecessary model calls is the biggest cost leverRouting decisions follow explicit rules; skipping the model also removes non-determinism
Judgment-based tasksImplement with ModelModel cost is justified where rules cannot do the jobModels excel where rules are difficult to define or tasks are inherently ambiguous

The reasoning path is straightforward: ask whether the task's correct output is fully determined by its input. If yes, code is cheaper, faster, and predictable. If no—if the task demands interpretation, creativity, or language understanding—the model lane applies. The result for any given task is a clear determination: "Implement as Code" or "Implement with Model."