The biggest cost lever in a generative AI system is whether you call a model at all. That is the core argument AWS engineers make: teams routinely route every task through a large language model—including input validation, formatting, and routing—when deterministic code could do the same work. While individual model calls are inexpensive, their cumulative effect at production scale can make them a significant architectural cost. Employing deterministic rules instead of models for suitable tasks can lead to cost savings, reduced latency, and increased predictability [1].
The Core Dilemma: Code for Rules, Models for Judgment
The AWS guidance starts one step earlier than most cost advice. Conventional cost optimization, including the AWS Well-Architected Generative AI Lens, begins at model selection—which model to use for a step. AWS argues the more important question goes unasked: does the step need a model at all?
The distinction comes down to the nature of the task. Some tasks are deterministic: the same input always yields the same correct output, and a rule can capture that mapping. Others require judgment, creativity, or handling of ambiguity that is hard to encode as rules. AWS frames the decision as a flowchart with three labeled lanes: "Runs as code," "Questions," and "Runs on a model." Tasks fall into the code lane when rules fully specify them; they fall into the model lane when rules are difficult to define or the task is inherently ambiguous.
The cost of getting this wrong compounds. AWS notes that when every step goes through a model, testing becomes harder and audits run slower, and each new model launch restarts the debate about which model to use—while the earlier question, whether the step needs a model, never gets asked.
Generative AI Task Decision Framework
Use the matrix below to classify any task in your workflow. Evaluate the task against the criteria in order: if a deterministic rule can fully specify the input-to-output mapping, implement it as code; if the task requires judgment under ambiguity, use a model.
| Task | Implementation | Cost/Performance Impact | Key Consideration |
|---|---|---|---|
| Input validation | Implement as Code | Eliminates per-call model cost and latency | Validation rules are fully specifiable; each LLM call is cheap in isolation but costs accumulate at production scale |
| Data formatting | Implement as Code | Reduces cost, latency, and unpredictability | Deterministic transformations need no judgment |
| Workflow routing | Implement as Code (if rule-based) | Avoiding unnecessary model calls is the biggest cost lever | Routing decisions follow explicit rules; skipping the model also removes non-determinism |
| Judgment-based tasks | Implement with Model | Model cost is justified where rules cannot do the job | Models excel where rules are difficult to define or tasks are inherently ambiguous |
The reasoning path is straightforward: ask whether the task's correct output is fully determined by its input. If yes, code is cheaper, faster, and predictable. If no—if the task demands interpretation, creativity, or language understanding—the model lane applies. The result for any given task is a clear determination: "Implement as Code" or "Implement with Model."




