A Candid Post-Mortem: Building Smart Contract Template Filling for an Independent Dev Team

A scenario walkthrough with honest lessons for indie developers and small teams.

The Three Versions I Botched First

Last year I took on a gig: helping a legal-tech startup build "smart template filling for contracts" — a user uploads an employment agreement or lease template, the system automatically identifies the blanks to fill in (Party A's name, amounts, dates, breach-of-contract clause parameters), then fills them in one by one based on information the user provides through conversation.

Sounds simple. My first version went straight to "throw the whole document at the LLM": stuff the entire contract into the prompt and ask the model to find all placeholders and return the filled results.

Three pitfalls appeared in quick succession:

Pitfall 1: Templates exceeding the context window broke everything. For a forty-page equity agreement, the model started producing "truncation hallucinations," inventing clauses out of thin air. Worse, it treated existing examples in the contract like "e.g., Company X" as blanks to fill.

Pitfall 2: Structured output was unstable. I asked the model to return JSON — one day it came back wrapped in a markdown code block, the next day the key names had inconsistent casing, and the day after it prefixed the JSON with "Sure, here are the results." The downstream parsing code needed daily patches.

Pitfall 3: A separate adapter for every model API. I started with OpenAI, then found that domestic Chinese models handled Chinese contract corpora better and integrated two more. Three SDKs, three authentication schemes, three error code systems, three rate-limiting implementations — with only two backend devs on the team, maintenance costs spiraled out of control.

The Right Approach: Split "Filling Blanks" into a Three-Stage Pipeline

Looking back, the root cause of failure was using a generative approach to solve a structured task. The right approach is to decompose the problem:

第一段：占位符抽取（规则优先）
  ├─ 正则匹配 {{...}}、____、【】等显式占位符
  └─ 仅对无标记的隐式空位调用小参数模型做 NER 抽取

第二段：信息收集（对话式，状态机驱动）
  ├─ 维护字段清单与填充状态表
  ├─ 每轮只问缺失字段，支持"一次性粘贴全部信息"的兜底解析
  └─ 模型仅负责"从用户自然语言中提取字段值"，单字段小请求

第三段：填充与校验
  ├─ 用模板引擎做字符串替换，AI 不碰最终文档
  ├─ 金额、日期、证号走正则校验，校验失败回问
  └─ 输出前跑一遍 diff 比对，确保未填区域零改动

The key principle in one sentence: let AI do understanding and extraction; let code do replacement and persistence. Each model request handles only a single field or a small chunk of context, minimizing the room for hallucination. JSON output appears only in the narrow "single-field extraction" scenario, strictly constrained with function calling / JSON mode — write the parsing layer once and never touch it again.

Why a Unified API Gateway Saves Most of the Maintenance Cost

Once the three-stage architecture was settled, model selection became a frequent operation — different field types suit different models: lightweight models for placeholder extraction, larger models for semantic matching of complex clauses, plus nightly batch jobs for historical templates that should exploit low-cost compute windows.

Direct connections to each vendor's API would mean:

• Auth, retry, and rate-limiting logic duplicated N times, with different error code formats per vendor;
• Switching models = changing code, while the cost-effectiveness ranking of models changes almost monthly;
• Bills scattered across multiple dashboards, cost attribution done by manually pulling spreadsheets.

After adopting a unified AI API gateway, these problems were resolved at once: all requests go through a single OpenAI-compatible endpoint, and switching models is just changing a model name parameter; the gateway layer uniformly handles retries, fallbacks, and multi-key rotation; billing and usage stats are naturally aggregated — you can see at a glance which feature burns how many tokens. For a two-person team like ours, this was equivalent to deleting "model ops" from our codebase entirely — our two backend devs could finally focus on improving field extraction accuracy instead of being SDK repairmen.

Results and a Few Reminders

In the refactored version, placeholder identification accuracy reached nearly 100% thanks to rule-based fallbacks, and the miss rate for implicit blank extraction dropped to an acceptable range. The most visible change: customer support tickets about "filled in the wrong place" essentially disappeared.

Three reminders:

1. Legal scenarios must retain a human confirmation step — after AI fills in the fields, force the user to verify each item, with the UI highlighting all AI-filled positions. This is both a compliance requirement and a liability shield;
2. Mask sensitive information before sending it to the model — mask ID numbers and bank account numbers before sending, and restore them locally after the response;
3. Build a small test set for each field type — a few dozen annotated samples let you quickly run regression tests when switching models, avoiding the "switched to a cheaper one and quality quietly dropped" trap.

If you're building a similar AI document application, rather than migrating code back and forth between multiple model APIs, go with a unified gateway from day one. Sign up here: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.