Building a High-Availability Intelligent Document Summarization System from Scratch

As an architect focused on the practical implementation of AI applications, I have seen too many independent developers and small teams fall into the cycle of "digging holes just to fill them" when building intelligent applications. Everyone wants to leverage the capabilities of LLMs (Large Language Models) to build valuable products, such as an "Intelligent Document Summarization System." This sounds like a classic entry-level AI application: upload a document, output a summary. It seems simple, but there are many nuances involved in building it into a stable, low-cost, and maintainable commercial-grade application.

Today, let's break down how to build a highly available intelligent document summarization system from scratch using a specific scenario case.

I. Business Pain Points: Why is "Document Summarization" Harder Than It Looks?

Assume you are an independent developer, and your target users are law firms, consulting companies, or research teams. These users face hundreds of pages of contracts, tender documents, or papers every day. They are willing to pay for your SaaS service, provided you solve the following three core pain points:

1. The "Forgetfulness" Problem of Long Texts: A user uploads a 200-page PDF tender document. If you stuff the entire text directly into an API, the vast majority of models will either throw an error due to Context Window limitations or suffer from the "Lost in the Middle" phenomenon. This causes the model to ignore key definitions at the beginning of the document and pricing sections at the end, rendering the generated summary completely unusable.
2. Multimodal and Unstructured Data: Real business documents are rarely pure text. They are interspersed with images of financial statements, complex tables, handwritten signatures, and even blurred text from scanned copies. Simple OCR (Optical Character Recognition) often has low recognition rates, leading to missing summary information.
3. The Trade-off Between Cost and Stability: If you force the use of high-end models like GPT-4-32k or Claude 3 Opus to ensure results, the cost of a single processing run could be several dollars, quickly eating into your profits. However, switching to cheaper models with smaller parameters leads to unstable results. As a small team, you don't have the energy to maintain error retry logic for various SDKs from OpenAI, Anthropic, Google Gemini, and others simultaneously.

II. Architecture Design: A Layered Approach to Simplify Complexity

To address the pain points mentioned above, we need to design a scalable architecture. For independent developers, the core of the architecture lies in being "lightweight" and "decoupled." We do not recommend introducing heavy Kubernetes clusters in the initial stage; instead, adopt a "Serverless + Gateway" model.

Core Architecture Layers:

1. Access and Preprocessing Layer:

• Responsible for file upload and format conversion (PDF to Text/Markdown).
• Key components: OCR engine (such as PaddleOCR or cloud vendor APIs).
• Strategy: Document slicing (chunking).

1. Intelligent Logic Layer:

• Responsible for summary generation and key information extraction.
• Introduce the "Map-Reduce" concept to handle long documents: first generate local summaries for each slice, then merge them to generate a global summary.

1. AI Gateway Layer:

• This is the "hub" of the entire architecture. It interfaces with various model vendors downstream and provides a unified interface upstream.

1. Data Storage Layer:

• Vector database (optional, for RAG enhancement): such as Pinecone or Milvus Lite.
• Object storage: stores raw files.

III. Key Implementation Steps

#### Step 1: Document Parsing and Cleaning

Do not blindly trust so-called "universal parsers." For independent developers, it is recommended to first extract text using open-source libraries (such as Python's PyMuPDF), and then call OCR APIs for scanned copies.

The key to implementation lies in the slicing strategy. You cannot simply slice by character count; you must slice by semantic paragraphs. For example, we need to preserve chapter titles so that the model understands the contextual relationships when generating summaries.

#### Step 2: Building the Summary Generation Pipeline

Here, we recommend using a "hierarchical summary" strategy.

1. Map Phase: Split the document into N chunks and concurrently call the LLM, asking it to "summarize the core facts of this segment, preserving data details."
2. Reduce Phase: Concatenate the summaries of all chunks and call the LLM again, asking it to "generate a structured full-text summary based on these segment summaries."

#### Step 3: Prompt Engineering

Prompts determine the output quality. You must not only tell the model "what to do" but also "what not to do."

For example:

> "You are a professional legal assistant. Please summarize the following contract terms, focusing on: breach of contract liabilities, compensation amounts, and effective dates. If not mentioned in the text, please reply directly with 'Not mentioned'; fabrication is strictly prohibited."

IV. Code Implementation and Process Checklist

To make the process clearer, here is a simplified core logic implementation checklist based on Python, demonstrating how to call models via a unified gateway to handle long document summarization.

import os
# Assume we are using a unified API gateway SDK, simulating a generic calling class here
from unified_client import AIGatewayClient

# Initialize client
# All API Key management is completed at the gateway layer, no need to hard-code keys from multiple vendors in the code
client = AIGatewayClient(base_url="https://api.thistoken.ai/v1", api_key="YOUR_GATEWAY_KEY")

CHUNK_SIZE = 2000  # Define slice character count

def read_and_chunk(file_path):
    """Simple text reading and slicing logic"""
    with open(file_path, 'r', encoding='utf-8') as f:
        text = f.read()
    
    chunks = []
    # In actual projects, split by paragraph or recursive characters; simplified here to fixed length
    for i in range(0, len(text), CHUNK_SIZE):
        chunks.append(text[i:i+CHUNK_SIZE])
    return chunks

def summarize_chunk(chunk_text):
    """Map phase: Summarize a single chunk"""
    prompt = f"请总结以下文本片段的关键信息：\n\n{chunk_text}"
    
    # Key point: Specify model via gateway
    # The gateway automatically routes to available models; no need to handle OpenAI or Claude exceptions in code
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # Can configure aliases in gateway, e.g., "smart-summary-model"
        messages=[{"role": "user", "content": prompt}],
        temperature=0.3
    )
    return response.choices[0].message.content

def generate_final_summary(chunk_summaries):
    """Reduce phase: Generate final summary"""
    combined_text = "\n".join(chunk_summaries)
    prompt = f"基于以下各部分的摘要，撰写一份结构清晰的全文总结：\n\n{combined_text}"
    
    # Final summary usually requires a stronger model
    response = client.chat.completions.create(
        model="gpt-4o", # Call a smarter model
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

# --- Main Process ---
def main(document_path):
    print(f"正在处理文档: {document_path}")
    
    # 1. Slice
    chunks = read_and_chunk(document_path)
    print(f"文档已切分为 {len(chunks)} 个片段。")
    
    # 2. Process slices concurrently (async concurrency recommended for actual production)
    partial_summaries = []
    for idx, chunk in enumerate(chunks):
        print(f"正在处理片段 {idx+1}...")
        summary = summarize_chunk(chunk)
        partial_summaries.append(summary)
    
    # 3. Merge to generate final summary
    final_result = generate_final_summary(partial_summaries)
    
    print("\n=== 最终摘要 ===")
    print(final_result)

if __name__ == "__main__":
    # Simulate a long document path
    main("contract_draft.txt")

Code Logic Analysis:

In this flow, we did not directly import the native libraries of openai or anthropic, but instead made calls through a unified client. This is the essence of the architecture design.

V. Why Can a Unified AI API Gateway Reduce Maintenance Costs?

In the code above, you may have noticed our emphasis on the "unified client." For independent developers and small teams, directly interfacing with the APIs of various large model vendors is a huge invisible maintenance burden. This is why I strongly recommend introducing a Unified AI API Gateway in the architecture design.

Specifically, it solves three core pain points:

1. Maintenance Costs of SDK and Interface Unification:

OpenAI, Claude, and Gemini all have different interface parameter formats (e.g., differences in the definition of max_tokens, differences in data formats for Stream streaming returns). Without a gateway, your code base would be filled with massive if-else logic to adapt to different models. Once a new model is released (like Llama 3 or Mistral updates), you need to modify business code.

Gateway Benefit: It provides a standardized OpenAI-compatible interface. Your business code only needs to be written once, and the gateway forwards requests to different models on the backend. Want to switch the summary model from GPT-3.5 to Claude Haiku? Just change the configuration in the gateway console, with zero code changes.

1. High Availability and Disaster Recovery Costs:

Single model vendors inevitably experience service downtime, rate limiting, or response timeouts. Small teams find it difficult to build complex retry and circuit breaker mechanisms on their own.

Gateway Benefit: Mature AI gate