Paper Abstract Generation: Model Selection Pitfalls and the Right Way to Do It

Paper abstract generation looks like the easiest AI application to get started with: input a long text, output a short text, and evaluation seems straightforward. But precisely because it "looks simple," many independent developers and small teams stumble hardest on model selection. This article doesn't intend to give you a "best models list"—such rankings are everywhere online, and most of them won't hold up against your own business. Instead, I want to start with a few real failure modes I've seen, and then work backward to derive an actionable set of selection dimensions.

Three Typical Failure Patterns

Failure one: Deploying the strongest model directly, and getting crushed by costs first.

A friend of mine building a literature management tool initially used a flagship-tier large model for summarization, and the results were indeed good. But paper abstract generation has a brutal characteristic: the per-paper cost isn't high, yet call volume grows linearly. When a user batch-imports two hundred PDFs, that single task means two hundred long-context calls. Multiply the flagship model's premium by that volume, and the bill quickly becomes unbearable. Worse, he later found in his post-mortem that a large share of those papers were highly structured journal articles—abstract, introduction, and conclusion clearly delineated. Text like this works perfectly fine with a lightweight model, meaning the flagship model's capabilities were largely wasted.

Failure two: Switching everything to small models to save money, with quality collapsing in places you can't see.

Others swing to the opposite extreme. Small models handle most well-structured papers without issue, but when they hit survey articles dense with cross-paragraph argumentation, or long papers where chart information is critical, small models start "copying the first and last paragraphs"—stitching the beginning and end together into something that looks like an abstract but actually misses the paper's core thesis. This kind of quality degradation doesn't throw errors, and users may not complain immediately—they just gradually stop trusting the feature.

Failure three: Testing only on English Arxiv papers, then encountering Chinese papers, conference posters, and OCR transcriptions in production.

Many developers' test sets are just a few English papers they happened to download. But real users' input distribution is far more complex: mixed Chinese-English text, math-heavy formulas, chaotically formatted conference papers, even scanned documents with questionable OCR quality. The "optimal model" picked from a single test set falls apart the moment the distribution changes.

The Right Path: Break Down Selection by Scenario Dimensions

The right approach isn't to ask "which model is best," but to ask "which scenario bucket does my input fall into, and what capabilities does each bucket need." I recommend splitting along at least these dimensions:

The key insight is: these buckets don't need to be filled by the same model. Route well-structured short texts through lightweight models to control costs, route long surveys and messy formats through more capable models to protect quality, and offline tasks can even be shifted to off-peak hours. This is the "bucketed routing" approach—but many teams stop here, because they find that every model switch means changing code, swapping keys, and re-testing API behavior all over again, making trial-and-error costs high enough that nobody bothers.

Why a Unified Gateway Is the Prerequisite for Bucketed Routing

This brings us to the engineering problem of model switching. If you connect directly to each provider's official API, then "switching models by scenario" means maintaining multiple SDKs, multiple authentication setups, multiple error-handling logics—and differences in API behavior across providers (streaming response formats, error codes, rate-limit policies) make every switch feel like a fresh integration. After tallying this up, most small teams retreat to "one model for everything"—and end up right back at failure one or failure two from earlier.

A unified API gateway eliminates exactly this friction. With an OpenAI-compatible gateway, for example, switching models is often just a matter of changing the model parameter:

# Well-structured short text, use a lightweight model
response = client.chat.completions.create(
    model="light-model-id",
    messages=[{"role": "user", "content": paper_text}]
)

# Long survey, switch to a capability-oriented model
response = client.chat.completions.create(
    model="strong-model-id",
    messages=[{"role": "user", "content": paper_text}]
)

This delivers practical value on three levels:

1. Trial-and-error costs approach zero. You can take the same batch of test papers and cheaply compare multiple models side by side on your real distribution, rather than relying on someone else's leaderboard.
2. Routing strategies can be adjusted at any time. If you run a certain document type on a small model today and find quality issues tomorrow, changing a model name completes the migration—no infrastructure changes needed.
3. Keys and billing are centrally managed. Team members don't each hold their own provider keys; usage, costs, and anomalies are all visible in one place—especially important for small teams' cash flow management.

One Reminder When Going to Production

After bucketing, don't forget to build a feedback loop: regularly sample outputs from each bucket for manual review, with particular attention to signals of quality collapse in the small-model bucket (user retry rates, overly short abstracts, obvious first-and-last-paragraph stitching artifacts). Bucket boundaries aren't drawn once—they're continuously fine-tuned based on the real quality data you observe. The gateway's unified call logs come in handy here again—they're your cheapest source of quality monitoring data.

Model selection for abstract generation is fundamentally not a technical question, but a question of "what your input distribution looks like." First understand the distribution, then assign models by bucket, then use a unified gateway to drive switching costs to nearly zero—once you've walked this path, you'll find that model selection transforms from a painful one-time decision into a routine operation you can continuously optimize.

If you're about to start building such a setup, begin by registering a unified gateway account: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.