Long Context Models: "Fitting It In" Doesn't Mean "Understanding It Well"

For indie developers and small teams integrating AI APIs, 2024 has undoubtedly been the year of the "long context" explosion. From the early 4k token limit to the now widespread 128k, 200k, or even million-level tokens, Large Language Models (LLMs) seem to be sprinting towards "infinite memory."

However, as a model selection consultant, I observe many developers falling into a misconception: believing