OctoLong trains LMs on cross-repo code context

OctoLong builds dependency-rich code contexts and uses them to improve long-context model training.

• Research org: Unspecified in arXiv abstract
• Core data: ~6.2B OctoLong tokens in a ~50B-token mixture
• Breakthrough: Recursive retrieval of code references with AST, language server, and package manager

Anyone who has tried to debug a large codebase knows the pain: the answer is rarely in the file you are staring at. It lives in another repository, behind a chain of imports, references, and package boundaries that ordinary context windows do not capture well.

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling tackles that problem by manufacturing better long-context training data for code. The paper is less about making models simply “longer” and more about making those extra tokens actually useful for repository-scale reasoning.

What problem this paper is trying to fix

Long-context language models have become important because developers are asking them to do more than answer short prompts. They need to support in-context learning, self-improvement loops, and long-horizon agent workflows. But the paper argues that the existing long-context corpora are a weak fit for that job because they are dominated by books, academic articles, and code repositories that are finite and often thin on long-distance dependencies.

That matters for engineering use cases. If a model can hold a lot of text but the training data does not force it to follow dependencies across files, packages, and repositories, it may still struggle when a task depends on cross-file state, API usage, or a chain of references. OctoLong is built to close that gap.

The core idea is straightforward: instead of relying on generic long text, create code contexts that are rich in dependency structure. The paper says these contexts can reach millions of tokens in length, which is a very different training signal from a normal snippet or even a single repository snapshot.

How OctoLong works in plain English

OctoLong is described as a context engineering pipeline. It instruments three pieces: an AST parser, a language server backend, and a package manager. Together, these components enable recursive retrieval of code references.

In practice, that means the pipeline can start from a piece of code and keep pulling in the things it depends on: definitions, referenced symbols, related files, and package-level context. The result is not just “more code,” but code that is connected in the same way real software systems are connected.

This is the key technical move in the paper. The authors are not proposing a new tokenizer or a new attention mechanism here. They are changing the training corpus so the model sees more dependency-rich, cross-repository structure during mid-training.

The paper then uses that data to train OctoLong-Instruct, a suite of open long-context language models derived from base models ranging from 600M to 14B parameters. Training happens in two stages: context-extension mid-training on a roughly 50B-token mixture, which includes about 6.2B tokens of OctoLong code contexts, followed by about 10B tokens of instruction tuning.

What the paper actually shows

The abstract does include concrete scale numbers, but it does not give benchmark scores in the way a leaderboard paper would. So if you are looking for exact pass rates or accuracy values, they are not present in the abstract itself.

What the paper does claim is that the training ablations and evaluations were run against 18 state-of-the-art open-weight long-context LMs. The headline result is that replacing just 12% of traditional context-extension corpora with OctoLong data yields substantial gains across several tasks.

Those tasks include long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks. The paper also says OctoLong improves API usage in short-context coding scenarios, which is interesting because it suggests the benefit is not limited to giant prompts. The model may learn better code structure even when the prompt is short.

That combination matters. A lot of long-context work focuses on whether a model can simply “remember more.” This paper suggests that what you feed into the long-context stage changes the kind of memory the model learns. Dependency-rich code seems to teach the model to track relationships, not just tokens.

Why developers should care

If you are building coding assistants, repo agents, or tools that need to reason over large codebases, this paper points to a practical lesson: long-context quality depends on data quality, not just context length. A model trained on richer code dependencies may be better at following references across files, preserving state over long interactions, and using APIs more reliably.

That is especially relevant for agentic workflows, where the model has to plan, inspect, revise, and continue working over time. In those settings, the failure mode is often not a total lack of context window. It is losing the thread of how one part of the repository relates to another.

For teams training or adapting models, the paper also suggests a more efficient knob than starting from scratch. The authors report gains from supplanting only 12% of traditional context-extension corpora with OctoLong data. If that holds up broadly, it implies you may not need to replace your entire long-context pipeline to get better repository reasoning.

What is still missing or uncertain

The abstract leaves several important questions open. It does not provide the exact benchmark numbers, so readers cannot judge the size of the gains from the abstract alone. It also does not spell out the full composition of the non-OctoLong portion of the 50B-token mixture.

Another open question is generalization. The paper clearly targets code contexts, but the abstract does not claim that the same approach solves long-context reasoning across all domains. The gains are described in code understanding, retrieval, state tracking, and agentic tasks, so that is where the evidence is strongest.

There is also a practical caveat in the method itself: the pipeline depends on AST parsing, language server integration, and package management. That makes it powerful, but also more specialized than generic web-scale data collection. It is a curated pipeline, not an off-the-shelf corpus.

Still, the engineering takeaway is clear. If you want models that handle large repositories well, you probably need training data that mirrors how repositories actually work. OctoLong’s contribution is to make that dependency structure explicit and train on it at scale.

Bottom line

OctoLong is a data-centric long-context paper with a very practical angle: make the training context look more like real software systems, and long-context models become better at code tasks that depend on those systems.

For developers, that means better odds that a model can follow cross-file references, keep track of repository state, and behave more like a useful coding agent instead of a token counter with a large window.