Our research agenda

Blue Sky Research

Base Labs finds itself in a special position to be able to truly conduct blue sky research for post-training (and training) at scale: we operate from a company intimately tied with the open source ecosystem and who itself has an already-established compute ecosystem ready for large-scale model research. We aim to use this unique position to conduct proper science where we imbue methodological understanding to the big problems facing training writ large. Throughout, we will openly share the experiments and recipes that allowed us to arrive at this understanding, creating an environment where researchers can collectively advance the open-source frontier. With this ethos, we have begun conducting research on a number of ambitious directions.

Continual learning is a wide, general, and famously difficult problem. It is also, as such, an often poorly defined term. The following are some directions we are currently focusing on and are more optimistic for.

  • On continual memory: the active memory of an agent in the KV cache is highly compressible, especially indicated by the ability of larger LLM’s to represent entire tasks through short examples via ICL. Learning memory models directly in the KV’s embedded space, directly manipulating an agents memory through its life to determine what information is compressible or irrelevant, is a direction we have been actively researching and find crucial for continual learning systems.
  • On learning continually: training runs generally have a lot of structure, sometimes explicitly extractable (e.g. low-rank trajectories in RLVR) and other times more subtly structured. Studying and learning models over the dynamical systems of model training, and “learning to learn” akin to what we see in natural intelligence, is another direction we find important for continual learning.

Beyond the craze of continual learning, we are pursuing a broader set of questions about the science of model training:

  • On the science of RL: we want to understand when RL creates new behavior rather than amplifying existing capabilities and how it interacts with different forms of distillation. We are further interested in quantifying when on-policy self-distillation helps or fails, and whether the low-dimensional structure of RL updates can enable more efficient training.

Separately, we will study the possible economic futures of open and closed models such as the possibility of closed-lab dominance, open-weight competitiveness, or a hybrid in which each serves different workloads. We aim to understand which technical and economic conditions make each outcome more likely.

BaseHub Data Foundry

Many people have spoken about the need to aggregate compute in order for open-source to remain competitive, but less people have spoken about the need to also aggregate data. We believe this is of utmost importance to the future health of the open-source ecosystem. In a world where the closed-source labs are able to spend billions on high quality RL environments every year, open-source falls behind if it is not injected with a similarly good training signal.

As such, a key commitment of Base Labs is the BaseHub Data Foundry. This commitment is twofold. First, we will have a dedicated team to produce the highest quality RL environments, for anyone to train and benchmark on. The aim of this is to bridge the gap between what closed-source providers can afford and what open-source model trainers can access (we also hope this will alleviate some of the pressure labs face to distill.) Second, we will build the canonical hub for environment storage and hosting, running benchmarks with different models, and versioning all of this in a transparent way. The HuggingFace Hub for environments doesn’t exist yet, and we want to give the field a standardized way to run this.

This also includes a commitment to generating high quality mid-training data. As mentioned, a concern with open-source models is their reliance on distillation from closed-source models. Fortunately, a completed RL environment has high utility in generating quality mid-training data. For instance, we can use rejection sampling or privileged information to generate successful trajectories, and we will also open-source this, as well as our recipes for generating them so that anyone can generate on-policy mid-training data for their model.

A final component of this is evaluation. A unique advantage of Baseten is that we see the full spectrum of tasks that people are using language models for (both open-source and closed-source). We can use this information to construct real-world benchmarks that track this usage, and the capabilities it relies on, over time. Doing so will give people a valuable index on capabilities of models in the real world, not just for benchmarks designed as one-offs without this touchpoint with reality.

Post-post training

Two barriers to the widespread deployment of frontier open-source – particularly in the US – remain evident. First, whether these models can be adapted to the capabilities real applications require, and second, whether organizations have sufficient confidence in their security. Most of what the Base Labs team has done over the past few years has been creating targeted training data and RL environments that imbue models with specific capabilities. We aim to extend this experience to the broad, ongoing improvement of frontier open-source models.

First, we will pursue broad capability improvement that we believe can arise from continued pre-training, with an initial focus on the efficiency of reasoning and reliable tool-use. Our Data Foundry will provide the starting point for this work, and over time will create a repeatable process for updating models as new tasks and data become available.

Second, we will build a safety stack for open-source models. Many organizations are hesitant to deploy open-source models because they do not trust them. By running this safety stack, we can not only ameliorate concerns around explicit negative characteristics of these models (such as trigger words or censorship), but also mold the models to values more aligned with American open-source (through constitutional alignment).

Together, we expect this effort to catalyze a new way of looking at the open-source model lifecycle, where an initial release checkpoint is only the base that many people build on top of and mold to their liking.

Make models affordable (Model Performance)

As models grow larger, latency, throughput, and serving cost increasingly determine where they can be deployed and which customers or individuals can afford to use them. We believe improving the efficiency of running a model at scale is therefore a critical task of open-source research.

Base Labs already has a team developing and shipping methods across distillation, quantization, and speculative decoding. We will build on this work by combining approaches where their gains are complementary and develop reproducible methods for use by the community.

We will also integrate these improvements directly into our training research and stack. Efficient inference reduces the cost of rollout generation in RL (as one example) and this integration will simultaneously creating a compounding loop, where improvements in inference make the experiments to achieve them cheaper. Collectively, this work will make open-source frontier models more practical to train and deploy at scale.