NON_INT SOFTWARE & ML

Open Weight Durability

I love that powerful open weight LLMs have been part of the AI ecosystem for the past several years. Originally, they were mostly artifacts which allowed small research groups to experiment with post-training, interpretability and other LLM-adjacent research. These days, they are downright powerful software tools. I once trained small generative models on a small GPU cluster in my basement. I look forward to one day in the near future when I can host a private assistant on a similar computer living in the same place1. That assistant will have most of the knowledge and capabilities that were only available through a Google account just a decade ago - it is an incredible leap forward for us all to capture that kind of power in something we can run locally.

But in the discourse on open weight models, I find very little discussion on how we will keep open weight releases happening. Open weight models are not an inevitability. Large LLMs are extremely expensive to train and only getting more expensive as time goes on. I can all but guarantee that they are not paying for themselves for any company currently engaged in the act of releasing them. Instead, they are a result of a conflux of factors that exist in the world today: overinvestment in AI, the lead OpenAI and Anthropic have over everyone else, the prestige associated with open weight development, and lax regulation2.

For example - if a Chinese lab started producing frontier models that beat the likes of Claude or GPT, I don't really believe that the weights for these models would be released. Likewise, Meta has shifted in and out of supporting open weights over the years - promoting open weights when they were behind and halting open development3 when it seemed they were ahead. I want to be clear that I am not calling anyone a hypocrite here - I believe that the leaders of "open labs" and other open-weight advocates are sincere in their support of open weight models - but I also suspect they would halt or diminish the process of releasing model weights if it was economically valuable for them to do so.

I think that finding the economic value of training open weights models is they key thing we should be looking for if we want open weight models to be durable. Pundits often claim there is value in leveraging OSS developers to build infrastructure for your models. Certainly that was true a few years ago - fast inference libraries like llama.cpp, post-training methodology like LoRA and agent harnesses are examples of this. But I think the argument doesn't hold as much water anymore given that most of these models are basically the same and can be swapped into the same harnesses and software that the closed models use.

There are good answers to the problem of ensuring open weight durability. I really like the idea of compute providers like NVIDIA building open-weights models4. NVIDIA gains a lot by keeping this ecosystem vibrant: it builds something to run on the GPUs it sells and it ensures models train and run efficiently on that hardware. I also like the "post-training as a service" approach being taken by Thinking Machines - though it remains to be seen whether this can actually be made profitable.

I hope we figure this out. We'll know it has happened when we can confidently say that the very idea of some of the top-tier companies training models not releasing the weights is unthinkable.

P.S. - I think open weights is particularly critical for robotics. Having to access my AI assistant through a third party is a privacy and alignment5 issue. But having a physical robot in my home that is aligned to something other than myself and my family feels radically worse. In other words - I don't want my robot to have access to the internet. I think robotics hardware producers should open weight the models (and training code). Their incentive is selling more hardware.
  1. Of the caliber of the current frontier models with max thinking and tools ↩
  2. I am not expressing an opinion on any of these factors, just stating their existence ↩
  3. Maybe it was always the intent of MSL to release open-weights models, but I honestly doubt it. ↩
  4. Hopefully they really bring their A-game here and give the Chinese labs a run for their money! ↩
  5. Alignment to me, not to humanity ↩