DeepSeek is the newest and hottest large language model (LLM) on the block. Many claim it is open source. But what does that really mean, and can we trust it? I argue: no.
We call it open source because its parameter weights are publicly available, making it self-hostable. But what are these weights? Simply put, they’re a bunch of numbers. We have no idea what they mean except through experimentation—sending in prompts and observing the outputs. This is less transparent than an executable binary because, at least with a binary, you can inspect the text strings inside. Instead of calling an LLM with published weights “open source,” we should think of it as a freely redistributable binary that is both encrypted and executable. Would you run that on your machine? In fact, Ben Thompson has also highlighted this issue and proposed the term “open weights” for LLMs like DeepSeek.
So, what would truly constitute “open source” for an LLM? Like traditional open source, it would need to be reproducible. A binary is considered open source if you can recompile it from human-readable source code and produce the same binary, byte for byte. Otherwise, you can’t be sure what went into that binary. Recompiling from source is relatively inexpensive, but recreating an LLM’s weights from its source—including all the inputs, training instructions, and more—would be prohibitively expensive in terms of computational and hardware costs. Most people wouldn’t (and couldn’t) do this just to satisfy their curiosity about verification.
If the organizations behind an LLM like DeepSeek share their weights under a freely redistributable license but don’t provide the exact instructions and inputs needed to recreate those weights (or if no trusted organization has verified the process), we must treat it like an encrypted executable. But why? What harm could an LLM possibly cause?
The potential harms are significant. There’s the human safety factor, censorship concerns, and more—issues that organizations like OpenAI regularly test for before releasing their models. But when we consider LLMs published by countries or organizations we don’t trust, there’s an even more nefarious possibility: code injection.
Yes, one of the most popular and growing uses of LLMs is to write source code for developers. Developers might prompt an LLM to write a program to accomplish a specific task, and the LLM can often do a surprisingly good job. But what if that LLM were trained to write programs with a hidden second purpose? What if it generated code with backdoors that only a meticulous audit could detect? After all, security vulnerabilities in software are already hard to find, and those are usually accidental. What if security holes were deliberately introduced, allowing the LLM’s creator to exploit programs running on computers, devices, and web apps?
There’s no way to vet a black-box LLM—even a freely redistributed one like DeepSeek—against this risk. You could reduce the risk by asking it to generate many programs and then carefully reviewing the code for signs of malicious training, but that’s like searching a 1-billion-pixel photo for a specific shade of blue, one pixel at a time. And it’s possible that only certain prompts, like “create a login page for a government website,” would trigger the generation of malicious backdoor code, making it unlikely to be detected during testing.
Another possibility is indirect poisoning of an LLM. Imagine commissioning a group of people to create hundreds of simple applications and publish their source code on GitHub. These apps could contain hidden backdoors, but their code comments and documentation would tout their high quality and functionality. Eventually, an LLM developer might use this open-source code to train their model, inadvertently incorporating the faulty, backdoored code. In this scenario, even “trusted” LLMs could end up producing exploitable code.
Now, I’m a huge fan of DeepSeek R1 (the reasoning version). I’m absolutely amazed at what it can do, and I appreciate that it’s freely available online. But it makes me wonder: who will we trust to create the LLMs that write our code, make health decisions, and handle other critical tasks that we eventually rely on for our safety and security? Will we ever develop a better way to directly vet the trustworthiness of LLMs, rather than relying solely on the reputation of their creators?
What do you think?