5. Open weights are not open source

The open-weight crowd is drafting on a truth that belongs to open source software: a million eyes on the code, so the bugs get found. With open weights you get the weights. You cannot see inside. Rory’s question, which nobody can answer today: can you prove that somewhere in a trillion parameters there isn’t training that says, once you determine you’re at one of these five companies and you’ve been handed these three pieces of information, quietly do A, B and C? A commenter on the episode added the part we missed.

Even when you have weights and code, nobody outside the labs is re-running training or post-training to verify anything. You can audit it in theory and essentially nobody does. My learning → “You can inspect it” only counts as an answer if someone can do it at a cost someone would pay.