Lessons learnedEditorial digest
Evaluate an agent at a budget, not in a vacuum
A METR blog co-author raises caveats about interpreting a NanoGPT experiment: possible contamination, models that have not reached a performance plateau, and the difference between capability limits and performance at a given cost. They propose comparing human and agent performance as a function of expenditure.
Original excerptOriginal discussionNanoGPT Speedrun Frontier“Cool post! I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest”
Read the full original