AI productivity. I’m pretty skeptical about AI enhancing productivity (see e.g. AI isn’t changing anything yet and the slow gains despite apparent capability bursts). This paper shows that each year of progress on frontier models cuts task-completion time by ~8%. Extrapolated, you end up with something like a 20% productivity increase over the decade.

Seems positive, but I’m not yet convinced. This paper is about the speed of task completion compounding, not the quality of it. And speed on 30-minute discrete tasks.

He also identifies a puzzle which worries me: when the AI does a task alone, the output quality scales with compute. Experts grade frontier model output at a 6-7 out of 7 for quality. In comparison, humans get ~3.5/7 on the same task when unassisted. Annoyingly, when the task is collaborative (a human-AI back and forth) the quality ends up in the middle—~4.3/7. That is, when humans are monitoring AI, the quality is better than when the human does the task herself, but when they collaborate on the task, it sort of averages out.

He says this is a puzzle, but it seems to actually imply that humans are a bottleneck on AI production for certain kinds of tasks.

That said, as we get better at making tasks ai-shaped, lots of this will be resolved one way or the other.