Why Memory Is AI's Biggest Bottleneck with Vikram Sekar | EP 170
Impossibile aggiungere al carrello
Rimozione dalla Lista desideri non riuscita.
Non è stato possibile aggiungere il titolo alla Libreria
Non è stato possibile seguire il Podcast
Esecuzione del comando Non seguire più non riuscita
Why Memory Is AI's Biggest Bottleneck with Vikram Sekar | EP 170
-
Letto da:
-
Di:
Vikram Sekar, writer of Vik's Newsletter and co-host of Semi Doped, joins me to discuss AI agents, memory, and the hardware behind the data center buildout.
We break down why training and inference need different hardware, why copper becomes harder to use as GPUs move farther apart, and why the industry is turning to optical interconnects.
Vik explains the tradeoffs across SRAM, DRAM, high bandwidth memory, and storage. We discuss why useful work per token matters more than speed alone, how much context an agent needs, and why he sees memory as AI's biggest bottleneck.
Enjoy.
00:00 Highlights
01:18 Vikram Sekar and Semi Doped
02:55 AI agents and hardware demand
18:57 AI spending and the risk of overbuilding
25:10 Training versus inference
36:25 Why distance limits copper bandwidth
41:18 Co-packaged optics and the push beyond copper
46:04 New approaches to optical interconnects
52:24 The AI memory hierarchy
56:39 Useful work per token
57:51 How much context does an agent need
1:03:35 DRAM, HBM, and memory on logic
1:05:54 The next generation of inference chips
1:08:25 AI's biggest engineering bottleneck
Follow Vik
https://x.com/vikramskr
Vik's Newsletter
https://www.viksnewsletter.com/
Semi Doped
https://www.semidoped.fm/
Vik's institutional research
https://www.semiexponent.com/