Why We Need Long-Term Memory Layers for Agentic Scaling