高级检索

大语言模型能力扩展的技术演进

The Technological Evolution of Scaling Large Language Models

  • 摘要: 自Transformer成为大语言模型的基础架构以来,其能力演进始终遵循一条清晰的技术主线,即通过系统性地扩展某一核心维度来驱动性能跃升。这一思想被形式化为缩放定律,揭示了模型性能与参数规模、数据规模及计算资源之间的幂律关系。大语言模型的能力扩展已从早期聚焦参数规模与数据规模这2个基础维度,逐步延伸至后训练阶段的推理能力增强以及运行时的动态演化。前者包括基于结果监督的强化学习与在可交互环境中进行学习,后者则涵盖推理时扩展、经验与工具整合以及多智能体协作网络等多个方向,分别体现为以计算换取推理精度、在特定领域中实现持续进步,以及从个体智能走向群体智能的跃迁。然而,当前大语言模型的进一步发展仍面临多重根本性挑战,包括传统预训练扩展带来的边际收益递减、后训练环境在简化性与开放性之间的设计平衡,以及人类对其内部认知机制的有限理解。未来研究亟须在模型架构创新、训练效率优化与环境设计范式等方面寻求突破,以推动大语言模型能力增长迈向更加科学高效的新阶段。

     

    Abstract: Since the Transformer emerged as the foundational architecture for large language models (LLMs), their capability evolution has followed a remarkably consistent technical trajectory−systematically scaling a core dimension to drive performance gains. This principle was later formalized as scaling laws, which establish power-law relationships between model performance and parameter size, data volume, and computational resources. Over time, the scaling paradigm has expanded beyond the two traditional axes of model size and training data to include post-training reasoning enhancement and runtime environmental adaptation. The former encompasses outcome-supervised reinforcement learning and learning within interactive settings, while the latter spans inference-time scaling, integration of external tools and experiential knowledge, and multi-agent collaboration—reflecting, respectively, the trade-off between computation and reasoning fidelity, sustained domain-specific progress, and the shift from individual to collective intelligence. Nevertheless, the continued advancement of LLMs faces significant challenges, including diminishing returns from conventional pre-training scaling, the delicate balance between simplicity and openness in post-training environment design, and the limited understanding of internal working mechanisms. To move forward, future research needs to pursue innovations in model architectures, training efficiency, and environmental design, thereby steering LLMs capability growth toward a more principled and sustainable trajectory.

     

/

返回文章
返回