v2.6.0
PaddlePaddle/Paddlev2.6.0Jan 8, 2024by XiaoguangHu01
AI Summary
PaddlePaddle 2.6.0 introduces PIR (Paddle Intermediate Representation), upgraded dynamic-to-static with 100% success rate, enhanced distributed training, and comprehensive API improvements including 109 new inplace APIs.
Key Highlights
- PIR Intermediate Representation - SSA-based IR system for flexible framework extension
- Dynamic-to-static 100% success rate on 700+ industry models
- Enhanced distributed training with TensorFusion and communication-computation overlap
- 109 new inplace APIs including paddle.abs_, paddle.sin_, comparison and logical operations
- CustomDevice plugin architecture for hardware vendors
Breaking Changes
- Removed 0-dim tensor compatibility flag - full 0-dim tensor support across 376 APIs
- paddle.nn.functional.diag_embed simplified to paddle.diag_embed
- paddle.static.setitem recommended for static graph tensor indexing
- paddle.fluid API completely removed - migrated to paddle.base
New Features
- scaled_dot_product_attention API for efficient attention computation
- Scientific computing APIs: cummax, cummin, index_fill, masked_fill, pca_lowrank, hypot
- Optimizer APIs: LinearLR, CosineAnnealingWarmRestarts learning rate schedulers
- SubsetRandomSampler for data subset sampling
- Advanced tensor indexing with view support and C++ level optimization
- Enhanced Inplace mechanism with automatic dependency detection
Full Release Notes
* [English Version](https://github.com/PaddlePaddle/Paddle/wiki/PaddlePaddle-2.%EF%BC%96.0-Release-Note-EN) # 1. 重要更新 - **新一代中间表示 PIR**：为了进一步提升飞桨框架的可扩展性，研制了新一代中间表示 PIR（Paddle Intermediate Representation）。实现系统性的抽象飞桨框架底层核心概念，如：Operation、Attribute 和 Type 等，为开发者提供了灵活、高效的基础组件。通过引入 Dialect 机制，可以全面、分层次地满足各模块对中间表示的需求，从而极大地提升了框架的扩展性。PIR 严格遵循 SSA（Static Single Assignment）原则，在实现了顶层结构的统一的同时，还确保了“算子顺序性”与“计算图语义”的和谐共存。此外，PIR 提供了更为简洁、低成本的 Pass 开发流程，内置了一系列丰富且功能完备的 Pass 优化策略，为大型模型的极致性能优化提供了技术支撑。 - **动转静编译优化架构**：为了进一步提升框架的模型开发性能，飞桨动转静训练能力全面升级，支持自适应的图构建能力，在 700 多个飞桨产业级模型上验证，一键动转静训练成功率达到 100%。同时，飞桨框架的神经网络编译器 CINN 整合入飞桨主 Repo，使得编译器与飞桨更加融为一体。CINN 完成了架构的梳理和扩展能力的完善，提升系统稳定性。基于 PIR 完成动转静、组合算子、执行器和编译器的紧密联动，为飞桨框架整体性能的提升提供了更大的空间。 - **增强动态图分布式能力**：大模型对框架的分布式训练性能提出了更高的需求。飞桨在通信库、图分析、分布式策略和任务启停等维度进行了全面优化，增强了飞桨动态图的分布式计算能力，为大型模型高效训练提供了支持。在性能方面，通过减少流水线 GPU 显存占用、采用 TensorFusion 技术、实现通信计算 overlap 以及减少非必要的数据同步拷贝等方式，进一步提升了训练性能。同时，通过环境变量控制 Optimizer 等方式提高了混合并行调试的灵活性。此外，通过相关 Bug 的修复，显著提升了系统的稳定性。 - **动静统一自动并行架构**：为了进一步降低大模型编程和优化难度，飞桨对动静统一的半自动并行（Auto Parallel）编程范式进行了全面的优化，简化了开发者的编程复杂度。开发者无需深入了解手动并行编程范式下的复杂概念和 API 接口，如行切分、列切分等，仅需通过少量的张量切分标注即可完成混合并行模型的构建，框架便能够自动推导出所有张量和算子的分布式切分状态，并添加合适的通信算子。同时支持一键动转静进行分布式训练，使开发者能够高效地实现任意混合并行策略，大幅简化了混合并行训练代码的开发过程。 - **硬件适配方案(CustomDevice)**：大模型场景下新硬件并行训练需求增加，飞桨新增了对分布式高级策略、自定义算子和自定义融合策略的支持。升级了分布式通信库，新增了对 MP、GroupShared、PP、SP 和 MOE 等多项高级分布式策略的支持。同时，支持厂商灵活接入不同颗粒度的 Transformer 算子库并通过融合 Pass 修改计算图进行性能加速。 - **安装和开发体验**：采用模块化编译的方式优化了 CMake 代码的逻辑，提升了飞桨全量编译和增量编译的效率，提升了 RD 开发效率，同时支持了 Python3.12，CUDA12，Hopper 架构编译，并引入 Clang 等工具全面优化了代码格式。此外，将 C++单测从链接静态库的方式转变为链接动态库，减小编译体积。这些改进措施为用户提供更加流畅、高效的安装和开发体验。 # 2. 不兼容升级 - 为了避免误用，去除了 0 维 Tensor 兼容态开关，实现 API 行为和业界主流习惯一致。在上一个版本中，我们已经支持 0 维 Tensor，但是考虑到尽量避免部分模型的报错，添加了兼容态开关。即在一些模型套件使用较多且没有修改完成的场景中还是默认使用只有 1 个元素的 1 维 Tensor 来替代 0 维 Tensor。这个版本去除了兼容态开关，在任何场景中都不会再使用只有 1 个元素的 1 维 Tensor 来替代 0 维 Tensor，应该支持 0 维 Tensor 的 376 个 API 的行为都完成了修正和统一，彻底完成对 0 维 Tensor 的支持。[#57036](https://github.com/PaddlePaddle/Paddle/pull/57036), [#54581](https://github.com/PaddlePaddle/Paddle/pull/54581), [#54500](https://github.com/PaddlePaddle/Paddle/pull/54500) - 为了提升 API 易用性，将 paddle.nn.functional.diag_embed 精简为 paddle.diag_embed，并支持 Tensor.diag_embed 方式使用。 [#58223](https://github.com/PaddlePaddle/Paddle/pull/58223) - 为了解决在静态图下 Tensor 索引写（如 tensor[0] = 10）导致的微分计算错误问题，并符合静态图的规范，本版本引入了 paddle.static.setitem API。在静态图环境中，更推荐使用此 API 来支持 tensor 的索引写操作，而非下标运算符。这一变化并不影响动态图环境，其中仍允许使用下标运算符进行索引写操作。[#53682](https://github.com/PaddlePaddle/Paddle/pull/53682) - 本版本中 paddle.fluid API 全面退出历史舞台。在本次更新中，我们彻底移除了所有 paddle.fluid API，并删除了 fluid 目录。同时，飞桨底层的少量公共组件已被整合至 paddle.base 目录中。使得飞桨用户无需再关注 fluid 相关概念和接口，进一步简化了飞桨 API 体系，提升可读性。[#56576](https://github.com/PaddlePaddle/Paddle/pull/56576), [#54424](https://github.com/PaddlePaddle/Paddle/pull/54424), [#54829](https://github.com/PaddlePaddle/Paddle/pull/54829), [#53992](https://github.com/PaddlePaddle/Paddle/pull/53992), [#54806](https://github.com/PaddlePaddle/Paddle/pull/54806), [#55754](https://github.com/PaddlePaddle/Paddle/pull/55754), [#55986](https://github.com/PaddlePaddle/Paddle/pull/55986), [#55345](https://github.com/PaddlePaddle/Paddle/pull/55345), [#56099](https://github.com/PaddlePaddle/Paddle/pull/56099), [#51717](https://github.com/PaddlePaddle/Paddle/pull/51717), [#54152](https://github.com/PaddlePaddle/Paddle/pull/54152), [#55522](https://github.com/PaddlePaddle/Paddle/pull/55522), [#55757](https://github.com/PaddlePaddle/Paddle/pull/55757), [#58521](https://github.com/PaddlePaddle/Paddle/pull/58521), [#54936](https://github.com/PaddlePaddle/Paddle/pull/54936), [#55007](https://github.com/PaddlePaddle/Paddle/pull/55007), [#55661](https://github.com/PaddlePaddle/Paddle/pull/55661), [#55970](https://github.com/PaddlePaddle/Paddle/pull/55970) # 3. 训练框架（含分布式） ## Python API ### 升级 Tensor 索引机制 本版本全面优化了 Tensor 的基础索引、高级索引以及联合索引功能，以更好地符合业界标准与用户习惯。具体包括：在基础索引中增加了对 view 的支持，修正了高级索引中的一些错误行为，并实现了联合索引的读取功能。此外，我们还将索引解析下沉到 C++层面，改进了高级索引算子的性能，并移除了 bool 索引中的冗余计算。通过这些优化措施，Tensor 的基础索引、高级索引和联合索引性能得到了全面提升。[#56893](https://github.com/PaddlePaddle/Paddle/pull/56893), [#58643](https://github.com/PaddlePaddle/Paddle/pull/58643), [#57986](https://github.com/PaddlePaddle/Paddle/pull/57986), [#56272](https://github.com/PaddlePaddle/Paddle/pull/56272), [#58856](https://github.com/PaddlePaddle/Paddle/pull/58856), [#55211](https://github.com/PaddlePaddle/Paddle/pull/55211), [#57023](https://github.com/PaddlePaddle/Paddle/pull/57023), [#56613](https://github.com/PaddlePaddle/Paddle/pull/56613), [#55602](https://github.com/PaddlePaddle/Paddle/pull/55602), [#59281](https://github.com/PaddlePaddle/Paddle/pull/59281), [#57737](https://github.com/PaddlePaddle/Paddle/pull/57737) ### 升级 Inplace 机制 在之前的版本中，为了确保反向微分计算的正确性，当某个 API 的反向计算依赖于其前向输入数据时，飞桨会避免使用 Inplace 操作方式，因为这种方法可能会覆盖原始输入数据。虽然这种机制简化了实现过程，但也限制了许多 API 实现 Inplace 功能，从而影响了用户体验。 在本版本中，飞桨对 Inplace 机制进行了全面升级。实现自动检测反向计算对前向输入的依赖关系，并在需要时保存这些输入数据，从而支持更多的 Inplace 操作。这一改进不仅提升了内存使用效率，还增强了 API 的功能性。 此外，我们新增了 109 个支持 Inplace 操作的 API，包括 paddle.abs_、paddle.sin_/cos_/tan_、比较操作如 paddle.greater_than_/less_than_/equal_、逻辑操作如 paddle.logical_and_/logical_or_/logical_not_，以及 paddle.neg_和 paddle.log_等。在丰富飞桨的功能集同时，提升了用户在数值计算和深度学习任务中的效率与便捷性。[#54683](https://github.com/PaddlePaddle/Paddle/pull/54683), [#55078](https://github.com/PaddlePaddle/Paddle/pull/55078), [#55576](https://github.com/PaddlePaddle/Paddle/pull/55576), [#56888](https://github.com/PaddlePaddle/Paddle/pull/56888), [#55509](https://github.com/PaddlePaddle/Paddle/pull/55509), [#57093](https://github.com/PaddlePaddle/Paddle/pull/57093) ### 其他新增 API - 新增 paddle.nn.functional.scaled_dot_product_attention，显著提升大模型中注意力（attention）机制的计算效率，更好地满足大规模深度学习模型对高性能计算的需求。。[#55242](https://github.com/PaddlePaddle/Paddle/pull/55242) - 新增了一系列科学计算相关 API，包括 paddle.cummax 和 paddle.cummin 用于累积最大值和最小值的计算，paddle.index_fill 和 paddle.masked_fill 用于按索引或掩码填充张量，paddle.linalg.pca_lowrank 用于低秩主成分分析，paddle.hypot 用于计算直角三角形的斜边长，以及 paddle.atleast_1d、paddle.atleast_2d 和 paddle.atleast_3d 用于确保张量至少有一维、二维或三维。同时，我们还提供了 paddle.select_scatter 和 paddle.diagonal_scatter 用于更灵活地选择和散列张量数据，以及 paddle.multigammaln 用于计算多伽马函数的自然对数。此外，本版本新增优化器相关 API，包括：paddle.optimizer.lr.LinearLR 和 paddle.optimizer.lr.CosineAnnealingWarmRestarts 学习率调度策略；引入了 paddle.io.SubsetRandomSampler 以支持从数据子集中进行随机采样。这些新增 API 将进一步提升飞桨在各类应用场景中的灵活性和高效性。。 [#57416](https://github.com/PaddlePaddle/Paddle/pull/57416), [#53546](https://github.com/PaddlePaddle/Paddle/pull/53546), [#53743](https://github.com/PaddlePaddle/Paddle/pull/53743), [#57295](https://github.com/PaddlePaddle/Paddle/pull/57295), [#57726](https://github.com/PaddlePaddle/Paddle/pull/57726), [#58764](https://github.com/PaddlePaddle/Paddle/pull/58764), [#58323](https://github.com/PaddlePaddle/Paddle/pull/58323), [#57720](https://github.com/PaddlePaddle/Paddle/pull/57720), [#58209](https://github.com/PaddlePaddle/Paddle/pull/58209), [#58214](https://github.com/PaddlePaddle/Paddle/pull/58214), [#57792](https://github.com/PaddlePaddle/Paddle/pull/57792), [#51395](https://github.com/PaddlePaddle/Paddle/pull/51395), [#57724](https://github.com/PaddlePaddle/Paddle/pull/57724), [#57355](https://github.com/PaddlePaddle/Paddle/pull/57355), [#57744](https://github.com/PaddlePaddle/Paddle/pull/57744), [#58244](https://github.com/PaddlePaddle/Paddle/pull/58244), [#57599](https://github.com/PaddlePaddle/Paddle/pull/57599), [#59343](https://github.com/PaddlePaddle/Paddle/pull/59343), [#57879](https://github.com/PaddlePaddle/Paddle/pull/57879) ## 新一代中间表示（PIR） PIR（Paddle Intermediate Representation）对底层的核心概念如 Operation、Attribute 和 Type 等进行了系统性的抽象，为开发者构建了一套灵活且强大的基础组件。此外，通过引入 Dialect 这一概念，飞桨框架能够全面且分层次地管理各模块对中间表示（IR）的需求，并支持开发者根据特定需求定制化扩展 Dialect，从而显著提升了框架的扩展性和适应性。在设计上，PIR 严格遵循 SSA（Static Single Assignment）原则，统一了顶层结构，实现了“算子顺序性”与“计算图语义”的兼容表示，为复杂的计算流程提供了清晰且一致的视图。为了进一步优化大模型的性能，PIR 还提供了一套更加简洁、低成本的 Pass 开发流程，包括 DRR（Declarative Rewrite Rule）和模式重写器（Pattern Rewriter）。同时，内置了一系列丰富且功能完备的 Pass 优化策略，这些策略能够针对大模型的特点进行深度优化，从而为大模型的极致性能提供了强有力支撑。通过这些创新设计和优化手段，PIR 为飞桨框架的高效运行和持续扩展奠定了坚实基础。 ### 新功能 - 系统抽象了 IR 底层的核心概念，为开发者提供了灵活的基础组件，如 Operation、Attribute、Value、Type、Trait、Interface 等。[#56354](https://github.com/PaddlePaddle/Paddle/pull/56354),[#57106](https://github.com/PaddlePaddle/Paddle/pull/57106),[#57349](https://github.com/PaddlePaddle/Paddle/pull/57349),[#54844](https://github.com/PaddlePaddle/Paddle/pull/54844),[#54984](https://github.com/PaddlePaddle/Paddle/pull/54984),[#54565](https://github.com/PaddlePaddle/Paddle/pull/54565),[#54562](https://github.com/PaddlePaddle/Paddle/pull/54562),[#57249](https://github.com/PaddlePaddle/Paddle/pull/57249),[#57550](https://github.com/PaddlePaddle/Paddle/pull/57550),[#59278](https://github.com/PaddlePaddle/Paddle/pull/59278),[#54875](https://github.com/PaddlePaddle/Paddle/pull/54875),[#55041](https://github.com/PaddlePaddle/Paddle/pull/55041),[#54987](https://github.com/PaddlePaddle/Paddle/pull/54987),[#55903](https://github.com/PaddlePaddle/Paddle/pull/55903),[#57582](https://github.com/PaddlePaddle/Paddle/pull/57582),[#57580](https://github.com/PaddlePaddle/Paddle/pull/57580),[#58052](https://github.com/PaddlePaddle/Paddle/pull/58052),[#55322](https://github.com/PaddlePaddle/Paddle/pull/55322),[#57418](https://github.com/PaddlePaddle/Paddle/pull/57418),[#57635](https://github.com/PaddlePaddle/Paddle/pull/57635),[#55328](https://github.com/PaddlePaddle/Paddle/pull/55328),[#57463](https://github.com/PaddlePaddle/Paddle/pull/57463),[#59791](https://github.com/PaddlePaddle/Paddle/pull/59791),[#59821](https://github.com/PaddlePaddle/Paddle/pull/59821),[#59115](https://github.com/PaddlePaddle/Paddle/pull/59115),[#57461](https://github.com/PaddlePaddle/Paddle/pull/57461),[#59392](https://github.com/PaddlePaddle/Paddle/pull/59392),[#57373](https://github.com/PaddlePaddle/Paddle/pull/57373),[#59118](https://github.com/PaddlePaddle/Paddle/pull/59118) - 新增引入 Dialect 机制，支持全面、分层次管理框架各个模块对中间表示的需求，且内置了 Builtin Dialect，支持开发者根据需求自定义化扩展 Dialect。 [#56325](https://github.com/PaddlePaddle/Paddle/pull/56325),[#57539](https://github.com/PaddlePaddle/Paddle/pull/57539),[#54682](https://github.com/PaddlePaddle/Paddle/pull/54682),[#55381](https://github.com/PaddlePaddle/Paddle/pull/55381),[#56156](https://github.com/PaddlePaddle/Paddle/pull/56156),[#56431](https://github.com/PaddlePaddle/Paddle/pull/56431),[#56615](https://github.com/PaddlePaddle/Paddle/pull/56615),[#57103](https://github.com/PaddlePaddle/Paddle/pull/57103),[#57209](https://github.com/PaddlePaddle/Paddle/pull/57209) - 规范化了飞桨静态图算子体系，新增 OperatorDialect、KernelDialect，以 Dialect 形式分层管理编译期和执行期的算子表示概念差异性，架构更加清晰。[#56284](https://github.com/PaddlePaddle/Paddle/pull/56284),[#54469](https://github.com/PaddlePaddle/Paddle/pull/54469),[#58660](https://github.com/PaddlePaddle/Paddle/pull/58660),[#58975](https://github.com/PaddlePaddle/Paddle/pull/58975),[#56680](https://github.com/PaddlePaddle/Paddle/pull/56680),[#54790](https://github.com/PaddlePaddle/Paddle/pull/54790),[#54826](https://github.com/PaddlePaddle/Paddle/pull/54826),[#54840](https://github.com/PaddlePaddle/Paddle/pull/54840),[#55699](https://github.com/PaddlePaddle/Paddle/pull/55699),[#55648](https://github.com/PaddlePaddle/Paddle/pull/55648),[#55880](https://github.com/PaddlePaddle/Paddle/pull/55880),[#56101](https://github.com/PaddlePaddle/Paddle/pull/56101),[#56754](https://github.com/PaddlePaddle/Paddle/pull/56754),[#54944](https://github.com/PaddlePaddle/Paddle/pull/54944),[#56836](https://github.com/PaddlePaddle/Paddle/pull/56836),[#57185](https://github.com/PaddlePaddle/Paddle/pull/57185),[#58757](https://github.com/PaddlePaddle/Paddle/pull/58757),[#56243](https://github.com/PaddlePaddle/Paddle/pull/56243),[#56436](https://github.com/PaddlePaddle/Paddle/pull/56436),[#57741](https://github.com/PaddlePaddle/Paddle/pull/57741),[#59124](https://github.com/PaddlePaddle/Paddle/pull/59124),[#57054](https://github.com/PaddlePaddle/Paddle/pull/57054),[#56984](https://github.com/PaddlePaddle/Paddle/pull/56984),[#57403](https://github.com/PaddlePaddle/Paddle/pull/57403),[#57904](https://github.com/PaddlePaddle/Paddle/pull/57904),[#58031](https://github.com/PaddlePaddle/Paddle/pull/58031),[#56924](https://github.com/PaddlePaddle/Paddle/pull/56924),[#59270](https://github.com/PaddlePaddle/Paddle/pull/59270),[#55343](https://github.com/PaddlePaddle/Paddle/pull/55343),[#56557](https://github.com/PaddlePaddle/Paddle/pull/56557),[#55693](https://github.com/PaddlePaddle/Paddle/pull/55693),[#54428](https://github.com/PaddlePaddle/Paddle/pull/54428) - 新增 ShapeDialect，内置了丰富的 Shape 操作算子，用于面向 AI 编译器的动态 Shape 约束关系和表达式的构建。[#56727](https://github.com/PaddlePaddle/Paddle/pull/56727),[#59254](https://github.com/PaddlePaddle/Paddle/pull/59254),[#58368](https://github.com/PaddlePaddle/Paddle/pull/58368),[#57069](https://github.com/PaddlePaddle/Paddle/pull/57069),[#57337](https://github.com/PaddlePaddle/Paddle/pull/57337),[#56351](https://github.com/PaddlePaddle/Paddle/pull/56351),[#57029](https://github.com/PaddlePaddle/Paddle/pull/57029),[#58036](https://github.com/PaddlePaddle/Paddle/pull/58036),[#59032](https://github.com/PaddlePaddle/Paddle/pull/59032),[#57961](https://github.com/PaddlePaddle/Paddle/pull/57961),[#56427](https://github.com/PaddlePaddle/Paddle/pull/56427),[#57459](https://github.com/PaddlePaddle/Paddle/pull/57459) - 统一了框架 Program 顶层结构，支持兼容表示“算子顺序性”和“计算图语义”，解耦对 ir::Graph 的依赖，且严格遵循 SSA （即 Static Single Assignment）原则。[#59369](https://github.com/PaddlePaddle/Paddle/pull/59369),[#54563](https://github.com/PaddlePaddle/Paddle/pull/54563),[#57051](https://github.com/PaddlePaddle/Paddle/pull/57051),[#57306](https://github.com/PaddlePaddle/Paddle/pull/57306),[#57857](https://github.com/PaddlePaddle/Paddle/pull/57857) - 新增了 IrPrinter 和 IrPaser 组件，支持 PIR Program 的序列化和反序列化功能，提供了友好的 PIR 开发调试体验。[#55695](https://github.com/PaddlePaddle/Paddle/pull/55695),[#59449](https://github.com/PaddlePaddle/Paddle/pull/59449),[#54369](https://github.com/PaddlePaddle/Paddle/pull/54369),[#54499](https://github.com/PaddlePaddle/Paddle/pull/54499),[#55518](https://github.com/PaddlePaddle/Paddle/pull/55518),[#55784](https://github.com/PaddlePaddle/Paddle/pull/55784),[#57180](https://github.com/PaddlePaddle/Paddle/pull/57180),[#57471](https://github.com/PaddlePaddle/Paddle/pull/57471),[#54859](https://github.com/PaddlePaddle/Paddle/pull/54859),[#54968](https://github.com/PaddlePaddle/Paddle/pull/54968),[#55209](https://github.com/PaddlePaddle/Paddle/pull/55209),[#57314](https://github.com/PaddlePaddle/Paddle/pull/57314),[#57969](https://github.com/PaddlePaddle/Paddle/pull/57969) - 基于 DRR（即 Declarative Rewrite Rule） 和 Pattern Rewriter 构建了全新、简洁、低成本的 Pass 开发体系，并内置了一系列丰富且功能完备的 Pass 优化策略，加速训练和推理执行过程。[#54385](https://github.com/PaddlePaddle/Paddle/pull/54385),[#54738](https://github.com/PaddlePaddle/Paddle/pull/54738),[#55859](https://github.com/PaddlePaddle/Paddle/pull/55859),[#56638](https://github.com/PaddlePaddle/Paddle/pull/56638),[#57090](https://github.com/PaddlePaddle/Paddle/pull/57090),[#58673](https://github.com/PaddlePaddle/Paddle/pull/58673),[#59415](https://github.com/PaddlePaddle/Paddle/pull/59415),[#56729](https://github.com/PaddlePaddle/Paddle/pull/56729),[#58655](https://github.com/PaddlePaddle/Paddle/pull/58655) - 新增 ProgramTranslator 组件，支持由 ProgramDesc 一键转换为飞桨新一代 IR 表示，并提供了易用的 C++和 Python 接口。[#55433](https://github.com/PaddlePaddle/Paddle/pull/55433),[#54470](https://github.com/PaddlePaddle/Paddle/pull/54470),[#58044](https://github.com/PaddlePaddle/Paddle/pull/58044),[#58390](https://github.com/PaddlePaddle/Paddle/pull/58390),[#58100](https://github.com/PaddlePaddle/Paddle/pull/58100),[#55403](https://github.com/PaddlePaddle/Paddle/pull/55403),[#55406](https://github.com/PaddlePaddle/Paddle/pull/55406),[#54719](https://github.com/PaddlePaddle/Paddle/pull/54719),[#56550](https://github.com/PaddlePaddle/Paddle/pull/56550),[#55448](https://github.com/PaddlePaddle/Paddle/pull/55448),[#55453](https://github.com/PaddlePaddle/Paddle/pull/55453),[#56294](https://github.com/PaddlePaddle/Paddle/pull/56294),[#56308](https://github.com/PaddlePaddle/Paddle/pull/56308),[#56842](https://github.com/PaddlePaddle/Paddle/pull/56842),[#58517](https://github.com/PaddlePaddle/Paddle/pull/58517) - 借助自动代码生成技术，一键生成飞桨框架全量静态图算子表示。将静态图组网逻辑下沉至 C++端，统一绑定到\_C_ops 模块，大幅精简 Python 端 API 代码，实现飞桨框架 API 的极致化动静统一，升级了诸多 Python API 以支持新 IR 静态图组网。[#56570](https://github.com/PaddlePaddle/Paddle/pull/56570),[#55745](https://github.com/PaddlePaddle/Paddle/pull/55745),[#56955](https://github.com/PaddlePaddle/Paddle/pull/56955),[#57298](https://github.com/PaddlePaddle/Paddle/pull/57298),[#57946](https://github.com/PaddlePaddle/Paddle/pull/57946),[#57248](https://github.com/PaddlePaddle/Paddle/pull/57248),[#56080](https://github.com/PaddlePaddle/Paddle/pull/56080),[#54396](https://github.com/PaddlePaddle/Paddle/pull/54396),[#54551](https://github.com/PaddlePaddle/Paddle/pull/54551),[#56520](https://github.com/PaddlePaddle/Paddle/pull/56520),[#55002](https://github.com/PaddlePaddle/Paddle/pull/55002),[#57067](https://github.com/PaddlePaddle/Paddle/pull/57067),[#59320](https://github.com/PaddlePaddle/Paddle/pull/59320),[#59348](https://github.com/PaddlePaddle/Paddle/pull/59348),[#57164](https://github.com/PaddlePaddle/Paddle/pull/57164),[#57267](https://github.com/PaddlePaddle/Paddle/pull/57267),[#59064](https://github.com/PaddlePaddle/Paddle/pull/59064),[#54340](https://github.com/PaddlePaddle/Paddle/pull/54340),[#54895](https://github.com/PaddlePaddle/Paddle/pull/54895),[#55004](https://github.com/PaddlePaddle/Paddle/pull/55004),[#56196](https://github.com/PaddlePaddle/Paddle/pull/56196),[#56862](https://github.com/PaddlePaddle/Paddle/pull/56862),[#58991](https://github.com/PaddlePaddle/Paddle/pull/58991),[#55428](https://github.com/PaddlePaddle/Paddle/pull/55428),[#55909](https://github.com/PaddlePaddle/Paddle/pull/55909),[#56241](https://github.com/PaddlePaddle/Paddle/pull/56241),[#56526](https://github.com/PaddlePaddle/Paddle/pull/56526),[#56571](https://github.com/PaddlePaddle/Paddle/pull/56571),[#56518](https://github.com/PaddlePaddle/Paddle/pull/56518),[#57016](https://github.com/PaddlePaddle/Paddle/pull/57016),[#56653](https://github.com/PaddlePaddle/Paddle/pull/56653),[#56809](https://github.com/PaddlePaddle/Paddle/pull/56809),[#57158](https://github.com/PaddlePaddle/Paddle/pull/57158),[#55422](https://github.com/PaddlePaddle/Paddle/pull/55422),[#55458](https://github.com/PaddlePaddle/Paddle/pull/55458),[#55432](https://github.com/PaddlePaddle/Paddle/pull/55432),[#55467](https://github.com/PaddlePaddle/Paddle/pull/55467),[#55483](https://github.com/PaddlePaddle/Paddle/pull/55483),[#55419](https://github.com/PaddlePaddle/Paddle/pull/55419),[#55517](https://github.com/PaddlePaddle/Paddle/pull/55517),[#55500](https://github.com/PaddlePaddle/Paddle/pull/55500),[#56674](https://github.com/PaddlePaddle/Paddle/pull/56674),[#57693](https://github.com/PaddlePaddle/Paddle/pull/57693),[#55008](https://github.com/PaddlePaddle/Paddle/pull/55008),[#57166](https://github.com/PaddlePaddle/Paddle/pull/57166),[#57157](https://github.com/PaddlePaddle/Paddle/pull/57157),[#57159](https://github.com/PaddlePaddle/Paddle/pull/57159),[#57175](https://github.com/PaddlePaddle/Paddle/pull/57175),[#57325](https://github.com/PaddlePaddle/Paddle/pull/57325),[#57330](https://github.com/PaddlePaddle/Paddle/pull/57330),[#57415](https://github.com/PaddlePaddle/Paddle/pull/57415),[#57122](https://github.com/PaddlePaddle/Paddle/pull/57122),[#57393](https://github.com/PaddlePaddle/Paddle/pull/57393),[#57344](https://github.com/PaddlePaddle/Paddle/pull/57344),[#57667](https://github.com/PaddlePaddle/Paddle/pull/57667),[#57348](https://github.com/PaddlePaddle/Paddle/pull/57348),[#57700](https://github.com/PaddlePaddle/Paddle/pull/57700),[#58093](https://github.com/PaddlePaddle/Paddle/pull/58093),[#58005](https://github.com/PaddlePaddle/Paddle/pull/58005),[#58081](https://github.com/PaddlePaddle/Paddle/pull/58081),[#58094](https://github.com/PaddlePaddle/Paddle/pull/58094),[#58137](https://github.com/PaddlePaddle/Paddle/pull/58137),[#58287](https://github.com/PaddlePaddle/Paddle/pull/58287),[#58352](https://github.com/PaddlePaddle/Paddle/pull/58352),[#58340](https://github.com/PaddlePaddle/Paddle/pull/58340),[#58363](https://github.com/PaddlePaddle/Paddle/pull/58363),[#58331](https://github.com/PaddlePaddle/Paddle/pull/58331),[#58343](https://github.com/PaddlePaddle/Paddle/pull/58343),[#58317](https://github.com/PaddlePaddle/Paddle/pull/58317),[#58450](https://github.com/PaddlePaddle/Paddle/pull/58450),[#58377](https://github.com/PaddlePaddle/Paddle/pull/58377),[#58466](https://github.com/PaddlePaddle/Paddle/pull/58466),[#58470](https://github.com/PaddlePaddle/Paddle/pull/58470),[#58491](https://github.com/PaddlePaddle/Paddle/pull/58491),[#58546](https://github.com/PaddlePaddle/Paddle/pull/58546),[#58587](https://github.com/PaddlePaddle/Paddle/pull/58587),[#58453](https://github.com/PaddlePaddle/Paddle/pull/58453),[#58634](https://github.com/PaddlePaddle/Paddle/pull/58634),[#58604](https://github.com/PaddlePaddle/Paddle/pull/58604),[#58605](https://github.com/PaddlePaddle/Paddle/pull/58605),[#58593](https://github.com/PaddlePaddle/Paddle/pull/58593),[#58675](https://github.com/PaddlePaddle/Paddle/pull/58675),[#58699](https://github.com/PaddlePaddle/Paddle/pull/58699),[#58384](https://github.com/PaddlePaddle/Paddle/pull/58384),[#58629](https://github.com/PaddlePaddle/Paddle/pull/58629),[#58579](https://github.com/PaddlePaddle/Paddle/pull/58579),[#58695](https://github.com/PaddlePaddle/Paddle/pull/58695),[#58548](https://github.com/PaddlePaddle/Paddle/pull/58548),[#58688](https://github.com/PaddlePaddle/Paddle/pull/58688),[#58792](https://github.com/PaddlePaddle/Paddle/pull/58792),[#58843](https://github.com/PaddlePaddle/Paddle/pull/58843),[#58840](https://github.com/PaddlePaddle/Paddle/pull/58840),[#58718](https://github.com/PaddlePaddle/Paddle/pull/58718),[#58883](https://github.com/PaddlePaddle/Paddle/pull/58883),[#58785](https://github.com/PaddlePaddle/Paddle/pull/58785),[#58608](https://github.com/PaddlePaddle/Paddle/pull/58608),[#58781](https://github.com/PaddlePaddle/Paddle/pull/58781),[#58783](https://github.com/PaddlePaddle/Paddle/pull/58783),[#58429](https://github.com/PaddlePaddle/Paddle/pull/58429),[#58685](https://github.com/PaddlePaddle/Paddle/pull/58685),[#58696](https://github.com/PaddlePaddle/Paddle/pull/58696),[#58690](https://github.com/PaddlePaddle/Paddle/pull/58690),[#58831](https://github.com/PaddlePaddle/Paddle/pull/58831),[#58929](https://github.com/PaddlePaddle/Paddle/pull/58929),[#58740](https://github.com/PaddlePaddle/Paddle/pull/58740),[#58937](https://github.com/PaddlePaddle/Paddle/pull/58937),[#58782](https://github.com/PaddlePaddle/Paddle/pull/58782),[#58833](https://github.com/PaddlePaddle/Paddle/pull/58833),[#58882](https://github.com/PaddlePaddle/Paddle/pull/58882),[#58935](https://github.com/PaddlePaddle/Paddle/pull/58935),[#58931](https://github.com/PaddlePaddle/Paddle/pull/58931),[#59041](https://github.com/PaddlePaddle/Paddle/pull/59041),[#59040](https://github.com/PaddlePaddle/Paddle/pull/59040),[#58877](https://github.com/PaddlePaddle/Paddle/pull/58877),[#58888](https://github.com/PaddlePaddle/Paddle/pull/58888),[#59042](https://github.com/PaddlePaddle/Paddle/pull/59042),[#58780](https://github.com/PaddlePaddle/Paddle/pull/58780),[#58682](https://github.com/PaddlePaddle/Paddle/pull/58682),[#58815](https://github.com/PaddlePaddle/Paddle/pull/58815),[#58676](https://github.com/PaddlePaddle/Paddle/pull/58676),[#58678](https://github.com/PaddlePaddle/Paddle/pull/58678),[#58446](https://github.com/PaddlePaddle/Paddle/pull/58446),[#59077](https://github.com/PaddlePaddle/Paddle/pull/59077),[#59091](https://github.com/PaddlePaddle/Paddle/pull/59091),[#58661](https://github.com/PaddlePaddle/Paddle/pull/58661),[#58832](https://github.com/PaddlePaddle/Paddle/pull/58832),[#58642](https://github.com/PaddlePaddle/Paddle/pull/58642),[#58698](https://github.com/PaddlePaddle/Paddle/pull/58698),[#59313](https://github.com/PaddlePaddle/Paddle/pull/59313),[#59371](https://github.com/PaddlePaddle/Paddle/pull/59371),[#58700](https://github.com/PaddlePaddle/Paddle/pull/58700),[#58953](https://github.com/PaddlePaddle/Paddle/pull/58953),[#58879](https://github.com/PaddlePaddle/Paddle/pull/58879),[#59469](https://github.com/PaddlePaddle/Paddle/pull/59469),[#59573](https://github.com/PaddlePaddle/Paddle/pull/59573),[#59481](https://github.com/PaddlePaddle/Paddle/pull/59481),[#59419](https://github.com/PaddlePaddle/Paddle/pull/59419),[#59509](https://github.com/PaddlePaddle/Paddle/pull/59509),[#58735](https://github.com/PaddlePaddle/Paddle/pull/58735),[#59616](https://github.com/PaddlePaddle/Paddle/pull/59616),[#59582](https://github.com/PaddlePaddle/Paddle/pull/59582),[#59420](https://github.com/PaddlePaddle/Paddle/pull/59420),[#59500](https://github.com/PaddlePaddle/Paddle/pull/59500),[#58911](https://github.com/PaddlePaddle/Paddle/pull/58911),[#59535](https://github.com/PaddlePaddle/Paddle/pull/59535),[#54891](https://github.com/PaddlePaddle/Paddle/pull/54891),[#56794](https://github.com/PaddlePaddle/Paddle/pull/56794),[#57477](https://github.com/PaddlePaddle/Paddle/pull/57477),[#57929](https://github.com/PaddlePaddle/Paddle/pull/57929),[#57765](https://github.com/PaddlePaddle/Paddle/pull/57765),[#58693](https://github.com/PaddlePaddle/Paddle/pull/58693),[#58603](https://github.com/PaddlePaddle/Paddle/pull/58603),[#56291](https://github.com/PaddlePaddle/Paddle/pull/56291),[#57123](https://github.com/PaddlePaddle/Paddle/pull/57123),[#57317](https://github.com/PaddlePaddle/Paddle/pull/57317),[#57341](https://github.com/PaddlePaddle/Paddle/pull/57341),[#57020](https://github.com/PaddlePaddle/Paddle/pull/57020),[#57324](https://github.com/PaddlePaddle/Paddle/pull/57324),[#57761](https://github.com/PaddlePaddle/Paddle/pull/57761),[#57762](https://github.com/PaddlePaddle/Paddle/pull/57762),[#57907](https://github.com/PaddlePaddle/Paddle/pull/57907),[#57909](https://github.com/PaddlePaddle/Paddle/pull/57909),[#58099](https://github.com/PaddlePaddle/Paddle/pull/58099),[#58110](https://github.com/PaddlePaddle/Paddle/pull/58110),[#58114](https://github.com/PaddlePaddle/Paddle/pull/58114),[#58139](https://github.com/PaddlePaddle/Paddle/pull/58139),[#58144](https://github.com/PaddlePaddle/Paddle/pull/58144),[#58165](https://github.com/PaddlePaddle/Paddle/pull/58165),[#58194](https://github.com/PaddlePaddle/Paddle/pull/58194),[#58138](https://github.com/PaddlePaddle/Paddle/pull/58138),[#58113](https://github.com/PaddlePaddle/Paddle/pull/58113),[#58245](https://github.com/PaddlePaddle/Paddle/pull/58245),[#58318](https://github.com/PaddlePaddle/Paddle/pull/58318),[#58105](https://github.com/PaddlePaddle/Paddle/pull/58105),[#58348](https://github.com/PaddlePaddle/Paddle/pull/58348),[#58235](https://github.com/PaddlePaddle/Paddle/pull/58235),[#58354](https://github.com/PaddlePaddle/Paddle/pull/58354),[#58341](https://github.com/PaddlePaddle/Paddle/pull/58341),[#58445](https://github.com/PaddlePaddle/Paddle/pull/58445),[#58418](https://github.com/PaddlePaddle/Paddle/pull/58418),[#58239](https://github.com/PaddlePaddle/Paddle/pull/58239),[#58473](https://github.com/PaddlePaddle/Paddle/pull/58473),[#58239](https://github.com/PaddlePaddle/Paddle/pull/58239),[#58391](https://github.com/PaddlePaddle/Paddle/pull/58391),[#58501](https://github.com/PaddlePaddle/Paddle/pull/58501),[#58519](https://github.com/PaddlePaddle/Paddle/pull/58519),[#58416](https://github.com/PaddlePaddle/Paddle/pull/58416),[#58588](https://github.com/PaddlePaddle/Paddle/pull/58588),[#58531](https://github.com/PaddlePaddle/Paddle/pull/58531),[#58730](https://github.com/PaddlePaddle/Paddle/pull/58730),[#58773](https://github.com/PaddlePaddle/Paddle/pull/58773),[#58862](https://github.com/PaddlePaddle/Paddle/pull/58862),[#58946](https://github.com/PaddlePaddle/Paddle/pull/58946),[#58500](https://github.com/PaddlePaddle/Paddle/pull/58500),[#56585](https://github.com/PaddlePaddle/Paddle/pull/56585),[#57480](https://github.com/PaddlePaddle/Paddle/pull/57480),[#57433](https://github.com/PaddlePaddle/Paddle/pull/57433),[#58498](https://github.com/PaddlePaddle/Paddle/pull/58498) ### 功能优化 - 升级了静态图执行器，扩展了更多 Kernel Instruction 类型，支持加载 PIR 并高效调度执行，在训练、推理环节都有显存和性能收益。[#54570](https://github.com/PaddlePaddle/Paddle/pull/54570),[#58665](https://github.com/PaddlePaddle/Paddle/pull/58665),[#57291](https://github.com/PaddlePaddle/Paddle/pull/57291),[#54452](https://github.com/PaddlePaddle/Paddle/pull/54452),[#57431](https://github.com/PaddlePaddle/Paddle/pull/57431),[#54692](https://github.com/PaddlePaddle/Paddle/pull/54692),[#55112](https://github.com/PaddlePaddle/Paddle/pull/55112),[#55210](https://github.com/PaddlePaddle/Paddle/pull/55210),[#55401](https://github.com/PaddlePaddle/Paddle/pull/55401),[#55772](https://github.com/PaddlePaddle/Paddle/pull/55772),[#55828](https://github.com/PaddlePaddle/Paddle/pull/55828),[#56148](https://github.com/PaddlePaddle/Paddle/pull/56148),[#54763](https://github.com/PaddlePaddle/Paddle/pull/54763),[#56886](https://github.com/PaddlePaddle/Paddle/pull/56886),[#57284](https://github.com/PaddlePaddle/Paddle/pull/57284),[#57268](https://github.com/PaddlePaddle/Paddle/pull/57268),[#57791](https://github.com/PaddlePaddle/Paddle/pull/57791),[#56789](https://github.com/PaddlePaddle/Paddle/pull/56789),[#56704](https://github.com/PaddlePaddle/Paddle/pull/56704),[#57594](https://github.com/PaddlePaddle/Paddle/pull/57594),[#58397](https://github.com/PaddlePaddle/Paddle/pull/58397),[#58337](https://github.com/PaddlePaddle/Paddle/pull/58337),[#58756](https://github.com/PaddlePaddle/Paddle/pull/58756),[#58371](https://github.com/PaddlePaddle/Paddle/pull/58371) - 面向 PIR 重构了自动微分模块，迁移适配了高阶自动微分功能，优化了 Stop Gradient 传递机制，逻辑更加清晰，功能更加鲁棒。[#55660](https://github.com/PaddlePaddle/Paddle/pull/55660),[#57084](https://github.com/PaddlePaddle/Paddle/pull/57084),[#56890](https://github.com/PaddlePaddle/Paddle/pull/56890),[#58942](https://github.com/PaddlePaddle/Paddle/pull/58942),[#59373](https://github.com/PaddlePaddle/Paddle/pull/59373),[#57206](https://github.com/PaddlePaddle/Paddle/pull/57206),[#58145](https://github.com/PaddlePaddle/Paddle/pull/58145),[#55235](https://github.com/PaddlePaddle/Paddle/pull/55235),[#57255](https://github.com/PaddlePaddle/Paddle/pull/57255),[#56925](https://github.com/PaddlePaddle/Paddle/pull/56925),[#55957](https://github.com/PaddlePaddle/Paddle/pull/55957),[#56163](https://github.com/PaddlePaddle/Paddle/pull/56163),[#56316](https://github.com/PaddlePaddle/Paddle/pull/56316),[#57294](https://github.com/PaddlePaddle/Paddle/pull/57294),[#57449](https://github.com/PaddlePaddle/Paddle/pull/57449),[#59520](https://github.com/PaddlePaddle/Paddle/pull/59520),[#59565](https://github.com/PaddlePaddle/Paddle/pull/59565),[#56265](https://github.com/PaddlePaddle/Paddle/pull/56265),[#56512](https://github.com/PaddlePaddle/Paddle/pull/56512),[#56650](https://github.com/PaddlePaddle/Paddle/pull/56650),[#57183](https://github.com/PaddlePaddle/Paddle/pull/57183),[#57956](https://github.com/PaddlePaddle/Paddle/pull/57956),[#59100](https://github.com/PaddlePaddle/Paddle/pull/59100) - 优化了控制流前向,反向算子的设计和表示，引入 ControlFlow Dialect，并支持 ProgramDesc 下控制流算子到 PIR 的转换和执行。[#58729](https://github.com/PaddlePaddle/Paddle/pull/58729),[#57364](https://github.com/PaddlePaddle/Paddle/pull/57364),[#58625](https://github.com/PaddlePaddle/Paddle/pull/58625),[#57475](https://github.com/PaddlePaddle/Paddle/pull/57475),[#57265](https://github.com/PaddlePaddle/Paddle/pull/57265),[#56799](https://github.com/PaddlePaddle/Paddle/pull/56799),[#59033](https://github.com/PaddlePaddle/Paddle/pull/59033),[#57342](https://github.com/PaddlePaddle/Paddle/pull/57342),[#57801](https://github.com/PaddlePaddle/Paddle/pull/57801),[#57958](https://github.com/PaddlePaddle/Paddle/pull/57958),[#57949](https://github.com/PaddlePaddle/Paddle/pull/57949),[#57937](https://github.com/PaddlePaddle/Paddle/pull/57937),[#59231](https://github.com/PaddlePaddle/Paddle/pull/59231),[#59496](https://github.com/PaddlePaddle/Paddle/pull/59496),[#59321](https://github.com/PaddlePaddle/Paddle/pull/59321),[#58088](https://github.com/PaddlePaddle/Paddle/pull/58088),[#58198](https://github.com/PaddlePaddle/Paddle/pull/58198),[#58024](https://github.com/PaddlePaddle/Paddle/pull/58024),[#58089](https://github.com/PaddlePaddle/Paddle/pull/58089),[#58086](https://github.com/PaddlePaddle/Paddle/pull/58086),[#59175](https://github.com/PaddlePaddle/Paddle/pull/59175),[#59423](https://github.com/PaddlePaddle/Paddle/pull/59423),[#59567](https://github.com/PaddlePaddle/Paddle/pull/59567),[#58098](https://github.com/PaddlePaddle/Paddle/pull/58098),[#58163](https://github.com/PaddlePaddle/Paddle/pull/58163),[#58250](https://github.com/PaddlePaddle/Paddle/pull/58250),[#58277](https://github.com/PaddlePaddle/Paddle/pull/58277),[#58355](https://github.com/PaddlePaddle/Paddle/pull/58355),[#59020](https://github.com/PaddlePaddle/Paddle/pull/59020),[#59200](https://github.com/PaddlePaddle/Paddle/pull/59200),[#59585](https://github.com/PaddlePaddle/Paddle/pull/59585),[#58109](https://github.com/PaddlePaddle/Paddle/pull/58109) - 动转静执行流程升级支持 PIR，优化了动转静子图 Pass 机制，支持用户在@to_static 功能下尝鲜使用 PIR 体系下功能。[#57566](https://github.com/PaddlePaddle/Paddle/pull/57566),[#55620](https://github.com/PaddlePaddle/Paddle/pull/55620),[#56791](https://github.com/PaddlePaddle/Paddle/pull/56791),[#57357](https://github.com/PaddlePaddle/Paddle/pull/57357),[#59152](https://github.com/PaddlePaddle/Paddle/pull/59152),[#59312](https://github.com/PaddlePaddle/Paddle/pull/59312),[#58630](https://github.com/PaddlePaddle/Paddle/pull/58630),[#56035](https://github.com/PaddlePaddle/Paddle/pull/56035),[#59447](https://github.com/PaddlePaddle/Paddle/pull/59447),[#57361](https://github.com/PaddlePaddle/Paddle/pull/57361),[#59261](https://github.com/PaddlePaddle/Paddle/pull/59261),[#59774](https://github.com/PaddlePaddle/Paddle/pull/59774) - 升级了组合算子功能，引入 Backend 概念分层管理动、静态图组合算子模块逻辑，将必要组件和算子拆分规则下沉至 C++，大幅降低了维护成本。[#58153](https://github.com/PaddlePaddle/Paddle/pull/58153),[#56391](https://github.com/PaddlePaddle/Paddle/pull/56391),[#56614](https://github.com/PaddlePaddle/Paddle/pull/56614),[#57030](https://github.com/PaddlePaddle/Paddle/pull/57030),[#57554](https://github.com/PaddlePaddle/Paddle/pull/57554),[#58018](https://github.com/PaddlePaddle/Paddle/pull/58018),[#58130](https://github.com/PaddlePaddle/Paddle/pull/58130),[#58581](https://github.com/PaddlePaddle/Paddle/pull/58581),[#58679](https://github.com/PaddlePaddle/Paddle/pull/58679),[#59054](https://github.com/PaddlePaddle/Paddle/pull/59054),[#55480](https://github.com/PaddlePaddle/Paddle/pull/55480),[#58451](https://github.com/PaddlePaddle/Paddle/pull/58451),[#55647](https://github.com/PaddlePaddle/Paddle/pull/55647),[#56342](https://github.com/PaddlePaddle/Paddle/pull/56342),[#56798](https://github.com/PaddlePaddle/Paddle/pull/56798),[#57561](https://github.com/PaddlePaddle/Paddle/pull/57561),[#58023](https://github.com/PaddlePaddle/Paddle/pull/58023),[#57722](https://github.com/PaddlePaddle/Paddle/pull/57722) ### 性能优化 - 新增 DCE、constant_folding_pass 等 PIR Program 算子和结构优化的 Pass。[#54935](https://github.com/PaddlePaddle/Paddle/pull/54935),[#59430](https://github.com/PaddlePaddle/Paddle/pull/59430),[#58753](https://github.com/PaddlePaddle/Paddle/pull/58753),[#58732](https://github.com/PaddlePaddle/Paddle/pull/58732) 2. 新增 fused_attention,fused_dropout_add,fused_gemm_epilogue_pass,fused_linear_param_grad_add_pass,fused_weight_only_linear_pass,fused_softmax_mask_upper_triangle 等优化算子融合类 Pass，提升训练和推理性能。[#57557](https://github.com/PaddlePaddle/Paddle/pull/57557),[#58272](https://github.com/PaddlePaddle/Paddle/pull/58272),[#58188](https://github.com/PaddlePaddle/Paddle/pull/58188),[#58401](https://github.com/PaddlePaddle/Paddle/pull/58401),[#59366](https://github.com/PaddlePaddle/Paddle/pull/59366),[#57655](https://github.com/PaddlePaddle/Paddle/pull/57655),[#57360](https://github.com/PaddlePaddle/Paddle/pull/57360),[#56672](https://github.com/PaddlePaddle/Paddle/pull/56672),[#58537](https://github.com/PaddlePaddle/Paddle/pull/58537),[#56247](https://github.com/PaddlePaddle/Paddle/pull/56247),[#59391](https://github.com/PaddlePaddle/Paddle/pull/59391),[#58897](https://github.com/PaddlePaddle/Paddle/pull/58897),[#54933](https://github.com/PaddlePaddle/Paddle/pull/54933) ## 动转静能力增强 动态图到静态图的转换是深度学习框架中的一项关键技术，它允许开发者在灵活性和训练效率之间找到最佳平衡。本版本飞桨对动转静核心功能进行了全面升级，在飞桨产业级模型库的 700 多个模型中，动转静训练的成功率高达 100%。 ### 新功能 - 采用 Python Eval Frame 和虚拟机模拟执行技术，创新性地实现了自适应的 Graph Break 机制。这一机制特别针对控制流场景，通过引入 CallLayer 机制，充分利用飞桨动静统一的优势，支持 AST（抽象语法树）与字节码模拟的混合模式，有效捕获控制流算子，从而大幅度提高了计算图的静态化能力。在缓存优化层面，融合了公共子表达式消除等高级优化技术，显著提升了 Guard 的执行效率。这些优化措施不仅减少了冗余计算，还提高了整体系统的运行速度。为了增强系统的鲁棒性，设计了一个简洁高效的数据中间层结构。这一结构支持 SideEffects 的正确性恢复，确保了系统在复杂环境下的稳定性和可靠性。此外，广泛兼容 Python 3.8 至 3.11 的主流解释器版本，为用户提供了广泛的适用性。[#57824](https://github.com/PaddlePaddle/Paddle/pull/57824),[#55887](https://github.com/PaddlePaddle/Paddle/pull/55887),[#58155](https://github.com/PaddlePaddle/Paddle/pull/58155),[#56107](https://github.com/PaddlePaddle/Paddle/pull/56107),[#57490](https://github.com/PaddlePaddle/Paddle/pull/57490),[#58829](https://github.com/PaddlePaddle/Paddle/pull/58829),[#57240](https://github.com/PaddlePaddle/Paddle/pull/57240),[#57588](https://github.com/PaddlePaddle/Paddle/pull/57588),[#58117](https://github.com/PaddlePaddle/Paddle/pull/58117),[#59823](https://github.com/PaddlePaddle/Paddle/pull/59823),[#56077](https://github.com/PaddlePaddle/Paddle/pull/56077),[#58956](https://github.com/PaddlePaddle/Paddle/pull/58956),[#57653](https://github.com/PaddlePaddle/Paddle/pull/57653),[#59855](https://github.com/PaddlePaddle/Paddle/pull/59855),[#59017](https://github.com/PaddlePaddle/Paddle/pull/59017),[#58424](https://github.com/PaddlePaddle/Paddle/pull/58424),[#58187](https://github.com/PaddlePaddle/Paddle/pull/58187),[#57793](https://github.com/PaddlePaddle/Paddle/pull/57793),[#59698](https://github.com/PaddlePaddle/Paddle/pull/59698),[#59747](https://github.com/PaddlePaddle/Paddle/pull/59747),[#59710](https://github.com/PaddlePaddle/Paddle/pull/59710),[#59297](https://github.com/PaddlePaddle/Paddle/pull/59297),[#58423](https://github.com/PaddlePaddle/Paddle/pull/58423),[#56262](https://github.com/PaddlePaddle/Paddle/pull/56262),[#58103](https://github.com/PaddlePaddle/Paddle/pull/58103),[#58538](https://github.com/PaddlePaddle/Paddle/pull/58538),[#58771](https://github.com/PaddlePaddle/Paddle/pull/58771),[#59191](https://github.com/PaddlePaddle/Paddle/pull/59191),[#57754](https://github.com/PaddlePaddle/Paddle/pull/57754),[#59439](https://github.com/PaddlePaddle/Paddle/pull/59439),[#59816](https://github.com/PaddlePaddle/Paddle/pull/59816),[#59035](https://github.com/PaddlePaddle/Paddle/pull/59035) - 新增对 PyLayer 功能的动转静语法转写解析，使得 PyLayer 在动态图与静态图之间的转换更加顺畅。现在，用户可以在 PyLayer 下无缝地进行动转静的训练，并轻松导出推理模型。[#56108](https://github.com/PaddlePaddle/Paddle/pull/56108),[#56531](https://github.com/PaddlePaddle/Paddle/pull/56531),[#57066](https://github.com/PaddlePaddle/Paddle/pull/57066),[#57633](https://github.com/PaddlePaddle/Paddle/pull/57633) ### Bug Fix - 修复了动转静在 is_test=True 模式部分场景下出现显存异常的问题。[#58350](https://github.com/PaddlePaddle/Paddle/pull/58350) - 修复了被@to_static 装饰的函数在类似 foo(x,x,y) 场景下 jit.save 模型导出的问题。[#55963](https://github.com/PaddlePaddle/Paddle/pull/55963) - 修复了部分 API 行为动静逻辑不统一问题，提升了动转静整图转换成功率和使用体验。[#56092](https://github.com/PaddlePaddle/Paddle/pull/56092) ### 漏洞修复 - 修复了动转静语法转写模块使用 eval()存在的潜在安全漏洞问题。[#60100](https://github.com/PaddlePaddle/Paddle/pull/60100) ## 动态图分布式能力增强 为了满足大型模型的需求，本版本重点提升了飞桨动态图的分布式计算能力。在通信库、图分析、分布式策略和任务启停等方面进行了多方面的改进，为大型模型训练提供了全面的支持。在性能方面，我们通过减少流水并行 GPU 显存占用、采用 TensorFusion 技术、实现通信计算 overlap 以及减少非必要的数据同步拷贝等方式，进一步提升了训练性能。同时，通过环境变量控制 Optimizer 等方式提高了混合并行调试的灵活性。此外，通过修复相关 Bug，进一步提升了系统的稳定性。 ### 新功能 - 通信库新增 TraceHang 功能，当集群训练出现 Hang 的问题时，能够快速的定位到出现问题的节点。[#59217](https://github.com/PaddlePaddle/Paddle/pull/59217) - 为了提升训练效率和降低显存，动态图支持 stride 机制。[#55156](https://github.com/PaddlePaddle/Paddle/pull/55156),[#54762](https://github.com/PaddlePaddle/Paddle/pull/54762),[#55850](https://github.com/PaddlePaddle/Paddle/pull/55850),[#59190](https://github.com/PaddlePaddle/Paddle/pull/59190),[#57005](https://github.com/PaddlePaddle/Paddle/pull/57005),[#57005](https://github.com/PaddlePaddle/Paddle/pull/57005),[#57331](https://github.com/PaddlePaddle/Paddle/pull/57331),[#58033](https://github.com/PaddlePaddle/Paddle/pull/58033),[#58033](https://github.com/PaddlePaddle/Paddle/pull/58033),[#58303](https://github.com/PaddlePaddle/Paddle/pull/58303),[#57835](https://github.com/PaddlePaddle/Paddle/pull/57835),[#57189](https://github.com/PaddlePaddle/Paddle/pull/57189) - 为了方便计算图的分析，增强 paddleviz 功能。[#56837](https://github.com/PaddlePaddle/Paddle/pull/56837),[#57626](https://github.com/PaddlePaddle/Paddle/pull/57626) - 分布式 Sharding 策略(Stage1,2,3)新增 main_grad 功能，以支持更高精度的梯度累加，减少低精度累加带来的精度损失。[#57972](https://github.com/PaddlePaddle/Paddle/pull/57972),[#57934](https://github.com/PaddlePaddle/Paddle/pull/57934),[#57473](https://github.com/PaddlePaddle/Paddle/pull/57473),[#57537](https://github.com/PaddlePaddle/Paddle/pull/57537),[#59611](https://github.com/PaddlePaddle/Paddle/pull/59611),[#57960](https://github.com/PaddlePaddle/Paddle/pull/57960) - Sharding Stage1 策略新增开关变量，可以控制是否对 Optimizer 进行 fusion 计算。[#58790](https://github.com/PaddlePaddle/Paddle/pull/58790) - Recompute 功能新增对 Tuple 输入参数的支持，增强了 Recompute 接口的调用能力。[#56793](https://github.com/PaddlePaddle/Paddle/pull/56793) - 增强 Launch 功能，动态图下无需指定 endpoints 也可以进行分布式训练。 [#54636](https://github.com/PaddlePaddle/Paddle/pull/54636) ### 功能优化 - 实现动静统一的新通信库，通信算子全面适配 PHI 算子体系，减少开发和维护成本，更好地支持动态图和自动并行架构升级。[#54417](https://github.com/PaddlePaddle/Paddle/pull/54417),[#57768](https://github.com/PaddlePaddle/Paddle/pull/57768),[#57897](https://github.com/PaddlePaddle/Paddle/pull/57897),[#55537](https://github.com/PaddlePaddle/Paddle/pull/55537),[#56604](https://github.com/PaddlePaddle/Paddle/pull/56604),[#57519](https://github.com/PaddlePaddle/Paddle/pull/57519),[#56088](https://github.com/PaddlePaddle/Paddle/pull/56088),[#57153](https://github.com/PaddlePaddle/Paddle/pull/57153),[#57161](https://github.com/PaddlePaddle/Paddle/pull/57161),[#57252](https://github.com/PaddlePaddle/Paddle/pull/57252),[#57251](https://github.com/PaddlePaddle/Paddle/pull/57251),[#57208](https://github.com/PaddlePaddle/Paddle/pull/57208),[#57305](https://github.com/PaddlePaddle/Paddle/pull/57305),[#57424](https://github.com/PaddlePaddle/Paddle/pull/57424),[#57548](https://github.com/PaddlePaddle/Paddle/pull/57548),[#57560](https://github.com/PaddlePaddle/Paddle/pull/57560),[#57564](https://github.com/PaddlePaddle/Paddle/pull/57564),[#57233](https://github.com/PaddlePaddle/Paddle/pull/57233),[#55726](https://github.com/PaddlePaddle/Paddle/pull/55726),[#58073](https://github.com/PaddlePaddle/Paddle/pull/58073) - TCPStore 改为单例以便更灵活地支持动态图和自动并行功能。[#55956](https://github.com/PaddlePaddle/Paddle/pull/55956) - 改善了 MP/PP/SP 等分布式策略的可维护性和灵活性，包含增加打印 warning、报错信息，对代码文件进行结构清理，梳理 PP 对输入的限制等。[#54448](https://github.com/PaddlePaddle/Paddle/pull/54448),[#59762](https://github.com/PaddlePaddle/Paddle/pull/59762),[#55462](https://github.com/PaddlePaddle/Paddle/pull/55462),[#54788](https://github.com/PaddlePaddle/Paddle/pull/54788),[#54664](https://github.com/PaddlePaddle/Paddle/pull/54664),[#56456](https://github.com/PaddlePaddle/Paddle/pull/56456),[#55540](https://github.com/PaddlePaddle/Paddle/pull/55540) - PP 策略中增加可以在计算流中进行 P2P 通信的支持，通信模式更加灵活。[#54747](https://github.com/PaddlePaddle/Paddle/pull/54747) - Sharding 策略支持对梯度进行 reduce 操作。[#58842](https://github.com/PaddlePaddle/Paddle/pull/58842),[#57967](https://github.com/PaddlePaddle/Paddle/pull/57967),[#55495](https://github.com/PaddlePaddle/Paddle/pull/55495) ### 性能优化 - 实现 PP 策略的最后一层及时释放 output，以节约显存。[#54505](https://github.com/PaddlePaddle/Paddle/pull/54505) - MP 策略 Tensor fusion 支持传入 params group，增强 Tensor fusion 功能；增加 allreduce 异步通信性能，通过计算和通信的 overlap 提升训练性能。[#57690](https://github.com/PaddlePaddle/Paddle/pull/57690),[#55662](https://github.com/PaddlePaddle/Paddle/pull/55662) - Sharding 策略反向计算和梯度通信进行 overlap 以提升训练性能。Sharding stage1 新增 Tensor fusion 和 fuse grad clip，optimizer 等优化提高计算效率。支持 VPP 与 DP/Sharding Stage1 的 overlap，提升通信计算并行度。优化 Sharding Stage1 在 FP16 下的性能，在 check finite 阶段只对本 sharding rank 负责的梯度进行检查，降低计算开销；增加环境变量，控制是否进行 Optimize，以节约显存支持，实现使用更少的资源进行模型训练调试。[#55598](https://github.com/PaddlePaddle/Paddle/pull/55598),[#55427](https://github.com/PaddlePaddle/Paddle/pull/55427),[#56063](https://github.com/PaddlePaddle/Paddle/pull/56063),[#55766](https://github.com/PaddlePaddle/Paddle/pull/55766),[#59848](https://github.com/PaddlePaddle/Paddle/pull/59848) - 混合并行策略将 PP/VPP 下的 Tensor fusion 提到运行前，解决运行时 fuse 对显存额外开销的问题。通过减少非必需的同步 memcpy，以提升模型训练性能。[#54403](https://github.com/PaddlePaddle/Paddle/pull/54403),[#57215](https://github.com/PaddlePaddle/Paddle/pull/57215) ### Bug Fix - 修复了 PP、Launch 功能、MP 策略以及 fuse_rope 等 13 个 bug，增强了分布式策略的稳定性；机制层面，修复 inplace，tensor 引用的错误，提升稳定性。[#55116](https://github.com/PaddlePaddle/Paddle/pull/55116),[#55782](https://github.com/PaddlePaddle/Paddle/pull/55782),[#59609](https://github.com/PaddlePaddle/Paddle/pull/59609),[#57394](https://github.com/PaddlePaddle/Paddle/pull/57394),[#55864](https://github.com/PaddlePaddle/Paddle/pull/55864),[#58482](https://github.com/PaddlePaddle/Paddle/pull/58482),[#54571](https://github.com/PaddlePaddle/Paddle/pull/54571),[#55896](https://github.com/PaddlePaddle/Paddle/pull/55896),[#54648](https://github.com/PaddlePaddle/Paddle/pull/54648),[#58307](https://github.com/PaddlePaddle/Paddle/pull/58307),[#55679](https://github.com/PaddlePaddle/Paddle/pull/55679),[#58133](https://github.com/PaddlePaddle/Paddle/pull/58133),[#58408](https://github.com/PaddlePaddle/Paddle/pull/58408),[#59707](https://github.com/PaddlePaddle/Paddle/pull/59707),[#55342](https://github.com/PaddlePaddle/Paddle/pull/55342),[#54703](https://github.com/PaddlePaddle/Paddle/pull/54703),[#54869](https://github.com/PaddlePaddle/Paddle/pull/54869),[#55568](https://github.com/PaddlePaddle/Paddle/pull/55568),[#55233](https://github.com/PaddlePaddle/Paddle/pull/55233),[#56418](https://github.com/PaddlePaddle/Paddle/pull/56418),[#56428](https://github.com/PaddlePaddle/Paddle/pull/56428),[#56892](https://github.com/PaddlePaddle/Paddle/pull/56892),[#57192](https://github.com/PaddlePaddle/Paddle/pull/57192),[#59161](https://github.com/PaddlePaddle/Paddle/pull/59161),[#59340](https://github.com/PaddlePaddle/Paddle/pull/59340),[#57006](https://github.com/PaddlePaddle/Paddle/pull/57006),[#57353](https://github.com/PaddlePaddle/Paddle/pull/57353),[#57352](https://github.com/PaddlePaddle/Paddle/pull/57352),[#59088](https://github.com/PaddlePaddle/Paddle/pull/59088) - 修复了 PP 策略无法及时释放单层 output 的 bug，以及初始化过程中可能会 Hang 的 bug。 [#54624](https://github.com/PaddlePaddle/Paddle/pull/54624),[#58844](https://github.com/PaddlePaddle/Paddle/pull/58844),[#54673](https://github.com/PaddlePaddle/Paddle/pull/54673),[#58376](https://github.com/PaddlePaddle/Paddle/pull/58376) - 修复了 MP 策略下，当输入数据类型不统一时计算出错的 bug，修复了 MP 策略下参数同步的 bug 和没有正确使用用户输入 config 的 bug。[#58858](https://github.com/PaddlePaddle/Paddle/pull/58858),[#57918](https://github.com/PaddlePaddle/Paddle/pull/57918),[#58037](https://github.com/PaddlePaddle/Paddle/pull/58037) - 统一 dygraph 和 dynamic 模式的判断方法。[#54633](https://github.com/PaddlePaddle/Paddle/pull/54633) - 修复了 fuse_rope 中 sin 和 cos 的 Shape 不对的 bug。[#56132](https://github.com/PaddlePaddle/Paddle/pull/56132) - 修复了 Luanch 功能分布式场景下 endpoints 太长导致不能启动任务的 bug，同时修复了 endpoints 可能乱序的 bug。 [#55011](https://github.com/PaddlePaddle/Paddle/pull/55011),[#55478](https://github.com/PaddlePaddle/Paddle/pull/55478) - 修复了 MEA 功能可能导致 segmentation fault error 的 bug。[#55408](https://github.com/PaddlePaddle/Paddle/pull/55408) ## 自动并行 本版本对动静统一自动并行（Auto Parallel）编程范式进行了全面的优化，简化了开发者的编程复杂度。开发者无需深入了解手动并行编程范式下的复杂概念和 API 接口，如行切分、列切分等。仅需通过少量的张量切分标注即可完成混合并行模型的构建。框架能够自动推导出所有张量和算子的分布式切分状态，并添加合适的通信算子。同时支持一键动转静进行分布式训练，使开发者能够高效轻松地实现任意混合并行策略，大幅降低了混合并行训练代码的开发成本。 ### 完善了自动并行核心功能 - 实现 process_mesh、placement、shard_tensor、reshard、dtensor_from_fn、unshard_dtensor、shard_layer、to_static 等自动并行核心接口 [#55494](https://github.com/PaddlePaddle/Paddle/pull/55494),[#59059](https://github.com/PaddlePaddle/Paddle/pull/59059),[#56561](https://github.com/PaddlePaddle/Paddle/pull/56561),[#54425](https://github.com/PaddlePaddle/Paddle/pull/54425),[#59557](https://github.com/PaddlePaddle/Paddle/pull/59557),[#59682](https://github.com/PaddlePaddle/Paddle/pull/59682),[#56565](https://github.com/PaddlePaddle/Paddle/pull/56565),[#59862](https://github.com/PaddlePaddle/Paddle/pull/59862),[#59856](https://github.com/PaddlePaddle/Paddle/pull/59856),[#59342](https://github.com/PaddlePaddle/Paddle/pull/59342),[#59575](https://github.com/PaddlePaddle/Paddle/pull/59575),[#57604](https://github.com/PaddlePaddle/Paddle/pull/57604),[#57293](https://github.com/PaddlePaddle/Paddle/pull/57293),[#57278](https://github.com/PaddlePaddle/Paddle/pull/57278) - 实现基于 Enisum 表达式的切分推导规则，并完成 20+类算子切分推导规则，覆盖 LLaMA、GPT 等主流生成式大语言模型。[#55196](https://github.com/PaddlePaddle/Paddle/pull/55196),[#53863](https://github.com/PaddlePaddle/Paddle/pull/53863),[#56257](https://github.com/PaddlePaddle/Paddle/pull/56257),[#55394](https://github.com/PaddlePaddle/Paddle/pull/55394),[#54810](https://github.com/PaddlePaddle/Paddle/pull/54810),[#55508](https://github.com/PaddlePaddle/Paddle/pull/55508),[#56257](https://github.com/PaddlePaddle/Paddle/pull/56257),[#57813](https://github.com/PaddlePaddle/Paddle/pull/57813),[#58149](https://github.com/PaddlePaddle/Paddle/pull/58149),[#58506](https://github.com/PaddlePaddle/Paddle/pull/58506),[#58563](https://github.com/PaddlePaddle/Paddle/pull/58563),[#58360](https://github.com/PaddlePaddle/Paddle/pull/58360),[#58920](https://github.com/PaddlePaddle/Paddle/pull/58920),[#59050](https://github.com/PaddlePaddle/Paddle/pull/59050),[#58760](https://github.com/PaddlePaddle/Paddle/pull/58760),[#59083](https://github.com/PaddlePaddle/Paddle/pull/59083),[#59236](https://github.com/PaddlePaddle/Paddle/pull/59236),[#59350](https://github.com/PaddlePaddle/Paddle/pull/59350),[#59411](https://github.com/PaddlePaddle/Paddle/pull/59411),[#59260](https://github.com/PaddlePaddle/Paddle/pull/59260),[#54373](https://github.com/PaddlePaddle/Paddle/pull/54373),[#54991](https://github.com/PaddlePaddle/Paddle/pull/54991),[#55397](https://github.com/PaddlePaddle/Paddle/pull/55397),[#55350](https://github.com/PaddlePaddle/Paddle/pull/55350),[#55177](https://github.com/PaddlePaddle/Paddle/pull/55177),[#56443](https://github.com/PaddlePaddle/Paddle/pull/56443),[#58097](https://github.com/PaddlePaddle/Paddle/pull/58097),[#56509](https://github.com/PaddlePaddle/Paddle/pull/56509),[#56502](https://github.com/PaddlePaddle/Paddle/pull/56502),[#56504](https://github.com/PaddlePaddle/Paddle/pull/56504),[#56506](https://github.com/PaddlePaddle/Paddle/pull/56506),[#56507](https://github.com/PaddlePaddle/Paddle/pull/56507),[#56505](https://github.com/PaddlePaddle/Paddle/pull/56505),[#57176](https://github.com/PaddlePaddle/Paddle/pull/57176),[#57374](https://github.com/PaddlePaddle/Paddle/pull/57374),[#57573](https://github.com/PaddlePaddle/Paddle/pull/57573),[#57545](https://github.com/PaddlePaddle/Paddle/pull/57545),[#57875](https://github.com/PaddlePaddle/Paddle/pull/57875),[#57866](https://github.com/PaddlePaddle/Paddle/pull/57866),[#58854](https://github.com/PaddlePaddle/Paddle/pull/58854),[#59109](https://github.com/PaddlePaddle/Paddle/pull/59109),[#59185](https://github.com/PaddlePaddle/Paddle/pull/59185),[#58913](https://github.com/PaddlePaddle/Paddle/pull/58913),[#59547](https://github.com/PaddlePaddle/Paddle/pull/59547),[#58296](https://github.com/PaddlePaddle/Paddle/pull/58296),[#59545](https://github.com/PaddlePaddle/Paddle/pull/59545),[#59039](https://github.com/PaddlePaddle/Paddle/pull/59039),[#59002](https://github.com/PaddlePaddle/Paddle/pull/59002),[#58087](https://github.com/PaddlePaddle/Paddle/pull/58087),[#56367](https://github.com/PaddlePaddle/Paddle/pull/56367),[#57877](https://github.com/PaddlePaddle/Paddle/pull/57877),[#56839](https://github.com/PaddlePaddle/Paddle/pull/56839),[#59003](https://github.com/PaddlePaddle/Paddle/pull/59003),[#57269](https://github.com/PaddlePaddle/Paddle/pull/57269),[#55130](https://github.com/PaddlePaddle/Paddle/pull/55130),[#58474](https://github.com/PaddlePaddle/Paddle/pull/58474),[#57197](https://github.com/PaddlePaddle/Paddle/pull/57197),[#57467](https://github.com/PaddlePaddle/Paddle/pull/57467),[#57259](https://github.com/PaddlePaddle/Paddle/pull/57259),[#57280](https://github.com/PaddlePaddle/Paddle/pull/57280),[#56508](https://github.com/PaddlePaddle/Paddle/pull/56508) - 实现动静统一的分布式 checkpoint 存储和加载，支持任意按切分状态存储和加载时重切分。[#59659](https://github.com/PaddlePaddle/Paddle/pull/59659),[#59843](https://github.com/PaddlePaddle/Paddle/pull/59843),[#60033](https://github.com/PaddlePaddle/Paddle/pull/60033),[#60034](https://github.com/PaddlePaddle/Paddle/pull/60034) ### 增强动态图半自动并行能力 - 基础数据结构补充：C++端新增 DistTensor、Placements 等分布式特有的基础数据结构，并暴露到 Python 端，支持对相关属性和值的调试打印。[#58930](https://github.com/PaddlePaddle/Paddle/pull/58930),[#59068](https://github.com/PaddlePaddle/Paddle/pull/59068),[#55436](https://github.com/PaddlePaddle/Paddle/pull/55436),[#56449](https://github.com/PaddlePaddle/Paddle/pull/56449),[#59683](https://github.com/PaddlePaddle/Paddle/pull/59683),[#55593](https://github.com/PaddlePaddle/Paddle/pull/55593),[#58032](https://github.com/PaddlePaddle/Paddle/pull/58032),[#56368](https://github.com/PaddlePaddle/Paddle/pull/56368),[#59086](https://github.com/PaddlePaddle/Paddle/pull/59086) - 在前、反向算子执行流程中添加 SPMD 推导与 Reshard 的生成逻辑，适配 vector、optional 等多类型输入输出以及 cpu fallback、多 kernel 选择等特殊机制。[#56602](https://github.com/PaddlePaddle/Paddle/pull/56602),[#57321](https://github.com/PaddlePaddle/Paddle/pull/57321),[#57092](https://github.com/PaddlePaddle/Paddle/pull/57092),[#56831](https://github.com/PaddlePaddle/Paddle/pull/56831),[#57119](https://github.com/PaddlePaddle/Paddle/pull/57119),[#58819](https://github.com/PaddlePaddle/Paddle/pull/58819),[#58254](https://github.com/PaddlePaddle/Paddle/pull/58254),[#55698](https://github.com/PaddlePaddle/Paddle/pull/55698),[#59241](https://github.com/PaddlePaddle/Paddle/pull/59241),[#59328](https://github.com/PaddlePaddle/Paddle/pull/59328),[#58644](https://github.com/PaddlePaddle/Paddle/pull/58644),[#56202](https://github.com/PaddlePaddle/Paddle/pull/56202),[#59159](https://github.com/PaddlePaddle/Paddle/pull/59159),[#58573](https://github.com/PaddlePaddle/Paddle/pull/58573),[#59246](https://github.com/PaddlePaddle/Paddle/pull/59246),[#59133](https://github.com/PaddlePaddle/Paddle/pull/59133),[#59186](https://github.com/PaddlePaddle/Paddle/pull/59186),[#57505](https://github.com/PaddlePaddle/Paddle/pull/57505),[#57241](https://github.com/PaddlePaddle/Paddle/pull/57241),[#58928](https://github.com/PaddlePaddle/Paddle/pull/58928) - 对 custom 算子、手写算子等特殊类型的算子，适配自动并行的执行逻辑。支持 DistTensor 和 DenseTensor 作为混合输入时的自动转换。[#57774](https://github.com/PaddlePaddle/Paddle/pull/57774),[#59108](https://github.com/PaddlePaddle/Paddle/pull/59108),[#58436](https://github.com/PaddlePaddle/Paddle/pull/58436),[#59523](https://github.com/PaddlePaddle/Paddle/pull/59523),[#59136](https://github.com/PaddlePaddle/Paddle/pull/59136),[#59352](https://github.com/PaddlePaddle/Paddle/pull/59352),[#59062](https://github.com/PaddlePaddle/Paddle/pull/59062),[#58434](https://github.com/PaddlePaddle/Paddle/pull/58434),[#59148](https://github.com/PaddlePaddle/Paddle/pull/59148),[#58553](https://github.com/PaddlePaddle/Paddle/pull/58553),[#58716](https://github.com/PaddlePaddle/Paddle/pull/58716),[#58369](https://github.com/PaddlePaddle/Paddle/pull/58369),[#59061](https://github.com/PaddlePaddle/Paddle/pull/59061),[#58841](https://github.com/PaddlePaddle/Paddle/pull/58841),[#59139](https://github.com/PaddlePaddle/Paddle/pull/59139),[#59141](https://github.com/PaddlePaddle/Paddle/pull/59141),[#58837](https://github.com/PaddlePaddle/Paddle/pull/58837),[#59137](https://github.com/PaddlePaddle/Paddle/pull/59137),[#59143](https://github.com/PaddlePaddle/Paddle/pull/59143) - 动态图执行体系完善：适配 Autograd 执行过程，支持动态图的反向梯度聚合、AMP、Hook、PyLayer、View、自定义算子等周围机制。[#58437](https://github.com/PaddlePaddle/Paddle/pull/58437),[#58769](https://github.com/PaddlePaddle/Paddle/pull/58769),[#58796](https://github.com/PaddlePaddle/Paddle/pull/58796),[#58339](https://github.com/PaddlePaddle/Paddle/pull/58339),[#58409](https://github.com/PaddlePaddle/Paddle/pull/58409),[#58772](https://github.com/PaddlePaddle/Paddle/pull/58772),[#58380](https://github.com/PaddlePaddle/Paddle/pull/58380),[#58447](https://github.com/PaddlePaddle/Paddle/pull/58447),[#58706](https://github.com/PaddlePaddle/Paddle/pull/58706),[#58656](https://github.com/PaddlePaddle/Paddle/pull/58656),[#58172](https://github.com/PaddlePaddle/Paddle/pull/58172),[#59401](https://github.com/PaddlePaddle/Paddle/pull/59401),[#58727](https://github.com/PaddlePaddle/Paddle/pull/58727),[#58238](https://github.com/PaddlePaddle/Paddle/pull/58238),[#59243](https://github.com/PaddlePaddle/Paddle/pull/59243),[#58469](https://github.com/PaddlePaddle/Paddle/pull/58469),[#58442](https://github.com/PaddlePaddle/Paddle/pull/58442),[#58487](https://github.com/PaddlePaddle/Paddle/pull/58487),[#58476](https://github.com/PaddlePaddle/Paddle/pull/58476),[#59706](https://github.com/PaddlePaddle/Paddle/pull/59706) - 新增对 PP、SP 等分布式策略的支持。[#58126](https://github.com/PaddlePaddle/Paddle/pull/58126),[#59766](https://github.com/PaddlePaddle/Paddle/pull/59766),[#59060](https://github.com/PaddlePaddle/Paddle/pull/59060),[#59841](https://github.com/PaddlePaddle/Paddle/pull/59841),[#58609](https://github.com/PaddlePaddle/Paddle/pull/58609),[#59688](https://github.com/PaddlePaddle/Paddle/pull/59688),[#58449](https://github.com/PaddlePaddle/Paddle/pull/58449)、[#59598](https://github.com/PaddlePaddle/Paddle/pull/59598) - 新增多种 Reshard 策略，支持张量在不同分布式状态间的转换。[#58592](https://github.com/PaddlePaddle/Paddle/pull/58592),[#59138](https://github.com/PaddlePaddle/Paddle/pull/59138),[#59367](https://github.com/PaddlePaddle/Paddle/pull/59367),[#59621](https://github.com/PaddlePaddle/Paddle/pull/59621),[#59758](https://github.com/PaddlePaddle/Paddle/pull/59758),[#59777](https://github.com/PaddlePaddle/Paddle/pull/59777),[#56975](https://github.com/PaddlePaddle/Paddle/pull/56975),[#58550](https://github.com/PaddlePaddle/Paddle/pull/58550),[#58703](https://github.com/PaddlePaddle/Paddle/pull/58703),[#57210](https://github.com/PaddlePaddle/Paddle/pull/57210),[#58734](https://github.com/PaddlePaddle/Paddle/pull/58734),[#56833](https://github.com/PaddlePaddle/Paddle/pull/56833),[#59292](https://github.com/PaddlePaddle/Paddle/pull/59292),[#57432](https://github.com/PaddlePaddle/Paddle/pull/57432),[#57568](https://github.com/PaddlePaddle/Paddle/pull/57568),[#56553](https://github.com/PaddlePaddle/Paddle/pull/56553),[#58284](https://github.com/PaddlePaddle/Paddle/pull/58284),[#56039](https://github.com/PaddlePaddle/Paddle/pull/56039),[#55552](https://github.com/PaddlePaddle/Paddle/pull/55552),[#56149](https://github.com/PaddlePaddle/Paddle/pull/56149) ### 静态图半自动并行能力增强 - 新增 Sequence Parallel 并行策略；流水线并行新增: FThenB、Interleaved 1F1B、Eager 1F1B、VPP 等调度模式，支持流水线调度的可视化，并支持上述策略与原有并行策略的混合并行；升级梯度同步机制，支持数据在任意 broadcast 维度后需要的梯度同步。[#57605](https://github.com/PaddlePaddle/Paddle/pull/57605),[#54727](https://github.com/PaddlePaddle/Paddle/pull/54727),[#54409](https://github.com/PaddlePaddle/Paddle/pull/54409),[#54787](https://github.com/PaddlePaddle/Paddle/pull/54787),[#58313](https://github.com/PaddlePaddle/Paddle/pull/58313),[#59179](https://github.com/PaddlePaddle/Paddle/pull/59179),[#59416](https://github.com/PaddlePaddle/Paddle/pull/59416),[#59719](https://github.com/PaddlePaddle/Paddle/pull/59719),[#59822](https://github.com/PaddlePaddle/Paddle/pull/59822),[#59057](https://github.com/PaddlePaddle/Paddle/pull/59057),[#59522](https://github.com/PaddlePaddle/Paddle/pull/59522),[#57061](https://github.com/PaddlePaddle/Paddle/pull/57061) - 执行体系与 PIR 进一步适配，打通 PIR 的优化 Pass，分布式场景下支持了 fuse_linear fuse 优化，实现性能提升。[#58459](https://github.com/PaddlePaddle/Paddle/pull/58459),[#58528](https://github.com/PaddlePaddle/Paddle/pull/58528),[#55555](https://github.com/PaddlePaddle/Paddle/pull/55555),[#59757](https://github.com/PaddlePaddle/Paddle/pull/59757),[#59102](https://github.com/PaddlePaddle/Paddle/pull/59102),[#57917](https://github.com/PaddlePaddle/Paddle/pull/57917) - 底层架构升级: 执行器升级支持图依赖信息复用和静态化 kernel 选择；整图切分补全机制升级，切换新切分推导规则并支持更多长尾 cases 的正确切分补全；优化了静态图分布式下对控制流的支持，适配更多场景；优化了整图编译速度、日志信息格式等提升用户体验。 [#55389](https://github.com/PaddlePaddle/Paddle/pull/55389),[#55650](https://github.com/PaddlePaddle/Paddle/pull/55650),[#54938](https://github.com/PaddlePaddle/Paddle/pull/54938),[#57447](https://github.com/PaddlePaddle/Paddle/pull/57447),[#57751](https://github.com/PaddlePaddle/Paddle/pull/57751),[#57742](https://github.com/PaddlePaddle/Paddle/pull/57742),[#59524](https://github.com/PaddlePaddle/Paddle/pull/59524),[#59526](https://github.com/PaddlePaddle/Paddle/pull/59526),[#58669](https://github.com/PaddlePaddle/Paddle/pull/58669),[#57616](https://github.com/PaddlePaddle/Paddle/pull/57616),[#56511](https://github.com/PaddlePaddle/Paddle/pull/56511),[#55727](https://github.com/PaddlePaddle/Paddle/pull/55727),[#58906](https://github.com/PaddlePaddle/Paddle/pull/58906),[#56016](https://github.com/PaddlePaddle/Paddle/pull/56016),[#54897](https://github.com/PaddlePaddle/Paddle/pull/54897) - 优化静态图显存管理，新增精细化重计算策略；优化混合精度适配，支持用户手动指定 cast 范围等场景；支持 Cross Entropy 的并行计算；支持 scaled_dot_product_attention、fuse_rope 等融合算子；执行调度优化，支持张量并行、流水线并行中通信计算间更好地 Overlap。[#58421](https://github.com/PaddlePaddle/Paddle/pull/58421),[#58533](https://github.com/PaddlePaddle/Paddle/pull/58533),[#59498](https://github.com/PaddlePaddle/Paddle/pull/59498),[#59498](https://github.com/PaddlePaddle/Paddle/pull/59498),[#59187](https://github.com/PaddlePaddle/Paddle/pull/59187),[#59188](https://github.com/PaddlePaddle/Paddle/pull/59188),[#58172](https://github.com/PaddlePaddle/Paddle/pull/58172),[#58628](https://github.com/PaddlePaddle/Paddle/pull/58628),[#56185](https://github.com/PaddlePaddle/Paddle/pull/56185),[#56696](https://github.com/PaddlePaddle/Paddle/pull/56696),[#59497](https://github.com/PaddlePaddle/Paddle/pull/59497),[#58304](https://github.com/PaddlePaddle/Paddle/pull/58304),[#58977](https://github.com/PaddlePaddle/Paddle/pull/58977) ### AutoTuner 本版本实现基于 Profiling 的并行策略自动搜索和调优工具 AutoTuner，能够在给定模型和硬件资源的条件下，自动将并行策略和优化策略进行组合，并选取有效的组合配置运行实验，从而搜索出大模型训练和推理的最佳配置。此外，AutoTuner 实现了多种剪枝优化策略，包括显存建模等，能够大幅度减少搜索空间和搜索时间。[#54460](https://github.com/PaddlePaddle/Paddle/pull/54460),[#54668](https://github.com/PaddlePaddle/Paddle/pull/54668),[#59794](https://github.com/PaddlePaddle/Paddle/pull/59794),[#59727](https://github.com/PaddlePaddle/Paddle/pull/59727),[#59782](https://github.com/PaddlePaddle/Paddle/pull/59782),[#54834](https://github.com/PaddlePaddle/Paddle/pull/54834),[#58127](https://github.com/PaddlePaddle/Paddle/pull/58127),[#56968](https://github.com/PaddlePaddle/Paddle/pull/56968),[#55466](https://github.com/PaddlePaddle/Paddle/pull/55466),[#56939](https://github.com/PaddlePaddle/Paddle/pull/56939),[#58183](https://github.com/PaddlePaddle/Paddle/pull/58183),[#58314](https://github.com/PaddlePaddle/Paddle/pull/58314),[#55499](https://github.com/PaddlePaddle/Paddle/pull/55499),[#59748](https://github.com/PaddlePaddle/Paddle/pull/59748) ## 算子库 ### 不兼容升级 为了提升飞桨框架的可维护性，删除框架中部分废弃的算子(如 diag_v1, isfinite_v1, pad2d_v1 等)，通过飞桨 1.x 版本训练所保存的使用到这些算子的模型将无法在飞桨新版本上进行推理。[#57895](https://github.com/PaddlePaddle/Paddle/pull/57895),[#57892](https://github.com/PaddlePaddle/Paddle/pull/57892),[#57898](https://github.com/PaddlePaddle/Paddle/pull/57898),[#57730](https://github.com/PaddlePaddle/Paddle/pull/57730),[#57732](https://github.com/PaddlePaddle/Paddle/pull/57732),[#57810](https://github.com/PaddlePaddle/Paddle/pull/57810),[#57884](https://github.com/PaddlePaddle/Paddle/pull/57884),[#57794](https://github.com/PaddlePaddle/Paddle/pull/57794),[#57926](https://github.com/PaddlePaddle/Paddle/pull/57926),[#57925](https://github.com/PaddlePaddle/Paddle/pull/57925),[#57807](https://github.com/PaddlePaddle/Paddle/pull/57807),[#57808](https://github.com/PaddlePaddle/Paddle/pull/57808) ### 算子库功能增强 - 飞桨 PHI 算子库复数计算功能进一步增强，累计新增支持复数计算 Kernel 40+。[#55380](https://github.com/PaddlePaddle/Paddle/pull/55380), [#56349](https://github.com/PaddlePaddle/Paddle/pull/56349), [#56412](https://github.com/PaddlePaddle/Paddle/pull/56412), [#56323](https://github.com/PaddlePaddle/Paddle/pull/56323), [#56723](https://github.com/PaddlePaddle/Paddle/pull/56723), [#56457](https://github.com/PaddlePaddle/Paddle/pull/56457), [#56903](https://github.com/PaddlePaddle/Paddle/pull/56903)[#56914](https://github.com/PaddlePaddle/Paddle/pull/56914), [#57116](https://github.com/PaddlePaddle/Paddle/pull/57116), [#56048](https://github.com/PaddlePaddle/Paddle/pull/56048), [#57244](https://github.com/PaddlePaddle/Paddle/pull/57244), [#57639](https://github.com/PaddlePaddle/Paddle/pull/57639), [#57638](https://github.com/PaddlePaddle/Paddle/pull/57638), [#57540](https://github.com/PaddlePaddle/Paddle/pull/57540), [#58545](https://github.com/PaddlePaddle/Paddle/pull/58545), [#58336](https://github.com/PaddlePaddle/Paddle/pull/58336), [#58532](https://github.com/PaddlePaddle/Paddle/pull/58532), [#58839](https://github.com/PaddlePaddle/Paddle/pull/58839), [#59079](https://github.com/PaddlePaddle/Paddle/pull/59079), [#59277](https://github.com/PaddlePaddle/Paddle/pull/59277), [#59122](https://github.com/PaddlePaddle/Paddle/pull/59122), [#57058](https://github.com/PaddlePaddle/Paddle/pull/57058) - 优化和新增部分算子的 XPU Kernel，并增强了 XPU Kernel 对 bfloat16 等数据类型的运算支持。[#54478](https://github.com/PaddlePaddle/Paddle/pull/54478), [#57740](https://github.com/PaddlePaddle/Paddle/pull/57740), [#58346](https://github.com/PaddlePaddle/Paddle/pull/58346), [#58456](https://github.com/PaddlePaddle/Paddle/pull/58456), [#58662](https://github.com/PaddlePaddle/Paddle/pull/58662), [#59066](https://github.com/PaddlePaddle/Paddle/pull/59066), [#59263](https://github.com/PaddlePaddle/Paddle/pull/59263)), [#59375](https://github.com/PaddlePaddle/Paddle/pull/59375), [#59505](https://github.com/PaddlePaddle/Paddle/pull/59505), [#59653](https://github.com/PaddlePaddle/Paddle/pull/59653), [#55001](https://github.com/PaddlePaddle/Paddle/pull/55001), [#57272](https://github.com/PaddlePaddle/Paddle/pull/57272), [#56169](https://github.com/PaddlePaddle/Paddle/pull/56169), [#59454](https://github.com/PaddlePaddle/Paddle/pull/59454), [#59480](https://github.com/PaddlePaddle/Paddle/pull/59480), [#55914](https://github.com/PaddlePaddle/Paddle/pull/55914), [#54758](https://github.com/PaddlePaddle/Paddle/pull/54758), [#54827](https://github.com/PaddlePaddle/Paddle/pull/54827), [#58364](https://github.com/PaddlePaddle/Paddle/pull/58364), [#58419](https://github.com/PaddlePaddle/Paddle/pull/58419), [#58982](https://github.com/PaddlePaddle/Paddle/pull/58982), [#57216](https://github.com/PaddlePaddle/Paddle/pull/57216), [#59166](https://github.com/PaddlePaddle/Paddle/pull/59166), [#55033](https://github.com/PaddlePaddle/Paddle/pull/55033), [#55375](https://github.com/PaddlePaddle/Paddle/pull/55375), [#58805](https://github.com/PaddlePaddle/Paddle/pull/58805), [#59389](https://github.com/PaddlePaddle/Paddle/pull/59389), [#57077](https://github.com/PaddlePaddle/Paddle/pull/57077), [#55166](https://github.com/PaddlePaddle/Paddle/pull/55166), [#56773](https://github.com/PaddlePaddle/Paddle/pull/56773) - 新增了用于优化大模型训练和推理性能的常见算子。[#55758](https://github.com/PaddlePaddle/Paddle/pull/55758), [#54998](https://github.com/PaddlePaddle/Paddle/pull/54998), [#55400](https://github.com/PaddlePaddle/Paddle/pull/55400), [#54630](https://github.com/PaddlePaddle/Paddle/pull/54630), [#55969](https://github.com/PaddlePaddle/Paddle/pull/55969), [#55026](https://github.com/PaddlePaddle/Paddle/pull/55026), [#58986](https://github.com/PaddlePaddle/Paddle/pull/58986) - 完善算子库 Tensor Strided 机制。[#59422](https://github.com/PaddlePaddle/Paddle/pull/59422), [#59325](https://github.com/PaddlePaddle/Paddle/pull/59325), [#56863](https://github.com/PaddlePaddle/Paddle/pull/56863), [#56882](https://github.com/PaddlePaddle/Paddle/pull/56882), [#56947](https://github.com/PaddlePaddle/Paddle/pull/56947) - 对算子 Kernel 中的函数实现以及模板调用接口进行了编译优化，降低算子库编包体积。[#57083](https://github.com/PaddlePaddle/Paddle/pull/57083), [#57299](https://github.com/PaddlePaddle/Paddle/pull/57299), [#57261](https://github.com/PaddlePaddle/Paddle/pull/57261), [#57290](https://github.com/PaddlePaddle/Paddle/pull/57290), [#57118](https://github.com/PaddlePaddle/Paddle/pull/57118), [#57551](https://github.com/PaddlePaddle/Paddle/pull/57551), [#57509](https://github.com/PaddlePaddle/Paddle/pull/57509), [#57558](https://github.com/PaddlePaddle/Paddle/pull/57558), [#57064](https://github.com/PaddlePaddle/Paddle/pull/57064), [#57365](https://github.com/PaddlePaddle/Paddle/pull/57365), [#57327](https://github.com/PaddlePaddle/Paddle/pull/57327), [#57603](https://github.com/PaddlePaddle/Paddle/pull/57603), [#57671](https://github.com/PaddlePaddle/Paddle/pull/57671), [#57672](https://github.com/PaddlePaddle/Paddle/pull/57672), [#57631](https://github.com/PaddlePaddle/Paddle/pull/57631), [#57082](https://github.com/PaddlePaddle/Paddle/pull/57082), [#57721](https://github.com/PaddlePaddle/Paddle/pull/57721), [#57823](https://github.com/PaddlePaddle/Paddle/pull/57823), [#57821](https://github.com/PaddlePaddle/Paddle/pull/57821), [#57815](https://github.com/PaddlePaddle/Paddle/pull/57815), [#57822](https://github.com/PaddlePaddle/Paddle/pull/57822), [#57541](https://github.com/PaddlePaddle/Paddle/pull/57541), [#57817](https://github.com/PaddlePaddle/Paddle/pull/57817), [#57838](https://github.com/PaddlePaddle/Paddle/pull/57838) ### Bug 修复 - 修复了飞桨框架适配 CUDA 12 的一些问题。[#54640](https://github.com/PaddlePaddle/Paddle/pull/54640), [#57820](https://github.com/PaddlePaddle/Paddle/pull/57820), [#58958](https://github.com/PaddlePaddle/Paddle/pull/58958), [#58179](https://github.com/PaddlePaddle/Paddle/pull/58179), [#55594](https://github.com/PaddlePaddle/Paddle/pull/55594) ## CUDA ### 新功能 - 新增调试类 API paddle.amp.debugging.check_check_numerics，计算并返回这个 Tensor 数值中异常值(NaN、Inf)和零元素的数量。[#54301](https://github.com/PaddlePaddle/Paddle/pull/54301) - 新增 fused_rope 融合算子，加速 LLaMA 类大模型训练。[#54351](https://github.com/PaddlePaddle/Paddle/pull/54351) - 更新 CUDNN Frontend API 版本到 v0.9.1，并新增加速 ResNet 网络的 fused_scale_bias_add_relu 融合算子。注意该功能处于实验期，默认不开启。[#58367](https://github.com/PaddlePaddle/Paddle/pull/58367), [#54949](https://github.com/PaddlePaddle/Paddle/pull/54949), [#58504](https://github.com/PaddlePaddle/Paddle/pull/58504) - 基于 Flash-Attention v2，添加 Tensor 类似 Mask 功能支持，反向算子支持确定性计算，便于调试。[#57276](https://github.com/PaddlePaddle/Paddle/pull/57276), [#56363](https://github.com/PaddlePaddle/Paddle/pull/56363) - 修改稀疏 conv3d 后端实现以支持 2d 形状，避免前端 reshape 的开销。[#54707](https://github.com/PaddlePaddle/Paddle/pull/54707) - 新增 matmul_int8 算子。([#55228](https://github.com/PaddlePaddle/Paddle/pull/55228)) ### 功能优化 - 优化 CUDA Graph 对随机数算子的支持。[#58310](https://github.com/PaddlePaddle/Paddle/pull/58310) - 自动混合精度训练默认功能加强，包括： - 优化自动混合精度训练接口的使用体验。[#58152](https://github.com/PaddlePaddle/Paddle/pull/58152),[#55364](https://github.com/PaddlePaddle/Paddle/pull/55364),[#57903](https://github.com/PaddlePaddle/Paddle/pull/57903) - 将 fused_attention、fused_feedforward、fused_gemm_epilogue 等矩阵计算类算子加入框架默认的白名单，并统一动静态图默认黑白名单设置。[#55373](https://github.com/PaddlePaddle/Paddle/pull/55373), [#55713](https://github.com/PaddlePaddle/Paddle/pull/55713) - argsort、dist、erfinv、nanmedian、poisson 算子和 lamb 优化器算子支持 FP16、BF16 低精度计算。[#51662](https://github.com/PaddlePaddle/Paddle/pull/51662), [#55105](https://github.com/PaddlePaddle/Paddle/pull/55105), [#55287](https://github.com/PaddlePaddle/Paddle/pull/55287), [#55824](https://github.com/PaddlePaddle/Paddle/pull/55824), [#56056](https://github.com/PaddlePaddle/Paddle/pull/56056), [#56184](https://github.com/PaddlePaddle/Paddle/pull/56184), [#55641](https://github.com/PaddlePaddle/Paddle/pull/55641) - 修复 elementwise_max 算子低精度实现，改成使用 FP32 类型进行数值计算，减少精度损失。[#54799](https://github.com/PaddlePaddle/Paddle/pull/54799) - 将 Reduce 类算子计算需要的临时结果 Tensor 改成 FP32 类型，避免将中间结果转换成低精度带来的精度损失。[#55709](https://github.com/PaddlePaddle/Paddle/pull/55709)) - flip、roll & roll_grad、index_put & index_put_grad 等算子 GPU 代码实现优化，在性能不下降的前提下移除不必要的 C++模板，优化算子编译耗时并减少编译生成的二进制体积。[#57309](https://github.com/PaddlePaddle/Paddle/pull/57309), [#57525](https://github.com/PaddlePaddle/Paddle/pull/57525) - bernoulli 算子增加对输入概率合法性的检查。[#59174](https://github.com/PaddlePaddle/Paddle/pull/59174) ### 性能优化 - 优化 BroadcastKernel 对大 Tensor 的支持，改成对大 Tensor 切片多次调用 INT32 版本实现的方式，算子性能提升 7.27x。[#57313](https://github.com/PaddlePaddle/Paddle/pull/57313), [#57996](https://github.com/PaddlePaddle/Paddle/pull/57996) - 优化 Tensor 保存接口的性能，通过先将 Tensor 拷贝到 CPU 再转 numpy，避免 Tensor 不连续时自动转换成连续 Tensor 的开销。[#57040](https://github.com/PaddlePaddle/Paddle/pull/57040) ### Bug Fix - 修复 memmory_efficient_attention 算子对 sm_90 的支持。[#58070](https://github.com/PaddlePaddle/Paddle/pull/58070) - 修复 softmax 算子，当 axis=-1 且长度大于 100000 的实现出现的 NaN 问题。[#57851](https://github.com/PaddlePaddle/Paddle/pull/57851) - 修复 set_constant 算子在一些情况下出现 GPU 访存错误问题。[#59905](https://github.com/PaddlePaddle/Paddle/pull/59905) - 修复 layer_norm 算子快速实现版本中出现的 GPU 存储读写竞争问题。[#56435](https://github.com/PaddlePaddle/Paddle/pull/56435) ## 拓展神经网络编译器 CINN 架构能力 在本次更新中，飞桨神经网络编译器 CINN 的重点在于架构的梳理和能力的全面扩展。鉴于大模型对动态 Shape 的需求日益增长，初步探索并实现了在动态 shape 下编译器的有效运行和优化策略。 在架构层面，引入了 Python DSL，这一举措显著提升了 CINN 的开发便捷性和 Debug 能力，使得开发者能够更高效地编写和调试代码。同时，对 Schedule 的逻辑进行了重构，以 GroupSchedule 为主导，从而在算子 Group 层面实现更加通用且稳定的优化策略。为了增强 CINN 的稳定性，探索并引入了强约束组件，这一组件能够有效减少系统中的不确定性和潜在错误。此外，对 CINN 的历史工具类和软件结构进行了系统性的整理、优化和改进，进一步提升了代码的可读性和可维护性。在与飞桨其他组件的整合方面，进一步加强了 CINN 与 PIR、Paddle 的紧密结合，使得编译器与飞桨整体框架更加协调一致。这一改进不仅提升了编译器的性能，还为开发者提供了更加流畅和统一的开发体验。 ### 兼容性升级 - 更新存储读取接口至兼容 Paddle 2.0。 [#55836](https://github.com/PaddlePaddle/Paddle/pull/55836) - 更新 relu6 Op Mapper 的兼容性。 [#55611](https://github.com/PaddlePaddle/Paddle/pull/55611) ### 改造废弃 - 删除旧的 Schedule 形式。 [#55566](https://github.com/PaddlePaddle/Paddle/pull/55566),[#55391](https://github.com/PaddlePaddle/Paddle/pull/55391) - 删除一些过时测试。 [#56245](https://github.com/PaddlePaddle/Paddle/pull/56245),[#57987](https://github.com/PaddlePaddle/Paddle/pull/57987) - 删除不再适用的 remove_nested_block Visitor 工具。 [#56972](https://github.com/PaddlePaddle/Paddle/pull/56972) - 删除其他无用代码。 [#55413](https://github.com/PaddlePaddle/Paddle/pull/55413) ### 新功能 - 增加飞桨端 CINN paddle.framework.core.is_run_with_cinn()运行接口。 [#54355](https://github.com/PaddlePaddle/Paddle/pull/54355) - 增加 CINN 相关算子逻辑，包括各种组合算子拆解逻辑。 [#56072](https://github.com/PaddlePaddle/Paddle/pull/56072),[#58210](https://github.com/PaddlePaddle/Paddle/pull/58210),[#58502](https://github.com/PaddlePaddle/Paddle/pull/58502), [#58591](https://github.com/PaddlePaddle/Paddle/pull/58591), [#58981](https://github.com/PaddlePaddle/Paddle/pull/58981), [#59135](https://github.com/PaddlePaddle/Paddle/pull/59135), [#59274](https://github.com/PaddlePaddle/Paddle/pull/59274), [#59306](https://github.com/PaddlePaddle/Paddle/pull/59306), [#59202](https://github.com/PaddlePaddle/Paddle/pull/59202), [#59176](https://github.com/PaddlePaddle/Paddle/pull/59176), [#59534](https://github.com/PaddlePaddle/Paddle/pull/59534), [#59713](https://github.com/PaddlePaddle/Paddle/pull/59713), [#59798](https://github.com/PaddlePaddle/Paddle/pull/59798)；支持 bf16、amp 等形式[#54399](https://github.com/PaddlePaddle/Paddle/pull/54399), [#54368](https://github.com/PaddlePaddle/Paddle/pull/54368), [#54608](https://github.com/PaddlePaddle/Paddle/pull/54608)；支持算子零维能力[#54892](https://github.com/PaddlePaddle/Paddle/pull/54892), [#54919](https://github.com/PaddlePaddle/Paddle/pull/54919), [#54907](https://github.com/PaddlePaddle/Paddle/pull/54907), [#54966](https://github.com/PaddlePaddle/Paddle/pull/54966) - CINN 和飞桨 PIR、组合算子交界运行方式，使新增 PIR 和 CINN 运行浑然一体。 [#54732](https://github.com/PaddlePaddle/Paddle/pull/54732), [#56074](https://github.com/PaddlePaddle/Paddle/pull/56074), [#58216](https://github.com/PaddlePaddle/Paddle/pull/58216), [#55680](https://github.com/PaddlePaddle/Paddle/pull/55680), [#56302](https://github.com/PaddlePaddle/Paddle/pull/56302), [#59037](https://github.com/PaddlePaddle/Paddle/pull/59037), [#55186](https://github.com/PaddlePaddle/Paddle/pull/55186), [#58641](https://github.com/PaddlePaddle/Paddle/pull/58641) - 对 CINN 变化起到稳定作用的强约束组件。 [#58719](https://github.com/PaddlePaddle/Paddle/pull/58719), [#59309](https://github.com/PaddlePaddle/Paddle/pull/59309), [#58993](https://github.com/PaddlePaddle/Paddle/pull/58993) - Group Schedule 相关的 CINN 架构流程添加。 [#58399](https://github.com/PaddlePaddle/Paddle/pull/58399), [#56444](https://github.com/PaddlePaddle/Paddle/pull/56444) - CINN 架构功能初步增加 CUTLASS、报错处理、NVRTC Cubin Fmad 选项。 [#58079](https://github.com/PaddlePaddle/Paddle/pull/58079), [#57198](https://github.com/PaddlePaddle/Paddle/pull/57198), [#58794](https://github.com/PaddlePaddle/Paddle/pull/58794) - CINN 增加 Python 界面语言。 [#57731](https://github.com/PaddlePaddle/Paddle/pull/57731), [#57515](https://github.com/PaddlePaddle/Paddle/pull/57515), [#57644](https://github.com/PaddlePaddle/Paddle/pull/57644), [#57981](https://github.com/PaddlePaddle/Paddle/pull/57981), [#58009](https://github.com/PaddlePaddle/Paddle/pull/58009) - CINN 增加动态 Shape 功能，涵盖 ASTGen 可以代替 ISL 产生动态 Shape 符号 [#56360](https://github.com/PaddlePaddle/Paddle/pull/56360), [#57207](https://github.com/PaddlePaddle/Paddle/pull/57207), [#57454](https://github.com/PaddlePaddle/Paddle/pull/57454)；增加分桶条件编译功能 [#59165](https://github.com/PaddlePaddle/Paddle/pull/59165)；增加 Schedule、Device、IR 层面支持动态 shape 的功能 [#58988](https://github.com/PaddlePaddle/Paddle/pull/58988), [#59493](https://github.com/PaddlePaddle/Paddle/pull/59493), [#58717](https://github.com/PaddlePaddle/Paddle/pull/58717), [#58602](https://github.com/PaddlePaddle/Paddle/pull/58602), [#59196](https://github.com/PaddlePaddle/Paddle/pull/59196) - CINN Group Schedule 算子 Group 层面做更通用稳定的 Schedule 优化。 [#56122](https://github.com/PaddlePaddle/Paddle/pull/56122), [#57777](https://github.com/PaddlePaddle/Paddle/pull/57777), [#57569](https://github.com/PaddlePaddle/Paddle/pull/57569) ### 功能优化 - 丰富或改善算子功能，包括修理反向、FP16、Infershape、算子单测等各种算子过程的改善。 [#56320](https://github.com/PaddlePaddle/Paddle/pull/56320), [#56845](https://github.com/PaddlePaddle/Paddle/pull/56845), [#54939](https://github.com/PaddlePaddle/Paddle/pull/54939),[#54378](https://github.com/PaddlePaddle/Paddle/pull/54378),[#55321](https://github.com/PaddlePaddle/Paddle/pull/55321),[#55336](https://github.com/PaddlePaddle/Paddle/pull/55336),[#55337](https://github.com/PaddlePaddle/Paddle/pull/55337),[#55442](https://github.com/PaddlePaddle/Paddle/pull/55442),[#55470](https://github.com/PaddlePaddle/Paddle/pull/55470),[#55489](https://github.com/PaddlePaddle/Paddle/pull/55489),[#55510](https://github.com/PaddlePaddle/Paddle/pull/55510),[#55547](https://github.com/PaddlePaddle/Paddle/pull/55547),[#55505](https://github.com/PaddlePaddle/Paddle/pull/55505),[#55563](https://github.com/PaddlePaddle/Paddle/pull/55563),[#54280](https://github.com/PaddlePaddle/Paddle/pull/54280),[#59650](https://github.com/PaddlePaddle/Paddle/pull/59650),[#54862](https://github.com/PaddlePaddle/Paddle/pull/54862),[#55135](https://github.com/PaddlePaddle/Paddle/pull/55135),[#55292](https://github.com/PaddlePaddle/Paddle/pull/55292),[#55333](https://github.com/PaddlePaddle/Paddle/pull/55333),[#55316](https://github.com/PaddlePaddle/Paddle/pull/55316),[#55379](https://github.com/PaddlePaddle/Paddle/pull/55379),[#55326](https://github.com/PaddlePaddle/Paddle/pull/55326) - CINN、飞桨、PIR、组合算子交界运行方式改善，主要包括各种和 PIR 及其执行器接口和 CINN 互相支持。 [#59170](https://github.com/PaddlePaddle/Paddle/pull/59170),[#58766](https://github.com/PaddlePaddle/Paddle/pull/58766),[#59255](https://github.com/PaddlePaddle/Paddle/pull/59255),[#59203](https://github.com/PaddlePaddle/Paddle/pull/59203),[#59024](https://github.com/PaddlePaddle/Paddle/pull/59024),[#57829](https://github.com/PaddlePaddle/Paddle/pull/57829),[#58135](https://github.com/PaddlePaddle/Paddle/pull/58135),[#58193](https://github.com/PaddlePaddle/Paddle/pull/58193),[#58207](https://github.com/PaddlePaddle/Paddle/pull/58207),[#58606](https://github.com/PaddlePaddle/Paddle/pull/58606),[#59437](https://github.com/PaddlePaddle/Paddle/pull/59437),[#59759](https://github.com/PaddlePaddle/Paddle/pull/59759),[#55075](https://github.com/PaddlePaddle/Paddle/pull/55075),[#56805](https://github.com/PaddlePaddle/Paddle/pull/56805),[#57764](https://github.com/PaddlePaddle/Paddle/pull/57764),[#58620](https://github.com/PaddlePaddle/Paddle/pull/58620),[#59769](https://github.com/PaddlePaddle/Paddle/pull/59769),[#58702](https://github.com/PaddlePaddle/Paddle/pull/58702),[#58749](https://github.com/PaddlePaddle/Paddle/pull/58749),[#59025](https://github.com/PaddlePaddle/Paddle/pull/59025),[#58820](https://github.com/PaddlePaddle/Paddle/pull/58820),[#58908](https://github.com/PaddlePaddle/Paddle/pull/58908),[#58169](https://github.com/PaddlePaddle/Paddle/pull/58169) - 对 CINN 改善稳定作用的强约束组件。 [#55090](https://github.com/PaddlePaddle/Paddle/pull/55090),[#55705](https://github.com/PaddlePaddle/Paddle/pull/55705),[#57587](https://github.com/PaddlePaddle/Paddle/pull/57587),[#59501](https://github.com/PaddlePaddle/Paddle/pull/59501) - CINN IR 和相关工具代码改善。 [#55145](https://github.com/PaddlePaddle/Paddle/pull/55145),[#55955](https://github.com/PaddlePaddle/Paddle/pull/55955),[#56307](https://github.com/PaddlePaddle/Paddle/pull/56307),[#55519](https://github.com/PaddlePaddle/Paddle/pull/55519),[#56958](https://github.com/PaddlePaddle/Paddle/pull/56958),[#57019](https://github.com/PaddlePaddle/Paddle/pull/57019),[#57230](https://github.com/PaddlePaddle/Paddle/pull/57230),[#57531](https://github.com/PaddlePaddle/Paddle/pull/57531),[#57532](https://github.com/PaddlePaddle/Paddle/pull/57532),[#57524](https://github.com/PaddlePaddle/Paddle/pull/57524),[#58770](https://github.com/PaddlePaddle/Paddle/pull/58770),[#59337](https://github.com/PaddlePaddle/Paddle/pull/59337),[#59096](https://github.com/PaddlePaddle/Paddle/pull/59096),[#56274](https://github.com/PaddlePaddle/Paddle/pull/56274),[#56350](https://github.com/PaddlePaddle/Paddle/pull/56350),[#57312](https://github.com/PaddlePaddle/Paddle/pull/57312),[#55171](https://github.com/PaddlePaddle/Paddle/pull/55171) - CINN Group Schedule 算子 Group 层面做更通用稳定的 Schedule 优化。 [#54982](https://github.com/PaddlePaddle/Paddle/pull/54982),[#57963](https://github.com/PaddlePaddle/Paddle/pull/57963),[#58220](https://github.com/PaddlePaddle/Paddle/pull/58220),[#55484](https://github.com/PaddlePaddle/Paddle/pull/55484),[#55935](https://github.com/PaddlePaddle/Paddle/pull/55935),[#55590](https://github.com/PaddlePaddle/Paddle/pull/55590),[#56530](https://github.com/PaddlePaddle/Paddle/pull/56530),[#58344](https://github.com/PaddlePaddle/Paddle/pull/58344),[#59810](https://github.com/PaddlePaddle/Paddle/pull/59810) - CINN 架构功能改善，包括并行编译、低层存储分配方式、打印信息、Group 结构、Pass 结构等。[#56282](https://github.com/PaddlePaddle/Paddle/pull/56282), [#59014](https://github.com/PaddlePaddle/Paddle/pull/59014),[#59209](https://github.com/PaddlePaddle/Paddle/pull/59209),[#52660](https://github.com/PaddlePaddle/Paddle/pull/52660),[#54749](https://github.com/PaddlePaddle/Paddle/pull/54749),[#58694](https://github.com/PaddlePaddle/Paddle/pull/58694),[#58940](https://github.com/PaddlePaddle/Paddle/pull/58940),[#59504](https://github.com/PaddlePaddle/Paddle/pull/59504),[#56123](https://github.com/PaddlePaddle/Paddle/pull/56123) - CINN 改善 codegen、jit instruction、dim args、host kernel 等以支持动态 Shape 功能。 [#58825](https://github.com/PaddlePaddle/Paddle/pull/58825),[#59395](https://github.com/PaddlePaddle/Paddle/pull/59395),[#59398](https://github.com/PaddlePaddle/Paddle/pull/59398),[#59540](https://github.com/PaddlePaddle/Paddle/pull/59540),[#59470](https://github.com/PaddlePaddle/Paddle/pull/59470),[#59640](https://github.com/PaddlePaddle/Paddle/pull/59640) - CINN 报错优化。 [#54983](https://github.com/PaddlePaddle/Paddle/pull/54983),[#55544](https://github.com/PaddlePaddle/Paddle/pull/55544) - CINN 其他代码清理改善、包括 CI、文件路径、C++17、Flags、第三方库、Docker 等 [#55018](https://github.com/PaddlePaddle/Paddle/pull/55018),[#55121](https://github.com/PaddlePaddle/Paddle/pull/55121),[#55009](https://github.com/PaddlePaddle/Paddle/pull/55009),[#55888](https://github.com/PaddlePaddle/Paddle/pull/55888),[#56168](https://github.com/PaddlePaddle/Paddle/pull/56168),[#56192](https://github.com/PaddlePaddle/Paddle/pull/56192),[#56896](https://github.com/PaddlePaddle/Paddle/pull/56896),[#53861](https://github.com/PaddlePaddle/Paddle/pull/53861),[#55208](https://github.com/PaddlePaddle/Paddle/pull/55208) ### 性能优化 - 对 vit attention 进行融合。 [#54139](https://github.com/PaddlePaddle/Paddle/pull/54139) - 优化 block reduce。 [#58196](https://github.com/PaddlePaddle/Paddle/pull/58196) ### bug 修复 - 算子相关 bug 修复。 [#56280](https://github.com/PaddlePaddle/Paddle/pull/56280),[#57767](https://github.com/PaddlePaddle/Paddle/pull/57767),[#58406](https://github.com/PaddlePaddle/Paddle/pull/58406),[#54406](https://github.com/PaddlePaddle/Paddle/pull/54406),[#54494](https://github.com/PaddlePaddle/Paddle/pull/54494),[#54751](https://github.com/PaddlePaddle/Paddle/pull/54751),[#55674](https://github.com/PaddlePaddle/Paddle/pull/55674),[#55684](https://github.com/PaddlePaddle/Paddle/pull/55684),[#55683](https://github.com/PaddlePaddle/Paddle/pull/55683),[#57798](https://github.com/PaddlePaddle/Paddle/pull/57798),[#57816](https://github.com/PaddlePaddle/Paddle/pull/57816),[#57687](https://github.com/PaddlePaddle/Paddle/pull/57687),[#56719](https://github.com/PaddlePaddle/Paddle/pull/56719),[#59756](https://github.com/PaddlePaddle/Paddle/pull/59756),[#59770](https://github.com/PaddlePaddle/Paddle/pull/59770),[#58811](https://github.com/PaddlePaddle/Paddle/pull/58811) - 流程架构相关 bug 修复。 [#54899](https://github.com/PaddlePaddle/Paddle/pull/54899),[#59737](https://github.com/PaddlePaddle/Paddle/pull/59737),[#59356](https://github.com/PaddlePaddle/Paddle/pull/59356),[#56105](https://github.com/PaddlePaddle/Paddle/pull/56105),[#56662](https://github.com/PaddlePaddle/Paddle/pull/56662),[#58146](https://github.com/PaddlePaddle/Paddle/pull/58146),[#58910](https://github.com/PaddlePaddle/Paddle/pull/58910),[#58121](https://github.com/PaddlePaddle/Paddle/pull/58121),[#58943](https://github.com/PaddlePaddle/Paddle/pull/58943),[#58886](https://github.com/PaddlePaddle/Paddle/pull/58886),[#59642](https://github.com/PaddlePaddle/Paddle/pull/59642),[#56164](https://github.com/PaddlePaddle/Paddle/pull/56164),[#56338](https://github.com/PaddlePaddle/Paddle/pull/56338),[#56966](https://github.com/PaddlePaddle/Paddle/pull/56966),[#59112](https://github.com/PaddlePaddle/Paddle/pull/59112),[#55820](https://github.com/PaddlePaddle/Paddle/pull/55820),[#56660](https://github.com/PaddlePaddle/Paddle/pull/56660),[#57307](https://github.com/PaddlePaddle/Paddle/pull/57307),[#57530](https://github.com/PaddlePaddle/Paddle/pull/57530),[#58236](https://github.com/PaddlePaddle/Paddle/pull/58236),[#55190](https://github.com/PaddlePaddle/Paddle/pull/55190),[#55043](https://github.com/PaddlePaddle/Paddle/pull/55043),[#55667](https://github.com/PaddlePaddle/Paddle/pull/55667) - 其他 bug 修复。 [#57239](https://github.com/PaddlePaddle/Paddle/pull/57239),[#55530](https://github.com/PaddlePaddle/Paddle/pull/55530),[#56605](https://github.com/PaddlePaddle/Paddle/pull/56605),[#58243](https://github.com/PaddlePaddle/Paddle/pull/58243),[#58197](https://github.com/PaddlePaddle/Paddle/pull/58197),[#58197](https://github.com/PaddlePaddle/Paddle/pull/58197),[#56086](https://github.com/PaddlePaddle/Paddle/pull/56086),[#56065](https://github.com/PaddlePaddle/Paddle/pull/56065),[#58775](https://github.com/PaddlePaddle/Paddle/pull/58775),[#54750](https://github.com/PaddlePaddle/Paddle/pull/54750),[#58595](https://github.com/PaddlePaddle/Paddle/pull/58595),[#58873](https://github.com/PaddlePaddle/Paddle/pull/58873) ### 文档 - 增加 README 文件。 [#58349](https://github.com/PaddlePaddle/Paddle/pull/58349) # 4. 部署方向（Paddle Inference） ## 通用推理优化 本版本升级提升了推理引擎在 GPU 和 CPU 上性能和易用性，降低了用户使用成本和线上推理的应用成本。在 GPU 上支持了高性能的多线程异步执行器，各模型推理性能提升 5%~10%；同时支持新版本 TensorRT 和 BF16 推理能力，TensorRT 推理性能和易用性进一步提升；在 CPU 上，支持最新版本的 OneDNN 高性能推理，在 SwinTransformer、FastRCNN 等系列模型上性能大幅提升。 - matmul 支持 transpose、broadcast 操作。 [#56827](https://github.com/PaddlePaddle/Paddle/pull/56827) - TruncatedNormal and Assign 支持 FP64 数据类型。[#57507](https://github.com/PaddlePaddle/Paddle/pull/57507) - 支持 conv2d 显式量化推理。[#57160](https://github.com/PaddlePaddle/Paddle/pull/57160),[#58015](https://github.com/PaddlePaddle/Paddle/pull/58015) - 新增 conv_fuse_pass，支持 conv + bn 融合，conv2d_fusion 融合重命名为 fused_conv2d_add_act。 [#58724](https://github.com/PaddlePaddle/Paddle/pull/58724),[#55374](https://github.com/PaddlePaddle/Paddle/pull/55374),[#54477](https://github.com/PaddlePaddle/Paddle/pull/54477),[#59431](https://github.com/PaddlePaddle/Paddle/pull/59431) - 混合精度推理支持 OP 白名单。[#56535](https://github.com/PaddlePaddle/Paddle/pull/56535) - 默认开启 OneDNN 优化，支持 SwinTransformer、FastRCNNd 等推理优化。[#58560](https://github.com/PaddlePaddle/Paddle/pull/58560),[#59394](https://github.com/PaddlePaddle/Paddle/pull/59394),[#59421](https://github.com/PaddlePaddle/Paddle/pull/59421),[#58435](https://github.com/PaddlePaddle/Paddle/pull/58435),[#58488](https://github.com/PaddlePaddle/Paddle/pull/58488),[#59259](https://github.com/PaddlePaddle/Paddle/pull/59259),[#56303](https://github.com/PaddlePaddle/Paddle/pull/56303),[#56782](https://github.com/PaddlePaddle/Paddle/pull/56782),[#57598](https://github.com/PaddlePaddle/Paddle/pull/57598),[#58361](https://github.com/PaddlePaddle/Paddle/pull/58361),[#59641](https://github.com/PaddlePaddle/Paddle/pull/59641),[#59527](https://github.com/PaddlePaddle/Paddle/pull/59527),[#59663](https://github.com/PaddlePaddle/Paddle/pull/59663),[#59744](https://github.com/PaddlePaddle/Paddle/pull/59744) - 新增 share_data 支持传入指定数据。[#57933](https://github.com/PaddlePaddle/Paddle/pull/57933) ## 大模型推理优化 实现了生成式大模型的细粒度融合推理优化，该优化方案既保证了高性能的推理能力，又具备良好的可拓展性。用户可以根据需要，灵活运用各种细粒度融合算子和飞桨原生算子，自由组合构建生成式大模型的网络结构，从而实现高效且低成本的推理。此外，我们的方案还支持主流的生成式大模型结构，显著降低了这类模型的推理部署成本，为生成式大模型的高效、低成本落地提供了有力支持。 - 支持 FMHA/MMHA 对 CacheKV 划分 block 调度。[#59462](https://github.com/PaddlePaddle/Paddle/pull/59462) - RoPE 编码融合算子支持输入 sin/cos 值。[#55415](https://github.com/PaddlePaddle/Paddle/pull/55415) - 新增细粒度融合算子支持生成式大模型高性能推理优化，新增 quant_linear、weight_quantize、linear_compress 等算子支持大模型量化推理。[#57852](https://github.com/PaddlePaddle/Paddle/pull/57852),[#55128](https://github.com/PaddlePaddle/Paddle/pull/55128),[#59090](https://github.com/PaddlePaddle/Paddle/pull/59090),[#56706](https://github.com/PaddlePaddle/Paddle/pull/56706),[#59951](https://github.com/PaddlePaddle/Paddle/pull/59951),[#55490](https://github.com/PaddlePaddle/Paddle/pull/55490),[#59291](https://github.com/PaddlePaddle/Paddle/pull/59291),[#59441](https://github.com/PaddlePaddle/Paddle/pull/59441),[#59778](https://github.com/PaddlePaddle/Paddle/pull/59778),[#59651](https://github.com/PaddlePaddle/Paddle/pull/59651)[#55301](https://github.com/PaddlePaddle/Paddle/pull/55301),[#58637](https://github.com/PaddlePaddle/Paddle/pull/58637),[#56673](https://github.com/PaddlePaddle/Paddle/pull/56673),[#56401](https://github.com/PaddlePaddle/Paddle/pull/56401) - 支持变长推理系列 API。[#57948](https://github.com/PaddlePaddle/Paddle/pull/57948) - 支持 GQA 推理。[#58472](https://github.com/PaddlePaddle/Paddle/pull/58472),[#58836](https://github.com/PaddlePaddle/Paddle/pull/58836) - 新增 masked multihead attention 支持高性能 MMHA 推理。[#55344](https://github.com/PaddlePaddle/Paddle/pull/55344),[#56411](https://github.com/PaddlePaddle/Paddle/pull/56411),[#58134](https://github.com/PaddlePaddle/Paddle/pull/58134),[#57936](https://github.com/PaddlePaddle/Paddle/pull/57936) - weight_quantize/weight_only_linear 支持 Volta 架构。[#58082](https://github.com/PaddlePaddle/Paddle/pull/58082) - 新增 weight_only_linear_grad 支持大模型 weight only 量化梯度回传。[#57685](https://github.com/PaddlePaddle/Paddle/pull/57685) - 修复大模型动转静问题，优化静态图卡间通信初始化逻辑。[#56390](https://github.com/PaddlePaddle/Paddle/pull/56390),[#57169](https://github.com/PaddlePaddle/Paddle/pull/57169),[#56688](https://github.com/PaddlePaddle/Paddle/pull/56688),[#56592](https://github.com/PaddlePaddle/Paddle/pull/56592),[#58868](https://github.com/PaddlePaddle/Paddle/pull/58868) - 优化 top_p_sampling 随机数生成逻辑。[#59494](https://github.com/PaddlePaddle/Paddle/pull/59494) ## Paddle-TensorRT 推理优化 - elementwise_add 融合支持 NHWC 格式。 [#56795](https://github.com/PaddlePaddle/Paddle/pull/56795) - conv2d 支持 filter 作为输入。[#55246](https://github.com/PaddlePaddle/Paddle/pull/55246)。 - 支持 BF16、FP64 推理。[#59765](https://github.com/PaddlePaddle/Paddle/pull/59765),[#55520](https://github.com/PaddlePaddle/Paddle/pull/55520) - 新增 MarkTrtEngineOutputs API 支持指定 TensorRT Engine 输出。 [#56858](https://github.com/PaddlePaddle/Paddle/pull/56858),[#56188](https://github.com/PaddlePaddle/Paddle/pull/56188),[#57407](https://github.com/PaddlePaddle/Paddle/pull/57407) - 支持自定义 OP 自动生成 TensorRT Plugin。[#58976](https://github.com/PaddlePaddle/Paddle/pull/58976),[#56037](https://github.com/PaddlePaddle/Paddle/pull/56037) - TensorRT 推理支持指定输入 hook，优化 shape 收集流程。[#59466](https://github.com/PaddlePaddle/Paddle/pull/59466),[#54841](https://github.com/PaddlePaddle/Paddle/pull/54841),[#57498](https://github.com/PaddlePaddle/Paddle/pull/57498),[#54861](https://github.com/PaddlePaddle/Paddle/pull/54861),[#54432](https://github.com/PaddlePaddle/Paddle/pull/54432),[#55503](https://github.com/PaddlePaddle/Paddle/pull/55503) - TensorRT 推理支持保存 Tuning 后的推理模型。[#55893](https://github.com/PaddlePaddle/Paddle/pull/55893),[#56952](https://github.com/PaddlePaddle/Paddle/pull/56952),[#57031](https://github.com/PaddlePaddle/Paddle/pull/57031) - 支持变长 Transformer 模型 PromptTuning。[#57034](https://github.com/PaddlePaddle/Paddle/pull/57034) - 新增 bitwise_and、bitwise_or、bitwise_not、cumsum、einsum、lookup_table、assign、flip、size、scatter、solve、unbind、reduce、argsort 算子支持，优化已有算子支持。[#59214](https://github.com/PaddlePaddle/Paddle/pull/59214),[#59293](https://github.com/PaddlePaddle/Paddle/pull/59293),[#54882](https://github.com/PaddlePaddle/Paddle/pull/54882),[#54097](https://github.com/PaddlePaddle/Paddle/pull/54097),[#54860](https://github.com/PaddlePaddle/Paddle/pull/54860),[#55426](https://github.com/PaddlePaddle/Paddle/pull/55426),[#54372](https://github.com/PaddlePaddle/Paddle/pull/54372),[#55688](https://github.com/PaddlePaddle/Paddle/pull/55688),[#56069](https://github.com/PaddlePaddle/Paddle/pull/56069),[#59563](https://github.com/PaddlePaddle/Paddle/pull/59563),[#59317](https://github.com/PaddlePaddle/Paddle/pull/59317),[#59424](https://github.com/PaddlePaddle/Paddle/pull/59424),[#55476](https://github.com/PaddlePaddle/Paddle/pull/55476),[#56043](https://github.com/PaddlePaddle/Paddle/pull/56043),[#58549](https://github.com/PaddlePaddle/Paddle/pull/58549),[#57326](https://github.com/PaddlePaddle/Paddle/pull/57326),[#59409](https://github.com/PaddlePaddle/Paddle/pull/59409)) - TensorRT 默认开启显存共享。[#59495](https://github.com/PaddlePaddle/Paddle/pull/59495),[#58251](https://github.com/PaddlePaddle/Paddle/pull/58251) - PrelnResidualBiasPluginDynamic 支持 4D 输入。[#56304](https://github.com/PaddlePaddle/Paddle/pull/56304) - 新增 SM80 以下架构 Paddle-TRT 推理对 FlashAttention 的支持。[#56492](https://github.com/PaddlePaddle/Paddle/pull/56492) ## 改造废弃 - OneDNN 中删除 fc_elementwise_add 融合。[#55504](https://github.com/PaddlePaddle/Paddle/pull/55504) - 删除 redunant op。 [#54442](https://github.com/PaddlePaddle/Paddle/pull/54442) ## Bug Fix - 修复 Inference so 链接 flags 冲突问题。[#59755](https://github.com/PaddlePaddle/Paddle/pull/59755) - 修复 constant_folding pass 执行报错。[#55556](https://github.com/PaddlePaddle/Paddle/pull/55556) - 修复 softmax 前向速度问题及反向精度问题。[#56036](https://github.com/PaddlePaddle/Paddle/pull/56036),[#57858](https://github.com/PaddlePaddle/Paddle/pull/57858)[#57538](https://github.com/PaddlePaddle/Paddle/pull/57538) - 修复自定义 OP while 报错及导出问题。[#58898](https://github.com/PaddlePaddle/Paddle/pull/58898),[#59318](https://github.com/PaddlePaddle/Paddle/pull/59318) - 修复 Windows 平台 CUDA 12.0 编译问题。[#59852](https://github.com/PaddlePaddle/Paddle/pull/59852) - 修复 TensorRT 版本大于等于 8.6 时推理部分算子报错问题。[#54379](https://github.com/PaddlePaddle/Paddle/pull/54379),[#54679](https://github.com/PaddlePaddle/Paddle/pull/54679),[#54251](https://github.com/PaddlePaddle/Paddle/pull/54251) - 修复、删除推理融合 Pass。[#54846](https://github.com/PaddlePaddle/Paddle/pull/54846),[#54887](https://github.com/PaddlePaddle/Paddle/pull/54887),[#55573](https://github.com/PaddlePaddle/Paddle/pull/55573),[#56434](https://github.com/PaddlePaddle/Paddle/pull/56434),[#56326](https://github.com/PaddlePaddle/Paddle/pull/56326),[#56753](https://github.com/PaddlePaddle/Paddle/pull/56753),[#57491](https://github.com/PaddlePaddle/Paddle/pull/57491),[#56909](https://github.com/PaddlePaddle/Paddle/pull/56909),[#54536](https://github.com/PaddlePaddle/Paddle/pull/54536),[#55073](https://github.com/PaddlePaddle/Paddle/pull/55073),[#55081](https://github.com/PaddlePaddle/Paddle/pull/55081),[#55240](https://github.com/PaddlePaddle/Paddle/pull/55240),[#56439](https://github.com/PaddlePaddle/Paddle/pull/56439),[#59009](https://github.com/PaddlePaddle/Paddle/pull/59009) - 修复多 Stream 推理上下文切换报错问题。[#57629](https://github.com/PaddlePaddle/Paddle/pull/57629),[#58048](https://github.com/PaddlePaddle/Paddle/pull/58048),[#54994](https://github.com/PaddlePaddle/Paddle/pull/54994) # 5. 硬件适配 ## 硬件适配方案 (Custom Device) 在本次更新中，新增了对分布式高级策略、自定义算子和自定义融合策略的支持。通过升级分布式通信库，新增了对 MP、GroupShared、PP、SP 和 MOE 等多项高级分布式策略的支持。同时支持厂商灵活接入不同颗粒度的 Transformer 算子库并通过融合 Pass 修改计算图进行性能加速。 ### 新功能 - CustomDevice 升级对 Paddle 最新分布式通信库 CommContext 的支持，并新增了多种高级分布式策略 GroupShared 和 MOE 等策略。[#56301](https://github.com/PaddlePaddle/Paddle/pull/56301),[#54671](https://github.com/PaddlePaddle/Paddle/pull/54671),[#57957](https://github.com/PaddlePaddle/Paddle/pull/57957),[#56669](https://github.com/PaddlePaddle/Paddle/pull/56669),[#54384](https://github.com/PaddlePaddle/Paddle/pull/54384),[#54572](https://github.com/PaddlePaddle/Paddle/pull/54572),[#54573](https://github.com/PaddlePaddle/Paddle/pull/54573),[#54676](https://github.com/PaddlePaddle/Paddle/pull/54676) - 新增 CustomDevice 对 CustomOP 的支持，并可注册 Paddle PHI 算子库中尚未定义的算子,同时新增 CustomDevice 通过 CAPI 支持 CustomOP。[#57038](https://github.com/PaddlePaddle/Paddle/pull/57038),[#55532](https://github.com/PaddlePaddle/Paddle/pull/55532),[#56755](https://github.com/PaddlePaddle/Paddle/pull/56755),[#55532](https://github.com/PaddlePaddle/Paddle/pull/55532),[#55533](https://github.com/PaddlePaddle/Paddle/pull/55533),[#55659](https://github.com/PaddlePaddle/Paddle/pull/55659) - 新增 CustomDevice 对 CustomPass 功能的功能，支持通过 Python API 修改计算图 IR。[#55511](https://github.com/PaddlePaddle/Paddle/pull/55511),[#55728](https://github.com/PaddlePaddle/Paddle/pull/55728) - 新增 CustomDevice 对 Paddle run_check 健康功能检查的支持。[#56318](https://github.com/PaddlePaddle/Paddle/pull/56318) - 新增 CustomDevice 对 StreamSafeAllocator 的支持。[#55393](https://github.com/PaddlePaddle/Paddle/pull/55393),[#56380](https://github.com/PaddlePaddle/Paddle/pull/56380),[#56536](https://github.com/PaddlePaddle/Paddle/pull/56536),[#58035](https://github.com/PaddlePaddle/Paddle/pull/58035) - 新增 CustomDevice 对 DataTransform 的支持。[#56627](https://github.com/PaddlePaddle/Paddle/pull/56627) ### 功能优化 - 新增 CustomDevice，支持飞桨更多的接口，包括 Variable.set_value,adamw,share_external_data,mp_allreduce_sum,tensor.numpy,get_paddle_place, GeneratorState。[#55272](https://github.com/PaddlePaddle/Paddle/pull/55272), [#56386](https://github.com/PaddlePaddle/Paddle/pull/56386), [#57253](https://github.com/PaddlePaddle/Paddle/pull/57253), [#56927](https://github.com/PaddlePaddle/Paddle/pull/56927),[#56189](https://github.com/PaddlePaddle/Paddle/pull/56189),[#55225](https://github.com/PaddlePaddle/Paddle/pull/55225),[#55247](https://github.com/PaddlePaddle/Paddle/pull/55247) - 修改 CustomDevice 动态库加载方式，从 RTLD_NOW 改为 RTLD_LAZY，方便后续检查 Custom Device 相关软件栈版本的兼容性。 [#57544](https://github.com/PaddlePaddle/Paddle/pull/57544) - 新增 CustomDevice 在混合精度训练下对 FP16 算子的检测功能。[#56053](https://github.com/PaddlePaddle/Paddle/pull/56053),[#56176](https://github.com/PaddlePaddle/Paddle/pull/56176) ### Bug Fix - 修复 CustomDevice 对分布式通信库支持上的一些问题。[#55293](https://github.com/PaddlePaddle/Paddle/pull/55293),[#58038](https://github.com/PaddlePaddle/Paddle/pull/58038),[#59800](https://github.com/PaddlePaddle/Paddle/pull/59800) - 修复 CustomDevice 在部分算子上的问题，包括 c_softmax_with_cross_entropy,data loader,SplitDenseTensor,grad accumulation,atan2 grad。[#56486](https://github.com/PaddlePaddle/Paddle/pull/56486),[#55541](https://github.com/PaddlePaddle/Paddle/pull/55541),[#55615](https://github.com/PaddlePaddle/Paddle/pull/55615),[#56052](https://github.com/PaddlePaddle/Paddle/pull/56052),[#56067](https://github.com/PaddlePaddle/Paddle/pull/56067) - 修复 CustomDevice 中设备管理的一些问题，包括设备异常 ([#56556](https://github.com/PaddlePaddle/Paddle/pull/56556),[#58639](https://github.com/PaddlePaddle/Paddle/pull/58639),[#55173](https://github.com/PaddlePaddle/Paddle/pull/55173)), 异常事件([#56745](https://github.com/PaddlePaddle/Paddle/pull/56745),[#58059](https://github.com/PaddlePaddle/Paddle/pull/58059)), 显存异常([#56977](https://github.com/PaddlePaddle/Paddle/pull/56977),[#59247](https://github.com/PaddlePaddle/Paddle/pull/59247),[#54606](https://github.com/PaddlePaddle/Paddle/pull/54606)), 设备初始化 ([#57099](https://github.com/PaddlePaddle/Paddle/pull/57099),[#57994](https://github.com/PaddlePaddle/Paddle/pull/57994))，设备释放([#54932](https://github.com/PaddlePaddle/Paddle/pull/54932),[#55351](https://github.com/PaddlePaddle/Paddle/pull/55351),[#55783](https://github.com/PaddlePaddle/Paddle/pull/55783))，和设备资源池等。([#55229](https://github.com/PaddlePaddle/Paddle/pull/55229),[#56580](https://github.com/PaddlePaddle/Paddle/pull/56580)) - 修复 CustomDevice 编译相关问题。[#56760](https://github.com/PaddlePaddle/Paddle/pull/56760),[#56766](https://github.com/PaddlePaddle/Paddle/pull/56766) ## 昆仑 XPU ### 新功能 - 新增 XPTI (XPU Profiling Tool Interface) 支持运行时性能数据的采集和分析功能。[#54685](https://github.com/PaddlePaddle/Paddle/pull/54685),[#54690](https://github.com/PaddlePaddle/Paddle/pull/54690),[#54800](https://github.com/PaddlePaddle/Paddle/pull/54800) - 完成对 Paddle 最新分布式通信库 CommContext 的支持。[#59418](https://github.com/PaddlePaddle/Paddle/pull/59418) - 新增 XPU 融合算子包括 fast_where。[#55628](https://github.com/PaddlePaddle/Paddle/pull/55628) - 新增 XPU Plugin 功能支持，方便用户可通过 XTDK 编程方式开发 XPU 自定义算子。[#55101](https://github.com/PaddlePaddle/Paddle/pull/55101),[#59326](https://github.com/PaddlePaddle/Paddle/pull/59326) - 新增 XPU 对 AutoGrowthAllocator 的支持。[#54121](https://github.com/PaddlePaddle/Paddle/pull/54121) - 新增昆仑 3 的算子支持列表。[#57683](https://github.com/PaddlePaddle/Paddle/pull/57683) ### 功能优化 - 对 XPU Inference API 进行升级。[#54342](https://github.com/PaddlePaddle/Paddle/pull/54342) - 优化部分 XPU 算子性能和新增部分 XPU 算子对 bf16 的的支持，包括 unique/index_put,squeeze/unsqueeze kernels,swish/swish_grad,scatter_nd_add_grad/slice,rsqrt/bitwise_or/arange_tensor,where,collective 算子等。[#56582](https://github.com/PaddlePaddle/Paddle/pull/56582),[#58161](https://github.com/PaddlePaddle/Paddle/pull/58161),[#58440](https://github.com/PaddlePaddle/Paddle/pull/58440),[#58580](https://github.com/PaddlePaddle/Paddle/pull/58580),[#58950](https://github.com/PaddlePaddle/Paddle/pull/58950),[#58616](https://github.com/PaddlePaddle/Paddle/pull/58616),[#59273](https://github.com/PaddlePaddle/Paddle/pull/59273) - 优化 XPU 内存管理，避免内存泄漏。[#59334](https://github.com/PaddlePaddle/Paddle/pull/59334),[#54847](https://github.com/PaddlePaddle/Paddle/pull/54847) - 支持 INT8 推理。[#57258](https://github.com/PaddlePaddle/Paddle/pull/57258) - 新增 FP16 系列推理算子支持。[#55642](https://github.com/PaddlePaddle/Paddle/pull/55642),[#54410](https://github.com/PaddlePaddle/Paddle/pull/54410) - 支持 share_external_memory 接口传入输入输出。[#55170](https://github.com/PaddlePaddle/Paddle/pull/55170) - 开源量化模型 XPU 推理支持。[#58568](https://github.com/PaddlePaddle/Paddle/pull/58568) - 新增 context_gm_size 配置代替在 Pass 中分配 global memory。[#54674](https://github.com/PaddlePaddle/Paddle/pull/54674) - 新增 embedding、fast_gather_nd plugin。[#56488](https://github.com/PaddlePaddle/Paddle/pull/56488),[#56103](https://github.com/PaddlePaddle/Paddle/pull/56103) - 支持 fast_layternorm + leaky_relu 融合。[#57113](https://github.com/PaddlePaddle/Paddle/pull/57113) - KL1 和 KL2 精度下 elementwise_min/max/floordiv/where 推理支持。[#58422](https://github.com/PaddlePaddle/Paddle/pull/58422) - 支持 fc 和 conv2d 算子 autotune 配置。[#58801](https://github.com/PaddlePaddle/Paddle/pull/58801) - 支持 conv 和 fc 动态量化。[#59307](https://github.com/PaddlePaddle/Paddle/pull/59307) - fc + act 融合支持 sigmoid, swish and relu6。[#54486](https://github.com/PaddlePaddle/Paddle/pull/54486) - elementwise_sub/elementwise_div 支持 int 数据类型。[#55920](https://github.com/PaddlePaddle/Paddle/pull/55920) ### Bug Fix - 修复 XPU 通信库问题和部分算子问题包括 rnn、layer_norm_grad、yolo_box。 [#55475](https://github.com/PaddlePaddle/Paddle/pull/55475),[#55515](https://github.com/PaddlePaddle/Paddle/pull/55515) [#55656](https://github.com/PaddlePaddle/Paddle/pull/55656),[#54669](https://github.com/PaddlePaddle/Paddle/pull/54669),[#55310](https://github.com/PaddlePaddle/Paddle/pull/55310) ## 海光 DCU ### Bug Fix - 修复海光 DCU 部分算子问题，包括 rnn,concat/split,fft 等。[#59402](https://github.com/PaddlePaddle/Paddle/pull/59402),[#55821](https://github.com/PaddlePaddle/Paddle/pull/55821),[#56340](https://github.com/PaddlePaddle/Paddle/pull/56340) - 修复海光 DCU 通信库相关问题。[#57110](https://github.com/PaddlePaddle/Paddle/pull/57110) - 修复海光 DCU 编译相关问题。[#59775](https://github.com/PaddlePaddle/Paddle/pull/59775),[#55507](https://github.com/PaddlePaddle/Paddle/pull/55507),[#55612](https://github.com/PaddlePaddle/Paddle/pull/55612),[#54952](https://github.com/PaddlePaddle/Paddle/pull/54952),[#55076](https://github.com/PaddlePaddle/Paddle/pull/55076),[#56079](https://github.com/PaddlePaddle/Paddle/pull/56079),[#54874](https://github.com/PaddlePaddle/Paddle/pull/54874) - 修复海光 DCU 对 BF16 数据类型的支持问题。[#56517](https://github.com/PaddlePaddle/Paddle/pull/56517) # 6. 环境适配 采用模块化编译的方式优化了 CMake 代码的逻辑，提升了飞桨全量编译和增量编译的效率，提升了 RD 本地开发效率，同时支持了 Python3.12，CUDA12，Hopper 架构编译，并引入 Clang 等工具全面优化了代码格式。此外，将 C++单测从链接静态库的方式转变为链接动态库，减小编译体积。这些改进措施为用户提供更加流畅、高效地安装和开发体验。 - CMake 代码优化：分模块和目录编译成独立的静态库，并减少编译依赖，提升增量编译效率。[#59095](https://github.com/PaddlePaddle/Paddle/pull/59095), [#58960](https://github.com/PaddlePaddle/Paddle/pull/58960),[#56591](https://github.com/PaddlePaddle/Paddle/pull/56591),[#58484](https://github.com/PaddlePaddle/Paddle/pull/58484) - CMake 编译分层：将公共组件拆分到公有 common 库，自下而上实现飞桨架构的编译分层，提高编译效率。[#56442](https://github.com/PaddlePaddle/Paddle/pull/56442),[#54729](https://github.com/PaddlePaddle/Paddle/pull/54729),[#55733](https://github.com/PaddlePaddle/Paddle/pull/55733),[#56352](https://github.com/PaddlePaddle/Paddle/pull/56352),[#55109](https://github.com/PaddlePaddle/Paddle/pull/55109),[#54992](https://github.com/PaddlePaddle/Paddle/pull/54992),[#57698](https://github.com/PaddlePaddle/Paddle/pull/57698),[#55147](https://github.com/PaddlePaddle/Paddle/pull/55147),[#55113](https://github.com/PaddlePaddle/Paddle/pull/55113),[#56691](https://github.com/PaddlePaddle/Paddle/pull/56691),[#58618](https://github.com/PaddlePaddle/Paddle/pull/58618),[#58899](https://github.com/PaddlePaddle/Paddle/pull/58899),[#59140](https://github.com/PaddlePaddle/Paddle/pull/59140),[#59129](https://github.com/PaddlePaddle/Paddle/pull/59129),[#59222](https://github.com/PaddlePaddle/Paddle/pull/59222),[#59105](https://github.com/PaddlePaddle/Paddle/pull/59105),[#59711](https://github.com/PaddlePaddle/Paddle/pull/59711) - 第三方库离线编译：将第三方依赖库离线编译，CI/CE 系统无需每次编译重复下载第三方库，提升 CI/CE 系统运行效率。[#54344](https://github.com/PaddlePaddle/Paddle/pull/54344),[#54370](https://github.com/PaddlePaddle/Paddle/pull/54370),[#54466](https://github.com/PaddlePaddle/Paddle/pull/54466),[#54438](https://github.com/PaddlePaddle/Paddle/pull/54438),[#54388](https://github.com/PaddlePaddle/Paddle/pull/54388),[#54436](https://github.com/PaddlePaddle/Paddle/pull/54436),[#54392](https://github.com/PaddlePaddle/Paddle/pull/54392),[#54646](https://github.com/PaddlePaddle/Paddle/pull/54646),[#54380](https://github.com/PaddlePaddle/Paddle/pull/54380),[#55501](https://github.com/PaddlePaddle/Paddle/pull/55501),[#55136](https://github.com/PaddlePaddle/Paddle/pull/55136),[#54451](https://github.com/PaddlePaddle/Paddle/pull/54451),[#55631](https://github.com/PaddlePaddle/Paddle/pull/55631),[#55549](https://github.com/PaddlePaddle/Paddle/pull/55549),[#56165](https://github.com/PaddlePaddle/Paddle/pull/56165),[#54391](https://github.com/PaddlePaddle/Paddle/pull/54391),[#54614](https://github.com/PaddlePaddle/Paddle/pull/54614),[#54522](https://github.com/PaddlePaddle/Paddle/pull/54522),[#54764](https://github.com/PaddlePaddle/Paddle/pull/54764),[#54400](https://github.com/PaddlePaddle/Paddle/pull/54400),[#54322](https://github.com/PaddlePaddle/Paddle/pull/54322) - 飞桨支持 Python 3.12。[#59396](https://github.com/PaddlePaddle/Paddle/pull/59396),[#58069](https://github.com/PaddlePaddle/Paddle/pull/58069) - 使用 Clang 等工具对于源代码进行优化，提升代码质量。[#59626](https://github.com/PaddlePaddle/Paddle/pull/59626),[#55895](https://github.com/PaddlePaddle/Paddle/pull/55895),[#56632](https://github.com/PaddlePaddle/Paddle/pull/56632),[#54449](https://github.com/PaddlePaddle/Paddle/pull/54449),[#54523](https://github.com/PaddlePaddle/Paddle/pull/54523),[#54796](https://github.com/PaddlePaddle/Paddle/pull/54796),[#55847](https://github.com/PaddlePaddle/Paddle/pull/55847),[#55807](https://github.com/PaddlePaddle/Paddle/pull/55807),[#56261](https://github.com/PaddlePaddle/Paddle/pull/56261),[#57522](https://github.com/PaddlePaddle/Paddle/pull/57522),[#57868](https://github.com/PaddlePaddle/Paddle/pull/57868),[#57809](https://github.com/PaddlePaddle/Paddle/pull/57809),[#55658](https://github.com/PaddlePaddle/Paddle/pull/55658),[#58285](https://github.com/PaddlePaddle/Paddle/pull/58285),[#55491](https://github.com/PaddlePaddle/Paddle/pull/55491),[#55506](https://github.com/PaddlePaddle/Paddle/pull/55506),[#55279](https://github.com/PaddlePaddle/Paddle/pull/55279),[#55741](https://github.com/PaddlePaddle/Paddle/pull/55741),[#55894](https://github.com/PaddlePaddle/Paddle/pull/55894),[#55704](https://github.com/PaddlePaddle/Paddle/pull/55704),[#55800](https://github.com/PaddlePaddle/Paddle/pull/55800),[#55799](https://github.com/PaddlePaddle/Paddle/pull/55799),[#55983](https://github.com/PaddlePaddle/Paddle/pull/55983),[#55954](https://github.com/PaddlePaddle/Paddle/pull/55954),[#55764](https://github.com/PaddlePaddle/Paddle/pull/55764),[#56246](https://github.com/PaddlePaddle/Paddle/pull/56246),[#56219](https://github.com/PaddlePaddle/Paddle/pull/56219),[#56217](https://github.com/PaddlePaddle/Paddle/pull/56217),[#56216](https://github.com/PaddlePaddle/Paddle/pull/56216),[#56208](https://github.com/PaddlePaddle/Paddle/pull/56208),[#56134](https://github.com/PaddlePaddle/Paddle/pull/56134),[#56253](https://github.com/PaddlePaddle/Paddle/pull/56253),[#56255](https://github.com/PaddlePaddle/Paddle/pull/56255),[#56693](https://github.com/PaddlePaddle/Paddle/pull/56693),[#56692](https://github.com/PaddlePaddle/Paddle/pull/56692),[#56637](https://github.com/PaddlePaddle/Paddle/pull/56637),[#56636](https://github.com/PaddlePaddle/Paddle/pull/56636),[#56647](https://github.com/PaddlePaddle/Paddle/pull/56647),[#56218](https://github.com/PaddlePaddle/Paddle/pull/56218),[#56640](https://github.com/PaddlePaddle/Paddle/pull/56640),[#56635](https://github.com/PaddlePaddle/Paddle/pull/56635),[#55675](https://github.com/PaddlePaddle/Paddle/pull/55675),[#56601](https://github.com/PaddlePaddle/Paddle/pull/56601),[#56485](https://github.com/PaddlePaddle/Paddle/pull/56485),[#56648](https://github.com/PaddlePaddle/Paddle/pull/56648),[#56747](https://github.com/PaddlePaddle/Paddle/pull/56747),[#56676](https://github.com/PaddlePaddle/Paddle/pull/56676),[#56649](https://github.com/PaddlePaddle/Paddle/pull/56649),[#56895](https://github.com/PaddlePaddle/Paddle/pull/56895),[#56994](https://github.com/PaddlePaddle/Paddle/pull/56994),[#56904](https://github.com/PaddlePaddle/Paddle/pull/56904),[#56744](https://github.com/PaddlePaddle/Paddle/pull/56744),[#56954](https://github.com/PaddlePaddle/Paddle/pull/56954),[#57114](https://github.com/PaddlePaddle/Paddle/pull/57114),[#57343](https://github.com/PaddlePaddle/Paddle/pull/57343),[#57483](https://github.com/PaddlePaddle/Paddle/pull/57483),[#57871](https://github.com/PaddlePaddle/Paddle/pull/57871),[#57861](https://github.com/PaddlePaddle/Paddle/pull/57861),[#58028](https://github.com/PaddlePaddle/Paddle/pull/58028),[#57627](https://github.com/PaddlePaddle/Paddle/pull/57627),[#59072](https://github.com/PaddlePaddle/Paddle/pull/59072) - C++从链接静态库转变为链接动态库，减小编译体积，提升编译效率。[#59477](https://github.com/PaddlePaddle/Paddle/pull/59477),[#56630](https://github.com/PaddlePaddle/Paddle/pull/56630),[#57789](https://github.com/PaddlePaddle/Paddle/pull/57789),[#54257](https://github.com/PaddlePaddle/Paddle/pull/54257),[#59620](https://github.com/PaddlePaddle/Paddle/pull/59620),[#59384](https://github.com/PaddlePaddle/Paddle/pull/59384),[#59619](https://github.com/PaddlePaddle/Paddle/pull/59619),[#58583](https://github.com/PaddlePaddle/Paddle/pull/58583),[#58821](https://github.com/PaddlePaddle/Paddle/pull/58821),[#58710](https://github.com/PaddlePaddle/Paddle/pull/58710)，[#58619](https://github.com/PaddlePaddle/Paddle/pull/58619) - 修复源代码编译相关的问题，提升编译安装效率。[#56617](https://github.com/PaddlePaddle/Paddle/pull/56617),[#58195](https://github.com/PaddlePaddle/Paddle/pull/58195),[#56136](https://github.com/PaddlePaddle/Paddle/pull/56136),[#54540](https://github.com/PaddlePaddle/Paddle/pull/54540),[#57172](https://github.com/PaddlePaddle/Paddle/pull/57172),[#54429](https://github.com/PaddlePaddle/Paddle/pull/54429),[#55603](https://github.com/PaddlePaddle/Paddle/pull/55603),[#54807](https://github.com/PaddlePaddle/Paddle/pull/54807),[#56102](https://github.com/PaddlePaddle/Paddle/pull/56102),[#56829](https://github.com/PaddlePaddle/Paddle/pull/56829),[#56951](https://github.com/PaddlePaddle/Paddle/pull/56951),[#56555](https://github.com/PaddlePaddle/Paddle/pull/56555),[#57781](https://github.com/PaddlePaddle/Paddle/pull/57781),[#57836](https://github.com/PaddlePaddle/Paddle/pull/57836),[#58807](https://github.com/PaddlePaddle/Paddle/pull/58807),[#54535](https://github.com/PaddlePaddle/Paddle/pull/54535),[#54946](https://github.com/PaddlePaddle/Paddle/pull/54946),[#54437](https://github.com/PaddlePaddle/Paddle/pull/54437),[#54411](https://github.com/PaddlePaddle/Paddle/pull/54411),[#54411](https://github.com/PaddlePaddle/Paddle/pull/54411),[#54391](https://github.com/PaddlePaddle/Paddle/pull/54391),[#54466](https://github.com/PaddlePaddle/Paddle/pull/54466),[#54480](https://github.com/PaddlePaddle/Paddle/pull/54480),[#54480](https://github.com/PaddlePaddle/Paddle/pull/54480),[#54724](https://github.com/PaddlePaddle/Paddle/pull/54724),[#59193](https://github.com/PaddlePaddle/Paddle/pull/59193),[#54735](https://github.com/PaddlePaddle/Paddle/pull/54735),[#54812](https://github.com/PaddlePaddle/Paddle/pull/54812),[#56430](https://github.com/PaddlePaddle/Paddle/pull/56430),[#56655](https://github.com/PaddlePaddle/Paddle/pull/56655),[#56684](https://github.com/PaddlePaddle/Paddle/pull/56684),[#56774](https://github.com/PaddlePaddle/Paddle/pull/56774),[#56936](https://github.com/PaddlePaddle/Paddle/pull/56936),[#56949](https://github.com/PaddlePaddle/Paddle/pull/56949),[#56974](https://github.com/PaddlePaddle/Paddle/pull/56974),[#57171](https://github.com/PaddlePaddle/Paddle/pull/57171),[#57712](https://github.com/PaddlePaddle/Paddle/pull/57712),[#56617](https://github.com/PaddlePaddle/Paddle/pull/56617),[#58181](https://github.com/PaddlePaddle/Paddle/pull/58181),[#58253](https://github.com/PaddlePaddle/Paddle/pull/58253),[#58268](https://github.com/PaddlePaddle/Paddle/pull/58268),[#59051](https://github.com/PaddlePaddle/Paddle/pull/59051),[#59048](https://github.com/PaddlePaddle/Paddle/pull/59048),[#59081](https://github.com/PaddlePaddle/Paddle/pull/59081),[#59076](https://github.com/PaddlePaddle/Paddle/pull/59076),[#59155](https://github.com/PaddlePaddle/Paddle/pull/59155),[#59253](https://github.com/PaddlePaddle/Paddle/pull/59253),[#59347](https://github.com/PaddlePaddle/Paddle/pull/59347),[#58957](https://github.com/PaddlePaddle/Paddle/pull/58957),[#59443](https://github.com/PaddlePaddle/Paddle/pull/59443),[#58998](https://github.com/PaddlePaddle/Paddle/pull/58998),[#57574](https://github.com/PaddlePaddle/Paddle/pull/57574),[#55889](https://github.com/PaddlePaddle/Paddle/pull/55889),[#59078](https://github.com/PaddlePaddle/Paddle/pull/59078),[#55762](https://github.com/PaddlePaddle/Paddle/pull/55762),[#56252](https://github.com/PaddlePaddle/Paddle/pull/56252),[#56715](https://github.com/PaddlePaddle/Paddle/pull/56715),[#54905](https://github.com/PaddlePaddle/Paddle/pull/54905),[#56978](https://github.com/PaddlePaddle/Paddle/pull/56978),[#57032](https://github.com/PaddlePaddle/Paddle/pull/57032),[#57179](https://github.com/PaddlePaddle/Paddle/pull/57179),[#57179](https://github.com/PaddlePaddle/Paddle/pull/57179),[#58996](https://github.com/PaddlePaddle/Paddle/pull/58996),[#59915](https://github.com/PaddlePaddle/Paddle/pull/59915),[#54883](https://github.com/PaddlePaddle/Paddle/pull/54883),[#56746](https://github.com/PaddlePaddle/Paddle/pull/56746),[#57674](https://github.com/PaddlePaddle/Paddle/pull/57674),[#60117](https://github.com/PaddlePaddle/Paddle/pull/60117),[#55627](https://github.com/PaddlePaddle/Paddle/pull/55627),[#54568](https://github.com/PaddlePaddle/Paddle/pull/54568),[#54450](https://github.com/PaddlePaddle/Paddle/pull/54450),[#54513](https://github.com/PaddlePaddle/Paddle/pull/54513),[#54615](https://github.com/PaddlePaddle/Paddle/pull/54615),[#54913](https://github.com/PaddlePaddle/Paddle/pull/54913),[#54916](https://github.com/PaddlePaddle/Paddle/pull/54916),[#55148](https://github.com/PaddlePaddle/Paddle/pull/55148),[#55125](https://github.com/PaddlePaddle/Paddle/pull/55125),[#55479](https://github.com/PaddlePaddle/Paddle/pull/55479),[#55723](https://github.com/PaddlePaddle/Paddle/pull/55723),[#55831](https://github.com/PaddlePaddle/Paddle/pull/55831),[#55904](https://github.com/PaddlePaddle/Paddle/pull/55904),[#56085](https://github.com/PaddlePaddle/Paddle/pull/56085),[#56259](https://github.com/PaddlePaddle/Paddle/pull/56259),[#56366](https://github.com/PaddlePaddle/Paddle/pull/56366),[#56366](https://github.com/PaddlePaddle/Paddle/pull/56366),[#56546](https://github.com/PaddlePaddle/Paddle/pull/56546),[#56679](https://github.com/PaddlePaddle/Paddle/pull/56679),[#57222](https://github.com/PaddlePaddle/Paddle/pull/57222),[#57387](https://github.com/PaddlePaddle/Paddle/pull/57387),[#57993](https://github.com/PaddlePaddle/Paddle/pull/57993),[#59556](https://github.com/PaddlePaddle/Paddle/pull/59556),[#57931](https://github.com/PaddlePaddle/Paddle/pull/57931),[#58112](https://github.com/PaddlePaddle/Paddle/pull/58112),[#54228](https://github.com/PaddlePaddle/Paddle/pull/54228),[#56913](https://github.com/PaddlePaddle/Paddle/pull/56913),[#56993](https://github.com/PaddlePaddle/Paddle/pull/56993),[#55042](https://github.com/PaddlePaddle/Paddle/pull/55042),[#55305](https://github.com/PaddlePaddle/Paddle/pull/55305),[#55286](https://github.com/PaddlePaddle/Paddle/pull/55286),[#56634](https://github.com/PaddlePaddle/Paddle/pull/56634),[#57778](https://github.com/PaddlePaddle/Paddle/pull/57778),[#58374](https://github.com/PaddlePaddle/Paddle/pull/58374),[#58640](https://github.com/PaddlePaddle/Paddle/pull/58640),[#58822](https://github.com/PaddlePaddle/Paddle/pull/58822),[#59055](https://github.com/PaddlePaddle/Paddle/pull/59055),[#59303](https://github.com/PaddlePaddle/Paddle/pull/59303),[#59487](https://github.com/PaddlePaddle/Paddle/pull/59487),[#58400](https://github.com/PaddlePaddle/Paddle/pull/58400),[#59283](https://github.com/PaddlePaddle/Paddle/pull/59283),[#54791](https://github.com/PaddlePaddle/Paddle/pull/54791),[#59134](https://github.com/PaddlePaddle/Paddle/pull/59134),[#56206](https://github.com/PaddlePaddle/Paddle/pull/56206),[#56199](https://github.com/PaddlePaddle/Paddle/pull/56199),[#56670](https://github.com/PaddlePaddle/Paddle/pull/56670),[#58923](https://github.com/PaddlePaddle/Paddle/pull/58923) - 修复 Paddle ARM 编译相关问题。[#55416](https://github.com/PaddlePaddle/Paddle/pull/55416),[#55548](https://github.com/PaddlePaddle/Paddle/pull/55548) # Thanks to Our Contributors Azure-Tang, zhaoyinglia, From00, JZ-LIANG, xysheng-baidu, SylarTiaNII, kuizhiqing, zhiqiu, FeixLiu, liuzhenhai93, GhostScreaming, pangengzheng, xiaoyewww, wanghuancoder, ForFishes, hitywt, danleifeng, tianshuo78520a, ykkk2333, houj04, lj970926, XiaociZhang, HarperCy, cqulilujia, runzhech, RuohengMa, Caozhou1995, kangguangli, heavyrain-lzy, zyfncg, SigureMo, YuanRisheng, lchdl, LiYuRio, AndSonder, Wennie396, zhangbo9674, liudongxue01, risemeup1, phlrain, winter-wang, yuanlehome, NALLEIN, Liujie0926, yuguo-Jack, gitliuyf, zh794390558, Aurelius84, 6clc, GGBond8488, xiaoguoguo626807, Wong4j, iosmers, xiaoxiaohehe001, LielinJiang, carryyu, Difers, yangxiaoyu14, xuxinyi389, cxxly, gongshaotian, jjyaoao, lijialin03, lxd-cumt, cyber-pioneer, HydrogenSulfate, MayYouBeProsperous, Charles-hit, Patrick-Star125, ScottWong98, huangjiyi, DrRyanHuang, jinyouzhi, BeingGod, Wanglongzhi2001, yangguohao, zyt1024, longranger2, 2742195759, megemini, thisjiang, kevincheng2, zhoutianzi666, Wangzheee, ming1753, tianhaodongbd, freeliuzc, zhenyun-li, MARD1NO, RichardWooSJTU, eee4017, leo0519, csy0225, wwbitejotunn, bukejiyu, jiweibo, iamsonderr, ckl117, ronny1996, zhanglirong1999, LLee233, ZHUI, wangxn12138, zhwesky2010, Courtesy-Xs, zoooo0820, llyyxx0413, Asthestarsfalll, zxcd, pkuzyc, idontkonwher, sneaxiy, hong19860320, ZibinGuo, leolishaohao, MuShangCC, zhupengyang, shentanyue, Travis-Lee, wz1qqx, frank-oops, newway, QingshuChen, zhangyk0314, HandSomeLEEw, Shixiaowei02, zhangyuqin1998, Xing-lil, zhhsplendid, jiahy0825, xinyu-intel, MarioLulab, 0x45f, Tom-Zheng, xingmingyyj, zhangbopd, gouzil, zeroRains, BiynXu, WintersMontagne10335, wuhuachaocoding, GreatV, chenwhql, deepllz, parap1uie-s, ozogxyz, FisherWY, changeyoung98, zhiboniu, YangQun1 dynamicheart, Xreki, liugddx, Lylinnnnn, YSF-A, zzjjay, YanhuiDua, lishicheng1996, USTCKAY, abenmao, cocoshe, HermitSun, ccsuzzh, sanbuphy, enkilee, RedContritio, Liyulingyue, zrr1999, chen2016013, Galaxy1458, chalsliu, mrcangye, XieYunshen, zhiheng-liu, haohongxiang, ZzSean, JamesLim-sy, yuehuayingxueluo, niuliling123, umiswing, sijunhe, littsk, SecretXV, zhurou603, zhangjun, caizejun, yangjianfengo1, vivienfanghuagood, Xinyu302, lizexu123, yghstill, Li-fAngyU, VigiZhang, co63oc, dhanush-2501, ooooo-create, PommesPeter, zeus2x7, akshatvishu, jzhang533, Sekiro-x, gumblex, BernieHuang2008, YibinLiu666, qiuwenbogdut, XavierZXY, MqLeet, zhangting2020, mingxu1067, Ainavo, SSKlearns, yuchen202, silverling, zade23, wenxiaohahaha, NKNaN, Tsaiyue, fsczz, Tomoko-hjf, rhmaaa, zbt78, Hhankyangg, wangzhen38, zhengqiwen1997, engineer1109, onepick, qili93, Rane2021, nemonameless, DesmonDay, RachelXu7, ceci3, lyuwenyu, liuruyan, LokeZhou, shiyutang, lanxianghit, feifei-111, Sahala08, sunzhongkai588, Kaedeharai, Candy2Tang, liyongchao911, whisky-12, InsaneOnion, yoyoIcy, KongAKun, linzeyang, MuhammadNizamani, eltociear, Ligoml, LUZY0726, Windfarer, FlyingQianMM, jeng1220, junelotus, zlsh80826, Vvsmile, Frida-a, TonibMw, guoshengCS, zhink, ZhangYulongg, AlbertVan, fengxin-hello, mjp9527, entired, DanGuge.