{"id":535,"date":"2026-07-08T03:09:21","date_gmt":"2026-07-07T19:09:21","guid":{"rendered":"https:\/\/tech-renaissance.cn\/blog\/?p=535"},"modified":"2026-07-08T21:42:48","modified_gmt":"2026-07-08T13:42:48","slug":"amp%e8%87%aa%e5%8a%a8%e6%b7%b7%e5%90%88%e7%b2%be%e5%ba%a6%e8%ae%ad%e7%bb%83%ef%bc%9afp16%e7%9a%84%e9%80%9f%e5%ba%a6%e4%b8%8e%e7%b2%be%e5%ba%a6%e5%b9%b3%e8%a1%a1","status":"publish","type":"post","link":"https:\/\/tech-renaissance.cn\/blog\/index.php\/2026\/07\/08\/535\/","title":{"rendered":"(19) AMP\u81ea\u52a8\u6df7\u5408\u7cbe\u5ea6\u8bad\u7ec3\uff1aFP16\u7684\u901f\u5ea6\u4e0e\u7cbe\u5ea6\u5e73\u8861"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">\u2014\u2014\u201c\u4e00\u4e2a\u4eba\u7528AI\u5982\u4f55\u5199\u51fa\u6bd4PyTorch\u66f4\u5feb\u7684\u81ea\u7814\u6df1\u5ea6\u5b66\u4e60\u6846\u67b6\u201d\u7cfb\u5217\u6587\u7ae0\u4e4b\u5341\u4e5d<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u6df1\u5ea6\u5b66\u4e60\u8bad\u7ec3\u957f\u671f\u9762\u4e34\u4e00\u4e2a&#8221;\u4e0d\u53ef\u80fd\u4e09\u89d2&#8221;\uff1a\u7b97\u5f97\u5feb\u3001\u7b97\u5f97\u51c6\u3001\u5360\u5f97\u5c11\uff0c\u4e09\u8005\u4f3c\u4e4e\u5f88\u96be\u540c\u65f6\u6ee1\u8db3\u3002FP32 \u7cbe\u5ea6\u9ad8\u3001\u52a8\u6001\u8303\u56f4\u5927\uff0c\u4f46\u663e\u5b58\u5360\u7528\u5927\u548c\u8ba1\u7b97\u5bc6\u5ea6\u4f4e\uff1bFP16 \u663e\u5b58\u51cf\u534a\u3001Tensor Core \u541e\u5410\u9ad8\uff0c\u5374\u5bb9\u6613\u56e0\u4e3a\u52a8\u6001\u8303\u56f4\u4e0d\u8db3\u800c\u628a\u68af\u5ea6&#8221;\u541e&#8221;\u6210\u96f6\u3002\u81ea\u52a8\u6df7\u5408\u7cbe\u5ea6\u8bad\u7ec3\uff08AMP\uff0cAutomatic Mixed Precision\uff09\u5c31\u662f\u5728\u4e24\u8005\u4e4b\u95f4\u642d\u4e00\u5ea7\u6865\uff1a\u8ba9\u5927\u90e8\u5206\u524d\u5411\/\u53cd\u5411\u8ba1\u7b97\u8d70 FP16\uff0c\u628a\u4f18\u5316\u5668\u66f4\u65b0\u548c\u7cbe\u5ea6\u654f\u611f\u7684\u4e2d\u95f4\u72b6\u6001\u7559\u5728 FP32\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u4ece 2018 \u5e74 NVIDIA \u7684 Apex\uff0c\u5230 PyTorch 1.6 \u6b63\u5f0f\u5f15\u5165 <code>torch.cuda.amp<\/code>\uff0cAMP \u5df2\u7ecf\u6210\u4e3a\u73b0\u4ee3\u8bad\u7ec3\u6846\u67b6\u7684\u6807\u914d\u80fd\u529b\u3002\u4f46 AMP \u5e76\u4e0d\u53ea\u662f\u628a\u6743\u91cd\u7c7b\u578b\u6539\u4e00\u4e0b\u90a3\u4e48\u7b80\u5355\u2014\u2014\u5b83\u6d89\u53ca\u4e3b\u6743\u91cd\u53cc\u8f68\u5b58\u50a8\u3001\u635f\u5931\u7f29\u653e\u3001NaN\/Inf \u68c0\u6d4b\u3001\u68af\u5ea6\u88c1\u526a\u3001FP16\u2194FP32 \u6279\u91cf\u8f6c\u6362\uff0c\u4ee5\u53ca\u4e0e\u5e95\u5c42\u7b97\u5b50\u878d\u5408\uff08\u5982 Conv+BN+ReLU\uff09\u7684\u6df1\u5ea6\u8026\u5408\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Tech-Renaissance \u9009\u62e9\u4e86\u4e00\u6761\u4e0e PyTorch \u52a8\u6001 AMP \u4e0d\u540c\u7684\u8def\uff1a<strong>\u56fe\u7ea7\u3001\u9759\u6001\u3001\u663e\u5f0f\u7684 AMP<\/strong>\u3002\u6240\u6709\u7cbe\u5ea6\u8f6c\u6362\u3001\u7f29\u653e\u68c0\u6d4b\u3001\u7c7b\u578b\u8f6c\u6362\u5728\u7f16\u8bd1\u671f\u5c31\u5199\u5165\u8ba1\u7b97\u56fe\uff0c\u8fd0\u884c\u671f\u6309\u56fa\u5b9a\u987a\u5e8f\u542f\u52a8 CUDA Graph\u3002\u8fd9\u7bc7\u6587\u7ae0\u5c31\u6765\u62c6\u89e3\u8fd9\u6761\u6d41\u6c34\u7ebf\u662f\u600e\u4e48\u5de5\u4f5c\u7684\uff0c\u4ee5\u53ca\u4e3a\u4ec0\u4e48\u6211\u4eec\u575a\u6301\u4f7f\u7528 FP16 \u548c\u56fa\u5b9a\u635f\u5931\u7f29\u653e\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u4e00\u3001\u4e3a\u4ec0\u4e48\u9700\u8981\u6df7\u5408\u7cbe\u5ea6\u8bad\u7ec3<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">NVIDIA \u4ece Volta \u67b6\u6784\u5f00\u59cb\u5728 GPU \u4e2d\u96c6\u6210 Tensor Core\uff0c\u4e13\u95e8\u52a0\u901f\u534a\u7cbe\u5ea6\u77e9\u9635\u4e58\u52a0\u3002\u4ee5 A100 \u4e3a\u4f8b\uff0cFP16 Tensor Core \u7684<strong>\u7a20\u5bc6\u5cf0\u503c<\/strong>\u7b97\u529b\u7ea6\u4e3a 312 TFLOPS\uff0c\u800c FP32 \u53ea\u6709\u7ea6 19.5 TFLOPS\uff0c\u76f8\u5dee\u6574\u6574\u4e00\u4e2a\u6570\u91cf\u7ea7\u3002\u5bf9\u5377\u79ef\u3001\u5168\u8fde\u63a5\u8fd9\u7c7b\u4ee5 GEMM\/\u5377\u79ef\u4e3a\u6838\u5fc3\u7684\u7b97\u5b50\uff0c\u4f7f\u7528 FP16 \u8ba1\u7b97\u610f\u5473\u7740\u7406\u8bba\u4e0a\u53ef\u4ee5\u628a\u5cf0\u503c\u7b97\u529b\u63d0\u5347\u6570\u500d\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u9664\u4e86\u5cf0\u503c\u7b97\u529b\uff0cFP16 \u8fd8\u5e26\u6765\u4e24\u4e2a\u66f4\u5b9e\u9645\u7684\u597d\u5904\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u663e\u5b58\u5360\u7528\u51cf\u534a<\/strong>\uff1a\u540c\u6837\u5c3a\u5bf8\u7684\u6743\u91cd\u3001\u6fc0\u6d3b\u3001\u68af\u5ea6\uff0cFP16 \u53ea\u9700\u8981 FP32 \u4e00\u534a\u7684\u7a7a\u95f4\uff1b<\/li>\n\n\n\n<li><strong>\u663e\u5b58\u5e26\u5bbd\u51cf\u534a<\/strong>\uff1a\u6570\u636e\u5728 HBM \u4e0e\u8ba1\u7b97\u5355\u5143\u4e4b\u95f4\u642c\u8fd0\u65f6\uff0cFP16 \u4f20\u8f93\u91cf\u4e5f\u53ea\u6709 FP32 \u7684\u4e00\u534a\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u5bf9 ResNet-50\u3001VGG16BN \u8fd9\u7c7b\u7ecf\u5178 CNN\uff0c\u8bad\u7ec3\u8fc7\u7a0b\u5728 A100 \u4e0a\u5f80\u5f80\u662f<strong>\u663e\u5b58\u5e26\u5bbd\u53d7\u9650<\/strong>\u800c\u975e\u7eaf\u7b97\u529b\u53d7\u9650\u3002\u51cf\u534a\u5e26\u5bbd\u80fd\u76f4\u63a5\u63d0\u5347\u6709\u6548\u541e\u5410\uff0c\u8fd9\u4e5f\u662f AMP \u5728 CNN \u8bad\u7ec3\u4e2d\u6536\u76ca\u660e\u663e\u7684\u539f\u56e0\u4e4b\u4e00\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u4f46 FP16 \u7684\u4ee3\u4ef7\u540c\u6837\u660e\u663e\u3002\u5b83\u53ea\u6709 1 \u4f4d\u7b26\u53f7\u4f4d\u30015 \u4f4d\u6307\u6570\u4f4d\u300110 \u4f4d\u5c3e\u6570\u4f4d\uff0c\u6700\u5c0f\u6b63\u6b63\u89c4\u6570\u7ea6 <code>6.1\u00d710\u207b\u2075<\/code>\uff0c\u6700\u5927\u53ef\u8868\u793a\u503c\u7ea6 <code>6.55\u00d710\u2074<\/code>\u3002\u8bad\u7ec3\u4e2d\u7684\u5f88\u591a\u68af\u5ea6\u503c\u5728 <code>10\u207b\u2075<\/code> \u751a\u81f3 <code>10\u207b\u2077<\/code> \u91cf\u7ea7\uff0c\u5df2\u7ecf\u63a5\u8fd1\u6216\u4f4e\u4e8e FP16 \u7684\u6b63\u89c4\u6570\u8303\u56f4\uff0c\u53ef\u80fd\u8fdb\u5165 subnormal \u533a\u95f4\uff0c\u7cbe\u5ea6\u5927\u5e45\u4e0b\u964d\uff0c\u751a\u81f3\u5728\u542f\u7528 flush-to-zero \u7684\u8def\u5f84\u4e0b\u88ab\u76f4\u63a5\u5f52\u96f6\uff0c\u5bfc\u81f4\u6709\u6548\u68af\u5ea6\u4fe1\u606f\u4e22\u5931\uff1b\u5076\u5c14\u51fa\u73b0\u7684\u5927\u68af\u5ea6\u53c8\u53ef\u80fd\u4e0a\u6ea2\u6210 <code>Inf<\/code> \u6216 <code>NaN<\/code>\uff0c\u5bfc\u81f4\u8bad\u7ec3\u5d29\u6e83\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u56e0\u6b64\u6df7\u5408\u7cbe\u5ea6\u8bad\u7ec3\u7684\u6838\u5fc3\u601d\u60f3\u662f\u4e24\u6761\u89c4\u5219\uff1a<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>FP32 \u4e3b\u6743\u91cd\uff08master weights\uff09<\/strong>\uff1a\u4f18\u5316\u5668\u5728 FP32 \u6743\u91cd\u4e0a\u66f4\u65b0\uff0c\u907f\u514d\u5fae\u5c0f\u66f4\u65b0\u88ab FP16 \u7cbe\u5ea6&#8221;\u541e\u6389&#8221;\uff1b<\/li>\n\n\n\n<li><strong>\u635f\u5931\u7f29\u653e\uff08loss scaling\uff09<\/strong>\uff1a\u524d\u5411\u65f6\u628a\u635f\u5931\u503c\u4e58\u4ee5\u4e00\u4e2a\u8f83\u5927\u7684\u7f29\u653e\u56e0\u5b50 <code>S<\/code>\uff0c\u8ba9\u53cd\u5411\u4f20\u64ad\u7684\u68af\u5ea6\u4e5f\u653e\u5927 <code>S<\/code> \u500d\uff0c\u79bb\u5f00\u4e0b\u6ea2\u533a\uff1b\u4f18\u5316\u5668\u66f4\u65b0\u524d\u518d\u9664\u4ee5 <code>S<\/code> \u8fd8\u539f\u3002<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">\u4e3b\u6d41\u6846\u67b6 PyTorch \u7684 <code>torch.cuda.amp<\/code> \u91c7\u7528<strong>\u52a8\u6001\u635f\u5931\u7f29\u653e<\/strong>\uff1a\u521d\u59cb scale \u901a\u5e38\u8bbe\u4e3a 65536\uff0c\u6bcf\u8f6e\u68c0\u6d4b\u68af\u5ea6\u662f\u5426\u6ea2\u51fa\uff0c\u65e0\u6ea2\u51fa\u5219\u8fde\u7eed\u6210\u529f\u82e5\u5e72\u6b65\u540e\u7ffb\u500d\uff0c\u6ea2\u51fa\u5219\u8df3\u8fc7\u66f4\u65b0\u5e76\u51cf\u534a\u3002\u8fd9\u79cd\u7b56\u7565\u5bf9\u7528\u6237\u900f\u660e\uff0c\u4f46\u7ef4\u62a4 per-device \u7684 scale \u72b6\u6001\u3001<code>FoundInf<\/code> \u6807\u5fd7\u548c\u589e\u957f\u8ba1\u6570\u5668\uff0c\u5bf9 eager \u6a21\u5f0f\u5f88\u81ea\u7136\uff0c\u5bf9\u8ffd\u6c42\u786e\u5b9a\u6027\u3001\u9759\u6001\u5316\u7684\u6846\u67b6\u5374\u4e0d\u662f\u6700\u4f18\u89e3\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u4e8c\u3001Tech-Renaissance \u7684 AMP \u9009\u62e9\uff1aFP16 + \u56fa\u5b9a\u7f29\u653e<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Tech-Renaissance \u9009\u62e9<strong>\u56fa\u5b9a\u635f\u5931\u7f29\u653e<\/strong>\uff0c\u521d\u59cb\u503c\u786c\u7f16\u7801\u5728 <code>include\/renaissance\/core\/global_config.h<\/code>\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">#define TR_AMP_INITIAL_SCALING  8192.0f<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u4e2a\u503c\u662f 2 \u7684\u5e42\uff08<code>2\u00b9\u00b3<\/code>\uff09\u3002\u53d6 2 \u7684\u5e42\u6709\u4e24\u4e2a\u597d\u5904\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u653e\u5927\u548c\u8fd8\u539f\u672c\u8d28\u4e0a\u662f\u6307\u6570\u4f4d\u52a0\u51cf\uff0c\u820d\u5165\u8bef\u5dee\u66f4\u5c0f\uff1b<\/li>\n\n\n\n<li>\u5bf9\u5178\u578b\u7684 CNN \u8bad\u7ec3\uff0c8192 \u8db3\u591f\u628a\u5927\u90e8\u5206\u5c0f\u68af\u5ea6\u62ac\u5347\u5230 FP16 \u53ef\u8868\u793a\u8303\u56f4\uff0c\u540c\u65f6\u53c8\u4e0d\u81f3\u4e8e\u628a\u635f\u5931\u653e\u5927\u5230\u63a5\u8fd1 FP16 \u4e0a\u9650\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u4e0e PyTorch \u7684\u52a8\u6001\u7b56\u7565\u4e0d\u540c\uff0c\u6211\u4eec\u7684\u7f29\u653e\u56e0\u5b50<strong>\u53ea\u51cf\u4e0d\u589e<\/strong>\u3002\u5f53 <code>RANGE_CHECK_NAN<\/code> \u68c0\u6d4b\u5230\u68af\u5ea6\u4e2d\u5b58\u5728 NaN \u65f6\uff0c<code>RANGE_GRAD_SCALING<\/code> \u4f1a\u628a <code>scaling<\/code> \u4e58\u4ee5 0.5\uff0c\u4e0b\u9650\u622a\u65ad\u5230 1.0\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ grad_scaling_op.cu\n__global__ void grad_scaling_kernel(const int32_t* __restrict__ has_nan,\n                                    float* __restrict__ scaling)\n{\n    if (*has_nan != 0) {\n        float new_scaling = (*scaling) * 0.5f;\n        *scaling = (new_scaling &lt; 1.0f) ? 1.0f : new_scaling;\n    }\n}<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>has_nan<\/code>\uff1a<code>RANGE_CHECK_NAN<\/code> \u5199\u5165\u7684 GPU \u6807\u91cf\u6807\u5fd7\uff0c0 \u8868\u793a\u65e0\u5f02\u5e38\uff0c\u975e 0 \u8868\u793a\u53d1\u73b0\u5f02\u5e38\uff1b<\/li>\n\n\n\n<li><code>scaling<\/code>\uff1a\u5f53\u524d\u635f\u5931\u7f29\u653e\u56e0\u5b50\uff0c\u4fdd\u5b58\u5728 <code>S_SCALAR_FP32<\/code> \u533a\u57df\uff0c\u53ef\u88ab CUDA Graph \u52a8\u6001\u66f4\u65b0\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u79cd&#8221;\u53ea\u51cf\u4e0d\u589e&#8221;\u7684\u7b56\u7565\u770b\u8d77\u6765\u4f1a\u635f\u5931\u4e00\u4e9b\u7f29\u653e\u7a7a\u95f4\uff0c\u4f46\u5b83\u6709\u4e24\u4e2a\u660e\u786e\u4f18\u52bf\uff1a<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>CUDA Graph \u53cb\u597d<\/strong>\uff1a\u7f29\u653e\u56e0\u5b50\u7684\u8c03\u6574\u4e0d\u9700\u8981 CPU \u91cd\u65b0\u751f\u6210 kernel \u53c2\u6570\u6216\u63a7\u5236\u6d41\uff0c\u5b8c\u5168\u5728 GPU \u4fa7\u4e00\u4e2a kernel \u5185\u5b8c\u6210\uff1b<\/li>\n\n\n\n<li><strong>\u786e\u5b9a\u6027<\/strong>\uff1a\u5728\u56fa\u5b9a\u968f\u673a\u79cd\u5b50\u3001\u56fa\u5b9a\u7b97\u5b50\u5b9e\u73b0\u7684\u524d\u63d0\u4e0b\uff0c\u662f\u5426\u51fa\u73b0 NaN \u662f\u786e\u5b9a\u7684\uff0c\u7f29\u653e\u8def\u5f84\u4e5f\u662f\u786e\u5b9a\u7684\u3002<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">\u5b83\u7684\u4ee3\u4ef7\u662f\u9c81\u68d2\u6027\u4e0d\u5982\u52a8\u6001\u7b56\u7565\uff1a\u5982\u679c\u8bad\u7ec3\u65e9\u671f\u56e0\u4e3a\u5f02\u5e38\u628a scale \u964d\u5230\u5f88\u4f4e\uff0c\u540e\u7eed\u4e0d\u4f1a\u81ea\u52a8\u6062\u590d\u3002\u4f46\u5728\u6211\u4eec\u5b9e\u6d4b\u7684 CNN \u8bad\u7ec3\u573a\u666f\u4e2d\uff0c\u6ea2\u51fa\u901a\u5e38\u53ea\u53d1\u751f\u5728\u6743\u91cd\u521d\u59cb\u5316\u4e0d\u7a33\u5b9a\u7684\u6700\u521d\u51e0\u4e2a batch\uff0cscale \u4ece 8192 \u964d\u5230 4096 \u6216 2048 \u540e\u5c31\u7a33\u5b9a\u4e0b\u6765\uff0c\u4ece\u672a\u89c2\u5bdf\u5230\u4e00\u8def\u964d\u5230 1 \u7684\u60c5\u51b5\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u4e3a\u4ec0\u4e48\u4e0d\u662f BF16\uff1f<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">BF16 \u7528 8 \u4f4d\u6307\u6570\u4f4d\u30017 \u4f4d\u5c3e\u6570\u4f4d\uff0c\u52a8\u6001\u8303\u56f4\u4e0e FP32 \u76f8\u540c\uff0c\u901a\u5e38\u4e0d\u9700\u8981 loss scaling\uff0c\u8bad\u7ec3\u66f4\u7a33\u5b9a\u3002\u4f46 Tech-Renaissance \u5f53\u524d\u7248\u672c\u53ea\u652f\u6301 FP16\uff0c\u539f\u56e0\u975e\u5e38\u5177\u4f53\uff1a<strong>cuDNN \u7684 Conv+GenStats \u7b97\u5b50\u76ee\u524d\u53ea\u652f\u6301 FP16<\/strong>\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u4e2a\u878d\u5408\u7b97\u5b50\u662f\u4e0b\u4e00\u7bc7\u8981\u8bb2\u7684 CBR\uff08Conv+BN+ReLU\uff09\u878d\u5408\u7684\u57fa\u7840\u3002\u5b83\u53ef\u4ee5\u5728\u5377\u79ef\u8ba1\u7b97\u7684\u540c\u65f6\uff0c\u76f4\u63a5\u751f\u6210 BN \u9700\u8981\u7684\u901a\u9053\u548c\u4e0e\u901a\u9053\u5e73\u65b9\u548c\uff08<code>sum<\/code> \u548c <code>sq_sum<\/code>\uff09\uff0c\u907f\u514d BN \u6267\u884c\u65f6\u91cd\u65b0\u8bfb\u53d6\u4e00\u6b21\u5377\u79ef\u8f93\u51fa\u8fd9\u4e2a\u5927\u5f20\u91cf\u3002\u5bf9\u5e26\u5bbd\u654f\u611f\u7684 CNN \u8bad\u7ec3\u6765\u8bf4\uff0c\u7701\u6389\u7684\u8fd9\u4e00\u6b21\u5927\u5f20\u91cf\u8bfb\u5199\u975e\u5e38\u53ef\u89c2\u3002BF16 \u76ee\u524d\u505a\u4e0d\u5230\u8fd9\u4e00\u70b9\uff0c\u6240\u4ee5\u6846\u67b6\u505a\u51fa\u4e86\u660e\u786e\u7684\u5de5\u7a0b\u53d6\u820d\uff1a\u5148\u62ff\u4e0b FP16 \u7684\u878d\u5408\u7ea2\u5229\uff0c\u800c\u4e0d\u662f\u4e3a\u4e86 BF16 \u7684\u7a33\u5b9a\u6027\u653e\u5f03 CBR \u878d\u5408\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u4e09\u3001AMP \u663e\u5b58\u5e03\u5c40\uff1aW\u3001A\u3001G\u3001E \u56db\u6761\u7ebf<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u5728 Tech-Renaissance \u91cc\uff0cAMP \u4e0d\u662f\u8fd0\u884c\u65f6\u4e34\u65f6\u51b3\u5b9a\u7cbe\u5ea6\uff0c\u800c\u662f\u5728\u7f16\u8bd1\u671f\u5c31\u5199\u8fdb\u4e86\u663e\u5b58\u5206\u533a\u3002\u6253\u5f00 <code>include\/renaissance\/core\/types.h<\/code>\uff0c\u53ef\u4ee5\u770b\u5230 68 \u4e2a\u547d\u540d\u8bed\u4e49 Region \u4e2d\u4e13\u95e8\u6709\u56db\u6761\u7ebf\u670d\u52a1\u4e8e AMP\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>W \u7cfb\u5217<\/strong>\uff08005\u2013012\uff09\uff1aFP32 \u4e3b\u6743\u91cd\uff0c\u5305\u62ec BN \u53c2\u6570\u3001FC \u53c2\u6570\u3001\u5377\u79ef\u53c2\u6570\uff1b<\/li>\n\n\n\n<li><strong>E \u7cfb\u5217<\/strong>\uff08013\u2013018\uff09\uff1aFP32 EMA \u6743\u91cd\uff1b019\u2013021 \u662f\u5bf9\u5e94\u7684 FP16 EMA \u526f\u672c\uff1b<\/li>\n\n\n\n<li><strong>A \u7cfb\u5217<\/strong>\uff08022\u2013024\uff09\uff1aFP16 AMP \u8ba1\u7b97\u6743\u91cd\uff1b<\/li>\n\n\n\n<li><strong>G \u7cfb\u5217<\/strong>\uff08025\u2013030\uff09\uff1aFP32 \u68af\u5ea6\uff1b032\u2013034 \u662f\u5bf9\u5e94\u7684 FP16 Conv\/CBR \u68af\u5ea6\u3002<\/li>\n<\/ul>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ include\/renaissance\/core\/types.h \u4e2d AMP \u76f8\u5173 Region \u8282\u9009\nA_FC_WEIGHT,         \/\/ 022\nA_FIRST_CONV,        \/\/ 023\nA_DEEP_CONV,         \/\/ 024\n\nG_BN_BIAS,           \/\/ 025\nG_BN_WEIGHT,         \/\/ 026\nG_FC_BIAS,           \/\/ 027\nG_FC_WEIGHT,         \/\/ 028\nG_FIRST_CONV,        \/\/ 029\nG_DEEP_CONV,         \/\/ 030\n\nG_FC_WEIGHT_FP16,    \/\/ 032\nG_FIRST_CONV_FP16,   \/\/ 033\nG_DEEP_CONV_FP16,    \/\/ 034<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u79cd\u5206\u533a\u8bbe\u8ba1\u7684\u5173\u952e\u5728\u4e8e\uff1a<strong>\u540c\u7c7b\u578b\u5f20\u91cf\u88ab\u8fde\u7eed\u6446\u653e<\/strong>\u3002\u6240\u6709\u6df1\u5c42\u5377\u79ef\u7684 FP32 \u6743\u91cd\u8fde\u7eed\u653e\u5728 <code>W_DEEP_CONV<\/code>\uff0c\u6240\u6709 FP16 \u8ba1\u7b97\u6743\u91cd\u8fde\u7eed\u653e\u5728 <code>A_DEEP_CONV<\/code>\uff0c\u6240\u6709 FP16 Conv \u68af\u5ea6\u8fde\u7eed\u653e\u5728 <code>G_DEEP_CONV_FP16<\/code>\u3002\u4e8e\u662f\u6846\u67b6\u53ef\u4ee5\u7528\u4e00\u6b21 <code>RANGE_CAST_FP32_TO_FP16<\/code> \u5c31\u628a\u6574\u4e2a <code>W_DEEP_CONV<\/code> \u533a\u57df cast \u5230 <code>A_DEEP_CONV<\/code>\uff0c\u800c\u4e0d\u7528\u9010\u5c42\u542f\u52a8 kernel\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>MemoryPlan<\/code> \u5728\u5206\u914d\u65f6\u4e5f\u663e\u5f0f\u68c0\u67e5\u4e86 AMP \u4e0e\u6570\u636e\u7c7b\u578b\u7684\u4e00\u81f4\u6027\u3002\u4ee5\u7279\u5f81\u56fe\u5206\u914d\u4e3a\u4f8b\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ src\/graph\/memory_plan.cpp\nDTensor MemoryPlan::alloc_feature(const Shape&amp; shape, DType dtype) {\n    bool amp = GlobalRegistry::instance().using_amp();\n    if (dtype == DType::FP16) {\n        TR_CHECK(amp, ValueError, \"FP16 feature requires amp_enabled\");\n        return alloc_impl(shape, DType::FP16, Region::F_FEATURE_FP16);\n    }\n    TR_CHECK(dtype == DType::FP32, ValueError,\n             \"Feature dtype must be FP32 or FP16, got ...\");\n    TR_CHECK(!amp, ValueError, \"FP32 feature requires !amp_enabled\");\n    return alloc_impl(shape, DType::FP32, Region::F_FEATURE_FP32);\n}<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>amp<\/code>\uff1a\u5168\u5c40 AMP \u5f00\u5173\uff0c\u7531 <code>GlobalRegistry::using_amp()<\/code> \u63d0\u4f9b\uff1b<\/li>\n\n\n\n<li><code>dtype<\/code>\uff1a\u8bf7\u6c42\u5206\u914d\u7684\u7279\u5f81\u56fe\u6570\u636e\u7c7b\u578b\uff1b<\/li>\n\n\n\n<li>\u8fd4\u56de\u503c <code>DTensor<\/code>\uff1a\u7eaf\u63cf\u8ff0\u7b26\uff0c\u53ea\u8bb0\u5f55 shape\/offset\/stride\/Region\uff0c\u4e0d\u6301\u6709\u5b9e\u9645\u663e\u5b58\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u7c7b\u4f3c\u7684\u65ad\u8a00\u4e5f\u5b58\u5728\u4e8e <code>alloc_grad_slot<\/code> \u548c <code>alloc_scalar<\/code> \u4e2d\u3002FP16 \u5f20\u91cf\u8981\u6c42 AMP \u5f00\u542f\uff0cFP32 \u7279\u5f81\u56fe\u8981\u6c42 AMP \u5173\u95ed\u3002\u8fd9\u79cd&#8221;\u7f16\u8bd1\u671f\u51b3\u5b9a\u4e00\u5207&#8221;\u7684\u504f\u6267\uff0c\u4e0e\u9759\u6001\u56fe\u3001CUDA Graph\u3001MemoryPlan \u7b49\u8bbe\u8ba1\u7406\u5ff5\u4e00\u8109\u76f8\u627f\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5728 <code>MemoryPlan::is_condition_enabled()<\/code> \u4e2d\uff0cA \u7cfb\u5217\u3001G \u7cfb\u5217\u7684 FP16 \u68af\u5ea6\u533a\u3001E \u7cfb\u5217\u7684 FP16 \u526f\u672c\u3001F \u7cfb\u5217\u7684 FP16 \u7279\u5f81\u56fe\uff0c\u4ee5\u53ca FP16 \u6807\u91cf\u533a\uff0c\u90fd\u88ab\u663e\u5f0f\u5730\u95e8\u63a7\u5728 <code>GlobalRegistry::using_amp()<\/code> \u4e0a\uff1b\u800c FP32 \u7279\u5f81\u56fe\u533a\u53ea\u5728 AMP \u5173\u95ed\u65f6\u624d\u542f\u7528\u3002\u8fd9\u610f\u5473\u7740 AMP \u5f00\u5173\u4e00\u65e6\u786e\u5b9a\uff0c\u6574\u5957\u663e\u5b58\u5e03\u5c40\u5c31\u662f\u9759\u6001\u4e14\u53ef\u9a8c\u8bc1\u7684\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u4e3a\u4ec0\u4e48\u4e00\u6b21 RangeOp \u80fd\u8f6c\u6362\u5168\u6a21\u578b\uff1f<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u4f9d\u8d56\u4e8e <code>DTensor::compute_slot_bytes<\/code> \u7684\u8bbe\u8ba1\u3002\u5bf9\u6743\u91cd\/\u68af\u5ea6\u533a\uff08W\u3001A\u3001G\u3001E \u7cfb\u5217\uff09\uff0cC \u901a\u9053\u5bf9\u9f50\u56e0\u5b50\u4e3a 1\uff0c\u56e0\u6b64\u540c\u4e00\u5c42\u540c\u4e00 region \u7684 FP32 \u69fd\u4f4d\u5927\u5c0f\u6070\u597d\u662f FP16 \u69fd\u4f4d\u5927\u5c0f\u7684\u4e24\u500d\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ \u7b80\u5316\u81ea include\/renaissance\/tensor\/distributed_tensor.h\nif (dtype == DType::FP16) {\n    return align_up_256(elems * 2 + 16);     \/\/ \u6bcf\u4e2a\u5143\u7d20 2 \u5b57\u8282\n} else if (dtype == DType::FP32) {\n    return 2 * align_up_256(elems * 2 + 16); \/\/ FP32 \u69fd\u4f4d = 2 \u00d7 FP16 \u69fd\u4f4d\n}<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>elems<\/code>\uff1a\u5f20\u91cf\u5143\u7d20\u603b\u6570\uff1b<\/li>\n\n\n\n<li><code>align_up_256<\/code>\uff1a\u5411\u4e0a\u5bf9\u9f50\u5230 256 \u5b57\u8282\uff1b<\/li>\n\n\n\n<li><code>+16<\/code>\uff1a\u672b\u5c3e\u4fdd\u7559 16 \u5b57\u8282\uff0c\u6ee1\u8db3 XNNPACK \u7b49\u540e\u7aef\u7684\u5bf9\u9f50\u8981\u6c42\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u7531\u4e8e FP32 \u4e0e FP16 \u69fd\u4f4d\u5927\u5c0f\u5448\u56fa\u5b9a 2:1 \u5173\u7cfb\uff0c\u4e14\u540c\u4e00\u5c42\u5728 W\/A \u6216 G\/G_FP16 \u4e2d\u7684\u987a\u5e8f\u5b8c\u5168\u4e00\u81f4\uff0c\u7f16\u8bd1\u5668\u53ef\u4ee5\u628a <code>W_FC_WEIGHT..W_DEEP_CONV<\/code> \u5230 <code>A_FC_WEIGHT..A_DEEP_CONV<\/code> \u7684\u8f6c\u6362\u8868\u8fbe\u4e3a\u4e00\u4e2a <code>RANGE_CAST_FP32_TO_FP16<\/code> \u8282\u70b9\uff0c\u4e00\u4e2a kernel \u8c03\u7528\u5b8c\u6210\u5168\u6a21\u578b\u6743\u91cd\u8f6c\u6362\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u56db\u3001\u8bad\u7ec3\u4e00\u6b65\u7684 AMP \u6d41\u6c34\u7ebf<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u5728 <code>src\/backend\/graph_executor.cpp<\/code> \u7684 <code>run_train_step()<\/code> \u4e2d\uff0c\u4e00\u6b21\u5e38\u89c4\u8bad\u7ec3\u8fed\u4ee3\u7684\u6267\u884c\u987a\u5e8f\u5927\u81f4\u5982\u4e0b\uff08\u7701\u7565\u90e8\u5206 A\/B \u53cc\u7f13\u51b2\u7ec6\u8282\uff09\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ src\/backend\/graph_executor.cpp\nlaunch(GraphId::TRANSFER_A);          \/\/ H2D \u5f02\u6b65\u4f20\u8f93\nlaunch(GraphId::ZERO_GRAD);           \/\/ \u68af\u5ea6\u533a\u6e05\u96f6\nlaunch(GraphId::FIRST_LAYER_FWD_A);   \/\/ \u9996\u5c42\u524d\u5411\nlaunch_dual(next_xfer, GraphId::DEEP_FWD_BWD); \/\/ \u4e0b\u4e00 batch \u4f20\u8f93 + \u6df1\u5c42\u6b63\u53cd\u5411\nsync_all();\n\nlaunch(GraphId::FIRST_LAYER_BWD_A);   \/\/ \u9996\u5c42\u53cd\u5411\nsync_all();\n\nlaunch(GraphId::CAST_DEEP_GRAD_FP16_TO_FP32); \/\/ \u6df1\u5c42\u5377\u79ef\u68af\u5ea6 FP16 \u2192 FP32\nlaunch(GraphId::DEEP_COMM);           \/\/ \u6df1\u5c42\u68af\u5ea6 AllReduce\nsync_all();\n\nlaunch(GraphId::CAST_FIRST_GRAD_FP16_TO_FP32); \/\/ \u9996\u5c42\u5377\u79ef\u68af\u5ea6 FP16 \u2192 FP32\nlaunch(GraphId::FIRST_COMM);          \/\/ \u9996\u5c42\u68af\u5ea6 AllReduce\n\nlaunch(GraphId::NAN_CHECK_AND_GRAD_SCALING);  \/\/ NaN \u68c0\u67e5 + \u7f29\u653e\u8c03\u6574\nlaunch(GraphId::STATS_COMM);          \/\/ BN \u7edf\u8ba1\u91cf\u540c\u6b65\nlaunch(GraphId::UPDATE_STATS);        \/\/ BN next \u2192 prev \u590d\u5236\n\nbool has_nan = check_nan_flag();\nif (!has_nan) {\n    launch(GraphId::OPTIMIZER);       \/\/ FP32 \u4e3b\u6743\u91cd\u66f4\u65b0\n    launch(GraphId::EMA_UPDATE);      \/\/ EMA \u66f4\u65b0\uff08\u542b FP32\u2192FP16 cast\uff09\n} else {\n    on_nan_detected();\n}\n\nlaunch_dual(GraphId::CAST_MAIN_FP32_TO_FP16,\n            GraphId::CAST_EMA_FP32_TO_FP16); \/\/ \u4e3b\u6743\u91cd \/ EMA FP32 \u2192 FP16\nsync_all();<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>launch<\/code>\uff1a\u5728\u6307\u5b9a CUDA Stream \u4e0a\u542f\u52a8\u4e00\u4e2a CUDA Graph\uff1b<\/li>\n\n\n\n<li><code>launch_dual<\/code>\uff1a\u540c\u65f6\u5728\u4e24\u4e2a stream \u4e0a\u542f\u52a8\u4e24\u4e2a\u56fe\uff0c\u5b9e\u73b0\u5e76\u53d1\uff1b<\/li>\n\n\n\n<li><code>sync_all<\/code>\uff1a\u7b49\u5f85\u6240\u6709\u6d41\u5b8c\u6210\u5f53\u524d\u9636\u6bb5\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u91cc\u6709\u4e24\u4e2a\u5b9e\u73b0\u7ec6\u8282\u503c\u5f97\u8865\u5145\u3002\u7b2c\u4e00\uff0c\u6240\u6709 FP32\u2194FP16 \u7684 cast\u3001\u4f18\u5316\u5668\u66f4\u65b0\u3001EMA \u66f4\u65b0\u3001<code>NAN_CHECK_AND_GRAD_SCALING<\/code> \u90fd\u8fd0\u884c\u5728 <code>StreamKind::UPDATE<\/code> \u4e0a\uff0c\u4e0e\u4e3b\u8ba1\u7b97\u6d41 <code>COMP_1<\/code> \u4e92\u4e0d\u963b\u585e\uff0c\u4fbf\u4e8e\u548c\u8ba1\u7b97\u9636\u6bb5\u91cd\u53e0\u3002\u7b2c\u4e8c\uff0c<code>EMA_UPDATE<\/code> \u56fe\u4e2d\u5df2\u7ecf\u5305\u542b\u4e86 EMA FP32\u2192FP16 \u7684 <code>RANGE_CAST_FP32_TO_FP16<\/code> \u8282\u70b9\uff1b<code>GraphExecutor<\/code> \u672b\u5c3e\u540c\u65f6\u542f\u52a8\u7684 <code>CAST_EMA_FP32_TO_FP16<\/code> \u56fe\u5f53\u524d\u4e3a\u7a7a\u64cd\u4f5c\uff0c\u4fdd\u7559\u5b83\u53ea\u662f\u4e3a\u4e86\u4fdd\u6301\u6267\u884c\u5668\u7ed3\u6784\u7684\u7edf\u4e00\u6027\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u4e2a\u6d41\u7a0b\u91cc\u6709\u51e0\u4e2a\u503c\u5f97\u5355\u72ec\u5c55\u5f00\u7684\u7ec6\u8282\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1. FC \u4e0e Conv \u7684\u68af\u5ea6\u8def\u5f84\u4e0d\u540c<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">FC \u5c42\u7684\u53cd\u5411\u8d70 cuBLAS\uff0c\u53ef\u4ee5\u76f4\u63a5\u8981\u6c42\u8f93\u51fa FP32\u3002\u5728 <code>src\/backend\/ops\/dtensor\/fc_op.cpp<\/code> \u4e2d\uff0c<code>cublasGemmEx<\/code> \u7684\u8f93\u51fa\u6570\u636e\u7c7b\u578b\u6307\u5b9a\u4e3a <code>CUDA_R_32F<\/code>\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ \u7b80\u5316\u81ea fc_op.cpp\nfloat* dw = ...;\ncublasGemmEx(handle, CUBLAS_OP_N, CUBLAS_OP_T, ...,\n    x,  CUDA_R_16F, x_ns,    \/\/ \u8f93\u5165 X\uff1aFP16\n    dy, CUDA_R_16F, dy_ns,   \/\/ \u8f93\u5165 dY\uff1aFP16\n    dw, CUDA_R_32F, dw_ns,   \/\/ \u8f93\u51fa dW\uff1aFP32\n    CUBLAS_COMPUTE_32F,\n    CUBLAS_GEMM_DEFAULT_TENSOR_OP);<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">\u56e0\u6b64 FC \u68af\u5ea6\u5929\u7136\u5c31\u662f FP32\uff0c\u76f4\u63a5\u5199\u5165 <code>G_FC_WEIGHT<\/code> \/ <code>G_FC_BIAS<\/code>\uff0c\u4e0d\u9700\u8981\u540e\u7eed\u8f6c\u6362\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Conv \u548c CBR \u7684\u60c5\u51b5\u66f4\u590d\u6742\u3002cuDNN \u7684\u5377\u79ef\u53cd\u5411 filter\uff08wgrad\uff09\u5728 FP16 \u6a21\u5f0f\u4e0b\u901a\u5e38\u8f93\u51fa FP16\u3002CBR \u53cd\u5411\u56fe\u5728 cuDNN Frontend Graph \u4e2d\u6784\u5efa\u65f6\uff0cdY\u3001W\u3001dX \u90fd\u662f FP16\uff0cwgrad \u8f93\u51fa\u81ea\u7136\u4e5f\u662f FP16\uff0c\u5148\u88ab\u5199\u5230 <code>G_DEEP_CONV_FP16<\/code> \u6216 <code>G_FIRST_CONV_FP16<\/code>\uff0c\u7136\u540e\u7531 <code>RANGE_CAST_FP16_TO_FP32<\/code> \u4e00\u6b21\u6027\u6279\u91cf\u8f6c\u6210 FP32\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ src\/graph\/compiler.cpp\nif (amp_on &amp;&amp; memory_plan.is_region_populated(Region::G_DEEP_CONV_FP16)) {\n    MemRange in_deep  = memory_plan.region_range(Region::G_DEEP_CONV_FP16);\n    MemRange out_deep = memory_plan.region_range(Region::G_DEEP_CONV);\n    train_cg.append_range(GraphId::CAST_DEEP_GRAD_FP16_TO_FP32,\n        RangeOp::RANGE_CAST_FP16_TO_FP32, {in_deep}, {out_deep});\n}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">\u4e3a\u4ec0\u4e48\u4e0d\u76f4\u63a5\u5728 cuDNN \u91cc\u8981\u6c42 wgrad \u8f93\u51fa FP32\uff1f\u56e0\u4e3a cuDNN Frontend Graph \u7684\u67d0\u4e9b\u878d\u5408\u8def\u5f84\u5bf9\u8f93\u51fa\u6570\u636e\u7c7b\u578b\u6709\u7ea6\u675f\uff0c\u5f3a\u5236\u6307\u5b9a FP32 \u8f93\u51fa\u53ef\u80fd\u5bfc\u81f4\u627e\u4e0d\u5230\u5408\u6cd5\u6267\u884c\u8ba1\u5212\uff0c\u6216\u8005\u6027\u80fd\u53cd\u800c\u4e0b\u964d\u3002\u63a5\u53d7 FP16 \u4e2d\u95f4\u7ed3\u679c\uff0c\u518d\u7528\u4e00\u6b21\u9ad8\u6548\u7684 RangeOp \u6279\u91cf\u8f6c\u6362\uff0c\u662f\u66f4\u52a1\u5b9e\u7684\u9009\u62e9\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. \u635f\u5931\u7f29\u653e\u5230\u5e95\u53d1\u751f\u5728\u54ea\u91cc\uff1f<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">\u635f\u5931\u7f29\u653e\u4e0d\u662f\u72ec\u7acb\u7684\u5916\u6302\uff0c\u800c\u662f\u5206\u6563\u5728 SoftmaxCE \u53cd\u5411\u548c\u4f18\u5316\u5668\u66f4\u65b0\u4e2d\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>SoftmaxCE BWD<\/strong> \u8f93\u51fa\u7684\u68af\u5ea6\u5df2\u7ecf\u5e26\u6709 <code>scaling \/ batch<\/code> \u7684\u7f29\u653e\uff1b<\/li>\n\n\n\n<li><strong>Optimizer<\/strong> \u5185\u90e8\u518d\u628a\u68af\u5ea6\u4e58\u4ee5 <code>1 \/ scaling<\/code>\uff0c\u5b8c\u6210\u8fd8\u539f\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">net effect \u662f\u68af\u5ea6\u6309 batch \u5e73\u5747\uff0c\u4e0e\u65e0\u7f29\u653e\u65f6\u4e00\u81f4\u3002\u628a <code>1\/scaling<\/code> \u653e\u5230\u4f18\u5316\u5668\u91cc\u505a\uff0c\u53ef\u4ee5\u7701\u4e0b\u4e00\u6b21\u5bf9\u68af\u5ea6\u533a\u7684\u5355\u72ec\u663e\u5b58\u904d\u5386\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ optimizer_op.cu \u4e2d SGD \u5185\u6838\u8282\u9009\nfloat _inv_scaling = (scaling &amp;&amp; *scaling != 0.0f) ? (1.0f \/ *scaling) : 1.0f;\nfor (...) {\n    float g_i = g[i] * _inv_scaling;\n    w[i] = w_i * (1.0f - _lr * _wd) - _lr * g_i;\n}<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>g[i]<\/code>\uff1a\u7b2c <code>i<\/code> \u4e2a FP32 \u68af\u5ea6\u5143\u7d20\uff1b<\/li>\n\n\n\n<li><code>scaling<\/code>\uff1a\u5f53\u524d\u635f\u5931\u7f29\u653e\u56e0\u5b50\uff1b<\/li>\n\n\n\n<li><code>_inv_scaling<\/code>\uff1a<code>1 \/ scaling<\/code>\uff1b<\/li>\n\n\n\n<li><code>w[i]<\/code>\uff1a\u7b2c <code>i<\/code> \u4e2a FP32 \u4e3b\u6743\u91cd\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u5982\u679c <code>has_nan<\/code> \u6807\u5fd7\u975e\u96f6\uff0c\u4f18\u5316\u5668\u5185\u6838\u7b2c\u4e00\u884c\u5c31\u76f4\u63a5 <code>return<\/code>\uff0c\u4e0d\u4f1a\u4fee\u6539\u4efb\u4f55\u4e3b\u6743\u91cd\u3002\u8fd9\u4fdd\u8bc1\u4e86 NaN \u68af\u5ea6\u88ab\u9694\u79bb\u5728\u5355\u6b21\u8fed\u4ee3\u5185\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. NaN \u68c0\u6d4b\u4e0e\u68af\u5ea6\u88c1\u526a<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><code>RANGE_CHECK_NAN<\/code> \u626b\u63cf\u6574\u4e2a FP32 \u68af\u5ea6\u533a <code>G_BN_BIAS..G_DEEP_CONV<\/code>\u3002\u7531\u4e8e MemoryPlan \u7684\u8bed\u4e49\u5206\u533a\uff0c\u6240\u6709\u53c2\u4e0e\u8bad\u7ec3\u7684\u68af\u5ea6\u5728\u5185\u5b58\u4e2d\u8fde\u7eed\u6392\u5e03\uff0c\u4e00\u4e2a RangeOp \u5c31\u80fd\u8986\u76d6\u5168\u90e8\u3002<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ src\/graph\/compiler.cpp\nnode.range_op = RangeOp::RANGE_CHECK_NAN;\nnode.input_ranges.push_back(\n    memory_plan.region_range(Region::G_BN_BIAS, Region::G_DEEP_CONV));\nnode.output_ids.push_back(nan_flag_id);<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><code>check_op.cu<\/code> \u5b9e\u73b0\u4e86\u4e24\u4e2a\u7248\u672c\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ \u4ec5\u68c0\u6d4b\n__global__ void check_nan_kernel(\n    volatile int32_t* __restrict__ has_nan_flag,\n    const float* __restrict__ data, size_t n)\n{\n    \/\/ \u4efb\u4e00\u5143\u7d20\u4e3a NaN \u6216 Inf\uff0c\u8bbe\u7f6e has_nan_flag\n}\n\n\/\/ \u68c0\u6d4b + \u88c1\u526a\n__global__ void check_nan_and_clip_kernel(\n    volatile int32_t* __restrict__ has_nan_flag,\n    float* __restrict__ data, size_t n, float clip_val)\n{\n    \/\/ NaN\uff1a\u8bbe\u7f6e\u6807\u5fd7\uff0c\u4e0d\u4fee\u6539\u503c\uff1b\n    \/\/ \u5176\u4ed6\u503c\uff1aclamp \u5230 [-clip_val, +clip_val]\uff0cInf \u4e5f\u4f1a\u88ab clamp\u3002\n}<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u5728<strong>\u4e0d\u5f00\u542f\u68af\u5ea6\u88c1\u526a<\/strong>\u65f6\uff0c<code>isnan<\/code> \u548c <code>isinf<\/code> \u90fd\u4f1a\u8bbe\u7f6e <code>has_nan<\/code>\uff1b<\/li>\n\n\n\n<li>\u5728<strong>\u5f00\u542f\u68af\u5ea6\u88c1\u526a<\/strong>\u65f6\uff0c\u53ea\u6709 <code>NaN<\/code> \u8bbe\u7f6e\u6807\u5fd7\uff0c<code>Inf<\/code> \u548c\u8d85\u51fa\u8fb9\u754c\u7684\u503c\u4f1a\u88ab <code>clamp<\/code> \u5230\u8fb9\u754c\uff0c\u56e0\u4e3a Inf \u901a\u5e38\u610f\u5473\u7740\u68af\u5ea6\u8fc7\u5927\u800c\u975e\u8ba1\u7b97\u9519\u8bef\uff0c\u88c1\u526a\u540e\u4ecd\u53ef\u66f4\u65b0\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u9700\u8981\u8bf4\u660e\u7684\u662f\uff0c\u4e0a\u8ff0\u628a <code>Inf<\/code> \u5f53\u4f5c\u53ef\u88c1\u526a\u8d85\u5927\u503c\u7684\u5904\u7406\u662f Tech-Renaissance \u5f53\u524d\u5b9e\u73b0\u7684\u9009\u62e9\uff1b\u5728\u66f4\u4fdd\u5b88\u7684\u8bad\u7ec3\u914d\u7f6e\u4e2d\uff0c<code>Inf<\/code> \u5e38\u4e0e <code>NaN<\/code> \u4e00\u6837\u88ab\u89c6\u4e3a\u6570\u503c\u6ea2\u51fa\uff0c\u7edf\u4e00\u89e6\u53d1\u635f\u5931\u7f29\u653e\u4e0b\u964d\u5e76\u8df3\u8fc7\u8be5\u6b65\u66f4\u65b0\uff0c\u4ee5\u907f\u514d\u53ef\u80fd\u7684\u65b9\u5411\u5931\u771f\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u5982\u679c\u7528\u6237\u914d\u7f6e\u4e86 <code>GradClipParams.max_abs &gt; 0<\/code>\uff0c\u7f16\u8bd1\u5668\u4f1a\u628a\u9608\u503c\u5d4c\u5165\u8282\u70b9\u53c2\u6570\uff0c\u540c\u4e00\u4e2a kernel \u540c\u65f6\u5b8c\u6210\u68c0\u6d4b\u4e0e\u88c1\u526a\uff0c\u907f\u514d\u7b2c\u4e8c\u6b21\u8bfb\u53d6\u68af\u5ea6\u533a\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u4e94\u3001CBR \u878d\u5408\uff1a\u4e3a\u4ec0\u4e48\u53ea\u652f\u6301 AMP\uff1f<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u7b2c 19 \u7bc7\u4f1a\u4e13\u95e8\u8bb2 CBR\uff08Conv + BatchNorm + ReLU\uff09\u878d\u5408\u7b97\u5b50\uff0c\u4f46\u8fd9\u91cc\u5fc5\u987b\u5148\u63d0\u4e00\u53e5\uff1aTech-Renaissance \u7684 CBR \u878d\u5408\u76ee\u524d\u53ea\u5b9e\u73b0\u4e86 AMP \u7248\u672c\uff0c\u6ca1\u6709 FP32 \u7248\u672c\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u539f\u56e0\u51fa\u5728 cuDNN Frontend Graph \u7684 <code>GenStats<\/code> \u8282\u70b9\u4e0a\u3002CBR \u524d\u5411\u8981\u628a\u5377\u79ef\u8f93\u51fa\u3001BN \u7edf\u8ba1\u91cf\u751f\u6210\u3001ReLU \u6fc0\u6d3b\u5c3d\u91cf\u878d\u5408\u5728\u4e00\u8d77\u3002\u5176\u4e2d BN \u9700\u8981\u7684\u901a\u9053\u548c\u4e0e\u901a\u9053\u5e73\u65b9\u548c\uff0c\u53ef\u4ee5\u901a\u8fc7 <code>genstats<\/code> \u8282\u70b9\u5728\u5377\u79ef\u8f93\u51fa\u4e0a\u76f4\u63a5\u751f\u6210\uff1a<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"cpp\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">\/\/ cbr_op.cpp \u4e2d CBR \u524d\u5411\u56fe\u6784\u5efa\u8282\u9009\nauto genstats_opts = Genstats_attributes()\n    .set_name(\"genstats\")\n    .set_compute_data_type(fe::DataType_t::FLOAT);  \/\/ \u5185\u90e8\u8ba1\u7b97\u7cbe\u5ea6\u4ecd\u4e3a FP32\nauto genstats_outputs = graph->genstats(conv_out, genstats_opts);\nauto sum    = genstats_outputs[0];\nauto sq_sum = genstats_outputs[1];<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>conv_out<\/code>\uff1a\u5377\u79ef\u8f93\u51fa\uff0c\u6570\u636e\u7c7b\u578b\u4e3a <code>HALF<\/code>\uff08FP16\uff09\uff1b<\/li>\n\n\n\n<li><code>compute_data_type<\/code>\uff1a\u5185\u90e8\u7d2f\u52a0\u7cbe\u5ea6\u4e3a <code>FLOAT<\/code>\uff08FP32\uff09\uff0c\u4fdd\u8bc1\u6570\u503c\u7a33\u5b9a\u6027\uff1b<\/li>\n\n\n\n<li><code>sum<\/code> \/ <code>sq_sum<\/code>\uff1aBN \u6240\u9700\u7684\u901a\u9053\u7edf\u8ba1\u91cf\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><code>genstats<\/code> \u5f53\u524d\u53ea\u652f\u6301 FP16 \u8f93\u5165\u3002\u8fd9\u610f\u5473\u7740\u5982\u679c\u60f3\u7528 CBR \u878d\u5408\uff0c\u5377\u79ef\u8f93\u5165\u3001\u8f93\u51fa\u3001\u6743\u91cd\u90fd\u5fc5\u987b\u662f FP16\u3002FP32 \u6a21\u5f0f\u4e0b\u5373\u4f7f\u5b9e\u73b0\u7c7b\u4f3c\u878d\u5408\uff0c\u4e5f\u4eab\u53d7\u4e0d\u5230 Conv+GenStats \u8fd9\u6761\u4f18\u5316\u8def\u5f84\uff0c\u6027\u80fd\u53cd\u800c\u672a\u5fc5\u6bd4\u5206\u7acb\u7b97\u5b50\u66f4\u597d\u3002\u56e0\u6b64\u6846\u67b6\u505a\u4e86\u4e00\u4e2a\u52a1\u5b9e\u7684\u9009\u62e9\uff1aCBR \u878d\u5408\u7b97\u5b50\u53ea\u670d\u52a1\u4e8e AMP \u8bad\u7ec3\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">CBR \u878d\u5408\u5e26\u6765\u7684\u6536\u76ca\u975e\u5e38\u76f4\u63a5\uff1a\u7701\u53bb\u4e86\u4e00\u6b21\u5b8c\u6574\u7684 <code>conv_output<\/code> \u8bfb\u5199\u3002\u4ee5 ResNet-50 \u5178\u578b\u5c42\u4e3a\u4f8b\uff0cFP16 \u7684 conv_output \u5927\u7ea6\u51e0 MB\uff0c\u8bad\u7ec3\u4e2d\u6709\u6570\u5341\u4e2a\u8fd9\u6837\u7684\u5c42\uff0c\u6bcf\u4e2a batch \u90fd\u91cd\u590d\u2014\u2014\u8282\u7701\u7684\u663e\u5b58\u5e26\u5bbd\u7d2f\u52a0\u8d77\u6765\u76f8\u5f53\u53ef\u89c2\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u516d\u3001\u4e0e PyTorch AMP \u7684\u672c\u8d28\u5dee\u5f02<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">\u5230\u8fd9\u91cc\uff0c\u53ef\u4ee5\u6e05\u6670\u770b\u51fa Tech-Renaissance \u7684 AMP \u548c PyTorch AMP \u8d70\u7684\u662f\u4e24\u6761\u4e0d\u540c\u7684\u8def\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>PyTorch AMP<\/strong> \u662f<strong>\u7b97\u5b50\u7ea7\u3001\u52a8\u6001\u3001\u81ea\u52a8<\/strong>\u7684\uff1a\u5728 <code>autocast<\/code> \u4e0a\u4e0b\u6587\u91cc\u5199\u6a21\u578b\u4ee3\u7801\uff0cPyTorch \u8fd0\u884c\u65f6\u81ea\u52a8\u51b3\u5b9a\u6bcf\u4e2a\u7b97\u5b50\u7684\u7cbe\u5ea6\uff0c\u5e76\u7528 <code>GradScaler<\/code> \u52a8\u6001\u8c03\u6574 loss scale\u3002\u597d\u5904\u662f\u5bf9\u7528\u6237\u900f\u660e\u3001\u51e0\u4e4e\u96f6\u4fb5\u5165\uff1b\u4ee3\u4ef7\u662f\u8fd0\u884c\u65f6\u8981\u505a\u5927\u91cf\u5224\u65ad\u3001\u72b6\u6001\u7ef4\u62a4\u548c CPU-GPU \u540c\u6b65\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Tech-Renaissance AMP<\/strong> \u662f<strong>\u56fe\u7ea7\u3001\u9759\u6001\u3001\u663e\u5f0f<\/strong>\u7684\uff1a\u7cbe\u5ea6\u8f6c\u6362\u3001NaN \u68c0\u67e5\u3001\u68af\u5ea6\u7f29\u653e\u3001AllReduce \u90fd\u662f\u7f16\u8bd1\u671f\u751f\u6210\u7684\u72ec\u7acb\u5b50\u56fe\uff0c\u8fd0\u884c\u65f6\u6309\u56fa\u5b9a\u987a\u5e8f launch\u3002\u7528\u6237\u901a\u8fc7 <code>GlobalRegistry::amp(true)<\/code> \u6253\u5f00 AMP \u540e\uff0c\u540e\u7eed\u4e00\u5207\u7531\u6846\u67b6\u81ea\u52a8\u5904\u7406\uff0c\u4e0d\u9700\u8981\u624b\u52a8\u5199 cast \u6216 scale\uff1b\u4f46\u6240\u6709\u9009\u62e9\u90fd\u662f\u9884\u5148\u51b3\u5b9a\u7684\uff0c\u65e0\u6cd5\u5728\u4e2d\u9014\u52a8\u6001\u63d2\u5165\u81ea\u5b9a\u4e49\u7cbe\u5ea6\u7b56\u7565\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u8fd9\u4e24\u6761\u8def\u7ebf\u6ca1\u6709\u7edd\u5bf9\u7684\u4f18\u52a3\uff0c\u53ea\u662f\u9002\u5408\u4e0d\u540c\u7684\u8bbe\u8ba1\u76ee\u6807\u3002PyTorch AMP \u9002\u5408\u7814\u7a76\u9636\u6bb5\u7684\u7075\u6d3b\u6027\uff1bTech-Renaissance \u7684 AMP \u9002\u5408\u628a\u8bad\u7ec3\u5faa\u73af captured \u6210 CUDA Graph\u3001\u628a\u663e\u5b58\u5e03\u5c40\u9759\u6001\u5316\u3001\u628a\u6bcf\u4e00\u70b9 CPU \u5f00\u9500\u90fd\u69a8\u5e72\u7684\u573a\u666f\u3002\u5982\u679c\u8bf4 PyTorch \u7684 AMP \u662f&#8221;\u4f18\u96c5\u7684\u81ea\u52a8\u6321&#8221;\uff0c\u90a3 Tech-Renaissance \u7684 AMP \u66f4\u50cf\u662f&#8221;\u4e3a\u8d5b\u9053\u8c03\u6821\u7684\u624b\u52a8\u6321&#8221;\u2014\u2014\u4f60\u9700\u8981\u5728\u51fa\u53d1\u524d\u6302\u597d\u6321\uff0c\u4f46\u8dd1\u8d77\u6765\u540e\u6bcf\u4e2a\u6321\u4f4d\u90fd\u54ac\u5408\u5f97\u4e25\u4e1d\u5408\u7f1d\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u7ed3\u8bed<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">AMP \u81ea\u52a8\u6df7\u5408\u7cbe\u5ea6\u8bad\u7ec3\uff0c\u672c\u8d28\u4e0a\u662f\u628a FP16 \u7684\u901f\u5ea6\u3001FP32 \u7684\u7cbe\u5ea6\u548c\u4e00\u4e2a\u806a\u660e\u7684\u68af\u5ea6\u7f29\u653e\u7b56\u7565\u7f1d\u5408\u5728\u4e00\u8d77\u3002Tech-Renaissance \u7684\u5b9e\u73b0\u601d\u8def\u53ef\u4ee5\u6982\u62ec\u4e3a\uff1a<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>\u7528\u663e\u5b58\u5206\u533a\u628a FP32 \u4e3b\u6743\u91cd\u3001FP16 \u8ba1\u7b97\u6743\u91cd\u3001FP16 \u68af\u5ea6\u3001EMA \u6743\u91cd\u7269\u7406\u9694\u79bb<\/strong>\uff1b<\/li>\n\n\n\n<li><strong>\u7528 RangeOp \u628a\u7cbe\u5ea6\u8f6c\u6362\u3001NaN \u68c0\u67e5\u3001\u68af\u5ea6\u7f29\u653e\u3001AllReduce \u8868\u8fbe\u4e3a\u6279\u91cf\u56fe\u9636\u6bb5<\/strong>\uff1b<\/li>\n\n\n\n<li><strong>\u7528\u56fa\u5b9a\u635f\u5931\u7f29\u653e\u6362\u53d6\u786e\u5b9a\u6027\u3001\u4f4e\u5f00\u9500\u548c CUDA Graph \u53cb\u597d\u6027<\/strong>\uff1b<\/li>\n\n\n\n<li><strong>\u7528 CBR \u878d\u5408\u628a AMP \u7684\u6027\u80fd\u7ea2\u5229\u53d1\u6325\u5230\u6781\u81f4<\/strong>\u3002<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">\u5b83\u4e0d\u50cf PyTorch AMP \u90a3\u6837\u5bf9\u7528\u6237\u5b8c\u5168\u900f\u660e\uff0c\u4f46\u5b83\u5728\u9759\u6001\u56fe\u3001\u9759\u6001\u663e\u5b58\u3001\u9759\u6001\u591a\u6d41\u7684\u6846\u67b6\u8bed\u5883\u91cc\uff0c\u627e\u5230\u4e86\u4e00\u6761\u81ea\u6d3d\u4e14\u9ad8\u6548\u7684\u5b9e\u73b0\u8def\u5f84\u3002\u4e0b\u4e00\u7bc7\uff0c\u6211\u4eec\u5c31\u6765\u804a\u804a AMP \u6a21\u5f0f\u4e0b\u6700\u5177\u4ee3\u8868\u6027\u7684\u7b97\u5b50\u4f18\u5316\u2014\u2014CBR \u878d\u5408\uff1a\u5982\u4f55\u628a\u5377\u79ef\u3001\u6279\u5f52\u4e00\u5316\u3001ReLU \u4e09\u4e2a\u7b97\u5b50\u63c9\u6210\u4e00\u6b21 cuDNN Graph \u6267\u884c\u3002<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u2014\u2014\u201c\u4e00\u4e2a\u4eba\u7528AI\u5982\u4f55\u5199\u51fa\u6bd4PyTorch\u66f4\u5feb\u7684\u81ea\u7814\u6df1\u5ea6\u5b66\u4e60\u6846\u67b6\u201d\u7cfb\u5217\u6587\u7ae0\u4e4b\u5341\u4e5d \u6df1\u5ea6\u5b66\u4e60\u8bad\u7ec3\u957f\u671f\u9762\u4e34\u4e00\u4e2a&#038;# [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":711,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-container-style":"default","site-container-layout":"default","site-sidebar-layout":"default","disable-article-header":"default","disable-site-header":"default","disable-site-footer":"default","disable-content-area-spacing":"default","footnotes":""},"categories":[15],"tags":[],"class_list":["post-535","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-15"],"_links":{"self":[{"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/posts\/535","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=535"}],"version-history":[{"count":3,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/posts\/535\/revisions"}],"predecessor-version":[{"id":710,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/posts\/535\/revisions\/710"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/media\/711"}],"wp:attachment":[{"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=535"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=535"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tech-renaissance.cn\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=535"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}