定义apply_rotary_pos_emb函数: 该函数接受两个参数:query_layer和rotary_pos_emb。 query_layer的形状通常为[seq_len, batch_size, num_heads, head_dim]。 rotary_pos_emb的形状通常为[seq_len, num_heads, head_dim // 2, 2],其中2代表复数的实部和虚部。 在函数内部实现旋转位置嵌入的逻辑: 首先...
Fix NameError and typo in ApplyRotaryEmbQKV_ (Dao-AILab#569) Browse files main crowsonkb committed Sep 25, 2023 Verified 1 parent 0a1d03c commit 4c8ff91 Showing 1 changed file with 3 additions and 3 deletions. Whitespace Ignore whitespace Split Unified 6 changes: 3 additions & ...
add apply_rotary_pos_emb_backward lines 235 to 237 were commented Eric-Russel added 2 commits June 24, 2024 07:06 Add apply_rotary_pos_emb_backward 3c7440b Add apply_rotary_pos_emb_backward 33e699f silencelamb approved these changes Jun 25, 2024 View reviewed changes silencelamb merge...