devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Transformer(cpu)代码复现_transformers cpu

作者：你好赵伟 | 2024-04-02 12:31:55

赞

踩

transformers cpu

逐行注释，逐行解析。可直接运行。
code from https://github.com/graykode/nlp-tutorial/tree/master/5-1.Transformer

  1. import numpy as np
  2. import torch
  3. import torch.nn as nn
  4. import math
  5. import time
  6. import torch.optim as optim
  7. from torch.utils.data import Dataset,DataLoader
  8.  
  9.  
  10. # 13. MyDataset
  11. class MyDataset(Dataset):
  12.     # 读数据
  13.     def __init__(self, enc_inputs, dec_inputs, target_batch):
  14.         self.enc_inputs = enc_inputs
  15.         self.dec_inputs = dec_inputs
  16.         self.target_batch = target_batch
  17.  
  18.     # 返回数据长度（有几行数据）
  19.     def __len__(self):
  20.         return len(self.enc_inputs)
  21.         # return self.enc_inputs.shape[0]
  22.  
  23.     # 返回相对位置上的元素，会比make_batch函数返回的tensor数据少一个维度
  24.     def __getitem__(self, idx):
  25.         return self.enc_inputs[idx], self.dec_inputs[idx], self.target_batch[idx]
  26.  
  27.  
  28. # 12. make_batch
  29. def make_batch(sentences):
  30.     input_batch = [[src_vocab[n] for n in sentences[0].split()]]  # [[1, 2, 3, 4, 0]]
  31.     output_batch = [[tgt_vocab[n] for n in sentences[1].split()]]  # [[5, 1, 2, 3, 4]]
  32.     target_batch = [[tgt_vocab[n] for n in sentences[2].split()]]  # [[1, 2, 3, 4, 6]]
  33.     return torch.LongTensor(input_batch), torch.LongTensor(output_batch), torch.LongTensor(target_batch)
  34.  
  35.  
  36. # 11. get_attn_subsequent_mask
  37. def get_attn_subsequent_mask(seq):
  38.     attn_shape = [seq.size(0), seq.size(1), seq.size(1)]  # [1, 5, 5]
  39.     subsequence_mask = np.triu(np.ones(attn_shape), k=1)  # ndarray [1, 5, 5]
  40.     # .byte() is equivalent to self.to(torch.uint8)
  41.     subsequence_mask = torch.from_numpy(subsequence_mask).byte()  # [1, 5, 5]
  42.     return subsequence_mask
  43.  
  44.  
  45. # 10. DecoderLayer：包含三个部分，带Mask的多头自注意力层、交互注意力层、前馈神经网络
  46. class DecoderLayer(nn.Module):
  47.     def __init__(self):
  48.         super(DecoderLayer, self).__init__()
  49.         self.dec_self_attn = MultiHeadAttention()
  50.         self.dec_enc_attn = MultiHeadAttention()
  51.         self.pos_fnn = PoswiseFeedForwardNet()
  52.  
  53.     #                [1, 5, 512]  [1, 5, 512]      [1, 5, 5]           [1, 5, 5]
  54.     def forward(self, dec_inputs, enc_outputs, dec_self_attn_mask, dec_enc_attn_mask):
  55.         # dec_self_attn===[1, 8, 5, 5]  dec_outputs===[1, 5, 512]
  56.         dec_outputs, dec_self_attn = self.dec_self_attn(dec_inputs, dec_inputs, dec_inputs, dec_self_attn_mask)
  57.         # dec_enc_attn===[1, 8, 5, 5]  dec_outputs===[1, 5, 512]
  58.         dec_outputs, dec_enc_attn = self.dec_enc_attn(dec_outputs, enc_outputs, enc_outputs, dec_enc_attn_mask)
  59.         dec_outputs = self.pos_fnn(dec_outputs)  # [1, 5, 512]
  60.         return dec_outputs, dec_self_attn, dec_enc_attn
  61.  
  62.  
  63. # 9. Decoder包含三个部分：词向量Embedding、位置编码部分、(带Masked自注意力层、交互注意力层、前馈神经网络)
  64. class Decoder(nn.Module):
  65.     def __init__(self):
  66.         super(Decoder, self).__init__()
  67.         self.tgt_emb = nn.Embedding(tgt_vocab_size, d_model)  # [7, 512]
  68.         self.pos_emb = PositionalEncoding(d_model)
  69.         self.layers = nn.ModuleList([DecoderLayer() for _ in range(n_layers)])  # 10.
  70.  
  71.     def forward(self, dec_inputs, enc_inputs, enc_outputs):
  72.         dec_outputs = self.tgt_emb(dec_inputs)  # [1, 5, 512]
  73.         dec_outputs = self.pos_emb(dec_outputs.transpose(0, 1)).transpose(0, 1)  # [1, 5, 512]--->[5, 1, 512]--->[1, 5, 512]
  74.         # 
    声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/352545
    推荐阅读
    相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。