[NLP][파이토치] seq2seq - Decoder(디코더)

김기현 강사님의 '자연어처리 딥러닝 캠프(파이토치편)' 책을 공부하면서 정리한 글입니다.

http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9791162241974&orderClick=LAG&Kc= 불러오는 중입니다...

[NLP][파이토치] seq2seq - Attention(어텐션)

 

* 수식으로 확인하는 Decoder

class Decoder(nn.Module):

	def __init__(self, word_vec_size, hidden_size, n_layers=4, dropout_p=.2):
        super(Decoder, self).__init__()

        # Be aware of value of 'batch_first' parameter and 'bidirectional' parameter.
        self.rnn = nn.LSTM(
            word_vec_size + hidden_size,
            hidden_size,
            num_layers=n_layers,
            dropout=dropout_p,
            bidirectional=False,
            batch_first=True,
        )

- Encoder의 LSTM과 다른점 

	차이점	Encoder	Decoder
1	nn.LSTM의 "input_size" parameter	word_vec_size	word_vec_size + hidden_size (이유: Decoder의 LSTM은에서는 t-1시점의 h~(틸다)도 함께 concat하여 계산하기 때문!)
2	bidirectional	True	False

 

 def forward(self, emb_t, h_t_1_tilde, h_t_1):
        # |emb_t| = (batch_size, 1, word_vec_size)
        # |h_t_1_tilde| = (batch_size, 1, hidden_size)
        # |h_t_1[0]| = (n_layers, batch_size, hidden_size)
        batch_size = emb_t.size(0)
        hidden_size = h_t_1[0].size(-1)

        if h_t_1_tilde is None:
            # If this is the first time-step,
            h_t_1_tilde = emb_t.new(batch_size, 1, hidden_size).zero_()

        # Input feeding trick.
        x = torch.cat([emb_t, h_t_1_tilde], dim=-1)

        # Unlike encoder, decoder must take an input for sequentially.
        y, h = self.rnn(x, h_t_1)

        return y, h