[Pytorch][BERT] 버트 소스코드 이해_⑦ Bert Pooler

[Pytorch][BERT] 버트 소스코드 이해 목차
BERT	📑 BERT Config
	📑 BERT Tokenizer
	📑 BERT Model	📑 BERT Input
		📑 BERT Output
		📑 BERT Embedding
		📑 BERT Pooler 👀
		📑 BERT Enocder	📑 BERT Layer	📑 BERT SelfAttention
				📑 BERT SelfOtput

 

 

BertPooler

: 마지막 hidden_state의 첫번째 토큰인 [CLS] 토큰의 embedding을 반환

• We "pool" the model by simply taking the hidden state corresponding to the first token.
• The pooling layer at the end of the BERT model. This returns an embedding for the [CLS] token, after passing it through a non-linear tanh activation; the non-linear layer is also part of the BERT model.
• https://docs.allennlp.org/main/api/modules/seq2vec_encoders/bert_pooler/

class BertPooler(nn.Module):
    def __init__(self, config):

        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states):

        first_token_tensor = **hidden_states[:, 0]**
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)

        return pooled_output