# Gpt2 token of specific string

**URL:** <https://discuss.huggingface.co/t/gpt2-token-of-specific-string/35128>\
**Category:** 🤗Transformers\
**Created:** [March 30, 2023, 6:26pm UTC](https://discuss.huggingface.co/t/gpt2-token-of-specific-string/35128 "2023-03-30T18:26:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Towen](https://avatars.discourse-cdn.com/v4/letter/t/bcef8e/32.png) [@Towen](https://discuss.huggingface.co/u/Towen)\
**Post date:** [March 30, 2023, 6:26pm UTC](https://discuss.huggingface.co/t/gpt2-token-of-specific-string/35128/1 "2023-03-30T18:26:47Z")

</div>

I have following texts to tokenize:

tokenizer = GPT2Tokenizer.from\_pretrained(‘gpt2’, bos\_token=‘\<|startoftext|\>’, eos\_token=‘\<|endoftext|\>’, pad\_token=‘\<|paf|\>’)

x =

text1 = tokenizer(‘\<|startoftext|\>’ + ‘Question. A’ + ‘\<|endoftext|\>’, truncation=True, max\_length=20, padding=“max\_length”)  
x.append(text1[‘input\_ids’])

text2 = tokenizer(‘\<|startoftext|\>’ + ‘Question. B’ + ‘\<|endoftext|\>’, truncation=True, max\_length=20, padding=“max\_length”)  
x.append(text2[‘input\_ids’])

text3 = tokenizer(‘\<|startoftext|\>’ + ‘Question. C’ + ‘\<|endoftext|\>’, truncation=True, max\_length=20, padding=“max\_length”)  
x.append(text3[‘input\_ids’])

print(x)

and this is what I got:  
[[50257, 24361, 13, 317, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258], [50257, 24361, 13, 347, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258], [50257, 24361, 13, 327, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258]]

text1, text2, text3 always have the same length. Is there anyway to output the token of ‘A’, ‘B’, ‘C’? Is it right before 50256 which is eos\_token i guess?
