Does Qwen support 16k context, what is the best config for max_new_tokens?
#22
by Cheshire94 - opened
I see that the default max_new_tokens in the configuration file is 512. May I ask how long the context length that Qwen currently supports?
8K is supported. 16K Qwen does not perform well, and we will figure it out.
yangapku changed discussion status to closed
Currently up to 8K sequence length is supported and you can find the corresponding specifications in both tokenizer_config.json (the model_max_length key) and config.json (the max_position_embeddings key). These specifications in config files will raise warnings in the tokenization process and model forwarding respectively if there is sequence longer than 8192.