Does Context Length Matters?
The stateof.ai 2023 report highlights (page 23,24) the importance of context length in language models (LMs) and its impact on their performance. Traditionally, parameter count has been used as a proxy for a model’s capabilities, but recent research indicates that the size of the input text can sometimes constrain these capabilities. This makes context length…