The model's input
Token splits, IDs and scores are illustrative, in the style of a vocabulary of about 100,000 tokens. Every model family has its own vocabulary, so the same text is cut differently from model to model.
The model's input
Token splits, IDs and scores are illustrative, in the style of a vocabulary of about 100,000 tokens. Every model family has its own vocabulary, so the same text is cut differently from model to model.