fix: update tokenizer config to support `add_generation_prompt=True` and clarify content

#57

Here I have updated it so that for each role it will return the correct tokens id of that role more clearly, trim the spaces and support add_generation_prompt config.

nguyenbh changed pull request status to closed

Sign up or log in to comment