tiiuae/falcon-mamba-7b-instruct · Inference examples not working

Aug 12

alan@duckyPC:~/falcon-mamba$ python3 main.py
The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████| 3/3 [00:17<00:00,  5.80s/it]
Traceback (most recent call last):
  File "/home/alan/falcon-mamba/main.py", line 11, in <module>
    input_ids = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True).to("cuda")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'to'
alan@duckyPC:~/falcon-mamba$

ybelkada

Technology Innovation Institute org Aug 12

Indeed, the script should be :

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-mamba-7b-instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/falcon-mamba-7b-instruct", device_map="auto")

# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
messages = [
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids, max_new_tokens=30)
print(tokenizer.decode(outputs[0]))

Will update all model cards

ybelkada changed discussion status to closed Aug 12