Ubuntu commited on
Commit
711d022
1 Parent(s): bc36086

first commit

Browse files
LICENSE ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The glm-4-9b License
2
+
3
+ 1. 定义
4
+
5
+ “许可方”是指分发其软件的 glm-4-9b 模型团队。
6
+ “软件”是指根据本许可提供的 glm-4-9b 模型参数。
7
+
8
+ 2. 许可授予
9
+
10
+ 根据本许可的条款和条件,许可方特此授予您非排他性、全球性、不可转让、不可再许可、可撤销、免版税的版权许可。
11
+ 本许可允许您免费使用本仓库中的所有开源模型进行学术研究,对于希望将模型用于商业目的的用户,需在[这里](https://open.bigmodel.cn/mla/form)完成登记。经过登记的用户可以免费使用本模型进行商业活动,但必须遵守本许可的所有条款和条件。
12
+ 上述版权声明和本许可声明应包含在本软件的所有副本或重要部分中。
13
+ 如果您分发或提供 THUDM / 智谱AI 关于 glm-4 开源模型的材料(或其任何衍生作品),或使用其中任何材料(包括 glm-4 系列的所有开源模型)的产品或服务,您应:
14
+
15
+ (A) 随任何此类 THUDM / 智谱AI 材料提供本协议的副本;
16
+ (B) 在相关网站、用户界面、博客文章、关于页面或产品文档上突出显示 “Built with glm-4”。
17
+ 如果您使用 THUDM / 智谱AI的 glm-4 开源模型的材料来创建、训练、微调或以其他方式改进已分发或可用的 AI 模型,您还应在任何此类 AI 模型名称的开头添加 “glm-4”。
18
+
19
+ 3. 限制
20
+
21
+ 您不得出于任何军事或非法目的使用、复制、修改、合并、发布、分发、复制或创建本软件的全部或部分衍生作品。
22
+ 您不得利用本软件从事任何危害国家安全和国家统一,危害社会公共利益及公序良俗,侵犯他人商业秘密、知识产权、名誉权、肖像权、财产权等权益的行为。
23
+ 您在使用中应遵循使用地所适用的法律法规政策、道德规范等要求。
24
+
25
+ 4. 免责声明
26
+
27
+ 本软件“按原样”提供,不提供任何明示或暗示的保证,包括但不限于对适销性、特定用途的适用性和非侵权性的保证。
28
+ 在任何情况下,作者或版权持有人均不对任何索赔、损害或其他责任负责,无论是在合同诉讼、侵权行为还是其他方面,由软件或软件的使用或其他交易引起、由软件引起或与之相关
29
+ 软件。
30
+
31
+ 5. 责任限制
32
+
33
+ 除适用法律禁止的范围外,在任何情况下且根据任何法律理论,无论是基于侵权行为、疏忽、合同、责任或其他原因,任何许可方均不对您承担任何直接、间接、特殊、偶然、示范性、
34
+ 或间接损害,或任何其他商业损失,即使许可人已被告知此类损害的可能性。
35
+
36
+ 6. 争议解决
37
+
38
+ 本许可受中华人民共和国法律管辖并按其解释。 因本许可引起的或与本许可有关的任何争议应提交北京市海淀区人民法院。
39
+ 请注意,许可证可能会更新到更全面的版本。 有关许可和版权的任何问题,请通过 [email protected] 与我们联系。
40
+
41
+ 1. Definitions
42
+
43
+ “Licensor” means the glm-4-9b Model Team that distributes its Software.
44
+ “Software” means the glm-4-9b model parameters made available under this license.
45
+
46
+ 2. License
47
+
48
+ Subject to the terms and conditions of this License, Licensor hereby grants you a non-exclusive, worldwide, irrevocable, non-sublicensable, revocable, photo-free copyright license.
49
+ This license allows you to use all open source models in this repository for free for academic research. For users who wish to use the models for commercial purposes, please do so [here](https://open.bigmodel.cn/mla/form)
50
+ Complete registration. Registered users are free to use this model for commercial activities, but must comply with all terms and conditions of this license.
51
+ The copyright notice and this license notice shall be included in all copies or substantial portions of the Software.
52
+ If you distribute or provide THUDM / Zhipu AI materials on the glm-4 open source model (or any derivative works thereof), or products or services that use any materials therein (including all open source models of the glm-4 series), you should:
53
+
54
+ (A) Provide a copy of this Agreement with any such THUDM/Zhipu AI Materials;
55
+ (B) Prominently display "Built with glm-4" on the relevant website, user interface, blog post, related page or product documentation.
56
+ If you use materials from THUDM/Zhipu AI's glm-4 model to create, train, operate, or otherwise improve assigned or available AI models, you should also add "glm-4" to the beginning of any such AI model name.
57
+
58
+ 3. Restrictions
59
+
60
+ You are not allowed to use, copy, modify, merge, publish, distribute, copy or create all or part of the derivative works of this software for any military or illegal purposes.
61
+ You are not allowed to use this software to engage in any behavior that endangers national security and unity, endangers social public interests and public order, infringes on the rights and interests of others such as trade secrets, intellectual property rights, reputation rights, portrait rights, and property rights.
62
+ You should comply with the applicable laws, regulations, policies, ethical standards, and other requirements in the place of use during use.
63
+
64
+ 4. Disclaimer
65
+
66
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
67
+ WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
68
+ COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
69
+ OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
70
+
71
+ 5. Limitation of Liability
72
+
73
+ EXCEPT TO THE EXTENT PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER BASED IN TORT,
74
+ NEGLIGENCE, CONTRACT, LIABILITY, OR OTHERWISE WILL ANY LICENSOR BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, SPECIAL,
75
+ INCIDENTAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES, OR ANY OTHER COMMERCIAL LOSSES, EVEN IF THE LICENSOR HAS BEEN ADVISED
76
+ OF THE POSSIBILITY OF SUCH DAMAGES.
77
+
78
+ 6. Dispute Resolution
79
+
80
+ This license shall be governed and construed in accordance with the laws of People’s Republic of China. Any dispute
81
+ arising from or in connection with this License shall be submitted to Haidian District People's Court in Beijing.
82
+
83
+ Note that the license is subject to update to a more comprehensive version. For any questions related to the license and
84
+ copyright, please contact us at [email protected].
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: glm-4
4
+ license_link: ./LICENSE
5
+
6
+ language:
7
+ - zh
8
+ - en
9
+ tags:
10
+ - glm
11
+ - chatglm
12
+ - thudm
13
+
14
+
15
+ inference: false
16
+ ---
17
+
18
+ # glm-4v-9b
19
+
20
+ GLM-4-9B 是智谱 AI 推出的最新一代预训练模型 GLM-4 系列中的开源版本。
21
+ 在语义、数学、推理、代码和知识等多方面的数据集测评中,GLM-4-9B 及其人类偏好对齐的版本 GLM-4-9B-Chat 均表现出较高的性能。
22
+ 除了能进行多轮对话,GLM-4-9B-Chat 还具备网页浏览、代码执行、自定义工具调用(Function Call)和长文本推理(支持最大 128K 上下文)等高级功能。
23
+ 本代模型增加了多语言支持,支持包括日语,韩语,德语在内的 26 种语言。我们还推出了支持 1M 上下文长度(约 200 万中文字符)的模型。
24
+ 本仓库是 GLM-4-9B 的多模态开源版本 GLM-4V-9B 模型。
25
+
26
+ ## 运行模型
27
+
28
+ ```python
29
+ import torch
30
+ from PIL import Image
31
+ from transformers import AutoModelForCausalLM, AutoTokenizer
32
+
33
+ device = "cuda"
34
+
35
+ tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-4v-9b", trust_remote_code=True)
36
+
37
+ query = '描述这张图片'
38
+ image = Image.open("your image").convert('RGB')
39
+ inputs = tokenizer.apply_chat_template([{"role": "user", "image": image, "content": "描述这张图片"}],
40
+ add_generation_prompt=True, tokenize=True, return_tensors="pt",
41
+ return_dict=True) # chat mode
42
+
43
+ inputs = inputs.to(device)
44
+ model = AutoModelForCausalLM.from_pretrained(
45
+ "THUDM/glm-4v-9b",
46
+ torch_dtype=torch.bfloat16,
47
+ low_cpu_mem_usage=True,
48
+ trust_remote_code=True
49
+ ).to(device).eval()
50
+
51
+ gen_kwargs = {"max_length": 2500, "do_sample": True, "top_k": 1}
52
+ with torch.no_grad():
53
+ outputs = model.generate(**inputs, **gen_kwargs)
54
+ outputs = outputs[:, inputs['input_ids'].shape[1]:]
55
+ print(tokenizer.decode(outputs[0]))
56
+ ```
57
+
58
+ ## 协议
59
+
60
+ GLM-4 模型的权重的使用则需要遵循 [LICENSE](LICENSE)。
61
+
62
+ Rhe use of the GLM-4 model weights needs to comply with the [LICENSE](LICENSE).
63
+
64
+ ## 引用
65
+
66
+ 如果你觉得我们的工作有帮助的话,请考虑引用下列论文。
67
+
68
+ ```
69
+ @article{zeng2022glm,
70
+ title={Glm-130b: An open bilingual pre-trained model},
71
+ author={Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and others},
72
+ journal={arXiv preprint arXiv:2210.02414},
73
+ year={2022}
74
+ }
75
+ ```
76
+
77
+ ```
78
+ @inproceedings{du2022glm,
79
+ title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling},
80
+ author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},
81
+ booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
82
+ pages={320--335},
83
+ year={2022}
84
+ }
85
+ ```
config.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "THUDM/glm-4v-9b",
3
+ "model_type": "chatglm",
4
+ "architectures": [
5
+ "ChatGLMModel"
6
+ ],
7
+ "auto_map": {
8
+ "AutoConfig": "configuration_chatglm.ChatGLMConfig",
9
+ "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
10
+ "AutoModelForCausalLM": "modeling_chatglm.ChatGLMForConditionalGeneration",
11
+ "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration",
12
+ "AutoModelForSequenceClassification": "modeling_chatglm.ChatGLMForSequenceClassification"
13
+ },
14
+ "vision_config": {
15
+ "dropout_prob": 0.0,
16
+ "hidden_act": "gelu",
17
+ "in_channels": 3,
18
+ "num_hidden_layers": 63,
19
+ "hidden_size": 1792,
20
+ "patch_size": 14,
21
+ "num_heads": 16,
22
+ "intermediate_size": 15360,
23
+ "layer_norm_eps": 1e-06,
24
+ "num_positions": 6401,
25
+ "image_size": 1120,
26
+ "scaling_factor": 8
27
+ },
28
+ "add_bias_linear": false,
29
+ "add_qkv_bias": true,
30
+ "apply_query_key_layer_scaling": true,
31
+ "apply_residual_connection_post_layernorm": false,
32
+ "attention_dropout": 0.0,
33
+ "attention_softmax_in_fp32": true,
34
+ "bias_dropout_fusion": true,
35
+ "ffn_hidden_size": 13696,
36
+ "fp32_residual_connection": false,
37
+ "hidden_dropout": 0.0,
38
+ "hidden_size": 4096,
39
+ "kv_channels": 128,
40
+ "layernorm_epsilon": 0.00000015625,
41
+ "multi_query_attention": true,
42
+ "multi_query_group_num": 2,
43
+ "num_attention_heads": 32,
44
+ "num_layers": 40,
45
+ "original_rope": true,
46
+ "padded_vocab_size": 151552,
47
+ "post_layer_norm": true,
48
+ "rmsnorm": true,
49
+ "seq_length": 8192,
50
+ "use_cache": true,
51
+ "torch_dtype": "bfloat16",
52
+ "transformers_version": "4.30.2",
53
+ "tie_word_embeddings": false,
54
+ "eos_token_id": [151329, 151336, 151338],
55
+ "pad_token_id": 151329,
56
+ "boi_token_id": 151339,
57
+ "eoi_token_id": 151340
58
+ }
configuration_chatglm.py ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+
3
+
4
+ class ChatGLMConfig(PretrainedConfig):
5
+ model_type = "chatglm"
6
+
7
+ def __init__(
8
+ self,
9
+ num_layers=28,
10
+ padded_vocab_size=65024,
11
+ hidden_size=4096,
12
+ ffn_hidden_size=13696,
13
+ kv_channels=128,
14
+ num_attention_heads=32,
15
+ seq_length=2048,
16
+ hidden_dropout=0.0,
17
+ classifier_dropout=None,
18
+ attention_dropout=0.0,
19
+ layernorm_epsilon=1e-5,
20
+ rmsnorm=True,
21
+ apply_residual_connection_post_layernorm=False,
22
+ post_layer_norm=True,
23
+ add_bias_linear=False,
24
+ add_qkv_bias=False,
25
+ bias_dropout_fusion=True,
26
+ multi_query_attention=False,
27
+ multi_query_group_num=1,
28
+ rope_ratio=1,
29
+ apply_query_key_layer_scaling=True,
30
+ attention_softmax_in_fp32=True,
31
+ fp32_residual_connection=False,
32
+ pre_seq_len=None,
33
+ prefix_projection=False,
34
+ boi_token_id=None,
35
+ eoi_token_id=None,
36
+ **kwargs
37
+ ):
38
+ self.num_layers = num_layers
39
+ self.vocab_size = padded_vocab_size
40
+ self.padded_vocab_size = padded_vocab_size
41
+ self.hidden_size = hidden_size
42
+ self.ffn_hidden_size = ffn_hidden_size
43
+ self.kv_channels = kv_channels
44
+ self.num_attention_heads = num_attention_heads
45
+ self.seq_length = seq_length
46
+ self.hidden_dropout = hidden_dropout
47
+ self.classifier_dropout = classifier_dropout
48
+ self.attention_dropout = attention_dropout
49
+ self.layernorm_epsilon = layernorm_epsilon
50
+ self.rmsnorm = rmsnorm
51
+ self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm
52
+ self.post_layer_norm = post_layer_norm
53
+ self.add_bias_linear = add_bias_linear
54
+ self.add_qkv_bias = add_qkv_bias
55
+ self.bias_dropout_fusion = bias_dropout_fusion
56
+ self.multi_query_attention = multi_query_attention
57
+ self.multi_query_group_num = multi_query_group_num
58
+ self.rope_ratio = rope_ratio
59
+ self.apply_query_key_layer_scaling = apply_query_key_layer_scaling
60
+ self.attention_softmax_in_fp32 = attention_softmax_in_fp32
61
+ self.fp32_residual_connection = fp32_residual_connection
62
+ self.pre_seq_len = pre_seq_len
63
+ self.prefix_projection = prefix_projection
64
+ self.boi_token_id = boi_token_id
65
+ self.eoi_token_id = eoi_token_id
66
+ super().__init__(**kwargs)
model-00001-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f5f7c4461ec292196389492c8ba97de591d9eaed6f214608f3ee875e0a5e3442
3
+ size 1945161760
model-00002-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5413314e163920b6a3563a5a8b5016204171cf78f00c60426eec503fa12273ee
3
+ size 1815217640
model-00003-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5f069db54d8e01d0c8126abccd5eba80ad1fd9f1ecbc9c83d422609d0a971a23
3
+ size 1968291912
model-00004-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7263d40c35f58a39c3d01bd35c7b43e7fc38a29e5cf18d4cd54baafcd5995312
3
+ size 1927406992
model-00005-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cba45f7b0bda438f74b95a9beb5bfc9d0bd80015c2b67d85f585be85a0e91b7c
3
+ size 1815217672
model-00006-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7c865a6bbb37362a13cefa732193ae6c5c1151106b61bfe6d7fd730778aca808
3
+ size 1968291952
model-00007-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a55c2cb3be400a256b441946fa9d8b6402fe6f7708467325569ed23f2e530ef8
3
+ size 1927406992
model-00008-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:03056c3ed47256a63021621daa4fe1d4b90bedda4b44c8e988193aced9f19c98
3
+ size 1815217672
model-00009-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1bff55c70098795914510320a7c994d8d8c108b9e2b95c429cddee5fcef3e5d4
3
+ size 1968291952
model-00010-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c08d3cc18a5fb1c9dd2e0d90e334bad0a38db4fdfc2604e5165f8fac58e61499
3
+ size 1971932440
model-00011-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:df12f8ce20711ef2f3aa93f9c5c75b47bdb31ce5a4baff967267a63fd63e9483
3
+ size 1957054176
model-00012-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d14427429d21c245eabd966d145069247cf7c721905b18181faae4e7ec90966c
3
+ size 1982747048
model-00013-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e583e560b38e6b69e3b6b6b0d6ac1d10bc4a63534fa48021eb7089feb0b9f7f8
3
+ size 1957054280
model-00014-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:36bf2fd8ba6b480964bf724a9027a7ffbf2930fd3bb2244d6df2469a12b7f02c
3
+ size 1982747048
model-00015-of-00015.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1eb45e9f635e09a3ea4b3ee9db6ae612fce6537dd82d20acc927eb8244f21858
3
+ size 810746296
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
modeling_chatglm.py ADDED
@@ -0,0 +1,1363 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ PyTorch ChatGLM model. """
2
+ import json
3
+ import math
4
+ import copy
5
+ import warnings
6
+ import re
7
+ import sys
8
+
9
+ import torch
10
+ import torch.utils.checkpoint
11
+ import torch.nn.functional as F
12
+ from torch import nn
13
+ from torch.nn import CrossEntropyLoss, LayerNorm, MSELoss, BCEWithLogitsLoss
14
+ from torch.nn.utils import skip_init
15
+ from typing import Optional, Tuple, Union, List, Callable, Dict, Any
16
+ from copy import deepcopy
17
+
18
+ from transformers.modeling_outputs import (
19
+ BaseModelOutputWithPast,
20
+ CausalLMOutputWithPast,
21
+ SequenceClassifierOutputWithPast,
22
+ )
23
+ from transformers.modeling_utils import PreTrainedModel
24
+ from transformers.utils import logging
25
+ from transformers.generation.logits_process import LogitsProcessor
26
+ from transformers.generation.utils import LogitsProcessorList, StoppingCriteriaList, GenerationConfig, ModelOutput
27
+
28
+ from .configuration_chatglm import ChatGLMConfig
29
+ from .visual import EVA2CLIPModel
30
+
31
+ # flags required to enable jit fusion kernels
32
+
33
+ if sys.platform != 'darwin':
34
+ torch._C._jit_set_profiling_mode(False)
35
+ torch._C._jit_set_profiling_executor(False)
36
+ torch._C._jit_override_can_fuse_on_cpu(True)
37
+ torch._C._jit_override_can_fuse_on_gpu(True)
38
+
39
+ logger = logging.get_logger(__name__)
40
+
41
+ LANGUAGE_TOKEN_TYPE = 0
42
+ VISION_TOKEN_TYPE = 1
43
+
44
+ _CHECKPOINT_FOR_DOC = "THUDM/ChatGLM"
45
+ _CONFIG_FOR_DOC = "ChatGLMConfig"
46
+
47
+ CHATGLM_6B_PRETRAINED_MODEL_ARCHIVE_LIST = [
48
+ "THUDM/chatglm3-6b",
49
+ # See all ChatGLM models at https://huggingface.co/models?filter=chatglm
50
+ ]
51
+
52
+
53
+ def default_init(cls, *args, **kwargs):
54
+ return cls(*args, **kwargs)
55
+
56
+
57
+ class InvalidScoreLogitsProcessor(LogitsProcessor):
58
+ def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
59
+ if torch.isnan(scores).any() or torch.isinf(scores).any():
60
+ scores.zero_()
61
+ scores[..., 198] = 5e4
62
+ return scores
63
+
64
+
65
+ class PrefixEncoder(torch.nn.Module):
66
+ """
67
+ The torch.nn model to encode the prefix
68
+ Input shape: (batch-size, prefix-length)
69
+ Output shape: (batch-size, prefix-length, 2*layers*hidden)
70
+ """
71
+
72
+ def __init__(self, config: ChatGLMConfig):
73
+ super().__init__()
74
+ self.prefix_projection = config.prefix_projection
75
+ if self.prefix_projection:
76
+ # Use a two-layer MLP to encode the prefix
77
+ kv_size = config.num_layers * config.kv_channels * config.multi_query_group_num * 2
78
+ self.embedding = torch.nn.Embedding(config.pre_seq_len, kv_size)
79
+ self.trans = torch.nn.Sequential(
80
+ torch.nn.Linear(kv_size, config.hidden_size),
81
+ torch.nn.Tanh(),
82
+ torch.nn.Linear(config.hidden_size, kv_size)
83
+ )
84
+ else:
85
+ self.embedding = torch.nn.Embedding(config.pre_seq_len,
86
+ config.num_layers * config.kv_channels * config.multi_query_group_num * 2)
87
+
88
+ def forward(self, prefix: torch.Tensor):
89
+ if self.prefix_projection:
90
+ prefix_tokens = self.embedding(prefix)
91
+ past_key_values = self.trans(prefix_tokens)
92
+ else:
93
+ past_key_values = self.embedding(prefix)
94
+ return past_key_values
95
+
96
+
97
+ def split_tensor_along_last_dim(
98
+ tensor: torch.Tensor,
99
+ num_partitions: int,
100
+ contiguous_split_chunks: bool = False,
101
+ ) -> List[torch.Tensor]:
102
+ """Split a tensor along its last dimension.
103
+
104
+ Arguments:
105
+ tensor: input tensor.
106
+ num_partitions: number of partitions to split the tensor
107
+ contiguous_split_chunks: If True, make each chunk contiguous
108
+ in memory.
109
+
110
+ Returns:
111
+ A list of Tensors
112
+ """
113
+ # Get the size and dimension.
114
+ last_dim = tensor.dim() - 1
115
+ last_dim_size = tensor.size()[last_dim] // num_partitions
116
+ # Split.
117
+ tensor_list = torch.split(tensor, last_dim_size, dim=last_dim)
118
+ # Note: torch.split does not create contiguous tensors by default.
119
+ if contiguous_split_chunks:
120
+ return tuple(chunk.contiguous() for chunk in tensor_list)
121
+
122
+ return tensor_list
123
+
124
+
125
+ class RotaryEmbedding(nn.Module):
126
+ def __init__(self, dim, rope_ratio=1, original_impl=False, device=None, dtype=None):
127
+ super().__init__()
128
+ inv_freq = 1.0 / (10000 ** (torch.arange(0, dim, 2, device=device).to(dtype=dtype) / dim))
129
+ self.register_buffer("inv_freq", inv_freq)
130
+ self.dim = dim
131
+ self.original_impl = original_impl
132
+ self.rope_ratio = rope_ratio
133
+
134
+ def impl(self, seq_length: int, dim: int, device: torch.device, dtype: torch.dtype):
135
+ base = 10000 * self.rope_ratio
136
+ inv_freq = 1.0 / (
137
+ base ** (torch.arange(0, dim, 2, device=device, dtype=torch.float32) / dim))
138
+ seq = torch.arange(seq_length, device=inv_freq.device, dtype=torch.float32)
139
+ freqs = torch.outer(seq, inv_freq)
140
+ # first part even vector components, second part odd vector components,
141
+ # 2 * dim in dimension size
142
+ emb = torch.cat((freqs, freqs), dim=-1)
143
+ return emb
144
+
145
+ def forward_impl(
146
+ self, seq_len: int, n_elem: int, dtype: torch.dtype, device: torch.device, base: int = 10000
147
+ ):
148
+ """Enhanced Transformer with Rotary Position Embedding.
149
+
150
+ Derived from: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/
151
+ transformers/rope/__init__.py. MIT License:
152
+ https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/license.
153
+ """
154
+ # $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$
155
+ base = base * self.rope_ratio
156
+ theta = 1.0 / (base ** (torch.arange(0, n_elem, 2, dtype=torch.float, device=device) / n_elem))
157
+
158
+ # Create position indexes `[0, 1, ..., seq_len - 1]`
159
+ seq_idx = torch.arange(seq_len, dtype=torch.float, device=device)
160
+
161
+ # Calculate the product of position index and $\theta_i$
162
+ idx_theta = torch.outer(seq_idx, theta).float()
163
+
164
+ cache = torch.stack([torch.cos(idx_theta), torch.sin(idx_theta)], dim=-1)
165
+
166
+ # this is to mimic the behaviour of complex32, else we will get different results
167
+ if dtype in (torch.float16, torch.bfloat16, torch.int8):
168
+ cache = cache.bfloat16() if dtype == torch.bfloat16 else cache.half()
169
+ return cache
170
+
171
+ def forward(self, max_seq_len, offset=0):
172
+ if self.original_impl:
173
+ return self.forward_impl(
174
+ max_seq_len, self.dim, dtype=self.inv_freq.dtype, device=self.inv_freq.device
175
+ )
176
+ else:
177
+ return self.impl(max_seq_len, self.dim, dtype=self.inv_freq.dtype, device=self.inv_freq.device)
178
+
179
+
180
+ @torch.jit.script
181
+ def apply_rotary_pos_emb(x: torch.Tensor, rope_cache: torch.Tensor) -> torch.Tensor:
182
+ # x: [b, np, sq, hn]
183
+ b, np, sq, hn = x.size(0), x.size(1), x.size(2), x.size(3)
184
+ rot_dim = rope_cache.shape[-2] * 2
185
+ x, x_pass = x[..., :rot_dim], x[..., rot_dim:]
186
+ # truncate to support variable sizes
187
+ rope_cache = rope_cache[:, :sq]
188
+ xshaped = x.reshape(b, np, sq, rot_dim // 2, 2)
189
+ rope_cache = rope_cache.view(-1, 1, sq, xshaped.size(3), 2)
190
+ x_out2 = torch.stack(
191
+ [
192
+ xshaped[..., 0] * rope_cache[..., 0] - xshaped[..., 1] * rope_cache[..., 1],
193
+ xshaped[..., 1] * rope_cache[..., 0] + xshaped[..., 0] * rope_cache[..., 1],
194
+ ],
195
+ -1,
196
+ )
197
+ x_out2 = x_out2.flatten(3)
198
+ return torch.cat((x_out2, x_pass), dim=-1)
199
+
200
+
201
+ class RMSNorm(torch.nn.Module):
202
+ def __init__(self, normalized_shape, eps=1e-5, device=None, dtype=None, **kwargs):
203
+ super().__init__()
204
+ self.weight = torch.nn.Parameter(torch.empty(normalized_shape, device=device, dtype=dtype))
205
+ self.eps = eps
206
+
207
+ def forward(self, hidden_states: torch.Tensor):
208
+ input_dtype = hidden_states.dtype
209
+ variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)
210
+ hidden_states = hidden_states * torch.rsqrt(variance + self.eps)
211
+
212
+ return (self.weight * hidden_states).to(input_dtype)
213
+
214
+
215
+ class CoreAttention(torch.nn.Module):
216
+ def __init__(self, config: ChatGLMConfig, layer_number):
217
+ super(CoreAttention, self).__init__()
218
+
219
+ self.apply_query_key_layer_scaling = config.apply_query_key_layer_scaling
220
+ self.attention_softmax_in_fp32 = config.attention_softmax_in_fp32
221
+ if self.apply_query_key_layer_scaling:
222
+ self.attention_softmax_in_fp32 = True
223
+ self.layer_number = max(1, layer_number)
224
+
225
+ projection_size = config.kv_channels * config.num_attention_heads
226
+
227
+ # Per attention head and per partition values.
228
+ self.hidden_size_per_partition = projection_size
229
+ self.hidden_size_per_attention_head = projection_size // config.num_attention_heads
230
+ self.num_attention_heads_per_partition = config.num_attention_heads
231
+
232
+ coeff = None
233
+ self.norm_factor = math.sqrt(self.hidden_size_per_attention_head)
234
+ if self.apply_query_key_layer_scaling:
235
+ coeff = self.layer_number
236
+ self.norm_factor *= coeff
237
+ self.coeff = coeff
238
+
239
+ self.attention_dropout = torch.nn.Dropout(config.attention_dropout)
240
+
241
+ def forward(self, query_layer, key_layer, value_layer, attention_mask):
242
+ pytorch_major_version = int(torch.__version__.split('.')[0])
243
+ if pytorch_major_version >= 2:
244
+ if attention_mask is None and query_layer.shape[2] == key_layer.shape[2]:
245
+ context_layer = torch.nn.functional.scaled_dot_product_attention(query_layer, key_layer, value_layer,
246
+ is_causal=True)
247
+ else:
248
+ if attention_mask is not None:
249
+ attention_mask = ~attention_mask
250
+ context_layer = torch.nn.functional.scaled_dot_product_attention(query_layer, key_layer, value_layer,
251
+ attention_mask)
252
+ context_layer = context_layer.transpose(1, 2).contiguous()
253
+ new_context_layer_shape = context_layer.size()[:-2] + (self.hidden_size_per_partition,)
254
+ context_layer = context_layer.reshape(*new_context_layer_shape)
255
+ else:
256
+ # Raw attention scores
257
+
258
+ # [b, np, sq, sk]
259
+ output_size = (query_layer.size(0), query_layer.size(1), query_layer.size(2), key_layer.size(2))
260
+
261
+ # [b, np, sq, hn] -> [b * np, sq, hn]
262
+ query_layer = query_layer.view(output_size[0] * output_size[1], output_size[2], -1)
263
+ # [b, np, sk, hn] -> [b * np, sk, hn]
264
+ key_layer = key_layer.view(output_size[0] * output_size[1], output_size[3], -1)
265
+
266
+ # preallocting input tensor: [b * np, sq, sk]
267
+ matmul_input_buffer = torch.empty(
268
+ output_size[0] * output_size[1], output_size[2], output_size[3], dtype=query_layer.dtype,
269
+ device=query_layer.device
270
+ )
271
+
272
+ # Raw attention scores. [b * np, sq, sk]
273
+ matmul_result = torch.baddbmm(
274
+ matmul_input_buffer,
275
+ query_layer, # [b * np, sq, hn]
276
+ key_layer.transpose(1, 2), # [b * np, hn, sk]
277
+ beta=0.0,
278
+ alpha=(1.0 / self.norm_factor),
279
+ )
280
+
281
+ # change view to [b, np, sq, sk]
282
+ attention_scores = matmul_result.view(*output_size)
283
+
284
+ # ===========================
285
+ # Attention probs and dropout
286
+ # ===========================
287
+
288
+ # attention scores and attention mask [b, np, sq, sk]
289
+ if self.attention_softmax_in_fp32:
290
+ attention_scores = attention_scores.float()
291
+ if self.coeff is not None:
292
+ attention_scores = attention_scores * self.coeff
293
+ if attention_mask is None and attention_scores.shape[2] == attention_scores.shape[3]:
294
+ attention_mask = torch.ones(output_size[0], 1, output_size[2], output_size[3],
295
+ device=attention_scores.device, dtype=torch.bool)
296
+ attention_mask.tril_()
297
+ attention_mask = ~attention_mask
298
+ if attention_mask is not None:
299
+ attention_scores = attention_scores.masked_fill(attention_mask, float("-inf"))
300
+ attention_probs = F.softmax(attention_scores, dim=-1)
301
+ attention_probs = attention_probs.type_as(value_layer)
302
+
303
+ # This is actually dropping out entire tokens to attend to, which might
304
+ # seem a bit unusual, but is taken from the original Transformer paper.
305
+ attention_probs = self.attention_dropout(attention_probs)
306
+ # =========================
307
+ # Context layer. [sq, b, hp]
308
+ # =========================
309
+
310
+ # value_layer -> context layer.
311
+ # [sk, b, np, hn] --> [b, np, sq, hn]
312
+
313
+ # context layer shape: [b, np, sq, hn]
314
+ output_size = (value_layer.size(1), value_layer.size(2), query_layer.size(0), value_layer.size(3))
315
+ # change view [b * np, sk, hn]
316
+ value_layer = value_layer.view(output_size[0] * output_size[1], value_layer.size(2), -1)
317
+ # change view [b * np, sq, sk]
318
+ attention_probs = attention_probs.view(output_size[0] * output_size[1], output_size[2], -1)
319
+ # matmul: [b * np, sq, hn]
320
+ context_layer = torch.bmm(attention_probs, value_layer)
321
+ # change view [b, np, sq, hn]
322
+ context_layer = context_layer.view(*output_size)
323
+ # [b, np, sq, hn] --> [b, sq, np, hn]
324
+ context_layer = context_layer.transpose(1, 2).contiguous()
325
+ # [b, sq, np, hn] --> [b, sq, hp]
326
+ new_context_layer_shape = context_layer.size()[:-2] + (self.hidden_size_per_partition,)
327
+ context_layer = context_layer.reshape(*new_context_layer_shape)
328
+
329
+ return context_layer
330
+
331
+
332
+ class SelfAttention(torch.nn.Module):
333
+ """Parallel self-attention layer abstract class.
334
+
335
+ Self-attention layer takes input with size [s, b, h]
336
+ and returns output of the same size.
337
+ """
338
+
339
+ def __init__(self, config: ChatGLMConfig, layer_number, device=None):
340
+ super(SelfAttention, self).__init__()
341
+ self.layer_number = max(1, layer_number)
342
+
343
+ self.projection_size = config.kv_channels * config.num_attention_heads
344
+
345
+ # Per attention head and per partition values.
346
+ self.hidden_size_per_attention_head = self.projection_size // config.num_attention_heads
347
+ self.num_attention_heads_per_partition = config.num_attention_heads
348
+
349
+ self.multi_query_attention = config.multi_query_attention
350
+ self.qkv_hidden_size = 3 * self.projection_size
351
+ self.original_rope = config.original_rope
352
+ if self.multi_query_attention:
353
+ self.num_multi_query_groups_per_partition = config.multi_query_group_num
354
+ self.qkv_hidden_size = (
355
+ self.projection_size + 2 * self.hidden_size_per_attention_head * config.multi_query_group_num
356
+ )
357
+ self.query_key_value = nn.Linear(config.hidden_size, self.qkv_hidden_size,
358
+ bias=config.add_bias_linear or config.add_qkv_bias,
359
+ device=device, **_config_to_kwargs(config)
360
+ )
361
+
362
+ self.core_attention = CoreAttention(config, self.layer_number)
363
+
364
+ # Output.
365
+ self.dense = nn.Linear(self.projection_size, config.hidden_size, bias=config.add_bias_linear,
366
+ device=device, **_config_to_kwargs(config)
367
+ )
368
+
369
+ def _allocate_memory(self, inference_max_sequence_len, batch_size, device=None, dtype=None):
370
+ if self.multi_query_attention:
371
+ num_attention_heads = self.num_multi_query_groups_per_partition
372
+ else:
373
+ num_attention_heads = self.num_attention_heads_per_partition
374
+ return torch.empty(
375
+ inference_max_sequence_len,
376
+ batch_size,
377
+ num_attention_heads,
378
+ self.hidden_size_per_attention_head,
379
+ dtype=dtype,
380
+ device=device,
381
+ )
382
+
383
+ def forward(
384
+ self, hidden_states, attention_mask, rotary_pos_emb, kv_cache=None, use_cache=True
385
+ ):
386
+ # hidden_states: [b, sq, h]
387
+
388
+ # =================================================
389
+ # Pre-allocate memory for key-values for inference.
390
+ # =================================================
391
+ # =====================
392
+ # Query, Key, and Value
393
+ # =====================
394
+
395
+ # Attention heads [b, sq, h] --> [b, sq, (np * 3 * hn)]
396
+ mixed_x_layer = self.query_key_value(hidden_states)
397
+
398
+ if self.multi_query_attention:
399
+ (query_layer, key_layer, value_layer) = mixed_x_layer.split(
400
+ [
401
+ self.num_attention_heads_per_partition * self.hidden_size_per_attention_head,
402
+ self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head,
403
+ self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head,
404
+ ],
405
+ dim=-1,
406
+ )
407
+ query_layer = query_layer.view(
408
+ query_layer.size()[:-1] + (self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
409
+ )
410
+ key_layer = key_layer.view(
411
+ key_layer.size()[:-1] + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head)
412
+ )
413
+ value_layer = value_layer.view(
414
+ value_layer.size()[:-1]
415
+ + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head)
416
+ )
417
+ else:
418
+ new_tensor_shape = mixed_x_layer.size()[:-1] + \
419
+ (self.num_attention_heads_per_partition,
420
+ 3 * self.hidden_size_per_attention_head)
421
+ mixed_x_layer = mixed_x_layer.view(*new_tensor_shape)
422
+
423
+ # [b, sq, np, 3 * hn] --> 3 [b, sq, np, hn]
424
+ (query_layer, key_layer, value_layer) = split_tensor_along_last_dim(mixed_x_layer, 3)
425
+
426
+ # [b, sq, np, hn] -> [b, np, sq, hn]
427
+ query_layer, key_layer, value_layer = [k.transpose(1, 2) for k in [query_layer, key_layer, value_layer]]
428
+
429
+ # apply relative positional encoding (rotary embedding)
430
+ if rotary_pos_emb is not None:
431
+ query_layer = apply_rotary_pos_emb(query_layer, rotary_pos_emb)
432
+ key_layer = apply_rotary_pos_emb(key_layer, rotary_pos_emb)
433
+
434
+ # adjust key and value for inference
435
+ if kv_cache is not None:
436
+ cache_k, cache_v = kv_cache
437
+ key_layer = torch.cat((cache_k, key_layer), dim=2)
438
+ value_layer = torch.cat((cache_v, value_layer), dim=2)
439
+ if use_cache:
440
+ kv_cache = (key_layer, value_layer)
441
+ else:
442
+ kv_cache = None
443
+
444
+ if self.multi_query_attention:
445
+ key_layer = key_layer.unsqueeze(2)
446
+ key_layer = key_layer.expand(
447
+ -1, -1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, -1, -1
448
+ )
449
+ key_layer = key_layer.contiguous().view(
450
+ key_layer.size()[:1] + (self.num_attention_heads_per_partition,) + key_layer.size()[3:]
451
+ )
452
+ value_layer = value_layer.unsqueeze(2)
453
+ value_layer = value_layer.expand(
454
+ -1, -1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, -1, -1
455
+ )
456
+ value_layer = value_layer.contiguous().view(
457
+ value_layer.size()[:1] + (self.num_attention_heads_per_partition,) + value_layer.size()[3:]
458
+ )
459
+
460
+ # ==================================
461
+ # core attention computation
462
+ # ==================================
463
+
464
+ context_layer = self.core_attention(query_layer, key_layer, value_layer, attention_mask)
465
+
466
+ # =================
467
+ # Output. [sq, b, h]
468
+ # =================
469
+
470
+ output = self.dense(context_layer)
471
+
472
+ return output, kv_cache
473
+
474
+
475
+ def _config_to_kwargs(args):
476
+ common_kwargs = {
477
+ "dtype": args.torch_dtype,
478
+ }
479
+ return common_kwargs
480
+
481
+
482
+ class MLP(torch.nn.Module):
483
+ """MLP.
484
+
485
+ MLP will take the input with h hidden state, project it to 4*h
486
+ hidden dimension, perform nonlinear transformation, and project the
487
+ state back into h hidden dimension.
488
+ """
489
+
490
+ def __init__(self, config: ChatGLMConfig, device=None):
491
+ super(MLP, self).__init__()
492
+
493
+ self.add_bias = config.add_bias_linear
494
+
495
+ # Project to 4h. If using swiglu double the output width, see https://arxiv.org/pdf/2002.05202.pdf
496
+ self.dense_h_to_4h = nn.Linear(
497
+ config.hidden_size,
498
+ config.ffn_hidden_size * 2,
499
+ bias=self.add_bias,
500
+ device=device,
501
+ **_config_to_kwargs(config)
502
+ )
503
+
504
+ def swiglu(x):
505
+ x = torch.chunk(x, 2, dim=-1)
506
+ return F.silu(x[0]) * x[1]
507
+
508
+ self.activation_func = swiglu
509
+
510
+ # Project back to h.
511
+ self.dense_4h_to_h = nn.Linear(
512
+ config.ffn_hidden_size,
513
+ config.hidden_size,
514
+ bias=self.add_bias,
515
+ device=device,
516
+ **_config_to_kwargs(config)
517
+ )
518
+
519
+ def forward(self, hidden_states):
520
+ # [s, b, 4hp]
521
+ intermediate_parallel = self.dense_h_to_4h(hidden_states)
522
+ intermediate_parallel = self.activation_func(intermediate_parallel)
523
+ # [s, b, h]
524
+ output = self.dense_4h_to_h(intermediate_parallel)
525
+ return output
526
+
527
+
528
+ class GLMBlock(torch.nn.Module):
529
+ """A single transformer layer.
530
+
531
+ Transformer layer takes input with size [s, b, h] and returns an
532
+ output of the same size.
533
+ """
534
+
535
+ def __init__(self, config: ChatGLMConfig, layer_number, device=None):
536
+ super(GLMBlock, self).__init__()
537
+ self.layer_number = layer_number
538
+
539
+ self.apply_residual_connection_post_layernorm = config.apply_residual_connection_post_layernorm
540
+
541
+ self.fp32_residual_connection = config.fp32_residual_connection
542
+
543
+ LayerNormFunc = RMSNorm if config.rmsnorm else LayerNorm
544
+ # Layernorm on the input data.
545
+ self.input_layernorm = LayerNormFunc(config.hidden_size, eps=config.layernorm_epsilon, device=device,
546
+ dtype=config.torch_dtype)
547
+
548
+ # Self attention.
549
+ self.self_attention = SelfAttention(config, layer_number, device=device)
550
+ self.hidden_dropout = config.hidden_dropout
551
+
552
+ # Layernorm on the attention output
553
+ self.post_attention_layernorm = LayerNormFunc(config.hidden_size, eps=config.layernorm_epsilon, device=device,
554
+ dtype=config.torch_dtype)
555
+
556
+ # MLP
557
+ self.mlp = MLP(config, device=device)
558
+
559
+ def forward(
560
+ self, hidden_states, attention_mask, rotary_pos_emb, kv_cache=None, use_cache=True,
561
+ ):
562
+ # hidden_states: [s, b, h]
563
+
564
+ # Layer norm at the beginning of the transformer layer.
565
+ layernorm_output = self.input_layernorm(hidden_states)
566
+ # Self attention.
567
+ attention_output, kv_cache = self.self_attention(
568
+ layernorm_output,
569
+ attention_mask,
570
+ rotary_pos_emb,
571
+ kv_cache=kv_cache,
572
+ use_cache=use_cache
573
+ )
574
+
575
+ # Residual connection.
576
+ if self.apply_residual_connection_post_layernorm:
577
+ residual = layernorm_output
578
+ else:
579
+ residual = hidden_states
580
+
581
+ layernorm_input = torch.nn.functional.dropout(attention_output, p=self.hidden_dropout, training=self.training)
582
+ layernorm_input = residual + layernorm_input
583
+
584
+ # Layer norm post the self attention.
585
+ layernorm_output = self.post_attention_layernorm(layernorm_input)
586
+
587
+ # MLP.
588
+ mlp_output = self.mlp(layernorm_output)
589
+
590
+ # Second residual connection.
591
+ if self.apply_residual_connection_post_layernorm:
592
+ residual = layernorm_output
593
+ else:
594
+ residual = layernorm_input
595
+
596
+ output = torch.nn.functional.dropout(mlp_output, p=self.hidden_dropout, training=self.training)
597
+ output = residual + output
598
+
599
+ return output, kv_cache
600
+
601
+
602
+ class GLMTransformer(torch.nn.Module):
603
+ """Transformer class."""
604
+
605
+ def __init__(self, config: ChatGLMConfig, device=None):
606
+ super(GLMTransformer, self).__init__()
607
+
608
+ self.fp32_residual_connection = config.fp32_residual_connection
609
+ self.post_layer_norm = config.post_layer_norm
610
+
611
+ # Number of layers.
612
+ self.num_layers = config.num_layers
613
+
614
+ # Transformer layers.
615
+ def build_layer(layer_number):
616
+ return GLMBlock(config, layer_number, device=device)
617
+
618
+ self.layers = torch.nn.ModuleList([build_layer(i + 1) for i in range(self.num_layers)])
619
+
620
+ if self.post_layer_norm:
621
+ LayerNormFunc = RMSNorm if config.rmsnorm else LayerNorm
622
+ # Final layer norm before output.
623
+ self.final_layernorm = LayerNormFunc(config.hidden_size, eps=config.layernorm_epsilon, device=device,
624
+ dtype=config.torch_dtype)
625
+
626
+ self.gradient_checkpointing = False
627
+
628
+ def _get_layer(self, layer_number):
629
+ return self.layers[layer_number]
630
+
631
+ def forward(
632
+ self, hidden_states, attention_mask, rotary_pos_emb, kv_caches=None,
633
+ use_cache: Optional[bool] = True,
634
+ output_hidden_states: Optional[bool] = False,
635
+ ):
636
+ if not kv_caches:
637
+ kv_caches = [None for _ in range(self.num_layers)]
638
+ presents = () if use_cache else None
639
+ if self.gradient_checkpointing and self.training:
640
+ if use_cache:
641
+ logger.warning_once(
642
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
643
+ )
644
+ use_cache = False
645
+
646
+ all_self_attentions = None
647
+ all_hidden_states = () if output_hidden_states else None
648
+ for index in range(self.num_layers):
649
+ if output_hidden_states:
650
+ all_hidden_states = all_hidden_states + (hidden_states,)
651
+
652
+ layer = self._get_layer(index)
653
+ if self.gradient_checkpointing and self.training:
654
+ layer_ret = torch.utils.checkpoint.checkpoint(
655
+ layer,
656
+ hidden_states,
657
+ attention_mask,
658
+ rotary_pos_emb,
659
+ kv_caches[index],
660
+ use_cache,
661
+ use_reentrant=False
662
+ )
663
+ else:
664
+ layer_ret = layer(
665
+ hidden_states,
666
+ attention_mask,
667
+ rotary_pos_emb,
668
+ kv_cache=kv_caches[index],
669
+ use_cache=use_cache
670
+ )
671
+ hidden_states, kv_cache = layer_ret
672
+ if use_cache:
673
+ presents = presents + (kv_cache,)
674
+
675
+ if output_hidden_states:
676
+ all_hidden_states = all_hidden_states + (hidden_states,)
677
+
678
+ # Final layer norm.
679
+ if self.post_layer_norm:
680
+ hidden_states = self.final_layernorm(hidden_states)
681
+
682
+ return hidden_states, presents, all_hidden_states, all_self_attentions
683
+
684
+
685
+ class ChatGLMPreTrainedModel(PreTrainedModel):
686
+ """
687
+ An abstract class to handle weights initialization and
688
+ a simple interface for downloading and loading pretrained models.
689
+ """
690
+
691
+ is_parallelizable = False
692
+ supports_gradient_checkpointing = True
693
+ config_class = ChatGLMConfig
694
+ base_model_prefix = "transformer"
695
+ _no_split_modules = ["GLMBlock"]
696
+
697
+ def _init_weights(self, module: nn.Module):
698
+ """Initialize the weights."""
699
+ return
700
+
701
+ def get_masks(self, input_ids, past_key_values, padding_mask=None):
702
+ batch_size, seq_length = input_ids.shape
703
+ full_attention_mask = torch.ones(batch_size, seq_length, seq_length, device=input_ids.device)
704
+ full_attention_mask.tril_()
705
+ past_length = 0
706
+ if past_key_values:
707
+ past_length = past_key_values[0][0].shape[2]
708
+ if past_length:
709
+ full_attention_mask = torch.cat((torch.ones(batch_size, seq_length, past_length,
710
+ device=input_ids.device), full_attention_mask), dim=-1)
711
+ if padding_mask is not None:
712
+ full_attention_mask = full_attention_mask * padding_mask.unsqueeze(1)
713
+ if not past_length and padding_mask is not None:
714
+ full_attention_mask -= padding_mask.unsqueeze(-1) - 1
715
+ full_attention_mask = (full_attention_mask < 0.5).bool()
716
+ full_attention_mask.unsqueeze_(1)
717
+ return full_attention_mask
718
+
719
+ def get_position_ids(self, input_ids, device):
720
+ batch_size, seq_length = input_ids.shape
721
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
722
+ return position_ids
723
+
724
+ def get_multimodal_position_ids(self, input_ids, device):
725
+ batch_size, seq_length = input_ids.shape
726
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
727
+
728
+ def gradient_checkpointing_enable(self, gradient_checkpointing_kwargs=None):
729
+ if not self.supports_gradient_checkpointing:
730
+ raise ValueError(f"{self.__class__.__name__} does not support gradient checkpointing.")
731
+
732
+
733
+ class Embedding(torch.nn.Module):
734
+ """Language model embeddings."""
735
+
736
+ def __init__(self, config: ChatGLMConfig, device=None):
737
+ super(Embedding, self).__init__()
738
+
739
+ self.hidden_size = config.hidden_size
740
+ # Word embeddings (parallel).
741
+ self.word_embeddings = nn.Embedding(
742
+ config.padded_vocab_size,
743
+ self.hidden_size,
744
+ dtype=config.torch_dtype,
745
+ device=device
746
+ )
747
+ self.fp32_residual_connection = config.fp32_residual_connection
748
+
749
+ def forward(self, input_ids):
750
+ # Embeddings.
751
+ words_embeddings = self.word_embeddings(input_ids)
752
+ embeddings = words_embeddings
753
+ # If the input flag for fp32 residual connection is set, convert for float.
754
+ if self.fp32_residual_connection:
755
+ embeddings = embeddings.float()
756
+ return embeddings
757
+
758
+
759
+ def is_empty(images_list: Optional[List[List[torch.Tensor]]]):
760
+ if images_list is None or len(images_list) == 0:
761
+ return True
762
+ for image_list in images_list:
763
+ if image_list is not None:
764
+ return False
765
+ return True
766
+
767
+
768
+ class ChatGLMModel(ChatGLMPreTrainedModel):
769
+ def __init__(self, config: ChatGLMConfig, device=None, empty_init=True):
770
+ super().__init__(config)
771
+ if empty_init:
772
+ init_method = skip_init
773
+ else:
774
+ init_method = default_init
775
+ init_kwargs = {}
776
+ if device is not None:
777
+ init_kwargs["device"] = device
778
+ self.embedding = init_method(Embedding, config, **init_kwargs)
779
+ self.num_layers = config.num_layers
780
+ self.multi_query_group_num = config.multi_query_group_num
781
+ self.kv_channels = config.kv_channels
782
+
783
+ # Rotary positional embeddings
784
+ self.seq_length = config.seq_length
785
+ rotary_dim = (
786
+ config.hidden_size // config.num_attention_heads if config.kv_channels is None else config.kv_channels
787
+ )
788
+
789
+ self.rotary_pos_emb = RotaryEmbedding(rotary_dim // 2, rope_ratio=config.rope_ratio,
790
+ original_impl=config.original_rope,
791
+ device=device, dtype=config.torch_dtype)
792
+ self.encoder = init_method(GLMTransformer, config, **init_kwargs)
793
+ self.output_layer = init_method(nn.Linear, config.hidden_size, config.padded_vocab_size, bias=False,
794
+ dtype=config.torch_dtype, **init_kwargs)
795
+ self.pre_seq_len = config.pre_seq_len
796
+ self.prefix_projection = config.prefix_projection
797
+ if self.pre_seq_len is not None:
798
+ for param in self.parameters():
799
+ param.requires_grad = False
800
+ self.prefix_tokens = torch.arange(self.pre_seq_len).long()
801
+ self.prefix_encoder = PrefixEncoder(config)
802
+ self.dropout = torch.nn.Dropout(0.1)
803
+
804
+ self.vision = EVA2CLIPModel(config)
805
+
806
+ def get_input_embeddings(self):
807
+ return self.embedding.word_embeddings
808
+
809
+ def set_input_embeddings(self, value):
810
+ self.embedding.word_embeddings = value
811
+
812
+ def get_prompt(self, batch_size, device, dtype=torch.half):
813
+ prefix_tokens = self.prefix_tokens.unsqueeze(0).expand(batch_size, -1).to(device)
814
+ past_key_values = self.prefix_encoder(prefix_tokens).type(dtype)
815
+ past_key_values = past_key_values.view(
816
+ batch_size,
817
+ self.pre_seq_len,
818
+ self.pre_seq_len,
819
+ self.num_layers * 2,
820
+ self.multi_query_group_num,
821
+ self.kv_channels
822
+ )
823
+ # seq_len, b, nh, hidden_size
824
+ past_key_values = self.dropout(past_key_values)
825
+ past_key_values = past_key_values.permute([2, 1, 0, 3, 4]).split(2)
826
+ return past_key_values
827
+
828
+ def forward(
829
+ self,
830
+ input_ids: torch.LongTensor = None,
831
+ images: torch.Tensor = None,
832
+ position_ids: Optional[torch.Tensor] = None,
833
+ attention_mask: Optional[torch.BoolTensor] = None,
834
+ full_attention_mask: Optional[torch.BoolTensor] = None,
835
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
836
+ inputs_embeds: Optional[torch.Tensor] = None,
837
+ use_cache: Optional[bool] = None,
838
+ output_hidden_states: Optional[bool] = None,
839
+ return_dict: Optional[bool] = None,
840
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
841
+ """take care of image_encode, position_ids and (attention_mask = None is fine)"""
842
+
843
+ # generate mode with past_key_values. the image features are already mapped
844
+ if past_key_values is None:
845
+ # not allow for inputs_embeds, because we want to process image feature
846
+ assert input_ids is not None and inputs_embeds is None, f"{input_ids} {inputs_embeds}"
847
+ if not is_empty(images): # multi-modality
848
+ image_size: int = self.config.vision_config['image_size']
849
+ patch_size: int = self.config.vision_config['patch_size']
850
+ num_patches = (image_size // patch_size // 2) ** 2
851
+ assert len(input_ids) == len(images), f"{len(input_ids)} {len(images)}"
852
+ inputs_embeds = self.embedding(input_ids)
853
+
854
+ images = images.to(dtype=inputs_embeds.dtype)
855
+ images_features = self.vision(images)
856
+
857
+ if position_ids is None:
858
+ position_ids = self.get_position_ids(input_ids, device=inputs_embeds.device)
859
+ new_input_embeds, new_position_ids = [], []
860
+
861
+ for i in range(len(input_ids)):
862
+ input_id = input_ids[i].tolist()
863
+ boi_token_pos, eoi_token_pos = input_id.index(self.config.boi_token_id), input_id.index(
864
+ self.config.eoi_token_id)
865
+ assert eoi_token_pos - boi_token_pos == 2
866
+ new_input_embeds.append(torch.cat(
867
+ (inputs_embeds[i, :boi_token_pos], images_features[i], inputs_embeds[i, eoi_token_pos + 1:])))
868
+ new_position_ids.append(torch.cat(
869
+ (position_ids[i, :boi_token_pos + 1], position_ids[i, boi_token_pos + 1].repeat(num_patches),
870
+ position_ids[i, eoi_token_pos:])
871
+ ))
872
+ inputs_embeds = torch.stack(new_input_embeds, dim=0)
873
+ position_ids = torch.stack(new_position_ids, dim=0)
874
+
875
+ output_hidden_states = (
876
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
877
+ )
878
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
879
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
880
+
881
+ batch_size, seq_length = input_ids.shape
882
+
883
+ if inputs_embeds is None:
884
+ inputs_embeds = self.embedding(input_ids)
885
+
886
+ if self.pre_seq_len is not None:
887
+ if past_key_values is None:
888
+ past_key_values = self.get_prompt(batch_size=batch_size, device=input_ids.device,
889
+ dtype=inputs_embeds.dtype)
890
+ if attention_mask is not None:
891
+ attention_mask = torch.cat([attention_mask.new_ones((batch_size, self.pre_seq_len)),
892
+ attention_mask], dim=-1)
893
+
894
+ if full_attention_mask is None:
895
+ if (attention_mask is not None and not attention_mask.all()) or (past_key_values and seq_length != 1):
896
+ full_attention_mask = self.get_masks(input_ids, past_key_values, padding_mask=attention_mask)
897
+
898
+ # Rotary positional embeddings
899
+ rotary_pos_emb = self.rotary_pos_emb(self.seq_length)
900
+ if position_ids is not None:
901
+ rotary_pos_emb = rotary_pos_emb[position_ids]
902
+ else:
903
+ rotary_pos_emb = rotary_pos_emb[None, :seq_length]
904
+
905
+ # Run encoder.
906
+ hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
907
+ inputs_embeds, full_attention_mask, rotary_pos_emb=rotary_pos_emb,
908
+ kv_caches=past_key_values, use_cache=use_cache, output_hidden_states=output_hidden_states
909
+ )
910
+
911
+ if not return_dict:
912
+ return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
913
+
914
+ return BaseModelOutputWithPast(
915
+ last_hidden_state=hidden_states,
916
+ past_key_values=presents,
917
+ hidden_states=all_hidden_states,
918
+ attentions=all_self_attentions,
919
+ )
920
+
921
+
922
+ def _history_to_prompt(history, query):
923
+ prompt = ''
924
+ flag = False
925
+ for i, (old_query, response) in enumerate(history):
926
+ prompt += ('<|user|>' if flag else '') + old_query + "<|assistant|>" + response + "<|endoftext|>"
927
+ flag = True
928
+ prompt += '{}{}<|assistant|>'.format('<|user|>' if flag else '', query)
929
+ return prompt
930
+
931
+
932
+ class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):
933
+ def __init__(self, config: ChatGLMConfig, empty_init=True, device=None):
934
+ super().__init__(config)
935
+
936
+ self.max_sequence_length = config.max_length
937
+ self.transformer = ChatGLMModel(config, empty_init=empty_init, device=device)
938
+ self.config = config
939
+
940
+ def _update_model_kwargs_for_generation(
941
+ self,
942
+ outputs: ModelOutput,
943
+ model_kwargs: Dict[str, Any],
944
+ is_encoder_decoder: bool = False,
945
+ standardize_cache_format: bool = False,
946
+ ) -> Dict[str, Any]:
947
+ # update past_key_values
948
+ model_kwargs["past_key_values"] = self._extract_past_from_model_output(
949
+ outputs, standardize_cache_format=standardize_cache_format
950
+ )
951
+
952
+ # update attention mask
953
+ if "attention_mask" in model_kwargs:
954
+ attention_mask = model_kwargs["attention_mask"]
955
+ model_kwargs["attention_mask"] = torch.cat(
956
+ [attention_mask, attention_mask.new_ones((attention_mask.shape[0], 1))], dim=-1
957
+ )
958
+
959
+ # update position ids
960
+ if "position_ids" in model_kwargs:
961
+ position_ids = model_kwargs["position_ids"]
962
+ new_position_id = position_ids[..., -1:].clone()
963
+ new_position_id += 1
964
+ model_kwargs["position_ids"] = torch.cat(
965
+ [position_ids, new_position_id], dim=-1
966
+ )
967
+
968
+ model_kwargs["is_first_forward"] = False
969
+ return model_kwargs
970
+
971
+ def prepare_inputs_for_generation(
972
+ self,
973
+ input_ids: torch.LongTensor,
974
+ images: Optional[torch.Tensor] = None,
975
+ past_key_values: Optional[torch.Tensor] = None,
976
+ attention_mask: Optional[torch.Tensor] = None,
977
+ position_ids: Optional[torch.Tensor] = None,
978
+ use_cache: Optional[bool] = None,
979
+ is_first_forward: bool = True,
980
+ **kwargs
981
+ ) -> dict:
982
+ # only last token for input_ids if past is not None
983
+ if position_ids is None:
984
+ position_ids = self.get_position_ids(input_ids, device=input_ids.device)
985
+ if not is_first_forward:
986
+ if past_key_values is not None:
987
+ position_ids = position_ids[..., -1:]
988
+ input_ids = input_ids[:, -1:]
989
+ return {
990
+ "input_ids": input_ids,
991
+ "images": images,
992
+ "past_key_values": past_key_values,
993
+ "position_ids": position_ids,
994
+ "attention_mask": attention_mask,
995
+ "return_last_logit": True,
996
+ "use_cache": use_cache
997
+ }
998
+
999
+ def forward(
1000
+ self,
1001
+ input_ids: Optional[torch.Tensor] = None,
1002
+ images: List[List[torch.Tensor]] = None,
1003
+ position_ids: Optional[torch.Tensor] = None,
1004
+ attention_mask: Optional[torch.Tensor] = None,
1005
+ past_key_values: Optional[Tuple[torch.FloatTensor]] = None,
1006
+ inputs_embeds: Optional[torch.Tensor] = None,
1007
+ labels: Optional[torch.Tensor] = None,
1008
+ use_cache: Optional[bool] = None,
1009
+ output_attentions: Optional[bool] = None,
1010
+ output_hidden_states: Optional[bool] = None,
1011
+ return_dict: Optional[bool] = None,
1012
+ return_last_logit: Optional[bool] = False,
1013
+ ):
1014
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
1015
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1016
+
1017
+ transformer_outputs = self.transformer(
1018
+ input_ids=input_ids,
1019
+ images=images,
1020
+ position_ids=position_ids,
1021
+ attention_mask=attention_mask,
1022
+ past_key_values=past_key_values,
1023
+ inputs_embeds=inputs_embeds,
1024
+ use_cache=use_cache,
1025
+ output_hidden_states=output_hidden_states,
1026
+ return_dict=return_dict,
1027
+ )
1028
+
1029
+ hidden_states = transformer_outputs[0]
1030
+ if return_last_logit:
1031
+ hidden_states = hidden_states[:, -1:]
1032
+ lm_logits = self.transformer.output_layer(hidden_states)
1033
+
1034
+ loss = None
1035
+ if labels is not None:
1036
+ lm_logits = lm_logits.to(torch.float32)
1037
+
1038
+ # Shift so that tokens < n predict n
1039
+ shift_logits = lm_logits[..., :-1, :].contiguous()
1040
+ shift_labels = labels[..., 1:].contiguous()
1041
+ # Flatten the tokens
1042
+ loss_fct = CrossEntropyLoss(ignore_index=-100)
1043
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
1044
+
1045
+ lm_logits = lm_logits.to(hidden_states.dtype)
1046
+ loss = loss.to(hidden_states.dtype)
1047
+
1048
+ if not return_dict:
1049
+ output = (lm_logits,) + transformer_outputs[1:]
1050
+ return ((loss,) + output) if loss is not None else output
1051
+
1052
+ return CausalLMOutputWithPast(
1053
+ loss=loss,
1054
+ logits=lm_logits,
1055
+ past_key_values=transformer_outputs.past_key_values,
1056
+ hidden_states=transformer_outputs.hidden_states,
1057
+ attentions=transformer_outputs.attentions,
1058
+ )
1059
+
1060
+ @staticmethod
1061
+ def _reorder_cache(
1062
+ past: Tuple[Tuple[torch.Tensor, torch.Tensor], ...], beam_idx: torch.LongTensor
1063
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor], ...]:
1064
+ """
1065
+ This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
1066
+ [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
1067
+ beam_idx at every generation step.
1068
+
1069
+ Output shares the same memory storage as `past`.
1070
+ """
1071
+ return tuple(
1072
+ (
1073
+ layer_past[0].index_select(0, beam_idx.to(layer_past[0].device)),
1074
+ layer_past[1].index_select(0, beam_idx.to(layer_past[1].device)),
1075
+ )
1076
+ for layer_past in past
1077
+ )
1078
+
1079
+ def process_response(self, output, history):
1080
+ content = ""
1081
+ history = deepcopy(history)
1082
+ for response in output.split("<|assistant|>"):
1083
+ if "\n" in response:
1084
+ metadata, content = response.split("\n", maxsplit=1)
1085
+ else:
1086
+ metadata, content = "", response
1087
+ if not metadata.strip():
1088
+ content = content.strip()
1089
+ history.append({"role": "assistant", "metadata": metadata, "content": content})
1090
+ content = content.replace("[[训练时间]]", "2023年")
1091
+ else:
1092
+ history.append({"role": "assistant", "metadata": metadata, "content": content})
1093
+ if history[0]["role"] == "system" and "tools" in history[0]:
1094
+ parameters = json.loads(content)
1095
+ content = {"name": metadata.strip(), "parameters": parameters}
1096
+ else:
1097
+ content = {"name": metadata.strip(), "content": content}
1098
+ return content, history
1099
+
1100
+ @torch.inference_mode()
1101
+ def chat(self, tokenizer, query: str, history: List[Dict] = None, role: str = "user", image=None,
1102
+ max_length: int = 8192, num_beams=1, do_sample=True, top_p=0.8, temperature=0.8, logits_processor=None,
1103
+ **kwargs):
1104
+ if history is None:
1105
+ history = []
1106
+ if logits_processor is None:
1107
+ logits_processor = LogitsProcessorList()
1108
+ logits_processor.append(InvalidScoreLogitsProcessor())
1109
+ gen_kwargs = {"max_length": max_length, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p,
1110
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
1111
+ message = {"role": role, "content": query}
1112
+ if image is not None:
1113
+ message["image"] = image
1114
+ history.append(message)
1115
+ inputs = tokenizer.apply_chat_template(history, add_generation_prompt=True, tokenize=True,
1116
+ return_tensors="pt", return_dict=True)
1117
+ inputs = inputs.to(self.device)
1118
+ eos_token_id = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|user|>"),
1119
+ tokenizer.convert_tokens_to_ids("<|observation|>")]
1120
+ outputs = self.generate(**inputs, **gen_kwargs, eos_token_id=eos_token_id)
1121
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):-1]
1122
+ response = tokenizer.decode(outputs)
1123
+ response, history = self.process_response(response, history)
1124
+ return response, history
1125
+
1126
+ @torch.inference_mode()
1127
+ def stream_chat(self, tokenizer, query: str, history: List[Dict] = None, role: str = "user", image=None,
1128
+ past_key_values=None, max_length: int = 8192, do_sample=True, top_p=0.8, temperature=0.8,
1129
+ logits_processor=None, return_past_key_values=False, **kwargs):
1130
+ if history is None:
1131
+ history = []
1132
+ if logits_processor is None:
1133
+ logits_processor = LogitsProcessorList()
1134
+ logits_processor.append(InvalidScoreLogitsProcessor())
1135
+ eos_token_id = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|user|>"),
1136
+ tokenizer.convert_tokens_to_ids("<|observation|>")]
1137
+ gen_kwargs = {"max_length": max_length, "do_sample": do_sample, "top_p": top_p,
1138
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
1139
+ message = {"role": role, "content": "query"}
1140
+ if image is not None:
1141
+ message["image"] = image
1142
+ if past_key_values is None:
1143
+ inputs = tokenizer.apply_chat_template(history + [message],
1144
+ add_generation_prompt=True, tokenize=True, return_tensors="pt",
1145
+ return_dict=True)
1146
+ else:
1147
+ inputs = tokenizer.apply_chat_template([message], add_special_tokens=False,
1148
+ add_generation_prompt=True, tokenize=True, return_tensors="pt",
1149
+ return_dict=True)
1150
+ inputs = inputs.to(self.device)
1151
+ if past_key_values is not None:
1152
+ past_length = past_key_values[0][0].shape[2]
1153
+ if self.transformer.pre_seq_len is not None:
1154
+ past_length -= self.transformer.pre_seq_len
1155
+ inputs.position_ids += past_length
1156
+ attention_mask = inputs.attention_mask
1157
+ attention_mask = torch.cat((attention_mask.new_ones(1, past_length), attention_mask), dim=1)
1158
+ inputs['attention_mask'] = attention_mask
1159
+ history.append({"role": role, "content": query})
1160
+ for outputs in self.stream_generate(**inputs, past_key_values=past_key_values,
1161
+ eos_token_id=eos_token_id, return_past_key_values=return_past_key_values,
1162
+ **gen_kwargs):
1163
+ if return_past_key_values:
1164
+ outputs, past_key_values = outputs
1165
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):-1]
1166
+ response = tokenizer.decode(outputs)
1167
+ if response and response[-1] != "�":
1168
+ response, new_history = self.process_response(response, history)
1169
+ if return_past_key_values:
1170
+ yield response, new_history, past_key_values
1171
+ else:
1172
+ yield response, new_history
1173
+
1174
+ @torch.inference_mode()
1175
+ def stream_generate(
1176
+ self,
1177
+ input_ids,
1178
+ generation_config: Optional[GenerationConfig] = None,
1179
+ logits_processor: Optional[LogitsProcessorList] = None,
1180
+ stopping_criteria: Optional[StoppingCriteriaList] = None,
1181
+ prefix_allowed_tokens_fn: Optional[Callable[[int, torch.Tensor], List[int]]] = None,
1182
+ return_past_key_values=False,
1183
+ **kwargs,
1184
+ ):
1185
+ batch_size, input_ids_seq_length = input_ids.shape[0], input_ids.shape[-1]
1186
+
1187
+ if generation_config is None:
1188
+ generation_config = self.generation_config
1189
+ generation_config = copy.deepcopy(generation_config)
1190
+ model_kwargs = generation_config.update(**kwargs)
1191
+ model_kwargs["use_cache"] = generation_config.use_cache
1192
+ bos_token_id, eos_token_id = generation_config.bos_token_id, generation_config.eos_token_id
1193
+
1194
+ if isinstance(eos_token_id, int):
1195
+ eos_token_id = [eos_token_id]
1196
+ eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None
1197
+
1198
+ has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
1199
+ if has_default_max_length and generation_config.max_new_tokens is None:
1200
+ warnings.warn(
1201
+ f"Using `max_length`'s default ({generation_config.max_length}) to control the generation length. "
1202
+ "This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we"
1203
+ " recommend using `max_new_tokens` to control the maximum length of the generation.",
1204
+ UserWarning,
1205
+ )
1206
+ elif generation_config.max_new_tokens is not None:
1207
+ generation_config.max_length = generation_config.max_new_tokens + input_ids_seq_length
1208
+ if not has_default_max_length:
1209
+ logger.warn(
1210
+ f"Both `max_new_tokens` (={generation_config.max_new_tokens}) and `max_length`(="
1211
+ f"{generation_config.max_length}) seem to have been set. `max_new_tokens` will take precedence. "
1212
+ "Please refer to the documentation for more information. "
1213
+ "(https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)",
1214
+ UserWarning,
1215
+ )
1216
+
1217
+ if input_ids_seq_length >= generation_config.max_length:
1218
+ input_ids_string = "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
1219
+ logger.warning(
1220
+ f"Input length of {input_ids_string} is {input_ids_seq_length}, but `max_length` is set to"
1221
+ f" {generation_config.max_length}. This can lead to unexpected behavior. You should consider"
1222
+ " increasing `max_new_tokens`."
1223
+ )
1224
+
1225
+ # 2. Set generation parameters if not already defined
1226
+ logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
1227
+ stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()
1228
+
1229
+ logits_processor = self._get_logits_processor(
1230
+ generation_config=generation_config,
1231
+ input_ids_seq_length=input_ids_seq_length,
1232
+ encoder_input_ids=input_ids,
1233
+ prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
1234
+ logits_processor=logits_processor,
1235
+ )
1236
+
1237
+ stopping_criteria = self._get_stopping_criteria(
1238
+ generation_config=generation_config, stopping_criteria=stopping_criteria
1239
+ )
1240
+ logits_warper = self._get_logits_warper(generation_config)
1241
+
1242
+ unfinished_sequences = input_ids.new(input_ids.shape[0]).fill_(1)
1243
+ scores = None
1244
+ while True:
1245
+ model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
1246
+ # forward pass to get next token
1247
+ outputs = self(
1248
+ **model_inputs,
1249
+ return_dict=True,
1250
+ output_attentions=False,
1251
+ output_hidden_states=False,
1252
+ )
1253
+
1254
+ next_token_logits = outputs.logits[:, -1, :]
1255
+
1256
+ # pre-process distribution
1257
+ next_token_scores = logits_processor(input_ids, next_token_logits)
1258
+ next_token_scores = logits_warper(input_ids, next_token_scores)
1259
+
1260
+ # sample
1261
+ probs = nn.functional.softmax(next_token_scores, dim=-1)
1262
+ if generation_config.do_sample:
1263
+ next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
1264
+ else:
1265
+ next_tokens = torch.argmax(probs, dim=-1)
1266
+ # update generated ids, model inputs, and length for next step
1267
+ input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
1268
+ model_kwargs = self._update_model_kwargs_for_generation(
1269
+ outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
1270
+ )
1271
+ unfinished_sequences = unfinished_sequences.mul(
1272
+ next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_tensor.unsqueeze(1)).prod(dim=0)
1273
+ )
1274
+ if return_past_key_values:
1275
+ yield input_ids, outputs.past_key_values
1276
+ else:
1277
+ yield input_ids
1278
+ # stop when each sentence is finished, or if we exceed the maximum length
1279
+ if unfinished_sequences.max() == 0 or stopping_criteria(input_ids, scores):
1280
+ break
1281
+
1282
+
1283
+ class ChatGLMForSequenceClassification(ChatGLMPreTrainedModel):
1284
+ def __init__(self, config: ChatGLMConfig, empty_init=True, device=None):
1285
+ super().__init__(config)
1286
+
1287
+ self.num_labels = config.num_labels
1288
+ self.transformer = ChatGLMModel(config, empty_init=empty_init, device=device)
1289
+
1290
+ self.classifier_head = nn.Linear(config.hidden_size, config.num_labels, bias=True, dtype=torch.half)
1291
+ if config.classifier_dropout is not None:
1292
+ self.dropout = nn.Dropout(config.classifier_dropout)
1293
+ else:
1294
+ self.dropout = None
1295
+ self.config = config
1296
+
1297
+ def forward(
1298
+ self,
1299
+ input_ids: Optional[torch.LongTensor] = None,
1300
+ position_ids: Optional[torch.LongTensor] = None,
1301
+ attention_mask: Optional[torch.Tensor] = None,
1302
+ full_attention_mask: Optional[torch.Tensor] = None,
1303
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
1304
+ inputs_embeds: Optional[torch.LongTensor] = None,
1305
+ labels: Optional[torch.LongTensor] = None,
1306
+ use_cache: Optional[bool] = None,
1307
+ output_hidden_states: Optional[bool] = None,
1308
+ return_dict: Optional[bool] = None,
1309
+ ) -> Union[Tuple[torch.Tensor, ...], SequenceClassifierOutputWithPast]:
1310
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
1311
+
1312
+ transformer_outputs = self.transformer(
1313
+ input_ids=input_ids,
1314
+ position_ids=position_ids,
1315
+ attention_mask=attention_mask,
1316
+ full_attention_mask=full_attention_mask,
1317
+ past_key_values=past_key_values,
1318
+ inputs_embeds=inputs_embeds,
1319
+ use_cache=use_cache,
1320
+ output_hidden_states=output_hidden_states,
1321
+ return_dict=return_dict,
1322
+ )
1323
+
1324
+ hidden_states = transformer_outputs[0]
1325
+ pooled_hidden_states = hidden_states[-1]
1326
+ if self.dropout is not None:
1327
+ pooled_hidden_states = self.dropout(pooled_hidden_states)
1328
+ logits = self.classifier_head(pooled_hidden_states)
1329
+
1330
+ loss = None
1331
+ if labels is not None:
1332
+ if self.config.problem_type is None:
1333
+ if self.num_labels == 1:
1334
+ self.config.problem_type = "regression"
1335
+ elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
1336
+ self.config.problem_type = "single_label_classification"
1337
+ else:
1338
+ self.config.problem_type = "multi_label_classification"
1339
+
1340
+ if self.config.problem_type == "regression":
1341
+ loss_fct = MSELoss()
1342
+ if self.num_labels == 1:
1343
+ loss = loss_fct(logits.squeeze().float(), labels.squeeze())
1344
+ else:
1345
+ loss = loss_fct(logits.float(), labels)
1346
+ elif self.config.problem_type == "single_label_classification":
1347
+ loss_fct = CrossEntropyLoss()
1348
+ loss = loss_fct(logits.view(-1, self.num_labels).float(), labels.view(-1))
1349
+ elif self.config.problem_type == "multi_label_classification":
1350
+ loss_fct = BCEWithLogitsLoss()
1351
+ loss = loss_fct(logits.float(), labels.view(-1, self.num_labels))
1352
+
1353
+ if not return_dict:
1354
+ output = (logits,) + transformer_outputs[1:]
1355
+ return ((loss,) + output) if loss is not None else output
1356
+
1357
+ return SequenceClassifierOutputWithPast(
1358
+ loss=loss,
1359
+ logits=logits,
1360
+ past_key_values=transformer_outputs.past_key_values,
1361
+ hidden_states=transformer_outputs.hidden_states,
1362
+ attentions=transformer_outputs.attentions,
1363
+ )
tokenization_chatglm.py ADDED
@@ -0,0 +1,361 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import regex as re
2
+ import base64
3
+ import os
4
+ import json
5
+ import tiktoken
6
+ import torch
7
+ from torch import TensorType
8
+ from typing import List, Optional, Union, Dict, Any
9
+ from torchvision import transforms
10
+ from transformers import PreTrainedTokenizer
11
+ from transformers.utils import logging, PaddingStrategy
12
+ from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
13
+
14
+
15
+ class ChatGLM4Tokenizer(PreTrainedTokenizer):
16
+ vocab_files_names = {"vocab_file": "tokenizer.model"}
17
+ model_input_names = ["input_ids", "attention_mask", "position_ids"]
18
+
19
+ def __init__(
20
+ self,
21
+ vocab_file,
22
+ padding_side="left",
23
+ clean_up_tokenization_spaces=False,
24
+ encode_special_tokens=False,
25
+ image_size=None,
26
+ **kwargs
27
+ ):
28
+ self.name = "GLM4Tokenizer"
29
+ self.vocab_file = vocab_file
30
+ pat_str = "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"
31
+ self.pat_str = re.compile(pat_str)
32
+ self.encode_special_tokens = encode_special_tokens
33
+ self.image_size = image_size
34
+
35
+ mergeable_ranks = {}
36
+ with open(vocab_file) as f:
37
+ for line in f:
38
+ token, rank = line.strip().split()
39
+ rank = int(rank)
40
+ token = base64.b64decode(token)
41
+ mergeable_ranks[token] = rank
42
+
43
+ self.mergeable_ranks = mergeable_ranks
44
+
45
+ self.tokenizer = tiktoken.Encoding(
46
+ name="my_tokenizer",
47
+ pat_str=pat_str,
48
+ mergeable_ranks=mergeable_ranks,
49
+ special_tokens={}
50
+ )
51
+ self.decoder = {rank: token for token, rank in mergeable_ranks.items()}
52
+ self.n_words = len(self.decoder)
53
+
54
+ super().__init__(
55
+ padding_side=padding_side,
56
+ clean_up_tokenization_spaces=clean_up_tokenization_spaces,
57
+ **kwargs
58
+ )
59
+
60
+ @property
61
+ def vocab_size(self):
62
+ return self.n_words
63
+
64
+ def get_vocab(self):
65
+ """ Returns vocab as a dict """
66
+ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
67
+ vocab.update(self.added_tokens_encoder)
68
+ return vocab
69
+
70
+ def convert_tokens_to_string(self, tokens: List[Union[bytes, str]]) -> str:
71
+ """
72
+ Converts a sequence of tokens in a single string.
73
+ """
74
+ text = ""
75
+ temp = b""
76
+ for t in tokens:
77
+ if isinstance(t, str):
78
+ if temp:
79
+ text += temp.decode("utf-8", errors="replace")
80
+ temp = b""
81
+ text += t
82
+ elif isinstance(t, bytes):
83
+ temp += t
84
+ else:
85
+ raise TypeError("token should only be of type types or str")
86
+ if temp:
87
+ text += temp.decode("utf-8", errors="replace")
88
+ return text
89
+
90
+ def _tokenize(self, text, **kwargs):
91
+ tokens = []
92
+ ids = self.tokenizer.encode(text)
93
+ for t in ids:
94
+ tokens.append(self.decoder[t])
95
+ return tokens
96
+
97
+ def _convert_token_to_id(self, token):
98
+ """ Converts a token (str) in an id using the vocab. """
99
+ return self.mergeable_ranks[token]
100
+
101
+ def _convert_id_to_token(self, index):
102
+ """Converts an index (integer) in a token (str) using the vocab."""
103
+ return self.decoder.get(index, "")
104
+
105
+ def save_vocabulary(self, save_directory, filename_prefix=None):
106
+ """
107
+ Save the vocabulary and special tokens file to a directory.
108
+
109
+ Args:
110
+ save_directory (`str`):
111
+ The directory in which to save the vocabulary.
112
+ filename_prefix (`str`, *optional*):
113
+ An optional prefix to add to the named of the saved files.
114
+
115
+ Returns:
116
+ `Tuple(str)`: Paths to the files saved.
117
+ """
118
+ if os.path.isdir(save_directory):
119
+ vocab_file = os.path.join(
120
+ save_directory, self.vocab_files_names["vocab_file"]
121
+ )
122
+ else:
123
+ vocab_file = save_directory
124
+
125
+ with open(self.vocab_file, 'rb') as fin:
126
+ proto_str = fin.read()
127
+
128
+ with open(vocab_file, "wb") as writer:
129
+ writer.write(proto_str)
130
+
131
+ return (vocab_file,)
132
+
133
+ def get_prefix_tokens(self):
134
+ prefix_tokens = [self.convert_tokens_to_ids("[gMASK]"), self.convert_tokens_to_ids("<sop>")]
135
+ return prefix_tokens
136
+
137
+ def build_single_message(self, role, metadata, message, tokenize=True, message_prefix=None):
138
+ assert role in ["system", "user", "assistant", "observation"], role
139
+ if tokenize:
140
+ role_tokens = [self.convert_tokens_to_ids(f"<|{role}|>")] + self.tokenizer.encode(f"{metadata}\n",
141
+ disallowed_special=())
142
+ message_tokens = self.tokenizer.encode(message, disallowed_special=())
143
+ if message_prefix is not None:
144
+ message_tokens = message_prefix + message_tokens
145
+ tokens = role_tokens + message_tokens
146
+ return tokens
147
+ else:
148
+ return str(f"<|{role}|>{metadata}\n{message}")
149
+
150
+ def apply_chat_template(
151
+ self,
152
+ conversation: Union[List[Dict[str, str]], List[List[Dict[str, str]]], "Conversation"],
153
+ add_generation_prompt: bool = False,
154
+ tokenize: bool = True,
155
+ padding: bool = False,
156
+ truncation: bool = False,
157
+ max_length: Optional[int] = None,
158
+ return_tensors: Optional[Union[str, TensorType]] = None,
159
+ return_dict: bool = False,
160
+ tokenizer_kwargs: Optional[Dict[str, Any]] = None,
161
+ add_special_tokens: bool = True,
162
+ **kwargs,
163
+ ) -> Union[str, List[int], List[str], List[List[int]], BatchEncoding]:
164
+
165
+ if return_dict and not tokenize:
166
+ raise ValueError(
167
+ "`return_dict=True` is incompatible with `tokenize=False`, because there is no dict "
168
+ "of tokenizer outputs to return."
169
+ )
170
+
171
+ def handle_single_conversation(conversation):
172
+ input_ids = self.get_prefix_tokens() if add_special_tokens else []
173
+ input_message = "[gMASK]<sop>" if add_special_tokens else ""
174
+ input_image = None
175
+ transform = transforms.Compose(
176
+ [
177
+ transforms.Resize(
178
+ (self.image_size, self.image_size), interpolation=transforms.InterpolationMode.BICUBIC
179
+ ),
180
+ transforms.ToTensor(),
181
+ transforms.Normalize((0.48145466, 0.4578275, 0.40821073), (0.26862954, 0.26130258, 0.27577711)),
182
+ ]
183
+ )
184
+ for item in conversation:
185
+ if item.get("tools"):
186
+ tools = item["tools"]
187
+ content = "你是一个名为 GLM-4 的人工智能助手。你是基于智谱AI训练的语言模型 GLM-4 模型开发的,你的任务是针对用户的问题和要求提供适当的答复和支持。"
188
+ for tool in tools:
189
+ if tool["type"] == "function":
190
+ function = tool["function"]
191
+ content += f"\n\n## {function['name']}\n\n{json.dumps(function, ensure_ascii=False, indent=4)}"
192
+ content += "\n在调用上述函数时,请使用 Json 格式表示调用的参数。"
193
+ elif tool["type"] == "python":
194
+ content += "\n\n## python\n\n当你向 `python` 发送包含 Python 代码的消息时,该代码将会在一个有状态的 Jupyter notebook 环境中执行。\n`python` 返回代码执行的输出,或在执行 60 秒后返回超时。\n`/mnt/data` 将会持久化存储你的文件。在此会话中,`python` 无法访问互联网。不要使用 `python` 进行任何网络请求或者在线 API 调用,这些在线内容的访问将不会成功。"
195
+ elif tool["type"] == "simple_browser":
196
+ content += "\n\n## simple_browser\n\n你可以使用 `simple_browser` 工具。该工具支持以下函数:\n`search(query: str, recency_days: int)`:使用搜索引擎进行查询并显示结果,可以使用 `recency_days` 参数控制搜索内容的时效性。\n`mclick(ids: list[int])`:获取一系列指定 id 的页面内容。每次调用时,须选择3-10个页面。选择多个角度的页面,同时尽可能选择可信任的信息来源。考虑到部分页面是无法加载的,你也可以多打开一些可能有用的页面而不用担心内容过多。\n`open_url(url: str)`:打开指定的 URL。\n\n使用 `【{引用 id}†{引用文本}】` 来引用内容。\n\n操作步骤:1. 使用 `search` 来获得信息列表; 2. 使用 `mclick` 来获取指定 ID 页面的内容; 3. 根据获得的内容进行回复。在回复中应当引用信息来源。\n 如果用户提供了 URL,也可以用 `open_url` 直接打开页面。\n如果初次搜索结果没有找到合适的信息,也可以再次使用 `search` 进行搜索。"
197
+ elif tool["type"] == "cogview":
198
+ content += "\n\n## cogview\n\n如果用户的请求中包含了对图像的描述,你可以使用 `cogview` 来生成图像并展示给用户。你需要向 `cogview` 发送图像描述,规则:\n- 发送给 `cogview` 的消息必须使用英语。用户的中文描述必须完全翻译为英语。\n- 应当尽可能详细地描述图像生成的需求,需求描述约 100 英文单词。\n- 保持用户原始描述的意图。不要虚构内容或者没见过的人物。\n- 如无特殊说明,所在地为中国,持有中国立场并遵循中国社会主义价值观。"
199
+ else:
200
+ raise NotImplementedError(f"Unknown tool type {tool['type']}")
201
+ input = self.build_single_message("system", "", content, tokenize=tokenize)
202
+ if tokenize:
203
+ input_ids.extend(input)
204
+ else:
205
+ input_message += input
206
+ message = ""
207
+ message_prefix = None
208
+ if item.get("image"):
209
+ assert input_image is None, "Multiple images are not supported"
210
+ input_image = transform(item["image"])
211
+ message_prefix = self.convert_tokens_to_ids(
212
+ ["<|begin_of_image|>", "<|endoftext|>", "<|end_of_image|>"])
213
+ if item.get("content"):
214
+ message += item["content"]
215
+ if message or message_prefix:
216
+ input = self.build_single_message(
217
+ item["role"],
218
+ item.get("metadata", ""),
219
+ message,
220
+ tokenize=tokenize,
221
+ message_prefix=message_prefix
222
+ )
223
+ if tokenize:
224
+ input_ids.extend(input)
225
+ else:
226
+ input_message += input
227
+ if add_generation_prompt:
228
+ if tokenize:
229
+ input_ids.extend([self.convert_tokens_to_ids("<|assistant|>")])
230
+ else:
231
+ input_message += "<|assistant|>"
232
+ return {"input": input_ids if tokenize else input_message, "image": input_image}
233
+
234
+ # Main logic to handle different conversation formats
235
+ if isinstance(conversation, list) and all(isinstance(i, dict) for i in conversation):
236
+ result = handle_single_conversation(conversation)
237
+ input_ids = result["input"]
238
+ input_images = [result["image"]]
239
+ elif isinstance(conversation, list) and all(isinstance(i, list) for i in conversation):
240
+ results = [handle_single_conversation(c) for c in conversation]
241
+ input_ids = [item["input"] for item in results]
242
+ input_images = [item["image"] for item in results]
243
+ elif hasattr(conversation, "messages"):
244
+ result = handle_single_conversation(conversation.messages)
245
+ input_ids = result["input"]
246
+ input_images = [result["image"]]
247
+ else:
248
+ raise ValueError("Invalid conversation format")
249
+
250
+ if tokenize:
251
+ output = self.batch_encode_plus(
252
+ [input_ids] if isinstance(input_ids[0], int) else input_ids,
253
+ padding=padding,
254
+ truncation=truncation,
255
+ max_length=max_length,
256
+ return_tensors=return_tensors,
257
+ is_split_into_words=True,
258
+ add_special_tokens=False
259
+ )
260
+ if return_dict:
261
+ found_image = False
262
+ for image in input_images:
263
+ if image is not None:
264
+ found_image = True
265
+ break
266
+ if found_image:
267
+ output["images"] = torch.stack(input_images)
268
+ return output
269
+ else:
270
+ return output["input_ids"]
271
+ else:
272
+ return input_ids
273
+
274
+
275
+ def build_inputs_with_special_tokens(
276
+ self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
277
+ ) -> List[int]:
278
+ """
279
+ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
280
+ adding special tokens. A BERT sequence has the following format:
281
+
282
+ - single sequence: `[CLS] X [SEP]`
283
+ - pair of sequences: `[CLS] A [SEP] B [SEP]`
284
+
285
+ Args:
286
+ token_ids_0 (`List[int]`):
287
+ List of IDs to which the special tokens will be added.
288
+ token_ids_1 (`List[int]`, *optional*):
289
+ Optional second list of IDs for sequence pairs.
290
+
291
+ Returns:
292
+ `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
293
+ """
294
+ prefix_tokens = self.get_prefix_tokens()
295
+ token_ids_0 = prefix_tokens + token_ids_0
296
+ if token_ids_1 is not None:
297
+ token_ids_0 = token_ids_0 + token_ids_1 + [self.convert_tokens_to_ids("<eos>")]
298
+ return token_ids_0
299
+
300
+ def _pad(
301
+ self,
302
+ encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
303
+ max_length: Optional[int] = None,
304
+ padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
305
+ pad_to_multiple_of: Optional[int] = None,
306
+ return_attention_mask: Optional[bool] = None,
307
+ ) -> dict:
308
+ """
309
+ Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
310
+
311
+ Args:
312
+ encoded_inputs:
313
+ Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
314
+ max_length: maximum length of the returned list and optionally padding length (see below).
315
+ Will truncate by taking into account the special tokens.
316
+ padding_strategy: PaddingStrategy to use for padding.
317
+
318
+ - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
319
+ - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
320
+ - PaddingStrategy.DO_NOT_PAD: Do not pad
321
+ The tokenizer padding sides are defined in self.padding_side:
322
+
323
+ - 'left': pads on the left of the sequences
324
+ - 'right': pads on the right of the sequences
325
+ pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
326
+ This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
327
+ `>= 7.5` (Volta).
328
+ return_attention_mask:
329
+ (optional) Set to False to avoid returning attention mask (default: set to model specifics)
330
+ """
331
+ # Load from model defaults
332
+ assert self.padding_side == "left"
333
+
334
+ required_input = encoded_inputs[self.model_input_names[0]]
335
+ seq_length = len(required_input)
336
+
337
+ if padding_strategy == PaddingStrategy.LONGEST:
338
+ max_length = len(required_input)
339
+
340
+ if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
341
+ max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
342
+
343
+ needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
344
+
345
+ # Initialize attention mask if not present.
346
+ if "attention_mask" not in encoded_inputs:
347
+ encoded_inputs["attention_mask"] = [1] * seq_length
348
+
349
+ if "position_ids" not in encoded_inputs:
350
+ encoded_inputs["position_ids"] = list(range(seq_length))
351
+
352
+ if needs_to_be_padded:
353
+ difference = max_length - len(required_input)
354
+
355
+ if "attention_mask" in encoded_inputs:
356
+ encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
357
+ if "position_ids" in encoded_inputs:
358
+ encoded_inputs["position_ids"] = [0] * difference + encoded_inputs["position_ids"]
359
+ encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
360
+
361
+ return encoded_inputs
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5a493598071550244b2ee7f26118f3edec2150b9dfa967929a99052ac83fe716
3
+ size 2623634
tokenizer_config.json ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_map": {
3
+ "AutoTokenizer": [
4
+ "tokenization_chatglm.ChatGLM4Tokenizer",
5
+ null
6
+ ]
7
+ },
8
+ "added_tokens_decoder": {
9
+ "151329": {
10
+ "content": "<|endoftext|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false,
15
+ "special": true
16
+ },
17
+ "151330": {
18
+ "content": "[MASK]",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false,
23
+ "special": true
24
+ },
25
+ "151331": {
26
+ "content": "[gMASK]",
27
+ "lstrip": false,
28
+ "normalized": false,
29
+ "rstrip": false,
30
+ "single_word": false,
31
+ "special": true
32
+ },
33
+ "151332": {
34
+ "content": "[sMASK]",
35
+ "lstrip": false,
36
+ "normalized": false,
37
+ "rstrip": false,
38
+ "single_word": false,
39
+ "special": true
40
+ },
41
+ "151333": {
42
+ "content": "<sop>",
43
+ "lstrip": false,
44
+ "normalized": false,
45
+ "rstrip": false,
46
+ "single_word": false,
47
+ "special": true
48
+ },
49
+ "151334": {
50
+ "content": "<eop>",
51
+ "lstrip": false,
52
+ "normalized": false,
53
+ "rstrip": false,
54
+ "single_word": false,
55
+ "special": true
56
+ },
57
+ "151335": {
58
+ "content": "<|system|>",
59
+ "lstrip": false,
60
+ "normalized": false,
61
+ "rstrip": false,
62
+ "single_word": false,
63
+ "special": true
64
+ },
65
+ "151336": {
66
+ "content": "<|user|>",
67
+ "lstrip": false,
68
+ "normalized": false,
69
+ "rstrip": false,
70
+ "single_word": false,
71
+ "special": true
72
+ },
73
+ "151337": {
74
+ "content": "<|assistant|>",
75
+ "lstrip": false,
76
+ "normalized": false,
77
+ "rstrip": false,
78
+ "single_word": false,
79
+ "special": true
80
+ },
81
+ "151338": {
82
+ "content": "<|observation|>",
83
+ "lstrip": false,
84
+ "normalized": false,
85
+ "rstrip": false,
86
+ "single_word": false,
87
+ "special": true
88
+ },
89
+ "151339": {
90
+ "content": "<|begin_of_image|>",
91
+ "lstrip": false,
92
+ "normalized": false,
93
+ "rstrip": false,
94
+ "single_word": false,
95
+ "special": true
96
+ },
97
+ "151340": {
98
+ "content": "<|end_of_image|>",
99
+ "lstrip": false,
100
+ "normalized": false,
101
+ "rstrip": false,
102
+ "single_word": false,
103
+ "special": true
104
+ },
105
+ "151341": {
106
+ "content": "<|begin_of_video|>",
107
+ "lstrip": false,
108
+ "normalized": false,
109
+ "rstrip": false,
110
+ "single_word": false,
111
+ "special": true
112
+ },
113
+ "151342": {
114
+ "content": "<|end_of_video|>",
115
+ "lstrip": false,
116
+ "normalized": false,
117
+ "rstrip": false,
118
+ "single_word": false,
119
+ "special": true
120
+ }
121
+ },
122
+ "additional_special_tokens": ["<|endoftext|>", "[MASK]", "[gMASK]", "[sMASK]", "<sop>", "<eop>", "<|system|>",
123
+ "<|user|>", "<|assistant|>", "<|observation|>", "<|begin_of_image|>", "<|end_of_image|>",
124
+ "<|begin_of_video|>", "<|end_of_video|>"],
125
+ "clean_up_tokenization_spaces": false,
126
+ "do_lower_case": false,
127
+ "eos_token": "<|endoftext|>",
128
+ "pad_token": "<|endoftext|>",
129
+ "model_max_length": 1000000000000000019884624838656,
130
+ "padding_side": "left",
131
+ "remove_space": false,
132
+ "tokenizer_class": "ChatGLM4Tokenizer",
133
+ "image_size": 1120
134
+ }
visual.py ADDED
@@ -0,0 +1,171 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from torch import nn
3
+ from argparse import Namespace
4
+ import torch.nn.functional as F
5
+ from transformers.activations import ACT2FN
6
+ import math
7
+ from torch.nn import LayerNorm
8
+
9
+ def standard_attention(query_layer, key_layer, value_layer, scaling_attention_score=True):
10
+ if scaling_attention_score:
11
+ query_layer = query_layer / math.sqrt(query_layer.shape[-1])
12
+ attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
13
+
14
+ attention_probs = F.softmax(attention_scores, dim=-1)
15
+
16
+ context_layer = torch.matmul(attention_probs, value_layer)
17
+ return context_layer
18
+
19
+ def attention_fn_default(query_layer, key_layer, value_layer, scaling_attention_score=True):
20
+ if int(torch.__version__.split('.')[0]) >= 2 and scaling_attention_score:
21
+ # Pytorch 2.0 attention uses very much memory if attention_mask is float, and has NaN bug if attention_mask is None.
22
+ attn_output = torch.nn.functional.scaled_dot_product_attention(
23
+ query_layer, key_layer, value_layer,
24
+ attn_mask=None,
25
+ dropout_p=0.,
26
+ is_causal=False
27
+ )
28
+ return attn_output
29
+ else:
30
+ return standard_attention(
31
+ query_layer, key_layer, value_layer, scaling_attention_score=scaling_attention_score
32
+ )
33
+
34
+ class PatchEmbedding(nn.Module):
35
+ def __init__(self, config):
36
+ super().__init__()
37
+ self.proj = nn.Conv2d(config.in_channels, config.hidden_size, kernel_size=config.patch_size, stride=config.patch_size)
38
+ self.cls_embedding = nn.Parameter(torch.zeros(1, config.hidden_size))
39
+ self.position_embedding = nn.Embedding(config.num_positions, config.hidden_size)
40
+
41
+ def forward(self, images: "tensor(B, C, H, W)") -> "tensor(B, L, D)":
42
+ x = self.proj(images)
43
+ x = x.flatten(2).transpose(1, 2)
44
+ cls_token = self.cls_embedding.expand(x.shape[0], -1, -1)
45
+ x = torch.cat((cls_token, x), dim=1)
46
+ x += self.position_embedding.weight.unsqueeze(0)
47
+ return x
48
+
49
+
50
+ class Attention(nn.Module):
51
+ def __init__(self, config):
52
+ super().__init__()
53
+ self.num_heads = config.num_heads
54
+ head_dim = config.hidden_size // config.num_heads
55
+ self.scale = head_dim ** -0.5
56
+ self.query_key_value = nn.Linear(config.hidden_size, config.hidden_size * 3)
57
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
58
+ self.output_dropout = torch.nn.Dropout(config.dropout_prob)
59
+
60
+ def forward(self, x: "tensor(B, L, D)") -> "tensor(B, L, D)":
61
+ B, L, _ = x.shape
62
+ qkv = self.query_key_value(x)
63
+ qkv = qkv.reshape(B, L, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4) # 3, B, H, L, D
64
+ q, k, v = qkv[0], qkv[1], qkv[2]
65
+
66
+ out = attention_fn_default(
67
+ q, k, v
68
+ )
69
+ output = self.dense(out.transpose(1, 2).view(B, L, -1))
70
+ output = self.output_dropout(output)
71
+ return output
72
+
73
+ def attention(self, q, k, v):
74
+ attn_weights = torch.matmul(q * self.scale, k.transpose(-2, -1))
75
+ attn_weights = attn_weights.softmax(dim=-1)
76
+ output = torch.matmul(attn_weights, v)
77
+ return output
78
+
79
+
80
+ class MLP(nn.Module):
81
+ def __init__(self, config):
82
+ super().__init__()
83
+ self.config = config
84
+ self.activation_fn = ACT2FN[config.hidden_act]
85
+ self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
86
+ self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
87
+
88
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
89
+ x = self.fc1(x)
90
+ x = self.activation_fn(x)
91
+ x = self.fc2(x)
92
+ return x
93
+
94
+
95
+ class TransformerLayer(nn.Module):
96
+ def __init__(self, config):
97
+ super().__init__()
98
+ self.input_layernorm = LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
99
+ self.attention = Attention(config)
100
+ self.mlp = MLP(config)
101
+ self.post_attention_layernorm = LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
102
+
103
+ def forward(self, hidden_states):
104
+ attention_input = hidden_states
105
+ attention_output = self.input_layernorm(self.attention(attention_input))
106
+ hidden_states = attention_input + attention_output
107
+ mlp_input = hidden_states
108
+ mlp_output = self.post_attention_layernorm(self.mlp(mlp_input))
109
+ output = mlp_input + mlp_output
110
+ return output
111
+
112
+
113
+ class Transformer(nn.Module):
114
+ def __init__(self, config):
115
+ super().__init__()
116
+ self.layers = nn.ModuleList([TransformerLayer(config) for _ in range(config.num_hidden_layers)])
117
+
118
+ def forward(self, hidden_states):
119
+ for layer_module in self.layers:
120
+ hidden_states = layer_module(hidden_states)
121
+ return hidden_states
122
+
123
+
124
+ class GLU(nn.Module):
125
+ def __init__(self, config, in_features):
126
+ super().__init__()
127
+ self.linear_proj = nn.Linear(in_features, config.hidden_size, bias=False)
128
+ self.norm1 = nn.LayerNorm(config.hidden_size)
129
+ self.act1 = nn.GELU()
130
+ self.act2 = nn.functional.silu
131
+ self.dense_h_to_4h = nn.Linear(config.hidden_size, config.ffn_hidden_size, bias=False)
132
+ self.gate_proj = nn.Linear(config.hidden_size, config.ffn_hidden_size, bias=False)
133
+ self.dense_4h_to_h = nn.Linear(config.ffn_hidden_size, config.hidden_size, bias=False)
134
+
135
+ def forward(self, x):
136
+ x = self.linear_proj(x)
137
+ x = self.act1(self.norm1(x))
138
+ x = self.act2(self.gate_proj(x)) * self.dense_h_to_4h(x)
139
+ x = self.dense_4h_to_h(x)
140
+ return x
141
+
142
+
143
+ class EVA2CLIPModel(nn.Module):
144
+ def __init__(self, config):
145
+ super().__init__()
146
+ vision_config = Namespace(**config.vision_config)
147
+ self.patch_embedding = PatchEmbedding(vision_config)
148
+ self.transformer = Transformer(vision_config)
149
+ self.linear_proj = GLU(config, in_features=config.hidden_size)
150
+ self.conv = nn.Conv2d(in_channels=vision_config.hidden_size, out_channels=config.hidden_size, kernel_size=2, stride=2)
151
+ self.boi = nn.Parameter(torch.zeros(1, 1, config.hidden_size))
152
+ self.eoi = nn.Parameter(torch.zeros(1, 1, config.hidden_size))
153
+ self.scaling_factor = vision_config.scaling_factor
154
+
155
+ def forward(self, images: "tensor(B, C, H, W)") -> "tensor(B, L, D)":
156
+ x = self.patch_embedding(images)
157
+ x = self.transformer(x)
158
+ x = x[:, 1:]
159
+
160
+ b, s, h = x.shape
161
+ grid_size = int(s**0.5)
162
+ x = x.view(b, grid_size, grid_size, h).permute(0, 3, 1, 2)
163
+ x = self.conv(x)
164
+
165
+ x = x.flatten(2).transpose(1, 2)
166
+ x = self.linear_proj(x)
167
+ boi = self.boi.expand(x.shape[0], -1, -1)
168
+ eoi = self.eoi.expand(x.shape[0], -1, -1)
169
+ x = torch.cat((boi, x, eoi), dim=1)
170
+ x = x / self.scaling_factor
171
+ return x