sanchit-gandhi HF staff commited on
Commit
e687bf7
1 Parent(s): 2cade16

Saving train state of step 90000

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitignore +1 -0
  2. accelerate_config.yaml +17 -0
  3. checkpoint-80000-epoch-5/optimizer.bin +3 -0
  4. checkpoint-80000-epoch-5/pytorch_model.bin +3 -0
  5. checkpoint-80000-epoch-5/random_states_0.pkl +3 -0
  6. checkpoint-80000-epoch-5/random_states_1.pkl +3 -0
  7. checkpoint-80000-epoch-5/random_states_2.pkl +3 -0
  8. checkpoint-80000-epoch-5/random_states_3.pkl +3 -0
  9. checkpoint-80000-epoch-5/random_states_4.pkl +3 -0
  10. checkpoint-80000-epoch-5/random_states_5.pkl +3 -0
  11. checkpoint-80000-epoch-5/random_states_6.pkl +3 -0
  12. checkpoint-80000-epoch-5/random_states_7.pkl +3 -0
  13. checkpoint-80000-epoch-5/scheduler.bin +3 -0
  14. checkpoint-90000-epoch-6/config.json +278 -0
  15. checkpoint-90000-epoch-6/generation_config.json +12 -0
  16. checkpoint-90000-epoch-6/optimizer.bin +3 -0
  17. checkpoint-90000-epoch-6/pytorch_model.bin +3 -0
  18. checkpoint-90000-epoch-6/random_states_0.pkl +3 -0
  19. checkpoint-90000-epoch-6/random_states_1.pkl +3 -0
  20. checkpoint-90000-epoch-6/random_states_2.pkl +3 -0
  21. checkpoint-90000-epoch-6/random_states_3.pkl +3 -0
  22. checkpoint-90000-epoch-6/random_states_4.pkl +3 -0
  23. checkpoint-90000-epoch-6/random_states_5.pkl +3 -0
  24. checkpoint-90000-epoch-6/random_states_6.pkl +3 -0
  25. checkpoint-90000-epoch-6/random_states_7.pkl +3 -0
  26. checkpoint-90000-epoch-6/scheduler.bin +3 -0
  27. config.json +278 -0
  28. parler_tts/__init__.py +16 -0
  29. parler_tts/__pycache__/__init__.cpython-311.pyc +0 -0
  30. parler_tts/__pycache__/configuration_parler_tts.cpython-311.pyc +0 -0
  31. parler_tts/__pycache__/modeling_parler_tts.cpython-311.pyc +0 -0
  32. parler_tts/configuration_parler_tts.py +249 -0
  33. parler_tts/dac_wrapper/__init__.py +2 -0
  34. parler_tts/dac_wrapper/__pycache__/__init__.cpython-311.pyc +0 -0
  35. parler_tts/dac_wrapper/__pycache__/configuration_dac.cpython-311.pyc +0 -0
  36. parler_tts/dac_wrapper/__pycache__/modeling_dac.cpython-311.pyc +0 -0
  37. parler_tts/dac_wrapper/configuration_dac.py +25 -0
  38. parler_tts/dac_wrapper/modeling_dac.py +137 -0
  39. parler_tts/modeling_parler_tts.py +0 -0
  40. preprocessor_config.json +10 -0
  41. slurm_job.slurm +74 -0
  42. special_tokens_map.json +125 -0
  43. spiece.model +3 -0
  44. starting_point_0.01_rope.json +79 -0
  45. tokenizer.json +0 -0
  46. tokenizer_config.json +938 -0
  47. training/README.md +211 -0
  48. training/__init__.py +0 -0
  49. training/__pycache__/__init__.cpython-311.pyc +0 -0
  50. training/__pycache__/arguments.cpython-311.pyc +0 -0
.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ wandb
accelerate_config.yaml ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ compute_environment: LOCAL_MACHINE
2
+ debug: false
3
+ distributed_type: MULTI_GPU
4
+ downcast_bf16: 'no'
5
+ enable_cpu_affinity: false
6
+ gpu_ids: all
7
+ machine_rank: 0
8
+ main_training_function: main
9
+ mixed_precision: bf16
10
+ num_machines: 1
11
+ num_processes: 8
12
+ rdzv_backend: static
13
+ same_network: true
14
+ tpu_env: []
15
+ tpu_use_cluster: false
16
+ tpu_use_sudo: false
17
+ use_cpu: false
checkpoint-80000-epoch-5/optimizer.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d85a6ebcfaea697deb81a9f43cf52b763722cc63c52559be99475194f3b1740
3
+ size 3652769047
checkpoint-80000-epoch-5/pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:64ed77445bae74c0a44de014644335672082db652c167e03307f5aa69f497029
3
+ size 2588465818
checkpoint-80000-epoch-5/random_states_0.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9bee040cadf66c07cc28951d8b63f5309317c467c935239899967dedde236356
3
+ size 16100
checkpoint-80000-epoch-5/random_states_1.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac2984f6c11e0d6baba6e153c36e02cf7d8849f435a8e1d10db3db9422a5e14f
3
+ size 16100
checkpoint-80000-epoch-5/random_states_2.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0328ba0db514b5427f0cf5728abf82349345bec8a02097c30900ec57f433da11
3
+ size 16100
checkpoint-80000-epoch-5/random_states_3.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:668cc4578435da562a838c3860a81ef62a34fdfed1920ccebb7657eb586c14cb
3
+ size 16100
checkpoint-80000-epoch-5/random_states_4.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c856097a2b0832cc85eba37f9cd1be76e0d395fe4b2c99ef3153199386dae5e5
3
+ size 16100
checkpoint-80000-epoch-5/random_states_5.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:01d11b3bb1f3959f58be8c638fb21de1e89e54c13aa716a178687cb1fc51ac99
3
+ size 16100
checkpoint-80000-epoch-5/random_states_6.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18447190b31cbfd6ac8565a46dfbdc82123f510bef0eb0a72313d1150001b8f4
3
+ size 16100
checkpoint-80000-epoch-5/random_states_7.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:025b2bf49a9a6fe2a6fd8cb7a98ba98f4eb439e04b5aa68fd3285c8c109bdc4f
3
+ size 16100
checkpoint-80000-epoch-5/scheduler.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fd55300ad4b9da07287969193b88215d1af9b5d03f7bf3a833f1b260aa6434c4
3
+ size 1000
checkpoint-90000-epoch-6/config.json ADDED
@@ -0,0 +1,278 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ParlerTTSForConditionalGeneration"
4
+ ],
5
+ "audio_encoder": {
6
+ "_name_or_path": "parler-tts/dac_44khZ_8kbps",
7
+ "add_cross_attention": false,
8
+ "architectures": [
9
+ "DACModel"
10
+ ],
11
+ "bad_words_ids": null,
12
+ "begin_suppress_tokens": null,
13
+ "bos_token_id": null,
14
+ "chunk_size_feed_forward": 0,
15
+ "codebook_size": 1024,
16
+ "cross_attention_hidden_size": null,
17
+ "decoder_start_token_id": null,
18
+ "diversity_penalty": 0.0,
19
+ "do_sample": false,
20
+ "early_stopping": false,
21
+ "encoder_no_repeat_ngram_size": 0,
22
+ "eos_token_id": null,
23
+ "exponential_decay_length_penalty": null,
24
+ "finetuning_task": null,
25
+ "forced_bos_token_id": null,
26
+ "forced_eos_token_id": null,
27
+ "frame_rate": 86,
28
+ "id2label": {
29
+ "0": "LABEL_0",
30
+ "1": "LABEL_1"
31
+ },
32
+ "is_decoder": false,
33
+ "is_encoder_decoder": false,
34
+ "label2id": {
35
+ "LABEL_0": 0,
36
+ "LABEL_1": 1
37
+ },
38
+ "latent_dim": 1024,
39
+ "length_penalty": 1.0,
40
+ "max_length": 20,
41
+ "min_length": 0,
42
+ "model_bitrate": 8,
43
+ "model_type": "dac",
44
+ "no_repeat_ngram_size": 0,
45
+ "num_beam_groups": 1,
46
+ "num_beams": 1,
47
+ "num_codebooks": 9,
48
+ "num_return_sequences": 1,
49
+ "output_attentions": false,
50
+ "output_hidden_states": false,
51
+ "output_scores": false,
52
+ "pad_token_id": null,
53
+ "prefix": null,
54
+ "problem_type": null,
55
+ "pruned_heads": {},
56
+ "remove_invalid_values": false,
57
+ "repetition_penalty": 1.0,
58
+ "return_dict": true,
59
+ "return_dict_in_generate": false,
60
+ "sampling_rate": 44100,
61
+ "sep_token_id": null,
62
+ "suppress_tokens": null,
63
+ "task_specific_params": null,
64
+ "temperature": 1.0,
65
+ "tf_legacy_loss": false,
66
+ "tie_encoder_decoder": false,
67
+ "tie_word_embeddings": true,
68
+ "tokenizer_class": null,
69
+ "top_k": 50,
70
+ "top_p": 1.0,
71
+ "torch_dtype": "float32",
72
+ "torchscript": false,
73
+ "typical_p": 1.0,
74
+ "use_bfloat16": false
75
+ },
76
+ "decoder": {
77
+ "_name_or_path": "./parler-tts-untrained-600M/decoder",
78
+ "activation_dropout": 0.0,
79
+ "activation_function": "gelu",
80
+ "add_cross_attention": true,
81
+ "architectures": [
82
+ "ParlerTTSForCausalLM"
83
+ ],
84
+ "attention_dropout": 0.0,
85
+ "bad_words_ids": null,
86
+ "begin_suppress_tokens": null,
87
+ "bos_token_id": 1025,
88
+ "chunk_size_feed_forward": 0,
89
+ "cross_attention_hidden_size": null,
90
+ "decoder_start_token_id": null,
91
+ "diversity_penalty": 0.0,
92
+ "do_sample": false,
93
+ "dropout": 0.1,
94
+ "early_stopping": false,
95
+ "encoder_no_repeat_ngram_size": 0,
96
+ "eos_token_id": 1024,
97
+ "exponential_decay_length_penalty": null,
98
+ "ffn_dim": 4096,
99
+ "finetuning_task": null,
100
+ "forced_bos_token_id": null,
101
+ "forced_eos_token_id": null,
102
+ "hidden_size": 1024,
103
+ "id2label": {
104
+ "0": "LABEL_0",
105
+ "1": "LABEL_1"
106
+ },
107
+ "initializer_factor": 0.02,
108
+ "is_decoder": true,
109
+ "is_encoder_decoder": false,
110
+ "label2id": {
111
+ "LABEL_0": 0,
112
+ "LABEL_1": 1
113
+ },
114
+ "layerdrop": 0.0,
115
+ "length_penalty": 1.0,
116
+ "max_length": 20,
117
+ "max_position_embeddings": 4096,
118
+ "min_length": 0,
119
+ "model_type": "parler_tts_decoder",
120
+ "no_repeat_ngram_size": 0,
121
+ "num_attention_heads": 16,
122
+ "num_beam_groups": 1,
123
+ "num_beams": 1,
124
+ "num_codebooks": 9,
125
+ "num_hidden_layers": 24,
126
+ "num_return_sequences": 1,
127
+ "output_attentions": false,
128
+ "output_hidden_states": false,
129
+ "output_scores": false,
130
+ "pad_token_id": 1024,
131
+ "prefix": null,
132
+ "problem_type": null,
133
+ "pruned_heads": {},
134
+ "remove_invalid_values": false,
135
+ "repetition_penalty": 1.0,
136
+ "return_dict": true,
137
+ "return_dict_in_generate": false,
138
+ "rope_embeddings": true,
139
+ "rope_theta": 10000.0,
140
+ "scale_embedding": false,
141
+ "sep_token_id": null,
142
+ "suppress_tokens": null,
143
+ "task_specific_params": null,
144
+ "temperature": 1.0,
145
+ "tf_legacy_loss": false,
146
+ "tie_encoder_decoder": false,
147
+ "tie_word_embeddings": false,
148
+ "tokenizer_class": null,
149
+ "top_k": 50,
150
+ "top_p": 1.0,
151
+ "torch_dtype": "float32",
152
+ "torchscript": false,
153
+ "typical_p": 1.0,
154
+ "use_bfloat16": false,
155
+ "use_cache": true,
156
+ "vocab_size": 1088
157
+ },
158
+ "decoder_start_token_id": 1025,
159
+ "is_encoder_decoder": true,
160
+ "model_type": "parler_tts",
161
+ "pad_token_id": 1024,
162
+ "prompt_cross_attention": true,
163
+ "text_encoder": {
164
+ "_name_or_path": "google/flan-t5-base",
165
+ "add_cross_attention": false,
166
+ "architectures": [
167
+ "T5ForConditionalGeneration"
168
+ ],
169
+ "bad_words_ids": null,
170
+ "begin_suppress_tokens": null,
171
+ "bos_token_id": null,
172
+ "chunk_size_feed_forward": 0,
173
+ "classifier_dropout": 0.0,
174
+ "cross_attention_hidden_size": null,
175
+ "d_ff": 2048,
176
+ "d_kv": 64,
177
+ "d_model": 768,
178
+ "decoder_start_token_id": 0,
179
+ "dense_act_fn": "gelu_new",
180
+ "diversity_penalty": 0.0,
181
+ "do_sample": false,
182
+ "dropout_rate": 0.1,
183
+ "early_stopping": false,
184
+ "encoder_no_repeat_ngram_size": 0,
185
+ "eos_token_id": 1,
186
+ "exponential_decay_length_penalty": null,
187
+ "feed_forward_proj": "gated-gelu",
188
+ "finetuning_task": null,
189
+ "forced_bos_token_id": null,
190
+ "forced_eos_token_id": null,
191
+ "id2label": {
192
+ "0": "LABEL_0",
193
+ "1": "LABEL_1"
194
+ },
195
+ "initializer_factor": 1.0,
196
+ "is_decoder": false,
197
+ "is_encoder_decoder": true,
198
+ "is_gated_act": true,
199
+ "label2id": {
200
+ "LABEL_0": 0,
201
+ "LABEL_1": 1
202
+ },
203
+ "layer_norm_epsilon": 1e-06,
204
+ "length_penalty": 1.0,
205
+ "max_length": 20,
206
+ "min_length": 0,
207
+ "model_type": "t5",
208
+ "n_positions": 512,
209
+ "no_repeat_ngram_size": 0,
210
+ "num_beam_groups": 1,
211
+ "num_beams": 1,
212
+ "num_decoder_layers": 12,
213
+ "num_heads": 12,
214
+ "num_layers": 12,
215
+ "num_return_sequences": 1,
216
+ "output_attentions": false,
217
+ "output_hidden_states": false,
218
+ "output_past": true,
219
+ "output_scores": false,
220
+ "pad_token_id": 0,
221
+ "prefix": null,
222
+ "problem_type": null,
223
+ "pruned_heads": {},
224
+ "relative_attention_max_distance": 128,
225
+ "relative_attention_num_buckets": 32,
226
+ "remove_invalid_values": false,
227
+ "repetition_penalty": 1.0,
228
+ "return_dict": true,
229
+ "return_dict_in_generate": false,
230
+ "sep_token_id": null,
231
+ "suppress_tokens": null,
232
+ "task_specific_params": {
233
+ "summarization": {
234
+ "early_stopping": true,
235
+ "length_penalty": 2.0,
236
+ "max_length": 200,
237
+ "min_length": 30,
238
+ "no_repeat_ngram_size": 3,
239
+ "num_beams": 4,
240
+ "prefix": "summarize: "
241
+ },
242
+ "translation_en_to_de": {
243
+ "early_stopping": true,
244
+ "max_length": 300,
245
+ "num_beams": 4,
246
+ "prefix": "translate English to German: "
247
+ },
248
+ "translation_en_to_fr": {
249
+ "early_stopping": true,
250
+ "max_length": 300,
251
+ "num_beams": 4,
252
+ "prefix": "translate English to French: "
253
+ },
254
+ "translation_en_to_ro": {
255
+ "early_stopping": true,
256
+ "max_length": 300,
257
+ "num_beams": 4,
258
+ "prefix": "translate English to Romanian: "
259
+ }
260
+ },
261
+ "temperature": 1.0,
262
+ "tf_legacy_loss": false,
263
+ "tie_encoder_decoder": false,
264
+ "tie_word_embeddings": false,
265
+ "tokenizer_class": null,
266
+ "top_k": 50,
267
+ "top_p": 1.0,
268
+ "torch_dtype": null,
269
+ "torchscript": false,
270
+ "typical_p": 1.0,
271
+ "use_bfloat16": false,
272
+ "use_cache": true,
273
+ "vocab_size": 32128
274
+ },
275
+ "torch_dtype": "float32",
276
+ "transformers_version": "4.40.2",
277
+ "vocab_size": 32128
278
+ }
checkpoint-90000-epoch-6/generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1025,
4
+ "decoder_start_token_id": 1025,
5
+ "do_sample": true,
6
+ "eos_token_id": 1024,
7
+ "guidance_scale": 1,
8
+ "key": 10,
9
+ "max_length": 2580,
10
+ "pad_token_id": 1024,
11
+ "transformers_version": "4.40.2"
12
+ }
checkpoint-90000-epoch-6/optimizer.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c8a5f279e2029513a27b4cded580e595a78c5fc272f40cb7f588433d7958d46c
3
+ size 3652769047
checkpoint-90000-epoch-6/pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8328177320aeb4953e3eec0bf9d69faaa0252580a421c044139beb4a6331ca7
3
+ size 2588465818
checkpoint-90000-epoch-6/random_states_0.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ce59f0c776bd98a152d59388c7b005bc4a240f161451d365033b376ffdbdf689
3
+ size 16100
checkpoint-90000-epoch-6/random_states_1.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9023097fa78daf107b4ab8d45d5c2428ce1dfcfa78f1325792412a2fc1422961
3
+ size 16100
checkpoint-90000-epoch-6/random_states_2.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ae413302aa9d35ab8fda74453e73117f5f7bb061667d47682aceeec97a48d1b
3
+ size 16100
checkpoint-90000-epoch-6/random_states_3.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:22951f7c252eb5c607a491f9a4aece64566703c5a3454daa80b5a9335031005d
3
+ size 16100
checkpoint-90000-epoch-6/random_states_4.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63052cdf7dd342cee03f75c8a78acd9ee45ca352bbfafa6fe46779ca27905489
3
+ size 16164
checkpoint-90000-epoch-6/random_states_5.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e4610b40691a1373dff81de30c9bc043ce4f6a4f82d18d910d9202ba688cba95
3
+ size 16100
checkpoint-90000-epoch-6/random_states_6.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9a333a3646432bc3d5f3cb8d6ccf35d3121cf07c5b6f5652c6c3b7e6086f748
3
+ size 16100
checkpoint-90000-epoch-6/random_states_7.pkl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72846ff2a7e97a862a1b449d5b5647f99109b910e1c68bc4a6410f2afeb6b5b3
3
+ size 16100
checkpoint-90000-epoch-6/scheduler.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ae6c681cda1d31cf3664adcef373dcad0d17378550a5e96a62ea22bcce6a8e0d
3
+ size 1000
config.json ADDED
@@ -0,0 +1,278 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ParlerTTSForConditionalGeneration"
4
+ ],
5
+ "audio_encoder": {
6
+ "_name_or_path": "parler-tts/dac_44khZ_8kbps",
7
+ "add_cross_attention": false,
8
+ "architectures": [
9
+ "DACModel"
10
+ ],
11
+ "bad_words_ids": null,
12
+ "begin_suppress_tokens": null,
13
+ "bos_token_id": null,
14
+ "chunk_size_feed_forward": 0,
15
+ "codebook_size": 1024,
16
+ "cross_attention_hidden_size": null,
17
+ "decoder_start_token_id": null,
18
+ "diversity_penalty": 0.0,
19
+ "do_sample": false,
20
+ "early_stopping": false,
21
+ "encoder_no_repeat_ngram_size": 0,
22
+ "eos_token_id": null,
23
+ "exponential_decay_length_penalty": null,
24
+ "finetuning_task": null,
25
+ "forced_bos_token_id": null,
26
+ "forced_eos_token_id": null,
27
+ "frame_rate": 86,
28
+ "id2label": {
29
+ "0": "LABEL_0",
30
+ "1": "LABEL_1"
31
+ },
32
+ "is_decoder": false,
33
+ "is_encoder_decoder": false,
34
+ "label2id": {
35
+ "LABEL_0": 0,
36
+ "LABEL_1": 1
37
+ },
38
+ "latent_dim": 1024,
39
+ "length_penalty": 1.0,
40
+ "max_length": 20,
41
+ "min_length": 0,
42
+ "model_bitrate": 8,
43
+ "model_type": "dac",
44
+ "no_repeat_ngram_size": 0,
45
+ "num_beam_groups": 1,
46
+ "num_beams": 1,
47
+ "num_codebooks": 9,
48
+ "num_return_sequences": 1,
49
+ "output_attentions": false,
50
+ "output_hidden_states": false,
51
+ "output_scores": false,
52
+ "pad_token_id": null,
53
+ "prefix": null,
54
+ "problem_type": null,
55
+ "pruned_heads": {},
56
+ "remove_invalid_values": false,
57
+ "repetition_penalty": 1.0,
58
+ "return_dict": true,
59
+ "return_dict_in_generate": false,
60
+ "sampling_rate": 44100,
61
+ "sep_token_id": null,
62
+ "suppress_tokens": null,
63
+ "task_specific_params": null,
64
+ "temperature": 1.0,
65
+ "tf_legacy_loss": false,
66
+ "tie_encoder_decoder": false,
67
+ "tie_word_embeddings": true,
68
+ "tokenizer_class": null,
69
+ "top_k": 50,
70
+ "top_p": 1.0,
71
+ "torch_dtype": "float32",
72
+ "torchscript": false,
73
+ "typical_p": 1.0,
74
+ "use_bfloat16": false
75
+ },
76
+ "decoder": {
77
+ "_name_or_path": "./parler-tts-untrained-600M/decoder",
78
+ "activation_dropout": 0.0,
79
+ "activation_function": "gelu",
80
+ "add_cross_attention": true,
81
+ "architectures": [
82
+ "ParlerTTSForCausalLM"
83
+ ],
84
+ "attention_dropout": 0.0,
85
+ "bad_words_ids": null,
86
+ "begin_suppress_tokens": null,
87
+ "bos_token_id": 1025,
88
+ "chunk_size_feed_forward": 0,
89
+ "cross_attention_hidden_size": null,
90
+ "decoder_start_token_id": null,
91
+ "diversity_penalty": 0.0,
92
+ "do_sample": false,
93
+ "dropout": 0.1,
94
+ "early_stopping": false,
95
+ "encoder_no_repeat_ngram_size": 0,
96
+ "eos_token_id": 1024,
97
+ "exponential_decay_length_penalty": null,
98
+ "ffn_dim": 4096,
99
+ "finetuning_task": null,
100
+ "forced_bos_token_id": null,
101
+ "forced_eos_token_id": null,
102
+ "hidden_size": 1024,
103
+ "id2label": {
104
+ "0": "LABEL_0",
105
+ "1": "LABEL_1"
106
+ },
107
+ "initializer_factor": 0.02,
108
+ "is_decoder": true,
109
+ "is_encoder_decoder": false,
110
+ "label2id": {
111
+ "LABEL_0": 0,
112
+ "LABEL_1": 1
113
+ },
114
+ "layerdrop": 0.0,
115
+ "length_penalty": 1.0,
116
+ "max_length": 20,
117
+ "max_position_embeddings": 4096,
118
+ "min_length": 0,
119
+ "model_type": "parler_tts_decoder",
120
+ "no_repeat_ngram_size": 0,
121
+ "num_attention_heads": 16,
122
+ "num_beam_groups": 1,
123
+ "num_beams": 1,
124
+ "num_codebooks": 9,
125
+ "num_hidden_layers": 24,
126
+ "num_return_sequences": 1,
127
+ "output_attentions": false,
128
+ "output_hidden_states": false,
129
+ "output_scores": false,
130
+ "pad_token_id": 1024,
131
+ "prefix": null,
132
+ "problem_type": null,
133
+ "pruned_heads": {},
134
+ "remove_invalid_values": false,
135
+ "repetition_penalty": 1.0,
136
+ "return_dict": true,
137
+ "return_dict_in_generate": false,
138
+ "rope_embeddings": true,
139
+ "rope_theta": 10000.0,
140
+ "scale_embedding": false,
141
+ "sep_token_id": null,
142
+ "suppress_tokens": null,
143
+ "task_specific_params": null,
144
+ "temperature": 1.0,
145
+ "tf_legacy_loss": false,
146
+ "tie_encoder_decoder": false,
147
+ "tie_word_embeddings": false,
148
+ "tokenizer_class": null,
149
+ "top_k": 50,
150
+ "top_p": 1.0,
151
+ "torch_dtype": "float32",
152
+ "torchscript": false,
153
+ "typical_p": 1.0,
154
+ "use_bfloat16": false,
155
+ "use_cache": true,
156
+ "vocab_size": 1088
157
+ },
158
+ "decoder_start_token_id": 1025,
159
+ "is_encoder_decoder": true,
160
+ "model_type": "parler_tts",
161
+ "pad_token_id": 1024,
162
+ "prompt_cross_attention": true,
163
+ "text_encoder": {
164
+ "_name_or_path": "google/flan-t5-base",
165
+ "add_cross_attention": false,
166
+ "architectures": [
167
+ "T5ForConditionalGeneration"
168
+ ],
169
+ "bad_words_ids": null,
170
+ "begin_suppress_tokens": null,
171
+ "bos_token_id": null,
172
+ "chunk_size_feed_forward": 0,
173
+ "classifier_dropout": 0.0,
174
+ "cross_attention_hidden_size": null,
175
+ "d_ff": 2048,
176
+ "d_kv": 64,
177
+ "d_model": 768,
178
+ "decoder_start_token_id": 0,
179
+ "dense_act_fn": "gelu_new",
180
+ "diversity_penalty": 0.0,
181
+ "do_sample": false,
182
+ "dropout_rate": 0.1,
183
+ "early_stopping": false,
184
+ "encoder_no_repeat_ngram_size": 0,
185
+ "eos_token_id": 1,
186
+ "exponential_decay_length_penalty": null,
187
+ "feed_forward_proj": "gated-gelu",
188
+ "finetuning_task": null,
189
+ "forced_bos_token_id": null,
190
+ "forced_eos_token_id": null,
191
+ "id2label": {
192
+ "0": "LABEL_0",
193
+ "1": "LABEL_1"
194
+ },
195
+ "initializer_factor": 1.0,
196
+ "is_decoder": false,
197
+ "is_encoder_decoder": true,
198
+ "is_gated_act": true,
199
+ "label2id": {
200
+ "LABEL_0": 0,
201
+ "LABEL_1": 1
202
+ },
203
+ "layer_norm_epsilon": 1e-06,
204
+ "length_penalty": 1.0,
205
+ "max_length": 20,
206
+ "min_length": 0,
207
+ "model_type": "t5",
208
+ "n_positions": 512,
209
+ "no_repeat_ngram_size": 0,
210
+ "num_beam_groups": 1,
211
+ "num_beams": 1,
212
+ "num_decoder_layers": 12,
213
+ "num_heads": 12,
214
+ "num_layers": 12,
215
+ "num_return_sequences": 1,
216
+ "output_attentions": false,
217
+ "output_hidden_states": false,
218
+ "output_past": true,
219
+ "output_scores": false,
220
+ "pad_token_id": 0,
221
+ "prefix": null,
222
+ "problem_type": null,
223
+ "pruned_heads": {},
224
+ "relative_attention_max_distance": 128,
225
+ "relative_attention_num_buckets": 32,
226
+ "remove_invalid_values": false,
227
+ "repetition_penalty": 1.0,
228
+ "return_dict": true,
229
+ "return_dict_in_generate": false,
230
+ "sep_token_id": null,
231
+ "suppress_tokens": null,
232
+ "task_specific_params": {
233
+ "summarization": {
234
+ "early_stopping": true,
235
+ "length_penalty": 2.0,
236
+ "max_length": 200,
237
+ "min_length": 30,
238
+ "no_repeat_ngram_size": 3,
239
+ "num_beams": 4,
240
+ "prefix": "summarize: "
241
+ },
242
+ "translation_en_to_de": {
243
+ "early_stopping": true,
244
+ "max_length": 300,
245
+ "num_beams": 4,
246
+ "prefix": "translate English to German: "
247
+ },
248
+ "translation_en_to_fr": {
249
+ "early_stopping": true,
250
+ "max_length": 300,
251
+ "num_beams": 4,
252
+ "prefix": "translate English to French: "
253
+ },
254
+ "translation_en_to_ro": {
255
+ "early_stopping": true,
256
+ "max_length": 300,
257
+ "num_beams": 4,
258
+ "prefix": "translate English to Romanian: "
259
+ }
260
+ },
261
+ "temperature": 1.0,
262
+ "tf_legacy_loss": false,
263
+ "tie_encoder_decoder": false,
264
+ "tie_word_embeddings": false,
265
+ "tokenizer_class": null,
266
+ "top_k": 50,
267
+ "top_p": 1.0,
268
+ "torch_dtype": null,
269
+ "torchscript": false,
270
+ "typical_p": 1.0,
271
+ "use_bfloat16": false,
272
+ "use_cache": true,
273
+ "vocab_size": 32128
274
+ },
275
+ "torch_dtype": "float32",
276
+ "transformers_version": "4.40.2",
277
+ "vocab_size": 32128
278
+ }
parler_tts/__init__.py ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ __version__ = "0.1"
2
+
3
+
4
+ from .configuration_parler_tts import ParlerTTSConfig, ParlerTTSDecoderConfig
5
+ from .modeling_parler_tts import (
6
+ ParlerTTSForCausalLM,
7
+ ParlerTTSForConditionalGeneration,
8
+ apply_delay_pattern_mask,
9
+ build_delay_pattern_mask,
10
+ )
11
+
12
+ from .dac_wrapper import DACConfig, DACModel
13
+ from transformers import AutoConfig, AutoModel
14
+
15
+ AutoConfig.register("dac", DACConfig)
16
+ AutoModel.register(DACConfig, DACModel)
parler_tts/__pycache__/__init__.cpython-311.pyc ADDED
Binary file (845 Bytes). View file
 
parler_tts/__pycache__/configuration_parler_tts.cpython-311.pyc ADDED
Binary file (11.5 kB). View file
 
parler_tts/__pycache__/modeling_parler_tts.cpython-311.pyc ADDED
Binary file (135 kB). View file
 
parler_tts/configuration_parler_tts.py ADDED
@@ -0,0 +1,249 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2024 and The HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ Parler-TTS model configuration"""
16
+
17
+ from transformers import AutoConfig, logging
18
+ from transformers.configuration_utils import PretrainedConfig
19
+
20
+
21
+ logger = logging.get_logger(__name__)
22
+
23
+ MUSICGEN_PRETRAINED_CONFIG_ARCHIVE_MAP = {
24
+ "facebook/parler_tts-small": "https://huggingface.co/facebook/parler_tts-small/resolve/main/config.json",
25
+ # See all ParlerTTS models at https://huggingface.co/models?filter=parler_tts
26
+ }
27
+
28
+
29
+ class ParlerTTSDecoderConfig(PretrainedConfig):
30
+ r"""
31
+ This is the configuration class to store the configuration of an [`ParlerTTSDecoder`]. It is used to instantiate a
32
+ Parler-TTS decoder according to the specified arguments, defining the model architecture. Instantiating a
33
+ configuration with the defaults will yield a similar configuration to that of the Parler-TTS
34
+ [facebook/parler_tts-small](https://huggingface.co/facebook/parler_tts-small) architecture.
35
+
36
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
37
+ documentation from [`PretrainedConfig`] for more information.
38
+
39
+
40
+ Args:
41
+ vocab_size (`int`, *optional*, defaults to 2049):
42
+ Vocabulary size of the ParlerTTSDecoder model. Defines the number of different tokens that can be
43
+ represented by the `inputs_ids` passed when calling [`ParlerTTSDecoder`].
44
+ hidden_size (`int`, *optional*, defaults to 1024):
45
+ Dimensionality of the layers and the pooler layer.
46
+ num_hidden_layers (`int`, *optional*, defaults to 24):
47
+ Number of decoder layers.
48
+ num_attention_heads (`int`, *optional*, defaults to 16):
49
+ Number of attention heads for each attention layer in the Transformer block.
50
+ ffn_dim (`int`, *optional*, defaults to 4096):
51
+ Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer block.
52
+ activation_function (`str` or `function`, *optional*, defaults to `"gelu"`):
53
+ The non-linear activation function (function or string) in the decoder and pooler. If string, `"gelu"`,
54
+ `"relu"`, `"silu"` and `"gelu_new"` are supported.
55
+ dropout (`float`, *optional*, defaults to 0.1):
56
+ The dropout probability for all fully connected layers in the embeddings, text_encoder, and pooler.
57
+ attention_dropout (`float`, *optional*, defaults to 0.0):
58
+ The dropout ratio for the attention probabilities.
59
+ activation_dropout (`float`, *optional*, defaults to 0.0):
60
+ The dropout ratio for activations inside the fully connected layer.
61
+ max_position_embeddings (`int`, *optional*, defaults to 2048):
62
+ The maximum sequence length that this model might ever be used with. Typically, set this to something large
63
+ just in case (e.g., 512 or 1024 or 2048).
64
+ initializer_factor (`float`, *optional*, defaults to 0.02):
65
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
66
+ layerdrop (`float`, *optional*, defaults to 0.0):
67
+ The LayerDrop probability for the decoder. See the [LayerDrop paper](see https://arxiv.org/abs/1909.11556)
68
+ for more details.
69
+ scale_embedding (`bool`, *optional*, defaults to `False`):
70
+ Scale embeddings by diving by sqrt(hidden_size).
71
+ use_cache (`bool`, *optional*, defaults to `True`):
72
+ Whether the model should return the last key/values attentions (not used by all models)
73
+ num_codebooks (`int`, *optional*, defaults to 4):
74
+ The number of parallel codebooks forwarded to the model.
75
+ tie_word_embeddings(`bool`, *optional*, defaults to `False`):
76
+ Whether input and output word embeddings should be tied.
77
+ rope_embeddings (`bool`, *optional*, defaults to `False`):
78
+ Whether to use ROPE or absolute positional embeddings.
79
+ rope_theta (`float`, *optional*, defaults to 100000.0):
80
+ The base period of the RoPE embeddings.
81
+ """
82
+
83
+ model_type = "parler_tts_decoder"
84
+ keys_to_ignore_at_inference = ["past_key_values"]
85
+
86
+ def __init__(
87
+ self,
88
+ vocab_size=2049, # vocab size = 2048 (encodec vocab size) + 1 (eos)
89
+ max_position_embeddings=2048,
90
+ num_hidden_layers=24,
91
+ ffn_dim=4096,
92
+ num_attention_heads=16,
93
+ layerdrop=0.0,
94
+ use_cache=True,
95
+ activation_function="gelu",
96
+ hidden_size=1024,
97
+ dropout=0.1,
98
+ attention_dropout=0.0,
99
+ activation_dropout=0.0,
100
+ initializer_factor=0.02,
101
+ scale_embedding=False,
102
+ num_codebooks=4,
103
+ pad_token_id=2048,
104
+ bos_token_id=2049,
105
+ eos_token_id=2048,
106
+ tie_word_embeddings=False,
107
+ rope_embeddings=False,
108
+ rope_theta=10_000.0,
109
+ **kwargs,
110
+ ):
111
+ self.vocab_size = vocab_size
112
+ self.max_position_embeddings = max_position_embeddings
113
+ self.hidden_size = hidden_size
114
+ self.ffn_dim = ffn_dim
115
+ self.num_hidden_layers = num_hidden_layers
116
+ self.num_attention_heads = num_attention_heads
117
+ self.dropout = dropout
118
+ self.attention_dropout = attention_dropout
119
+ self.activation_dropout = activation_dropout
120
+ self.activation_function = activation_function
121
+ self.initializer_factor = initializer_factor
122
+ self.layerdrop = layerdrop
123
+ self.use_cache = use_cache
124
+ self.scale_embedding = scale_embedding # scale factor will be sqrt(d_model) if True
125
+ self.num_codebooks = num_codebooks
126
+ self.rope_embeddings = rope_embeddings
127
+ self.rope_theta = rope_theta
128
+
129
+ super().__init__(
130
+ pad_token_id=pad_token_id,
131
+ bos_token_id=bos_token_id,
132
+ eos_token_id=eos_token_id,
133
+ tie_word_embeddings=tie_word_embeddings,
134
+ **kwargs,
135
+ )
136
+
137
+
138
+ class ParlerTTSConfig(PretrainedConfig):
139
+ r"""
140
+ This is the configuration class to store the configuration of a [`ParlerTTSModel`]. It is used to instantiate a
141
+ Parler-TTS model according to the specified arguments, defining the text encoder, audio encoder and Parler-TTS decoder
142
+ configs.
143
+
144
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
145
+ documentation from [`PretrainedConfig`] for more information.
146
+
147
+ Args:
148
+ vocab_size (`int`, *optional*, defaults to 1024):
149
+ Vocabulary size of the prompt token ids. Defines the number of different tokens that can be
150
+ represented by the `prompt_inputs_ids`.
151
+ prompt_cross_attention (`bool`, *optional*, defaults to `False`):
152
+ Whether to use cross-attention conditioning for the prompt (as well as the description).
153
+ kwargs (*optional*):
154
+ Dictionary of keyword arguments. Notably:
155
+
156
+ - **text_encoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that
157
+ defines the text encoder config.
158
+ - **audio_encoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that
159
+ defines the audio encoder config.
160
+ - **decoder** ([`PretrainedConfig`], *optional*) -- An instance of a configuration object that defines
161
+ the decoder config.
162
+
163
+ Example:
164
+
165
+ ```python
166
+ >>> from transformers import (
167
+ ... ParlerTTSConfig,
168
+ ... ParlerTTSDecoderConfig,
169
+ ... T5Config,
170
+ ... EncodecConfig,
171
+ ... ParlerTTSForConditionalGeneration,
172
+ ... )
173
+
174
+ >>> # Initializing text encoder, audio encoder, and decoder model configurations
175
+ >>> text_encoder_config = T5Config()
176
+ >>> audio_encoder_config = EncodecConfig()
177
+ >>> decoder_config = ParlerTTSDecoderConfig()
178
+
179
+ >>> configuration = ParlerTTSConfig.from_sub_models_config(
180
+ ... text_encoder_config, audio_encoder_config, decoder_config
181
+ ... )
182
+
183
+ >>> # Initializing a ParlerTTSForConditionalGeneration (with random weights) from the facebook/parler_tts-small style configuration
184
+ >>> model = ParlerTTSForConditionalGeneration(configuration)
185
+
186
+ >>> # Accessing the model configuration
187
+ >>> configuration = model.config
188
+ >>> config_text_encoder = model.config.text_encoder
189
+ >>> config_audio_encoder = model.config.audio_encoder
190
+ >>> config_decoder = model.config.decoder
191
+
192
+ >>> # Saving the model, including its configuration
193
+ >>> model.save_pretrained("parler_tts-model")
194
+
195
+ >>> # loading model and config from pretrained folder
196
+ >>> parler_tts_config = ParlerTTSConfig.from_pretrained("parler_tts-model")
197
+ >>> model = ParlerTTSForConditionalGeneration.from_pretrained("parler_tts-model", config=parler_tts_config)
198
+ ```"""
199
+
200
+ model_type = "parler_tts"
201
+ is_composition = True
202
+
203
+ def __init__(self, vocab_size=1024, prompt_cross_attention=False, **kwargs):
204
+ super().__init__(**kwargs)
205
+ if "text_encoder" not in kwargs or "audio_encoder" not in kwargs or "decoder" not in kwargs:
206
+ raise ValueError("Config has to be initialized with text_encoder, audio_encoder and decoder config")
207
+
208
+ text_encoder_config = kwargs.pop("text_encoder")
209
+ text_encoder_model_type = text_encoder_config.pop("model_type")
210
+
211
+ audio_encoder_config = kwargs.pop("audio_encoder")
212
+ audio_encoder_model_type = audio_encoder_config.pop("model_type")
213
+
214
+ decoder_config = kwargs.pop("decoder")
215
+
216
+ self.vocab_size = vocab_size
217
+ self.prompt_cross_attention = prompt_cross_attention
218
+ self.text_encoder = AutoConfig.for_model(text_encoder_model_type, **text_encoder_config)
219
+ self.audio_encoder = AutoConfig.for_model(audio_encoder_model_type, **audio_encoder_config)
220
+ self.decoder = ParlerTTSDecoderConfig(**decoder_config)
221
+ self.is_encoder_decoder = True
222
+
223
+ @classmethod
224
+ def from_sub_models_config(
225
+ cls,
226
+ text_encoder_config: PretrainedConfig,
227
+ audio_encoder_config: PretrainedConfig,
228
+ decoder_config: ParlerTTSDecoderConfig,
229
+ **kwargs,
230
+ ):
231
+ r"""
232
+ Instantiate a [`ParlerTTSConfig`] (or a derived class) from text encoder, audio encoder and decoder
233
+ configurations.
234
+
235
+ Returns:
236
+ [`ParlerTTSConfig`]: An instance of a configuration object
237
+ """
238
+
239
+ return cls(
240
+ text_encoder=text_encoder_config.to_dict(),
241
+ audio_encoder=audio_encoder_config.to_dict(),
242
+ decoder=decoder_config.to_dict(),
243
+ **kwargs,
244
+ )
245
+
246
+ @property
247
+ # This is a property because you might want to change the codec model on the fly
248
+ def sampling_rate(self):
249
+ return self.audio_encoder.sampling_rate
parler_tts/dac_wrapper/__init__.py ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ from .configuration_dac import DACConfig
2
+ from .modeling_dac import DACModel
parler_tts/dac_wrapper/__pycache__/__init__.cpython-311.pyc ADDED
Binary file (293 Bytes). View file
 
parler_tts/dac_wrapper/__pycache__/configuration_dac.cpython-311.pyc ADDED
Binary file (1.33 kB). View file
 
parler_tts/dac_wrapper/__pycache__/modeling_dac.cpython-311.pyc ADDED
Binary file (6.28 kB). View file
 
parler_tts/dac_wrapper/configuration_dac.py ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+ from typing import List
3
+
4
+
5
+ class DACConfig(PretrainedConfig):
6
+ model_type = "dac"
7
+
8
+ def __init__(
9
+ self,
10
+ num_codebooks: int = 9,
11
+ model_bitrate: int = 8, # kbps
12
+ codebook_size: int = 1024,
13
+ latent_dim: int = 1024,
14
+ frame_rate: int = 86,
15
+ sampling_rate: int = 44100,
16
+ **kwargs,
17
+ ):
18
+ self.codebook_size = codebook_size
19
+ self.model_bitrate = model_bitrate
20
+ self.latent_dim = latent_dim
21
+ self.num_codebooks = num_codebooks
22
+ self.frame_rate = frame_rate
23
+ self.sampling_rate = sampling_rate
24
+
25
+ super().__init__(**kwargs)
parler_tts/dac_wrapper/modeling_dac.py ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+
3
+ from transformers import PreTrainedModel
4
+ from transformers.models.encodec.modeling_encodec import EncodecEncoderOutput, EncodecDecoderOutput
5
+ from .configuration_dac import DACConfig
6
+
7
+ from dac.model import DAC
8
+
9
+
10
+ # model doesn't support batching yet
11
+
12
+
13
+ class DACModel(PreTrainedModel):
14
+ config_class = DACConfig
15
+
16
+ def __init__(self, config):
17
+ super().__init__(config)
18
+
19
+ self.model = DAC(
20
+ n_codebooks=config.num_codebooks,
21
+ latent_dim=config.latent_dim,
22
+ codebook_size=config.codebook_size,
23
+ )
24
+
25
+ def encode(
26
+ self, input_values, padding_mask=None, bandwidth=None, return_dict=None, n_quantizers=None, sample_rate=None
27
+ ):
28
+ """
29
+ Encodes the input audio waveform into discrete codes.
30
+
31
+ Args:
32
+ input_values (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`):
33
+ Float values of the input audio waveform.
34
+ padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`):
35
+ Padding mask used to pad the `input_values`.
36
+ bandwidth (`float`, *optional*):
37
+ Not used, kept to have the same inferface as HF encodec.
38
+ n_quantizers (`int`, *optional*) :
39
+ Number of quantizers to use, by default None
40
+ If None, all quantizers are used.
41
+ sample_rate (`int`, *optional*) :
42
+ Signal sampling_rate
43
+
44
+ Returns:
45
+ A list of frames containing the discrete encoded codes for the input audio waveform, along with rescaling
46
+ factors for each chunk when `normalize` is True. Each frames is a tuple `(codebook, scale)`, with
47
+ `codebook` of shape `[batch_size, num_codebooks, frames]`.
48
+ Scale is not used here.
49
+
50
+ """
51
+ _, channels, input_length = input_values.shape
52
+
53
+ if channels < 1 or channels > 2:
54
+ raise ValueError(f"Number of audio channels must be 1 or 2, but got {channels}")
55
+
56
+ audio_data = self.model.preprocess(input_values, sample_rate)
57
+
58
+ return_dict = return_dict if return_dict is not None else self.config.return_dict
59
+
60
+ # TODO: for now, no chunk length
61
+
62
+ chunk_length = None # self.config.chunk_length
63
+ if chunk_length is None:
64
+ chunk_length = input_length
65
+ stride = input_length
66
+ else:
67
+ stride = self.config.chunk_stride
68
+
69
+ if padding_mask is None:
70
+ padding_mask = torch.ones_like(input_values).bool()
71
+
72
+ encoded_frames = []
73
+ scales = []
74
+
75
+ step = chunk_length - stride
76
+ if (input_length % stride) - step != 0:
77
+ raise ValueError(
78
+ "The input length is not properly padded for batched chunked decoding. Make sure to pad the input correctly."
79
+ )
80
+
81
+ for offset in range(0, input_length - step, stride):
82
+ mask = padding_mask[..., offset : offset + chunk_length].bool()
83
+ frame = audio_data[:, :, offset : offset + chunk_length]
84
+
85
+ scale = None
86
+
87
+ _, encoded_frame, _, _, _ = self.model.encode(frame, n_quantizers=n_quantizers)
88
+ encoded_frames.append(encoded_frame)
89
+ scales.append(scale)
90
+
91
+ encoded_frames = torch.stack(encoded_frames)
92
+
93
+ if not return_dict:
94
+ return (encoded_frames, scales)
95
+
96
+ return EncodecEncoderOutput(encoded_frames, scales)
97
+
98
+ def decode(
99
+ self,
100
+ audio_codes,
101
+ audio_scales,
102
+ padding_mask=None,
103
+ return_dict=None,
104
+ ):
105
+ """
106
+ Decodes the given frames into an output audio waveform.
107
+
108
+ Note that the output might be a bit bigger than the input. In that case, any extra steps at the end can be
109
+ trimmed.
110
+
111
+ Args:
112
+ audio_codes (`torch.FloatTensor` of shape `(batch_size, nb_chunks, chunk_length)`, *optional*):
113
+ Discret code embeddings computed using `model.encode`.
114
+ audio_scales (`torch.Tensor` of shape `(batch_size, nb_chunks)`, *optional*):
115
+ Not used, kept to have the same inferface as HF encodec.
116
+ padding_mask (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`):
117
+ Padding mask used to pad the `input_values`.
118
+ Not used yet, kept to have the same inferface as HF encodec.
119
+ return_dict (`bool`, *optional*):
120
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
121
+
122
+ """
123
+ return_dict = return_dict or self.config.return_dict
124
+
125
+ # TODO: for now, no chunk length
126
+
127
+ if len(audio_codes) != 1:
128
+ raise ValueError(f"Expected one frame, got {len(audio_codes)}")
129
+
130
+ audio_values = self.model.quantizer.from_codes(audio_codes.squeeze(0))[0]
131
+ audio_values = self.model.decode(audio_values)
132
+ if not return_dict:
133
+ return (audio_values,)
134
+ return EncodecDecoderOutput(audio_values)
135
+
136
+ def forward(self, tensor):
137
+ raise ValueError(f"`DACModel.forward` not implemented yet")
parler_tts/modeling_parler_tts.py ADDED
The diff for this file is too large to render. See raw diff
 
preprocessor_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "chunk_length_s": null,
3
+ "feature_extractor_type": "EncodecFeatureExtractor",
4
+ "feature_size": 1,
5
+ "overlap": null,
6
+ "padding_side": "right",
7
+ "padding_value": 0.0,
8
+ "return_attention_mask": true,
9
+ "sampling_rate": 44100
10
+ }
slurm_job.slurm ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+ #SBATCH --job-name=parler-tts
3
+ #SBATCH --nodes=1
4
+ # set 48h for job wall time limit
5
+ #SBATCH --time=48:00:00
6
+ #SBATCH --ntasks-per-node=1 # crucial - only 1 task per dist per node!
7
+ #SBATCH --cpus-per-task=32
8
+ #SBATCH --gres=gpu:8
9
+ #SBATCH --partition=hopper-prod
10
+ #SBATCH --output=/fsx/sanchit/logs/%x-%j.out
11
+
12
+ set -x -e
13
+
14
+ # START EDIT
15
+ source ~/.bashrc
16
+ source /fsx/sanchit/miniconda3/bin/activate venv
17
+
18
+ LOG_PATH="/fsx/sanchit/logs/main_log.txt"
19
+ SAVE_DIR="/fsx/sanchit"
20
+ # END EDIT
21
+
22
+ echo "START TIME: $(date)"
23
+
24
+ GPUS_PER_NODE=8
25
+ NNODES=$SLURM_NNODES
26
+
27
+ # so processes know who to talk to
28
+ MASTER_ADDR=`scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1`
29
+
30
+ # From https://i.hsfzxjy.site/2021-03-10-obtain-a-random-unused-tcp-port-with-bash/
31
+ function unused_port() {
32
+ N=${1:-1}
33
+ comm -23 \
34
+ <(seq "1025" "65535" | sort) \
35
+ <(ss -Htan |
36
+ awk '{print $4}' |
37
+ cut -d':' -f2 |
38
+ sort -u) |
39
+ shuf |
40
+ head -n "$N"
41
+ }
42
+ MASTER_PORT=$(unused_port)
43
+
44
+ # export TORCH_CPP_LOG_LEVEL=INFO
45
+ # export TORCH_DISTRIBUTED_DEBUG=DETAIL
46
+
47
+ export LAUNCHER="python -u -m accelerate.commands.launch --config_file ./accelerate_config.yaml"
48
+
49
+ export PROGRAM="./training/run_parler_tts_training.py ./starting_point_0.01_rope.json"
50
+ export CMD="$LAUNCHER $PROGRAM"
51
+ echo $CMD
52
+
53
+ SRUN_ARGS=" \
54
+ --wait=60 \
55
+ --kill-on-bad-exit=1 \
56
+ "
57
+
58
+ # py-spy top -s -i -n -- $LAUNCHER --node_rank $SLURM_PROCID --role $SLURMD_NODENAME: $CMD
59
+ clear; srun $SRUN_ARGS --jobid $SLURM_JOB_ID bash -c "$CMD" 2>&1 | tee -a $SAVE_DIR/logs/main_log.txt
60
+
61
+
62
+ # srun error handling:
63
+ # --wait=60: wait 60 sec after the first task terminates before terminating all remaining tasks
64
+ # --kill-on-bad-exit=1: terminate a step if any task exits with a non-zero exit code
65
+
66
+ # SRUN_ARGS=" \
67
+ # --wait=60 \
68
+ # --kill-on-bad-exit=1 \
69
+ # "
70
+ #
71
+ # # py-spy top -s -i -n -- $LAUNCHER --node_rank $SLURM_PROCID --role $SLURMD_NODENAME: $CMD
72
+ # clear; srun $SRUN_ARGS --jobid $SLURM_JOBID bash -c "$CMD" 2>&1 | tee -a $SAVE_DIR/logs/main_log.txt
73
+
74
+ echo "END TIME: $(date)"
special_tokens_map.json ADDED
@@ -0,0 +1,125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<extra_id_0>",
4
+ "<extra_id_1>",
5
+ "<extra_id_2>",
6
+ "<extra_id_3>",
7
+ "<extra_id_4>",
8
+ "<extra_id_5>",
9
+ "<extra_id_6>",
10
+ "<extra_id_7>",
11
+ "<extra_id_8>",
12
+ "<extra_id_9>",
13
+ "<extra_id_10>",
14
+ "<extra_id_11>",
15
+ "<extra_id_12>",
16
+ "<extra_id_13>",
17
+ "<extra_id_14>",
18
+ "<extra_id_15>",
19
+ "<extra_id_16>",
20
+ "<extra_id_17>",
21
+ "<extra_id_18>",
22
+ "<extra_id_19>",
23
+ "<extra_id_20>",
24
+ "<extra_id_21>",
25
+ "<extra_id_22>",
26
+ "<extra_id_23>",
27
+ "<extra_id_24>",
28
+ "<extra_id_25>",
29
+ "<extra_id_26>",
30
+ "<extra_id_27>",
31
+ "<extra_id_28>",
32
+ "<extra_id_29>",
33
+ "<extra_id_30>",
34
+ "<extra_id_31>",
35
+ "<extra_id_32>",
36
+ "<extra_id_33>",
37
+ "<extra_id_34>",
38
+ "<extra_id_35>",
39
+ "<extra_id_36>",
40
+ "<extra_id_37>",
41
+ "<extra_id_38>",
42
+ "<extra_id_39>",
43
+ "<extra_id_40>",
44
+ "<extra_id_41>",
45
+ "<extra_id_42>",
46
+ "<extra_id_43>",
47
+ "<extra_id_44>",
48
+ "<extra_id_45>",
49
+ "<extra_id_46>",
50
+ "<extra_id_47>",
51
+ "<extra_id_48>",
52
+ "<extra_id_49>",
53
+ "<extra_id_50>",
54
+ "<extra_id_51>",
55
+ "<extra_id_52>",
56
+ "<extra_id_53>",
57
+ "<extra_id_54>",
58
+ "<extra_id_55>",
59
+ "<extra_id_56>",
60
+ "<extra_id_57>",
61
+ "<extra_id_58>",
62
+ "<extra_id_59>",
63
+ "<extra_id_60>",
64
+ "<extra_id_61>",
65
+ "<extra_id_62>",
66
+ "<extra_id_63>",
67
+ "<extra_id_64>",
68
+ "<extra_id_65>",
69
+ "<extra_id_66>",
70
+ "<extra_id_67>",
71
+ "<extra_id_68>",
72
+ "<extra_id_69>",
73
+ "<extra_id_70>",
74
+ "<extra_id_71>",
75
+ "<extra_id_72>",
76
+ "<extra_id_73>",
77
+ "<extra_id_74>",
78
+ "<extra_id_75>",
79
+ "<extra_id_76>",
80
+ "<extra_id_77>",
81
+ "<extra_id_78>",
82
+ "<extra_id_79>",
83
+ "<extra_id_80>",
84
+ "<extra_id_81>",
85
+ "<extra_id_82>",
86
+ "<extra_id_83>",
87
+ "<extra_id_84>",
88
+ "<extra_id_85>",
89
+ "<extra_id_86>",
90
+ "<extra_id_87>",
91
+ "<extra_id_88>",
92
+ "<extra_id_89>",
93
+ "<extra_id_90>",
94
+ "<extra_id_91>",
95
+ "<extra_id_92>",
96
+ "<extra_id_93>",
97
+ "<extra_id_94>",
98
+ "<extra_id_95>",
99
+ "<extra_id_96>",
100
+ "<extra_id_97>",
101
+ "<extra_id_98>",
102
+ "<extra_id_99>"
103
+ ],
104
+ "eos_token": {
105
+ "content": "</s>",
106
+ "lstrip": false,
107
+ "normalized": false,
108
+ "rstrip": false,
109
+ "single_word": false
110
+ },
111
+ "pad_token": {
112
+ "content": "<pad>",
113
+ "lstrip": false,
114
+ "normalized": false,
115
+ "rstrip": false,
116
+ "single_word": false
117
+ },
118
+ "unk_token": {
119
+ "content": "<unk>",
120
+ "lstrip": false,
121
+ "normalized": false,
122
+ "rstrip": false,
123
+ "single_word": false
124
+ }
125
+ }
spiece.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d60acb128cf7b7f2536e8f38a5b18a05535c9e14c7a355904270e15b0945ea86
3
+ size 791656
starting_point_0.01_rope.json ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_name_or_path": "parler-tts/parler-tts-untrained-600M-cross-attention-rope",
3
+ "save_to_disk": "/fsx/sanchit/10k_hours_processed_punctuated",
4
+ "temporary_save_to_disk": "/scratch/tmp_dataset_audio/",
5
+ "push_to_hub": true,
6
+
7
+
8
+ "feature_extractor_name":"ylacombe/dac_44khZ_8kbps",
9
+ "description_tokenizer_name":"google/flan-t5-base",
10
+ "prompt_tokenizer_name":"google/flan-t5-base",
11
+
12
+ "report_to": ["wandb"],
13
+ "wandb_run_name": "parler-tts-600M-cross-attention-rope-decayed",
14
+ "overwrite_output_dir": false,
15
+ "save_total_limit": 2,
16
+ "output_dir": "./",
17
+
18
+ "train_dataset_name": "blabble-io/libritts_r+blabble-io/libritts_r+blabble-io/libritts_r+parler-tts/mls_eng_10k",
19
+ "train_metadata_dataset_name": "parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/mls-eng-10k-tags_tagged_10k_generated",
20
+ "train_dataset_config_name": "clean+clean+other+default",
21
+ "train_split_name": "train.clean.360+train.clean.100+train.other.500+train",
22
+
23
+ "eval_dataset_name": "blabble-io/libritts_r+parler-tts/mls_eng_10k",
24
+ "eval_metadata_dataset_name": "parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/mls-eng-10k-tags_tagged_10k_generated",
25
+ "eval_dataset_config_name": "other+default",
26
+ "eval_split_name": "test.other+test",
27
+
28
+ "target_audio_column_name": "audio",
29
+ "description_column_name": "text_description",
30
+ "prompt_column_name": "text",
31
+
32
+ "max_eval_samples": 96,
33
+
34
+ "max_duration_in_seconds": 30,
35
+ "min_duration_in_seconds": 2.0,
36
+ "max_text_length": 400,
37
+
38
+ "group_by_length": true,
39
+
40
+ "add_audio_samples_to_wandb": true,
41
+ "id_column_name": "id",
42
+
43
+ "preprocessing_num_workers": 8,
44
+
45
+ "do_train": true,
46
+ "num_train_epochs": 15,
47
+ "gradient_accumulation_steps": 8,
48
+ "gradient_checkpointing": false,
49
+ "per_device_train_batch_size": 3,
50
+ "learning_rate": 0.00095,
51
+ "adam_beta1": 0.9,
52
+ "adam_beta2": 0.99,
53
+ "weight_decay": 0.01,
54
+
55
+ "lr_scheduler_type": "cosine",
56
+ "warmup_steps": 20000,
57
+
58
+
59
+ "logging_steps": 1000,
60
+ "freeze_text_encoder": true,
61
+
62
+
63
+ "do_eval": true,
64
+ "predict_with_generate": true,
65
+ "include_inputs_for_metrics": true,
66
+ "evaluation_strategy": "steps",
67
+ "eval_steps": 10000,
68
+ "save_steps": 10000,
69
+
70
+ "per_device_eval_batch_size": 12,
71
+
72
+ "audio_encoder_per_device_batch_size":20,
73
+ "dtype": "bfloat16",
74
+ "seed": 456,
75
+ "ddp_timeout": 7200,
76
+
77
+ "dataloader_num_workers":8
78
+ }
79
+
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,938 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<pad>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "</s>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "<unk>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "32000": {
28
+ "content": "<extra_id_99>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "32001": {
36
+ "content": "<extra_id_98>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "32002": {
44
+ "content": "<extra_id_97>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ },
51
+ "32003": {
52
+ "content": "<extra_id_96>",
53
+ "lstrip": false,
54
+ "normalized": false,
55
+ "rstrip": false,
56
+ "single_word": false,
57
+ "special": true
58
+ },
59
+ "32004": {
60
+ "content": "<extra_id_95>",
61
+ "lstrip": false,
62
+ "normalized": false,
63
+ "rstrip": false,
64
+ "single_word": false,
65
+ "special": true
66
+ },
67
+ "32005": {
68
+ "content": "<extra_id_94>",
69
+ "lstrip": false,
70
+ "normalized": false,
71
+ "rstrip": false,
72
+ "single_word": false,
73
+ "special": true
74
+ },
75
+ "32006": {
76
+ "content": "<extra_id_93>",
77
+ "lstrip": false,
78
+ "normalized": false,
79
+ "rstrip": false,
80
+ "single_word": false,
81
+ "special": true
82
+ },
83
+ "32007": {
84
+ "content": "<extra_id_92>",
85
+ "lstrip": false,
86
+ "normalized": false,
87
+ "rstrip": false,
88
+ "single_word": false,
89
+ "special": true
90
+ },
91
+ "32008": {
92
+ "content": "<extra_id_91>",
93
+ "lstrip": false,
94
+ "normalized": false,
95
+ "rstrip": false,
96
+ "single_word": false,
97
+ "special": true
98
+ },
99
+ "32009": {
100
+ "content": "<extra_id_90>",
101
+ "lstrip": false,
102
+ "normalized": false,
103
+ "rstrip": false,
104
+ "single_word": false,
105
+ "special": true
106
+ },
107
+ "32010": {
108
+ "content": "<extra_id_89>",
109
+ "lstrip": false,
110
+ "normalized": false,
111
+ "rstrip": false,
112
+ "single_word": false,
113
+ "special": true
114
+ },
115
+ "32011": {
116
+ "content": "<extra_id_88>",
117
+ "lstrip": false,
118
+ "normalized": false,
119
+ "rstrip": false,
120
+ "single_word": false,
121
+ "special": true
122
+ },
123
+ "32012": {
124
+ "content": "<extra_id_87>",
125
+ "lstrip": false,
126
+ "normalized": false,
127
+ "rstrip": false,
128
+ "single_word": false,
129
+ "special": true
130
+ },
131
+ "32013": {
132
+ "content": "<extra_id_86>",
133
+ "lstrip": false,
134
+ "normalized": false,
135
+ "rstrip": false,
136
+ "single_word": false,
137
+ "special": true
138
+ },
139
+ "32014": {
140
+ "content": "<extra_id_85>",
141
+ "lstrip": false,
142
+ "normalized": false,
143
+ "rstrip": false,
144
+ "single_word": false,
145
+ "special": true
146
+ },
147
+ "32015": {
148
+ "content": "<extra_id_84>",
149
+ "lstrip": false,
150
+ "normalized": false,
151
+ "rstrip": false,
152
+ "single_word": false,
153
+ "special": true
154
+ },
155
+ "32016": {
156
+ "content": "<extra_id_83>",
157
+ "lstrip": false,
158
+ "normalized": false,
159
+ "rstrip": false,
160
+ "single_word": false,
161
+ "special": true
162
+ },
163
+ "32017": {
164
+ "content": "<extra_id_82>",
165
+ "lstrip": false,
166
+ "normalized": false,
167
+ "rstrip": false,
168
+ "single_word": false,
169
+ "special": true
170
+ },
171
+ "32018": {
172
+ "content": "<extra_id_81>",
173
+ "lstrip": false,
174
+ "normalized": false,
175
+ "rstrip": false,
176
+ "single_word": false,
177
+ "special": true
178
+ },
179
+ "32019": {
180
+ "content": "<extra_id_80>",
181
+ "lstrip": false,
182
+ "normalized": false,
183
+ "rstrip": false,
184
+ "single_word": false,
185
+ "special": true
186
+ },
187
+ "32020": {
188
+ "content": "<extra_id_79>",
189
+ "lstrip": false,
190
+ "normalized": false,
191
+ "rstrip": false,
192
+ "single_word": false,
193
+ "special": true
194
+ },
195
+ "32021": {
196
+ "content": "<extra_id_78>",
197
+ "lstrip": false,
198
+ "normalized": false,
199
+ "rstrip": false,
200
+ "single_word": false,
201
+ "special": true
202
+ },
203
+ "32022": {
204
+ "content": "<extra_id_77>",
205
+ "lstrip": false,
206
+ "normalized": false,
207
+ "rstrip": false,
208
+ "single_word": false,
209
+ "special": true
210
+ },
211
+ "32023": {
212
+ "content": "<extra_id_76>",
213
+ "lstrip": false,
214
+ "normalized": false,
215
+ "rstrip": false,
216
+ "single_word": false,
217
+ "special": true
218
+ },
219
+ "32024": {
220
+ "content": "<extra_id_75>",
221
+ "lstrip": false,
222
+ "normalized": false,
223
+ "rstrip": false,
224
+ "single_word": false,
225
+ "special": true
226
+ },
227
+ "32025": {
228
+ "content": "<extra_id_74>",
229
+ "lstrip": false,
230
+ "normalized": false,
231
+ "rstrip": false,
232
+ "single_word": false,
233
+ "special": true
234
+ },
235
+ "32026": {
236
+ "content": "<extra_id_73>",
237
+ "lstrip": false,
238
+ "normalized": false,
239
+ "rstrip": false,
240
+ "single_word": false,
241
+ "special": true
242
+ },
243
+ "32027": {
244
+ "content": "<extra_id_72>",
245
+ "lstrip": false,
246
+ "normalized": false,
247
+ "rstrip": false,
248
+ "single_word": false,
249
+ "special": true
250
+ },
251
+ "32028": {
252
+ "content": "<extra_id_71>",
253
+ "lstrip": false,
254
+ "normalized": false,
255
+ "rstrip": false,
256
+ "single_word": false,
257
+ "special": true
258
+ },
259
+ "32029": {
260
+ "content": "<extra_id_70>",
261
+ "lstrip": false,
262
+ "normalized": false,
263
+ "rstrip": false,
264
+ "single_word": false,
265
+ "special": true
266
+ },
267
+ "32030": {
268
+ "content": "<extra_id_69>",
269
+ "lstrip": false,
270
+ "normalized": false,
271
+ "rstrip": false,
272
+ "single_word": false,
273
+ "special": true
274
+ },
275
+ "32031": {
276
+ "content": "<extra_id_68>",
277
+ "lstrip": false,
278
+ "normalized": false,
279
+ "rstrip": false,
280
+ "single_word": false,
281
+ "special": true
282
+ },
283
+ "32032": {
284
+ "content": "<extra_id_67>",
285
+ "lstrip": false,
286
+ "normalized": false,
287
+ "rstrip": false,
288
+ "single_word": false,
289
+ "special": true
290
+ },
291
+ "32033": {
292
+ "content": "<extra_id_66>",
293
+ "lstrip": false,
294
+ "normalized": false,
295
+ "rstrip": false,
296
+ "single_word": false,
297
+ "special": true
298
+ },
299
+ "32034": {
300
+ "content": "<extra_id_65>",
301
+ "lstrip": false,
302
+ "normalized": false,
303
+ "rstrip": false,
304
+ "single_word": false,
305
+ "special": true
306
+ },
307
+ "32035": {
308
+ "content": "<extra_id_64>",
309
+ "lstrip": false,
310
+ "normalized": false,
311
+ "rstrip": false,
312
+ "single_word": false,
313
+ "special": true
314
+ },
315
+ "32036": {
316
+ "content": "<extra_id_63>",
317
+ "lstrip": false,
318
+ "normalized": false,
319
+ "rstrip": false,
320
+ "single_word": false,
321
+ "special": true
322
+ },
323
+ "32037": {
324
+ "content": "<extra_id_62>",
325
+ "lstrip": false,
326
+ "normalized": false,
327
+ "rstrip": false,
328
+ "single_word": false,
329
+ "special": true
330
+ },
331
+ "32038": {
332
+ "content": "<extra_id_61>",
333
+ "lstrip": false,
334
+ "normalized": false,
335
+ "rstrip": false,
336
+ "single_word": false,
337
+ "special": true
338
+ },
339
+ "32039": {
340
+ "content": "<extra_id_60>",
341
+ "lstrip": false,
342
+ "normalized": false,
343
+ "rstrip": false,
344
+ "single_word": false,
345
+ "special": true
346
+ },
347
+ "32040": {
348
+ "content": "<extra_id_59>",
349
+ "lstrip": false,
350
+ "normalized": false,
351
+ "rstrip": false,
352
+ "single_word": false,
353
+ "special": true
354
+ },
355
+ "32041": {
356
+ "content": "<extra_id_58>",
357
+ "lstrip": false,
358
+ "normalized": false,
359
+ "rstrip": false,
360
+ "single_word": false,
361
+ "special": true
362
+ },
363
+ "32042": {
364
+ "content": "<extra_id_57>",
365
+ "lstrip": false,
366
+ "normalized": false,
367
+ "rstrip": false,
368
+ "single_word": false,
369
+ "special": true
370
+ },
371
+ "32043": {
372
+ "content": "<extra_id_56>",
373
+ "lstrip": false,
374
+ "normalized": false,
375
+ "rstrip": false,
376
+ "single_word": false,
377
+ "special": true
378
+ },
379
+ "32044": {
380
+ "content": "<extra_id_55>",
381
+ "lstrip": false,
382
+ "normalized": false,
383
+ "rstrip": false,
384
+ "single_word": false,
385
+ "special": true
386
+ },
387
+ "32045": {
388
+ "content": "<extra_id_54>",
389
+ "lstrip": false,
390
+ "normalized": false,
391
+ "rstrip": false,
392
+ "single_word": false,
393
+ "special": true
394
+ },
395
+ "32046": {
396
+ "content": "<extra_id_53>",
397
+ "lstrip": false,
398
+ "normalized": false,
399
+ "rstrip": false,
400
+ "single_word": false,
401
+ "special": true
402
+ },
403
+ "32047": {
404
+ "content": "<extra_id_52>",
405
+ "lstrip": false,
406
+ "normalized": false,
407
+ "rstrip": false,
408
+ "single_word": false,
409
+ "special": true
410
+ },
411
+ "32048": {
412
+ "content": "<extra_id_51>",
413
+ "lstrip": false,
414
+ "normalized": false,
415
+ "rstrip": false,
416
+ "single_word": false,
417
+ "special": true
418
+ },
419
+ "32049": {
420
+ "content": "<extra_id_50>",
421
+ "lstrip": false,
422
+ "normalized": false,
423
+ "rstrip": false,
424
+ "single_word": false,
425
+ "special": true
426
+ },
427
+ "32050": {
428
+ "content": "<extra_id_49>",
429
+ "lstrip": false,
430
+ "normalized": false,
431
+ "rstrip": false,
432
+ "single_word": false,
433
+ "special": true
434
+ },
435
+ "32051": {
436
+ "content": "<extra_id_48>",
437
+ "lstrip": false,
438
+ "normalized": false,
439
+ "rstrip": false,
440
+ "single_word": false,
441
+ "special": true
442
+ },
443
+ "32052": {
444
+ "content": "<extra_id_47>",
445
+ "lstrip": false,
446
+ "normalized": false,
447
+ "rstrip": false,
448
+ "single_word": false,
449
+ "special": true
450
+ },
451
+ "32053": {
452
+ "content": "<extra_id_46>",
453
+ "lstrip": false,
454
+ "normalized": false,
455
+ "rstrip": false,
456
+ "single_word": false,
457
+ "special": true
458
+ },
459
+ "32054": {
460
+ "content": "<extra_id_45>",
461
+ "lstrip": false,
462
+ "normalized": false,
463
+ "rstrip": false,
464
+ "single_word": false,
465
+ "special": true
466
+ },
467
+ "32055": {
468
+ "content": "<extra_id_44>",
469
+ "lstrip": false,
470
+ "normalized": false,
471
+ "rstrip": false,
472
+ "single_word": false,
473
+ "special": true
474
+ },
475
+ "32056": {
476
+ "content": "<extra_id_43>",
477
+ "lstrip": false,
478
+ "normalized": false,
479
+ "rstrip": false,
480
+ "single_word": false,
481
+ "special": true
482
+ },
483
+ "32057": {
484
+ "content": "<extra_id_42>",
485
+ "lstrip": false,
486
+ "normalized": false,
487
+ "rstrip": false,
488
+ "single_word": false,
489
+ "special": true
490
+ },
491
+ "32058": {
492
+ "content": "<extra_id_41>",
493
+ "lstrip": false,
494
+ "normalized": false,
495
+ "rstrip": false,
496
+ "single_word": false,
497
+ "special": true
498
+ },
499
+ "32059": {
500
+ "content": "<extra_id_40>",
501
+ "lstrip": false,
502
+ "normalized": false,
503
+ "rstrip": false,
504
+ "single_word": false,
505
+ "special": true
506
+ },
507
+ "32060": {
508
+ "content": "<extra_id_39>",
509
+ "lstrip": false,
510
+ "normalized": false,
511
+ "rstrip": false,
512
+ "single_word": false,
513
+ "special": true
514
+ },
515
+ "32061": {
516
+ "content": "<extra_id_38>",
517
+ "lstrip": false,
518
+ "normalized": false,
519
+ "rstrip": false,
520
+ "single_word": false,
521
+ "special": true
522
+ },
523
+ "32062": {
524
+ "content": "<extra_id_37>",
525
+ "lstrip": false,
526
+ "normalized": false,
527
+ "rstrip": false,
528
+ "single_word": false,
529
+ "special": true
530
+ },
531
+ "32063": {
532
+ "content": "<extra_id_36>",
533
+ "lstrip": false,
534
+ "normalized": false,
535
+ "rstrip": false,
536
+ "single_word": false,
537
+ "special": true
538
+ },
539
+ "32064": {
540
+ "content": "<extra_id_35>",
541
+ "lstrip": false,
542
+ "normalized": false,
543
+ "rstrip": false,
544
+ "single_word": false,
545
+ "special": true
546
+ },
547
+ "32065": {
548
+ "content": "<extra_id_34>",
549
+ "lstrip": false,
550
+ "normalized": false,
551
+ "rstrip": false,
552
+ "single_word": false,
553
+ "special": true
554
+ },
555
+ "32066": {
556
+ "content": "<extra_id_33>",
557
+ "lstrip": false,
558
+ "normalized": false,
559
+ "rstrip": false,
560
+ "single_word": false,
561
+ "special": true
562
+ },
563
+ "32067": {
564
+ "content": "<extra_id_32>",
565
+ "lstrip": false,
566
+ "normalized": false,
567
+ "rstrip": false,
568
+ "single_word": false,
569
+ "special": true
570
+ },
571
+ "32068": {
572
+ "content": "<extra_id_31>",
573
+ "lstrip": false,
574
+ "normalized": false,
575
+ "rstrip": false,
576
+ "single_word": false,
577
+ "special": true
578
+ },
579
+ "32069": {
580
+ "content": "<extra_id_30>",
581
+ "lstrip": false,
582
+ "normalized": false,
583
+ "rstrip": false,
584
+ "single_word": false,
585
+ "special": true
586
+ },
587
+ "32070": {
588
+ "content": "<extra_id_29>",
589
+ "lstrip": false,
590
+ "normalized": false,
591
+ "rstrip": false,
592
+ "single_word": false,
593
+ "special": true
594
+ },
595
+ "32071": {
596
+ "content": "<extra_id_28>",
597
+ "lstrip": false,
598
+ "normalized": false,
599
+ "rstrip": false,
600
+ "single_word": false,
601
+ "special": true
602
+ },
603
+ "32072": {
604
+ "content": "<extra_id_27>",
605
+ "lstrip": false,
606
+ "normalized": false,
607
+ "rstrip": false,
608
+ "single_word": false,
609
+ "special": true
610
+ },
611
+ "32073": {
612
+ "content": "<extra_id_26>",
613
+ "lstrip": false,
614
+ "normalized": false,
615
+ "rstrip": false,
616
+ "single_word": false,
617
+ "special": true
618
+ },
619
+ "32074": {
620
+ "content": "<extra_id_25>",
621
+ "lstrip": false,
622
+ "normalized": false,
623
+ "rstrip": false,
624
+ "single_word": false,
625
+ "special": true
626
+ },
627
+ "32075": {
628
+ "content": "<extra_id_24>",
629
+ "lstrip": false,
630
+ "normalized": false,
631
+ "rstrip": false,
632
+ "single_word": false,
633
+ "special": true
634
+ },
635
+ "32076": {
636
+ "content": "<extra_id_23>",
637
+ "lstrip": false,
638
+ "normalized": false,
639
+ "rstrip": false,
640
+ "single_word": false,
641
+ "special": true
642
+ },
643
+ "32077": {
644
+ "content": "<extra_id_22>",
645
+ "lstrip": false,
646
+ "normalized": false,
647
+ "rstrip": false,
648
+ "single_word": false,
649
+ "special": true
650
+ },
651
+ "32078": {
652
+ "content": "<extra_id_21>",
653
+ "lstrip": false,
654
+ "normalized": false,
655
+ "rstrip": false,
656
+ "single_word": false,
657
+ "special": true
658
+ },
659
+ "32079": {
660
+ "content": "<extra_id_20>",
661
+ "lstrip": false,
662
+ "normalized": false,
663
+ "rstrip": false,
664
+ "single_word": false,
665
+ "special": true
666
+ },
667
+ "32080": {
668
+ "content": "<extra_id_19>",
669
+ "lstrip": false,
670
+ "normalized": false,
671
+ "rstrip": false,
672
+ "single_word": false,
673
+ "special": true
674
+ },
675
+ "32081": {
676
+ "content": "<extra_id_18>",
677
+ "lstrip": false,
678
+ "normalized": false,
679
+ "rstrip": false,
680
+ "single_word": false,
681
+ "special": true
682
+ },
683
+ "32082": {
684
+ "content": "<extra_id_17>",
685
+ "lstrip": false,
686
+ "normalized": false,
687
+ "rstrip": false,
688
+ "single_word": false,
689
+ "special": true
690
+ },
691
+ "32083": {
692
+ "content": "<extra_id_16>",
693
+ "lstrip": false,
694
+ "normalized": false,
695
+ "rstrip": false,
696
+ "single_word": false,
697
+ "special": true
698
+ },
699
+ "32084": {
700
+ "content": "<extra_id_15>",
701
+ "lstrip": false,
702
+ "normalized": false,
703
+ "rstrip": false,
704
+ "single_word": false,
705
+ "special": true
706
+ },
707
+ "32085": {
708
+ "content": "<extra_id_14>",
709
+ "lstrip": false,
710
+ "normalized": false,
711
+ "rstrip": false,
712
+ "single_word": false,
713
+ "special": true
714
+ },
715
+ "32086": {
716
+ "content": "<extra_id_13>",
717
+ "lstrip": false,
718
+ "normalized": false,
719
+ "rstrip": false,
720
+ "single_word": false,
721
+ "special": true
722
+ },
723
+ "32087": {
724
+ "content": "<extra_id_12>",
725
+ "lstrip": false,
726
+ "normalized": false,
727
+ "rstrip": false,
728
+ "single_word": false,
729
+ "special": true
730
+ },
731
+ "32088": {
732
+ "content": "<extra_id_11>",
733
+ "lstrip": false,
734
+ "normalized": false,
735
+ "rstrip": false,
736
+ "single_word": false,
737
+ "special": true
738
+ },
739
+ "32089": {
740
+ "content": "<extra_id_10>",
741
+ "lstrip": false,
742
+ "normalized": false,
743
+ "rstrip": false,
744
+ "single_word": false,
745
+ "special": true
746
+ },
747
+ "32090": {
748
+ "content": "<extra_id_9>",
749
+ "lstrip": false,
750
+ "normalized": false,
751
+ "rstrip": false,
752
+ "single_word": false,
753
+ "special": true
754
+ },
755
+ "32091": {
756
+ "content": "<extra_id_8>",
757
+ "lstrip": false,
758
+ "normalized": false,
759
+ "rstrip": false,
760
+ "single_word": false,
761
+ "special": true
762
+ },
763
+ "32092": {
764
+ "content": "<extra_id_7>",
765
+ "lstrip": false,
766
+ "normalized": false,
767
+ "rstrip": false,
768
+ "single_word": false,
769
+ "special": true
770
+ },
771
+ "32093": {
772
+ "content": "<extra_id_6>",
773
+ "lstrip": false,
774
+ "normalized": false,
775
+ "rstrip": false,
776
+ "single_word": false,
777
+ "special": true
778
+ },
779
+ "32094": {
780
+ "content": "<extra_id_5>",
781
+ "lstrip": false,
782
+ "normalized": false,
783
+ "rstrip": false,
784
+ "single_word": false,
785
+ "special": true
786
+ },
787
+ "32095": {
788
+ "content": "<extra_id_4>",
789
+ "lstrip": false,
790
+ "normalized": false,
791
+ "rstrip": false,
792
+ "single_word": false,
793
+ "special": true
794
+ },
795
+ "32096": {
796
+ "content": "<extra_id_3>",
797
+ "lstrip": false,
798
+ "normalized": false,
799
+ "rstrip": false,
800
+ "single_word": false,
801
+ "special": true
802
+ },
803
+ "32097": {
804
+ "content": "<extra_id_2>",
805
+ "lstrip": false,
806
+ "normalized": false,
807
+ "rstrip": false,
808
+ "single_word": false,
809
+ "special": true
810
+ },
811
+ "32098": {
812
+ "content": "<extra_id_1>",
813
+ "lstrip": false,
814
+ "normalized": false,
815
+ "rstrip": false,
816
+ "single_word": false,
817
+ "special": true
818
+ },
819
+ "32099": {
820
+ "content": "<extra_id_0>",
821
+ "lstrip": false,
822
+ "normalized": false,
823
+ "rstrip": false,
824
+ "single_word": false,
825
+ "special": true
826
+ }
827
+ },
828
+ "additional_special_tokens": [
829
+ "<extra_id_0>",
830
+ "<extra_id_1>",
831
+ "<extra_id_2>",
832
+ "<extra_id_3>",
833
+ "<extra_id_4>",
834
+ "<extra_id_5>",
835
+ "<extra_id_6>",
836
+ "<extra_id_7>",
837
+ "<extra_id_8>",
838
+ "<extra_id_9>",
839
+ "<extra_id_10>",
840
+ "<extra_id_11>",
841
+ "<extra_id_12>",
842
+ "<extra_id_13>",
843
+ "<extra_id_14>",
844
+ "<extra_id_15>",
845
+ "<extra_id_16>",
846
+ "<extra_id_17>",
847
+ "<extra_id_18>",
848
+ "<extra_id_19>",
849
+ "<extra_id_20>",
850
+ "<extra_id_21>",
851
+ "<extra_id_22>",
852
+ "<extra_id_23>",
853
+ "<extra_id_24>",
854
+ "<extra_id_25>",
855
+ "<extra_id_26>",
856
+ "<extra_id_27>",
857
+ "<extra_id_28>",
858
+ "<extra_id_29>",
859
+ "<extra_id_30>",
860
+ "<extra_id_31>",
861
+ "<extra_id_32>",
862
+ "<extra_id_33>",
863
+ "<extra_id_34>",
864
+ "<extra_id_35>",
865
+ "<extra_id_36>",
866
+ "<extra_id_37>",
867
+ "<extra_id_38>",
868
+ "<extra_id_39>",
869
+ "<extra_id_40>",
870
+ "<extra_id_41>",
871
+ "<extra_id_42>",
872
+ "<extra_id_43>",
873
+ "<extra_id_44>",
874
+ "<extra_id_45>",
875
+ "<extra_id_46>",
876
+ "<extra_id_47>",
877
+ "<extra_id_48>",
878
+ "<extra_id_49>",
879
+ "<extra_id_50>",
880
+ "<extra_id_51>",
881
+ "<extra_id_52>",
882
+ "<extra_id_53>",
883
+ "<extra_id_54>",
884
+ "<extra_id_55>",
885
+ "<extra_id_56>",
886
+ "<extra_id_57>",
887
+ "<extra_id_58>",
888
+ "<extra_id_59>",
889
+ "<extra_id_60>",
890
+ "<extra_id_61>",
891
+ "<extra_id_62>",
892
+ "<extra_id_63>",
893
+ "<extra_id_64>",
894
+ "<extra_id_65>",
895
+ "<extra_id_66>",
896
+ "<extra_id_67>",
897
+ "<extra_id_68>",
898
+ "<extra_id_69>",
899
+ "<extra_id_70>",
900
+ "<extra_id_71>",
901
+ "<extra_id_72>",
902
+ "<extra_id_73>",
903
+ "<extra_id_74>",
904
+ "<extra_id_75>",
905
+ "<extra_id_76>",
906
+ "<extra_id_77>",
907
+ "<extra_id_78>",
908
+ "<extra_id_79>",
909
+ "<extra_id_80>",
910
+ "<extra_id_81>",
911
+ "<extra_id_82>",
912
+ "<extra_id_83>",
913
+ "<extra_id_84>",
914
+ "<extra_id_85>",
915
+ "<extra_id_86>",
916
+ "<extra_id_87>",
917
+ "<extra_id_88>",
918
+ "<extra_id_89>",
919
+ "<extra_id_90>",
920
+ "<extra_id_91>",
921
+ "<extra_id_92>",
922
+ "<extra_id_93>",
923
+ "<extra_id_94>",
924
+ "<extra_id_95>",
925
+ "<extra_id_96>",
926
+ "<extra_id_97>",
927
+ "<extra_id_98>",
928
+ "<extra_id_99>"
929
+ ],
930
+ "clean_up_tokenization_spaces": true,
931
+ "eos_token": "</s>",
932
+ "extra_ids": 100,
933
+ "model_max_length": 512,
934
+ "pad_token": "<pad>",
935
+ "sp_model_kwargs": {},
936
+ "tokenizer_class": "T5Tokenizer",
937
+ "unk_token": "<unk>"
938
+ }
training/README.md ADDED
@@ -0,0 +1,211 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Training Parler-TTS
2
+
3
+ <a target="_blank" href="https://colab.research.google.com/github/ylacombe/scripts_and_notebooks/blob/main/Finetuning_Parler_TTS_on_a_single_speaker_dataset.ipynb">
4
+ <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
5
+ </a>
6
+
7
+ **TL;DR:** After having followed the [installation steps](#requirements), you can reproduce the [Parler-TTS Mini v0.1](https://huggingface.co/parler-tts/parler_tts_mini_v0.1) training recipe with the following command line:
8
+
9
+ ```sh
10
+ accelerate launch ./training/run_parler_tts_training.py ./helpers/training_configs/starting_point_0.01.json
11
+ ```
12
+
13
+ -------------
14
+
15
+ This sub-folder contains all the information to train or fine-tune your own Parler-TTS model. It consists of:
16
+ - [1. An introduction to the Parler-TTS architecture](#a-architecture)
17
+ - [2. First steps to get started](#b-getting-started)
18
+ - [3. Training guide](#c-training)
19
+
20
+ > [!IMPORTANT]
21
+ > You can also follow [this fine-tuning guide](https://colab.research.google.com/github/ylacombe/scripts_and_notebooks/blob/main/Finetuning_Parler_TTS_on_a_single_speaker_dataset.ipynb) on a mono-speaker dataset example.
22
+
23
+ ## 1. Architecture
24
+
25
+ At the moment, Parler-TTS architecture is a carbon copy of the [MusicGen architecture](https://huggingface.co/docs/transformers/v4.39.3/en/model_doc/musicgen#model-structure) and can be decomposed into three distinct stages:
26
+ 1. Text encoder: maps the text descriptions to a sequence of hidden-state representations. Parler-TTS uses a frozen text encoder initialised entirely from Flan-T5
27
+ 2. Parler-TTS decoder: a language model (LM) that auto-regressively generates audio tokens (or codes) conditional on the encoder hidden-state representations
28
+ 3. Audio codec: used to recover the audio waveform from the audio tokens predicted by the decoder. We use the [DAC model](https://github.com/descriptinc/descript-audio-codec) from Descript, although other codec models, such as [EnCodec](https://huggingface.co/facebook/encodec_48khz), can also be used
29
+
30
+ Parler-TTS however introduces some small tweaks:
31
+ - The text **description** is passed through the text encoder and used in the cross-attention layers of the decoder.
32
+ - The text **prompt** is simply passed through an embedding layer and concatenated to the decoder input hidden states.
33
+ - The audio encoder used is [**DAC**](https://descript.notion.site/Descript-Audio-Codec-11389fce0ce2419891d6591a68f814d5) instead of [Encodec](https://github.com/facebookresearch/encodec), as it exhibits better quality.
34
+
35
+
36
+ ## 2. Getting started
37
+
38
+ To get started, you need to follow a few steps:
39
+ 1. Install the requirements.
40
+ 2. Find or initialize the model you'll train on.
41
+ 3. Find and/or annotate the dataset you'll train your model on.
42
+
43
+ ### Requirements
44
+
45
+ The Parler-TTS code is written in [PyTorch](https://pytorch.org) and [Accelerate](https://huggingface.co/docs/accelerate/index). It uses some additional requirements, like [wandb](https://wandb.ai/), especially for logging and evaluation.
46
+
47
+ To install the package for training, you need to clone the repository from source...
48
+
49
+ ```bash
50
+ git clone https://github.com/huggingface/parler-tts.git
51
+ cd parler-tts
52
+ ```
53
+
54
+ ... And then install the requirements:
55
+
56
+ ```bash
57
+ pip install -e .[train]
58
+ ```
59
+
60
+ Optionally, you can create a wandb account and login to it by following [this guide](https://docs.wandb.ai/quickstart). [`wandb`](https://docs.wandb.ai/) allows for better tracking of the experiments metrics and losses.
61
+
62
+ You also have the option to configure Accelerate by running the following command. Note that you should set the number of GPUs you wish to use for training, and also the data type (dtype) to your preferred dtype for training/inference (e.g. `bfloat16` on A100 GPUs, `float16` on V100 GPUs, etc.):
63
+
64
+ ```bash
65
+ accelerate config
66
+ ```
67
+
68
+ Lastly, you can link you Hugging Face account so that you can push model repositories on the Hub. This will allow you to save your trained models on the Hub so that you can share them with the community. Run the command:
69
+
70
+ ```bash
71
+ git config --global credential.helper store
72
+ huggingface-cli login
73
+ ```
74
+ And then enter an authentication token from https://huggingface.co/settings/tokens. Create a new token if you do not have one already. You should make sure that this token has "write" privileges.
75
+
76
+ ### Initialize a model from scratch or use a pre-trained one.
77
+
78
+ Depending on your compute resources and your dataset, you need to choose between fine-tuning a pre-trained model and training a new model from scratch.
79
+
80
+ In that sense, we released a 600M checkpoint trained on 10.5K hours of annotated data under the repository id: [`parler-tts/parler_tts_mini_v0.1`](https://huggingface.co/parler-tts/parler_tts_mini_v0.1), that you can fine-tune for your own use-case.
81
+
82
+ You can also train you own model from scratch. You can find [here](/helpers/model_init_scripts/) examples on how to initialize a model from scratch. For example, you can initialize a dummy model with:
83
+
84
+ ```sh
85
+ python helpers/model_init_scripts/init_dummy_model.py ./parler-tts-untrained-dummy --text_model "google-t5/t5-small" --audio_model "parler-tts/dac_44khZ_8kbps"
86
+ ```
87
+
88
+ In the rest of this guide, and to reproduce the Parler-TTS Mini v0.1 training recipe, we'll use a 600M parameters model that we'll initialize with:
89
+
90
+ ```sh
91
+ python helpers/model_init_scripts/init_model_600M.py ./parler-tts-untrained-600M --text_model "google/flan-t5-base" --audio_model "parler-tts/dac_44khZ_8kbps"
92
+ ```
93
+
94
+
95
+ ### Create or find datasets
96
+
97
+ To train your own Parler-TTS, you need datasets with 3 main features:
98
+ - speech data
99
+ - text transcription of the speech data
100
+ - conditionning text description - that you can create using [Data-Speech](https://github.com/huggingface/dataspeech), a library that allows you to annotate the speaker and utterance characteristics with natural language description.
101
+
102
+ Note that we made the choice to use description of the main speech characteristics (speaker pitch, speaking rate, level of noise, etc.) but that you are free to use any handmade or generated text description that makes sense.
103
+
104
+ To train Parler-TTS Mini v0.1, we used:
105
+ * The full [LibriTTS-R dataset](https://huggingface.co/datasets/blabble-io/libritts_r), a 1K hours high-quality speech dataset.
106
+ * A [10K hours subset](https://huggingface.co/datasets/parler-tts/mls_eng_10k) of [Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech).
107
+
108
+ Both datasets have been annotated using the [Data-Speech](https://github.com/huggingface/dataspeech) recipe, respectively [here](https://huggingface.co/datasets/parler-tts/libritts_r_tags_tagged_10k_generated) and [here](https://huggingface.co/datasets/parler-tts/mls-eng-10k-tags_tagged_10k_generated).
109
+
110
+
111
+ ## 3. Training
112
+
113
+ The script [`run_parler_tts_training.py`](/training/run_parler_tts_training.py) is an end-to-end script that:
114
+ 1. load dataset(s) and merge them to the annotation dataset(s) if necessary
115
+ 2. pre-compute audio tokens
116
+ 3. train Parler-TTS
117
+
118
+ To train Parler-TTS Mini v0.1, we roughly used:
119
+
120
+ ```sh
121
+ accelerate launch ./training/run_parler_tts_training.py \
122
+ --model_name_or_path "./parler-tts-untrained-600M/parler-tts-untrained-600M/" \
123
+ --feature_extractor_name "parler-tts/dac_44khZ_8kbps" \
124
+ --description_tokenizer_name "google/flan-t5-base" \
125
+ --prompt_tokenizer_name "google/flan-t5-base" \
126
+ --report_to "wandb" \
127
+ --overwrite_output_dir true \
128
+ --train_dataset_name "blabble-io/libritts_r+blabble-io/libritts_r+blabble-io/libritts_r+parler-tts/mls_eng_10k" \
129
+ --train_metadata_dataset_name "parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/mls-eng-10k-tags_tagged_10k_generated" \
130
+ --train_dataset_config_name "clean+clean+other+default" \
131
+ --train_split_name "train.clean.360+train.clean.100+train.other.500+train" \
132
+ --eval_dataset_name "blabble-io/libritts_r+parler-tts/mls_eng_10k" \
133
+ --eval_metadata_dataset_name "parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/mls-eng-10k-tags_tagged_10k_generated" \
134
+ --eval_dataset_config_name "other+default" \
135
+ --eval_split_name "test.other+test" \
136
+ --target_audio_column_name "audio" \
137
+ --description_column_name "text_description" \
138
+ --prompt_column_name "text" \
139
+ --max_duration_in_seconds 30 \
140
+ --min_duration_in_seconds 2.0 \
141
+ --max_text_length 400 \
142
+ --add_audio_samples_to_wandb true \
143
+ --id_column_name "id" \
144
+ --preprocessing_num_workers 8 \
145
+ --do_train true \
146
+ --num_train_epochs 40 \
147
+ --gradient_accumulation_steps 8 \
148
+ --gradient_checkpointing false \
149
+ --per_device_train_batch_size 3 \
150
+ --learning_rate 0.00095 \
151
+ --adam_beta1 0.9 \
152
+ --adam_beta2 0.99 \
153
+ --weight_decay 0.01 \
154
+ --lr_scheduler_type "constant_with_warmup" \
155
+ --warmup_steps 20000 \
156
+ --logging_steps 1000 \
157
+ --freeze_text_encoder true \
158
+ --do_eval true \
159
+ --predict_with_generate true \
160
+ --include_inputs_for_metrics true \
161
+ --evaluation_strategy steps \
162
+ --eval_steps 10000 \
163
+ --save_steps 10000 \
164
+ --per_device_eval_batch_size 12 \
165
+ --audio_encoder_per_device_batch_size 20 \
166
+ --dtype "bfloat16" \
167
+ --seed 456 \
168
+ --output_dir "./output_dir_training/" \
169
+ --temporary_save_to_disk "./audio_code_tmp/" \
170
+ --save_to_disk "./tmp_dataset_audio/" \
171
+ --max_eval_samples 96 \
172
+ --dataloader_num_workers 8 \
173
+ --group_by_length true
174
+ ```
175
+
176
+ In particular, note how multiple training datasets, metadataset, configurations and splits can be loaded by separating the dataset arguments by + symbols:
177
+ ```sh
178
+ "train_dataset_name": "blabble-io/libritts_r+blabble-io/libritts_r+blabble-io/libritts_r+parler-tts/mls_eng_10k",
179
+ "train_metadata_dataset_name": "parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/libritts_r_tags_tagged_10k_generated+parler-tts/mls-eng-10k-tags_tagged_10k_generated",
180
+ "train_dataset_config_name": "clean+clean+other+default",
181
+ "train_split_name": "train.clean.360+train.clean.100+train.other.500+train",
182
+ ```
183
+
184
+
185
+ Additionally, you can also write a JSON config file. Here, [starting_point_0.01.json](helpers/training_configs/starting_point_0.01.json) contains the exact same hyper-parameters than above and can be launched like that:
186
+ ```sh
187
+ accelerate launch ./training/run_parler_tts_training.py ./helpers/training_configs/starting_point_0.01.json
188
+ ```
189
+
190
+ Training logs will be reported to wandb, provided that you passed `--report_to "wandb"` to the arguments. An example of what a training log from the above training looks like can be found [here](https://wandb.ai/ylacombe/parler-tts-300M-punctuated/runs/q6h7hspc?nw=nwuserylacombe).
191
+
192
+ > [!TIP]
193
+ > Starting training a new model from scratch can easily be overwhelming, so here's what training looked like for v0.1: [logs](https://api.wandb.ai/links/ylacombe/ea449l81)
194
+
195
+ Scaling to multiple GPUs using [distributed data parallelism (DDP)](https://pytorch.org/tutorials/beginner/ddp_series_theory.html) is trivial: simply run `accelerate config` and select the multi-GPU option, specifying the IDs of the GPUs you wish to use. The above script can then be run using DDP with no code changes. In our case, we used a node of 8 H100 80GB to train Parler-TTS v0.1 for around 4 days.
196
+
197
+
198
+ There are a few other noteworthy arguments:
199
+ 1. `train_metadata_dataset_name` and `eval_metadata_dataset_name` specify, if necessary, the names of the dataset(s) that contain(s) the conditionning text descriptions. For example, this [dataset resulting from the Data-Speech annotation process](https://huggingface.co/datasets/parler-tts/libritts_r_tags_tagged_10k_generated) is saved without the audio column, as it's costly to write and push audio data, so it needs to be concatenated back to the original LibriTTS-R dataset.
200
+ 2. As noted above, the script pre-computes audio tokens as computing audio codes is costly and only needs to be done once, since we're freezing the audio encoder. `audio_encoder_per_device_batch_size` is used to precise the per devie batch size for this pre-processing step.
201
+ 3. Additionnally, when scaling up the training data and iterating on the hyper-parameters or the model architecture, we might want to avoid recomputing the audio tokens at each training run. That's why we introduced two additional parameters, `save_to_disk` and `temporary_save_to_disk` that serves as temporary buffers to save intermediary datasets. Note that processed data is made of text and audio tokens which are much more memory efficient, so the additional required space is negligible.
202
+ 4. `predict_with_generate` and `add_audio_samples_to_wandb` are required to store generated audios and to compute WER and CLAP similarity.
203
+ 5. `freeze_text_encoder`: which allows to freeze the text encoder, to save compute resources.
204
+
205
+ And finally, two additional comments:
206
+ 1. `lr_scheduler_stype`: defines the learning rate schedule, one of `constant_with_warmup` or `cosine`. When experimenting with a training set-up or training for very few epochs, using `constant_with_warmup` is typically beneficial, since the learning rate remains high over the short training run. When performing longer training runs, using a `cosine` schedule shoud give better results.
207
+ 2. `dtype`: data type (dtype) in which the model computation should be performed. Note that this only controls the dtype of the computations (forward and backward pass), and not the dtype of the parameters or optimiser states.
208
+
209
+ > [!TIP]
210
+ > Fine-tuning is as easy as modifying `model_name_or_path` to a pre-trained model.
211
+ > For example: `--model_name_or_path parler-tts/parler_tts_mini_v0.1`.
training/__init__.py ADDED
File without changes
training/__pycache__/__init__.cpython-311.pyc ADDED
Binary file (153 Bytes). View file
 
training/__pycache__/arguments.cpython-311.pyc ADDED
Binary file (13.7 kB). View file