When using -pc output in the terminal, some Chinese characters cannot be displayed normally

[chenqianhe]

just like #25, when transcribed in zh(chinese), there are still some characters missing, and the model is from ggml-large.bin in hugging-face(https://huggingface.co/datasets/ggerganov/whisper.cpp/tree/main).

Maybe the large model and the large-v1 model still need to be repaired?

error example:

[00:00:13.000 --> 00:00:15.000] 各种AI的应用��出不��

ground truth:

[00:00:13.000 --> 00:00:15.000] 各种AI的应用层出不穷

[ggerganov]

It's already converted using the solution in #25. Maybe there is some other problem with the encoding - not sure

[chenqianhe]

I'm very sorry that I may have caused some misunderstanding. Through experiments, I found that using -pc output on the terminal will lead to garbled code; And I initially determined that this was caused by the fixed length of char:

const char * text = whisper_full_get_token_text(ctx, i, j);

Like #25, sometimes the Chinese code length should be 2. I guess this should be the problem.
And I modified the output statement and added spaces to get such results, which may prove my guess.

printf("%s%s %s%s", speaker.c_str(), k_colors[3].c_str(), text, "\033[0m");

[00:00:13.000 --> 00:00:15.000] 各 种 AI 的 应 用 � � 出 不 � �

Finally, I would like to apologize for the trouble that my wrong description may cause again.

[ggerganov]

No problems. Just to make sure I understand - when you remove the -pc flag, do you get the correct characters?

[chenqianhe]

No problems. Just to make sure I understand - when you remove the -pc flag, do you get the correct characters?

Yes, it outputs normally

[chenqianhe]

No problems. Just to make sure I understand - when you remove the -pc flag, do you get the correct characters?

At the same time, when I output to the file, it is normal.
Call Whisper_full_get_segment_text(ctx, i); I will get the correct output

[giannhskp]

Any news on this issue? I am facing the same problem in Greek. The bug also occurs when using -ml with a small value, for example -ml 5.

[chenqianhe]

Any news on this issue? I am facing the same problem in Greek. The bug also occurs when using -ml with a small value, for example -ml 5.

I judge that this is caused by the uncertainty of the number correspondence between words and tokens. I think this will also lead to errors in obtaining the timestamp of each word. But I'm sorry, I haven't found a good way to fix this problem

[giannhskp]

I indeed intend to get word-level timestamps. This error does not occur during normal transcription (without -ml argument).

@ggerganov Any idea on what is causing this bug, or how it may be fixed?

[ggerganov]

If you add the --split-on-word argument does it fix the issue?