o j @svdZddlZddlZddlmZddlmZddlmZddlm Z ddl m Z dd Z d d Z ed kr9e dSdS) z+Find all the unique characters in a datasetN)RawTextHelpFormatter) process_map) load_config)load_tts_samples)GruutcCs&|d}t|dd}tt|S)Ntext|) phonemizer phonemizereplacesetlist)itemrphrO/home/kuhnn/.local/lib/python3.10/site-packages/TTS/bin/find_unique_phonemes.pycompute_phonemes s rc Csntjdtd}|jdtddd|}t|jat tj dtj tj d\}}||}t dt|d d |D}t|}tjrA|sEtd ||d t|ksTtd t|d ddatt|tdd}g}|D]} || qjt|}tdd|} dd |D} t| } t dt|t ddt|t ddt| t ddt| dS)NzFind all the unique characters or phonemes in a dataset. Example runs: python TTS/bin/find_unique_phonemes.py --config_path config.json ) descriptionformatter_classz --config_pathzPath to dataset config file.T)typehelprequired) eval_spliteval_split_max_sizeeval_split_sizez Num items:cSsg|]}|dqS)languager).0rrrr ,zmain..z+Phoneme language must be defined in config.rzCurrently, just one phoneme language per config file is supported !! Please split the dataset config into different configs and run it individually for each language !!)r keep_puncs) max_workers chunksizecSs|S)N)islower)crrr?szmain..cSsg|]}|qSr)lower)rr%rrrr@rz > Number of unique phonemes: z > Unique phonemes: r z > Unique lower phonemes: z( > Unique all forced to lower phonemes: )argparseArgumentParserr add_argumentstr parse_argsr config_pathr%rdatasetsrrprintlenallphoneme_language ValueErrorcountrr rrmultiprocessing cpu_countextendr filterjoinsorted) parserargs train_items eval_itemsitems language_list is_lang_defphonemesphonesr lower_phonesphones_force_lowerrrrmainsB     rF__main__)__doc__r(r5rtqdm.contrib.concurrentr TTS.configrTTS.tts.datasetsrTTS.tts.utils.text.phonemizersrrrF__name__rrrrs     6