o j @sddlZddlZddlmZddlmZddlmZmZm Z m Z m Z ddl Z ddlTddlTdddZd d Z   dd e e eefd ede e e e e ffddZddZddZdddZdS)N)Counter)Path)CallableDictListTupleUnion)*{Gz?c Cs6dd|D}tt|dk}|dkrt|}n|r&t|tt||}ntt||}|dks=Jddt|tjdtj||rg}dd|D}t |}t||krtj dt|}||d}||dkr| ||||d8<||=t||ks^||fS|d|||dfS) ajSplit a dataset into train and eval. Consider speaker distribution in multi-speaker training. Args: items (List[List]): A list of samples. Each sample is a list of `[audio_path, text, speaker_id]`. eval_split_max_size (int): Number maximum of samples to be used for evaluation in proportion split. Defaults to None (Disabled). eval_split_size (float): If between 0.0 and 1.0 represents the proportion of the dataset to include in the evaluation set. If > 1, represents the absolute number of evaluation samples. Defaults to 0.01 (1%). cSg|]}|dqS speaker_name.0itemrrL/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/datasets/__init__.py z!split_dataset..rz [!] You do not have enough samples for the evaluation set. You can work around this setting the 'eval_split_size' parameter to a minimum of {}cSr r rrrrrr.rr N) lensetintminformatnprandomseedshufflerrandintappend) itemseval_split_max_sizeeval_split_sizespeakersis_multi_speaker items_evalspeaker_counteritem_idxspeaker_to_be_removedrrr split_dataset s8         r*cCsN|D]"}||d<tjtj|d|dd}|d|}||d<q|S)Nlanguage audio_file root_pathr#audio_unique_name)ospathsplitextrelpath)metadatar+ dataset_namer relfilepathr/rrradd_extra_keys;s " r7Tdatasets formatterreturncCsg}|rgnd}t|ts|g}|D]}|d}|d} |d} |d} |d} |d} |d}|dur8t|}|| | | d }t|d ksOJd | d | t||| }td t|dt| |r| rw|| | | d }t||| }n|r|t|nd}t|||\}}||7}||7}|j rt t |d}t |D]\}}||d }||d|iq|rt |D]\}}||d }||d|iqd}q||fS)aParse the dataset from the datasets config, load the samples as a List and load the attention alignments if provided. If `formatter` is not None, apply the formatter to the samples else pick the formatter from the available ones based on the dataset name. Args: datasets (List[Dict], Dict): A list of datasets or a single dataset dictionary. If multiple datasets are in the list, they are all merged. eval_split (bool, optional): If true, create a evaluation split. If an eval split provided explicitly, generate an eval split automatically. Defaults to True. formatter (Callable, optional): The preprocessing function to be applied to create the list of samples. It must take the root_path and the meta_file name and return a list of samples in the format of `[[text, audio_path, speaker_id], ...]]`. See the available formatters in `TTS.tts.dataset.formatter` as example. Defaults to None. eval_split_max_size (int): Number maximum of samples to be used for evaluation in proportion split. Defaults to None (Disabled). eval_split_size (float): If between 0.0 and 1.0 represents the proportion of the dataset to include in the evaluation set. If > 1, represents the absolute number of evaluation samples. Defaults to 0.01 (1%). Returns: Tuple[List[List], List[List]: training and evaluation splits of the dataset. Nr9r5r1meta_file_train meta_file_valignored_speakersr+)r=rz" [!] No training samples found in /z | > Found z files in meta_file_attn_maskr,alignment_file) isinstancelist_get_formatter_by_namerr7printrresolver*r?dictload_attention_mask_meta_data enumeratestripupdate)r8 eval_splitr9r"r#meta_data_train_allmeta_data_eval_alldatasetformatter_namer5r-r;r<r=r+meta_data_trainmeta_data_evaleval_size_per_dataset meta_dataidxins attn_filerrrload_tts_samplesFsJ!     rWcCsdt|ddd }|}Wdn1swYg}|D]}|d\}}|||gq|S)z9Load meta data file created by compute_attention_masks.pyrzutf-8)encodingN|)open readlinessplitr ) metafile_pathflinesrSlinewav_filerVrrrrGs rGcCstjt}t||S)z.Returns the respective preprocessing function.)sysmodules__name__getattrlower)name thismodulerrrrCs rCcCsddd|D}t|}tdd|}dd|D}t|}|rOtdt|td dt|td dt|td dt||S) Ncss|]}|dVqdS)rNrrrrr sz$find_unique_chars..cSs|S)N)islower)crrrsz#find_unique_chars..cSsg|]}|qSr)rg)rrmrrrrrz%find_unique_chars..z > Number of unique characters: z > Unique characters: z > Unique lower characters: z* > Unique all forced to lower characters: )joinrfilterrDrsorted) data_samplesverbosetextschars lower_charschars_force_lowerrrrfind_unique_charssrx)Nr )TNNr )T)r0rc collectionsrpathlibrtypingrrrrrnumpyrTTS.tts.datasets.datasetTTS.tts.datasets.formattersr*r7rWrGrCrxrrrrs0   .  Q