o j} @s ddlZddlmZddlZddlZddlmmZddl Z ddl m Z ddl m Z ddlmZddlmZddlmZmZddlmZmZdd lmZdd lmZed ded d dd ddddddf ddZddZddZeGddde Z eGddde Z!GdddeZ"dS) N) dataclass)Coqpit)GPT) HifiDecoder)init_stream_support)VoiceBpeTokenizersplit_sentence)SpeakerManagerLanguageManager)BaseTTS) load_fsspecz"../experiments/clips_mel_norms.pthcpuF"V@Pc  Csztjj|||||| | | | dd |} ||}| |}ttj|dd}|dur1tj||d}||dd}|S) a Convert waveform to mel-spectrogram with hard-coded parameters for cloning. Args: wav (torch.Tensor): Input waveform tensor. mel_norms_file (str): Path to mel-spectrogram normalization file. mel_norms (torch.Tensor): Mel-spectrogram normalization tensor. device (torch.device): Device to use for computation. Returns: torch.Tensor: Mel-spectrogram tensor. slaney) n_fft hop_length win_lengthpower normalized sample_ratef_minf_maxn_melsnormgh㈵>)minN map_locationr) torchaudio transformsMelSpectrogramtotorchlogclampload unsqueeze)wavmel_norms_file mel_normsdevicerrrrrrrrrmel_stftmelr2F/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/models/xtts.pywav_to_mel_clonings*  r4cCst|\}}|ddkrtj|ddd}||kr"tj|||}t|dks0t|dksAtd|d| d| | d d|S) NrT)dimkeepdim z Error with z. Max=z min=r") r#r*sizer'mean functionalresampleanyprintmaxrclip_) audiopath sampling_rateaudiolsrr2r2r3 load_audioEs" rEcCsR|dd|f}|jd|kr|}|S|jd|kr't|d||jdf}|S)a> Ensure a given tensor t has a specified sequence length by either padding it with zeros or clipping it. Args: t (torch.Tensor): The input tensor to be padded or truncated. length (int): The desired length of the tensor. Returns: torch.Tensor: The padded or truncated tensor. .Nr"r)shapeFpad)tlengthtpr2r2r3pad_or_truncate[s rLc@s*eZdZUdZdZeed<dZeed<dS)XttsAudioConfigz Configuration class for audio-related parameters in the XTTS model. Args: sample_rate (int): The sample rate in which the GPT operates. output_sample_rate (int): The sample rate of the output audio waveform. rr]output_sample_rateN)__name__ __module__ __qualname____doc__rint__annotations__rOr2r2r2r3rMns  rMc@szeZdZUdZdZeed<dZeed<dZ eed<dZ e ed <dZ e ed <dZ e ed <d Zeed <dZe ed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed <d!Zeed"<d#Zeed$<dZeed%<dZeed&<dZeed'<d(Zeed)<d*Z eed+<d,Z!eed-<dZ"eed.<d/Z#eed0<dZ$eed1<d2Z%eed3<dS)4XttsArgsa_ A dataclass to represent XTTS model arguments that define the model structure. Args: gpt_batch_size (int): The size of the auto-regressive batch. enable_redaction (bool, optional): Whether to enable redaction. Defaults to True. kv_cache (bool, optional): Whether to use the kv_cache. Defaults to True. gpt_checkpoint (str, optional): The checkpoint for the autoregressive model. Defaults to None. clvp_checkpoint (str, optional): The checkpoint for the ConditionalLatentVariablePerseq model. Defaults to None. decoder_checkpoint (str, optional): The checkpoint for the DiffTTS model. Defaults to None. num_chars (int, optional): The maximum number of characters to generate. Defaults to 255. For GPT model: gpt_max_audio_tokens (int, optional): The maximum mel tokens for the autoregressive model. Defaults to 604. gpt_max_text_tokens (int, optional): The maximum text tokens for the autoregressive model. Defaults to 402. gpt_max_prompt_tokens (int, optional): The maximum prompt tokens or the autoregressive model. Defaults to 70. gpt_layers (int, optional): The number of layers for the autoregressive model. Defaults to 30. gpt_n_model_channels (int, optional): The model dimension for the autoregressive model. Defaults to 1024. gpt_n_heads (int, optional): The number of heads for the autoregressive model. Defaults to 16. gpt_number_text_tokens (int, optional): The number of text tokens for the autoregressive model. Defaults to 255. gpt_start_text_token (int, optional): The start text token for the autoregressive model. Defaults to 255. gpt_checkpointing (bool, optional): Whether to use checkpointing for the autoregressive model. Defaults to False. gpt_train_solo_embeddings (bool, optional): Whether to train embeddings for the autoregressive model. Defaults to False. gpt_code_stride_len (int, optional): The hop_size of dvae and consequently of the gpt output. Defaults to 1024. gpt_use_masking_gt_prompt_approach (bool, optional): If True, it will use ground truth as prompt and it will mask the loss to avoid repetition. Defaults to True. gpt_use_perceiver_resampler (bool, optional): If True, it will use perceiver resampler from flamingo paper - https://arxiv.org/abs/2204.14198. Defaults to False. r5gpt_batch_sizeFenable_redactionTkv_cacheNgpt_checkpointclvp_checkpointdecoder_checkpoint num_charstokenizer_filei]gpt_max_audio_tokensigpt_max_text_tokensFgpt_max_prompt_tokens gpt_layersrgpt_n_model_channels gpt_n_headsgpt_number_text_tokensgpt_start_text_tokengpt_stop_text_tokeni gpt_num_audio_tokensi gpt_start_audio_tokeni gpt_stop_audio_tokengpt_code_stride_len"gpt_use_masking_gt_prompt_approachgpt_use_perceiver_resamplerrinput_sample_raterNrOoutput_hop_lengthdecoder_input_dimi d_vector_dim&cond_d_vector_in_each_upsampling_layeriduration_const)&rPrQrRrSrWrTrUrXboolrYrZstrr[r\r^r`rarbrdrfrgrirjrkrlrmrnrorprqrrrsrOrurvrwrxryr2r2r2r3rV|s@                              rVcsXeZdZdZdeffdd ZddZeddZe d9d e d e fd dZ e ddZ e   d:ddZd;ddZe        dd(d)Zd*d+Zd,d-Zed?d/d0Zfd1d2Zd3d4Z       d@d5d6Zd7d8ZZS)AXttsuⓍTTS model implementation. ❗ Currently it only supports inference. Examples: >>> from TTS.tts.configs.xtts_config import XttsConfig >>> from TTS.tts.models.xtts import Xtts >>> config = XttsConfig() >>> model = Xtts.inif_from_config(config) >>> model.load_checkpoint(config, checkpoint_dir="paths/to/models_dir/", eval=True) configcsptj|dddd|_||_|jj|_|jj|_|j|_|jj |_ t |_ d|_ | |dtddS)N)ap tokenizer mel_statsr)super__init__mel_stats_pathr}argsrZr\ model_dir models_dirrWrrgpt init_modelsregister_bufferr'ones)selfr} __class__r2r3rs   z Xtts.__init__cCs|jjdur|j|j_|jjd|j_|jjd|j_|jjrRt|jj|jj |jj|jj|jj |jj |jj |jj |jj|jj|jj|jj|jj|jjd|_t|jj|jj|jj|jj|jj|jj|jjd|_dS)zOInitialize the models. We do it here since we need to load the tokenizer first.Nz[START]z[STOP])layers model_dimstart_text_tokenstop_text_tokenheadsmax_text_tokensmax_mel_tokensmax_prompt_tokensnumber_text_tokensnum_audio_tokensstart_audio_tokenstop_audio_tokenuse_perceiver_resamplercode_stride_len)rsrOruar_mel_length_compressionrvrwrx)rget_number_tokensrrj token_to_idrkrlrrfrgrirbrardrmrnrorrrprrrsrOrurvrwrxhifigan_decoderrr2r2r3rs<  zXtts.init_modelscCst|jSN)next parametersr/rr2r2r3r/sz Xtts.devicererJ chunk_lengthc Cs(|dkr tj||d}|dkr|dddd|f}|jjrrg}td|jdd|D]:}|dd||d|f}|ddkrEq-t||j ddd d d ddd d d }|j | |jd} || q-t|jdd} nt||j dd dd d ddd d d } |j | |j} | dd S)aCompute the conditioning latents for the GPT model from the given audio. Args: audio (tensor): audio tensor. sr (int): Sample rate of the audio. length (int): Length of the audio in seconds. If < 0, use the whole audio. Defaults to 30. chunk_length (int): Length of the audio chunks in seconds. When `length == chunk_length`, the whole audio is being used without chunking. It must be < `length`. Defaults to 6. rrNr5r"gl@irtrrFrr) r.rrrrrrrrrr6r)r#r;r<rrrrangerFr9r4rr r get_style_embr&r/appendr'stackr: transpose) rrCsrrJr style_embsi audio_chunk mel_chunk style_emb cond_latentr1r2r2r3get_gpt_cond_latentssR   zXtts.get_gpt_cond_latentscCs8tj||d}|jjj||jddd|jS)Ni>T)l2_normr") r#r;r<rspeaker_encoderforwardr&r/r+)rrCr audio_16kr2r2r3get_speaker_embedding<s zXtts.get_speaker_embeddingNFrcCst|ts |g}n|}g} g} d} |D]B} t| |} | ddd||f|j} |r7| t| d} |durEtj j | |dd} | | |} | | | | qtj | dd}|j||||d}| rst| } | jdd} || fS)a Get the conditioning latents for the GPT model from the given audio. Args: audio_path (str or List[str]): Path to reference audio file(s). max_ref_length (int): Maximum length of each reference audio in seconds. Defaults to 30. gpt_cond_len (int): Length of the audio used for gpt latents. Defaults to 6. gpt_cond_chunk_len (int): Chunk length used for gpt latents. It must be <= gpt_conf_len. Defaults to 6. librosa_trim_db (int, optional): Trim the audio using this value. If None, not trimming. Defaults to None. sound_norm_refs (bool, optional): Whether to normalize the audio. Defaults to False. load_sr (int, optional): Sample rate to load the audio. Defaults to 24000. N?)top_dbrr"r)rJr) isinstancelistrEr&r/r'absr?librosaeffectstrimrrcatrrr:)r audio_pathmax_ref_length gpt_cond_lengpt_cond_chunk_lenlibrosa_trim_dbsound_norm_refsload_sr audio_pathsspeaker_embeddingsaudiosspeaker_embedding file_pathrC full_audiogpt_cond_latentsr2r2r3get_conditioning_latentsEs0        zXtts.get_conditioning_latentsc Ks|dkrn||jjvsJd|d|jj|j|j|j|j|jd}|||durC|jj | \}} |j |||| fi|S||j |j |j|jd|j|||fi|S)aSynthesize speech with the given input text. Args: text (str): Input text. config (XttsConfig): Config with inference parameters. speaker_wav (list): List of paths to the speaker audio files to be used for cloning. language (str): Language ID of the speaker. **kwargs: Inference settings. See `inference()`. Returns: A dictionary of the output values with `wav` as output waveform, `deterministic_seed` as seed used at inference, `text_input` as text token IDs after tokenizer, `voice_samples` as samples used for cloning, `conditioning_latents` as latents used at inference. zhzzh-cnu ❗ Language z+ is not supported. Supported languages are ) temperaturelength_penaltyrepetition_penaltytop_ktop_pN)rr max_ref_lenr)r} languagesrrrrrupdatespeaker_managerspeakersvalues inferencerrrrfull_inference) rtextr} speaker_wavlanguage speaker_idkwargssettingsgpt_cond_latentrr2r2r3 synthesize~s( zXtts.synthesizer?$@2333333?Tr8c Ks>|j|| | | | d\}}|j||||f|||||| d|S)aq This function produces an audio clip of the given text being spoken with the given reference voice. Args: text: (str) Text to be spoken. ref_audio_path: (str) Path to a reference audio file to be used for cloning. This audio file should be >3 seconds long. language: (str) Language of the voice to be generated. temperature: (float) The softmax temperature of the autoregressive model. Defaults to 0.65. length_penalty: (float) A length penalty applied to the autoregressive decoder. Higher settings causes the model to produce more terse outputs. Defaults to 1.0. repetition_penalty: (float) A penalty that prevents the autoregressive decoder from repeating itself during decoding. Can be used to reduce the incidence of long silences or "uhhhhhhs", etc. Defaults to 2.0. top_k: (int) K value used in top-k sampling. [0,inf]. Lower values mean the decoder produces more "likely" (aka boring) outputs. Defaults to 50. top_p: (float) P value used in nucleus sampling. (0,1]. Lower values mean the decoder produces more "likely" (aka boring) outputs. Defaults to 0.8. gpt_cond_len: (int) Length of the audio used for cloning. If audio is shorter, then audio length is used else the first `gpt_cond_len` secs is used. Defaults to 30 seconds. gpt_cond_chunk_len: (int) Chunk length used for cloning. It must be <= `gpt_cond_len`. If gpt_cond_len == gpt_cond_chunk_len, no chunking. Defaults to 6 seconds. hf_generate_kwargs: (**kwargs) The huggingface Transformers generate API is used for the autoregressive transformer. Extra keyword args fed to this function get forwarded directly to that API. Documentation here: https://huggingface.co/docs/transformers/internal/generation_utils Returns: Generated audio clip(s) as a torch tensor. Shape 1,S if k=1 else, (k,1,S) where S is the sample length. Sample rate is 24kHz. )rrrrr)rrrrr do_sample)rr)rrref_audio_pathrrrrrrrrrrrhf_generate_kwargsrrr2r2r3rs,;  zXtts.full_inferencer5cKs|dd}dt| d}||j}||j}| r't|||jj|}n|g}g}g}|D]}|}t |jj ||d d|j}|j d|jjksWJdt p|jjd||d| | |||j| ||dd |}t j|j d|jjg|jd }t j|j dg|jd }|j|||||dd d }|dkrtj|d d|ddd d}||||j||dWdn1swYq0t j|ddt j|d d|dS)N-rr皙?langr"> ❗ XTTS can only generate text with a maximum of 400 tokens.F) cond_latents text_inputs input_tokensrrrrnum_return_sequences num_beamsrroutput_attentions)r/T)rreturn_attentions return_latentr5rlinear scale_factormodegr)r, gpt_latentsrr2)splitr?r&r/rr char_limitsstriplowerr' IntTensorencoder+rFrrbno_gradrgeneraterWtensorrrG interpolaterrr rsqueezernumpy)rrrrrrrrrrrrspeedenable_text_splittingr length_scalewavsgpt_latents_listsent text_tokens gpt_codesexpected_output_lentext_lenrr2r2r3rsv   $   (zXtts.inferencecCs|d| }|dur||jd|| }|durk|t|kr=|dur1||jd|d}n|| d}||dfS|d|}|tdd||j}|tdd||j|d|<|d||7<|| d}|}|||fS)z)Handle chunk formatting in streaming modeNrgr)rFlenr'linspacer&r/)rwav_gen wav_gen_prev wav_overlap overlap_len wav_chunk crossfade_wavr2r2r3 handle_chunksIs    " zXtts.handle_chunksrcks|dd}dt| d}||j}||j}|r(t|||jj|}n|g}|D]}|}t |jj ||d d|j}|j d|jjksTJd|j||j|}|jjd|| | || ddt|t| d d d |}g}g}d}d}d }|szt|\}}||g7}||g7}Wn tyd }Ynw|s|dkrt||krt j|dd dddf}|dkrtj|dd |dddd }|j|||jd}|||||\}}}g}|V|rq-dS)Nrrrrrr"rr5FT) fake_inputsrrrrrrrrroutput_hidden_statesrrrrrr2)rr?r&r/rrrrrr'rrr+rFrrbrcompute_embeddings get_generatorfloatr StopIterationrrrGr rrr r )rrrrrstream_chunk_sizeoverlap_wav_lenrrrrrrrrrrrrr" gpt_generator last_tokens all_latentsrris_endxlatentrrrr2r2r3inference_streambs~   $       zXtts.inference_streamcCtdNzuXTTS has a dedicated trainer, please check the XTTS docs: https://tts.readthedocs.io/en/dev/models/xtts.html#trainingNotImplementedErrorrr2r2r3rz Xtts.forwardcCr1r2r3rr2r2r3 eval_stepr5zXtts.eval_step XttsConfigcKst|Sr)r|)r}rr2r2r3init_from_configszXtts.init_from_configcs|jtdS)znSets the model to evaluation mode. Overrides the default eval() method to also set the GPT model to eval mode.N)rinit_gpt_for_inferencerevalrrr2r3r:s z Xtts.evalcCsxt|tddd}gd}t|D]$}|dr-|dd}||||<||=|}|dd|vr9||=q|S) Nr r model)#torch_mel_spectrogram_style_encodertorch_mel_spectrogram_dvaedvaezxtts.r_.r)r r'r/rkeys startswithreplacer)r model_path checkpoint ignore_keyskeynew_keyr2r2r3$get_compatible_checkpoint_state_dicts   z)Xtts.get_compatible_checkpoint_state_dictc Cs|ptj|d} |ptj|d}|ptj|d}t||_d|_tj|r.t||_tj|r:t|d|_ | | | } z |j | |dWn|r[|j j|jjd|j | |dY|r{|j|j j|jj|d|j dSdS) a Loads a checkpoint from disk and initializes the model's state and tokenizer. Args: config (dict): The configuration dictionary for the model. checkpoint_dir (str, optional): The directory where the checkpoint is stored. Defaults to None. checkpoint_path (str, optional): The path to the checkpoint file. Defaults to None. vocab_path (str, optional): The path to the vocabulary file. Defaults to None. eval (bool, optional): Whether to set the model to evaluation mode. Defaults to True. strict (bool, optional): Whether to strictly enforce that the keys in the checkpoint match the keys in the model. Defaults to True. Returns: None z model.pthz vocab.jsonzspeakers_xtts.pthN) vocab_file)strict)rY)rY use_deepspeed)ospathjoinr language_managerrexistsr rrrrHload_state_dictrr9rrYrr:) rr}checkpoint_dircheckpoint_path vocab_pathr:rJrKspeaker_file_pathrCrDr2r2r3load_checkpoints,       zXtts.load_checkpointcCr1r2r3rr2r2r3 train_stepr5zXtts.train_step)rer)rerrNFrr) rrrrrTrerr8F) rrrrrTr5rF) r!rrrrrrTrF)r}r7)NNNTTFN)rPrQrRrSrrrpropertyr/r'inference_moderTrrrrrrr r0rr6 staticmethodr8r:rHrVrW __classcell__r2r2rr3r|s # :  8' P R S   7r|)#rL dataclassesrrr'torch.nn.functionalnnr;rGr#coqpitrTTS.tts.layers.xtts.gptr#TTS.tts.layers.xtts.hifigan_decoderr$TTS.tts.layers.xtts.stream_generatorrTTS.tts.layers.xtts.tokenizerrr TTS.tts.layers.xtts.xtts_managerr r TTS.tts.models.base_ttsr TTS.utils.ior r/r4rErLrMrVr|r2r2r2r3sF        0 B