o j@sddlmZmZddlmZmZmZmZddlZddl m Z ddlm Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)eGddde Z*GdddeZ+dS)) dataclassfield)DictListTupleUnionN)Coqpit)nn)autocast)Decoder)Encoder)AlignmentNetwork)PositionalEncoding)DurationPredictor)BaseTTS)average_over_durations generate_path maximum_path sequence_mask)SpeakerManager) TTSTokenizer)plot_alignmentplot_avg_energyplot_avg_pitchplot_spectrogram) load_fsspecc@seZdZUdZdZeed<dZeed<dZeed<dZ e ed <dZ e ed <d Z eed <d Z eed<dZeed<d Zeed<dZe ed<d Zeed<d Zeed<dZeed<d Zeed<d Zeed<d Zeed<dZeed<dZe ed<dZe ed<dZeed<dZeed <ed!d"d#Ze ed$<dZ!eed%<ed&d"d#Z"e ed'<dZ#e ed(<d)Z$eed*<dZ%eed+<dZ&e ed,<dZ'eed-<dZ(e ed.<dZ)eed/<dZ*eed0<dS)1ForwardTTSArgsaForwardTTS Model arguments. Args: num_chars (int): Number of characters in the vocabulary. Defaults to 100. out_channels (int): Number of output channels. Defaults to 80. hidden_channels (int): Number of base hidden channels of the model. Defaults to 512. use_aligner (bool): Whether to use aligner network to learn the text to speech alignment or use pre-computed durations. If set False, durations should be computed by `TTS/bin/compute_attention_masks.py` and path to the pre-computed durations must be provided to `config.datasets[0].meta_file_attn_mask`. Defaults to True. use_pitch (bool): Use pitch predictor to learn the pitch. Defaults to True. use_energy (bool): Use energy predictor to learn the energy. Defaults to True. duration_predictor_hidden_channels (int): Number of hidden channels in the duration predictor. Defaults to 256. duration_predictor_dropout_p (float): Dropout rate for the duration predictor. Defaults to 0.1. duration_predictor_kernel_size (int): Kernel size of conv layers in the duration predictor. Defaults to 3. pitch_predictor_hidden_channels (int): Number of hidden channels in the pitch predictor. Defaults to 256. pitch_predictor_dropout_p (float): Dropout rate for the pitch predictor. Defaults to 0.1. pitch_predictor_kernel_size (int): Kernel size of conv layers in the pitch predictor. Defaults to 3. pitch_embedding_kernel_size (int): Kernel size of the projection layer in the pitch predictor. Defaults to 3. energy_predictor_hidden_channels (int): Number of hidden channels in the energy predictor. Defaults to 256. energy_predictor_dropout_p (float): Dropout rate for the energy predictor. Defaults to 0.1. energy_predictor_kernel_size (int): Kernel size of conv layers in the energy predictor. Defaults to 3. energy_embedding_kernel_size (int): Kernel size of the projection layer in the energy predictor. Defaults to 3. positional_encoding (bool): Whether to use positional encoding. Defaults to True. positional_encoding_use_scale (bool): Whether to use a learnable scale coeff in the positional encoding. Defaults to True. length_scale (int): Length scale that multiplies the predicted durations. Larger values result slower speech. Defaults to 1.0. encoder_type (str): Type of the encoder module. One of the encoders available in :class:`TTS.tts.layers.feed_forward.encoder`. Defaults to `fftransformer` as in the paper. encoder_params (dict): Parameters of the encoder module. Defaults to ```{"hidden_channels_ffn": 1024, "num_heads": 1, "num_layers": 6, "dropout_p": 0.1}``` decoder_type (str): Type of the decoder module. One of the decoders available in :class:`TTS.tts.layers.feed_forward.decoder`. Defaults to `fftransformer` as in the paper. decoder_params (str): Parameters of the decoder module. Defaults to ```{"hidden_channels_ffn": 1024, "num_heads": 1, "num_layers": 6, "dropout_p": 0.1}``` detach_duration_predictor (bool): Detach the input to the duration predictor from the earlier computation graph so that the duraiton loss does not pass to the earlier layers. Defaults to True. max_duration (int): Maximum duration accepted by the model. Defaults to 75. num_speakers (int): Number of speakers for the speaker embedding layer. Defaults to 0. speakers_file (str): Path to the speaker mapping file for the Speaker Manager. Defaults to None. speaker_embedding_channels (int): Number of speaker embedding channels. Defaults to 256. use_d_vector_file (bool): Enable/Disable the use of d-vectors for multi-speaker training. Defaults to False. d_vector_dim (int): Number of d-vector channels. Defaults to 0. N num_charsP out_channelsihidden_channelsT use_aligner use_pitchpitch_predictor_hidden_channelspitch_predictor_kernel_size皙?pitch_predictor_dropout_ppitch_embedding_kernel_sizeF use_energy energy_predictor_hidden_channelsenergy_predictor_kernel_sizeenergy_predictor_dropout_penergy_embedding_kernel_size"duration_predictor_hidden_channelsduration_predictor_kernel_sizeduration_predictor_dropout_ppositional_encodingpoisitonal_encoding_use_scale length_scale fftransformer encoder_typecCdddddSNir4r')hidden_channels_ffn num_heads num_layers dropout_pr?r?r?M/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/models/forward_tts.pyzForwardTTSArgs.)default_factoryencoder_params decoder_typecCr8r9r?r?r?r?r@rArBdecoder_paramsdetach_duration_predictorK max_duration num_speakersuse_speaker_embedding speakers_fileuse_d_vector_file d_vector_dim d_vector_file)+__name__ __module__ __qualname____doc__rint__annotations__rr r!boolr"r$r&r(floatr)r*r+r,r-r.r/r0r1r2r3r5r7strrrDdictrErFrGrIrJrKrLrMrNrOr?r?r?r@rsL  h                            rcseZdZdZ   dMdedddddeffd d Zdefd d ZedNd dZ ddZ ddZ dNde j de jde jdee je je je je jffddZde jde jde jde jde jdee je jff ddZ  dOde jde jde jde jdee je jff dd Z  dOde jde jd!e jde jdee je jff d"d#Zde jd$e jde jd%e jdee je je je jff d&d'Zd(efd)d*Zddddddd+fde j d,e j de j d$e jde jde jd!e jd(edefd-d.Ze ddd+fd/d0Zd1ed2ejfd3d4Zd5d6Zd1ed7ed8d9d:ed;e ddf dd?Z"d1ed7ed8d9d:ed;e ddf d@dAZ# BdPdCdDZ$dEdFZ%dGdHZ&edNddIdJe'e(e(e(effdKdLZ)Z*S)Q ForwardTTSaGeneral forward TTS model implementation that uses an encoder-decoder architecture with an optional alignment network and a pitch predictor. If the alignment network is used, the model learns the text-to-speech alignment from the data instead of using pre-computed durations. If the pitch predictor is used, the model trains a pitch predictor that predicts average pitch value for each input character as in the FastPitch model. `ForwardTTS` can be configured to one of these architectures, - FastPitch - SpeedySpeech - FastSpeech - FastSpeech2 (requires average speech energy predictor) Args: config (Coqpit): Model coqpit class. speaker_manager (SpeakerManager): Speaker manager for multi-speaker training. Only used for multi-speaker models. Defaults to None. Examples: >>> from TTS.tts.models.fast_pitch import ForwardTTS, ForwardTTSArgs >>> config = ForwardTTSArgs() >>> model = ForwardTTS(config) NconfigapAudioProcessor tokenizerrspeaker_managercst|||||||||jj|_|jj|_|jj|_|jj|_d|_ t |jj t r7t |jj n|jj |_ t|jj|jj|_t|jj|jj|jj|jj|j|_|jjrdt|jj|_t|jj|jj|jj|jj|_t |jj|jj!|jj"|jj#|_$|jjrt |jj|jj%|jj&|jj'|_(tj)d|jj|jj*t |jj*ddd|_+|jjrt |jj|jj,|jj-|jj.|_/tj)d|jj|jj0t |jj0ddd|_1|jjrt2|jj|jjd|_3dSdS)Ngr4) kernel_sizepadding)in_query_channelsin_key_channels)4super__init___set_model_argsinit_multispeakerargsrIr!r"r*binary_loss_weight isinstancer5rTrWr Embeddingrr embr r7rDembedded_speaker_dimencoderr2r pos_encoderr rrErFdecoderrr/r0r1duration_predictorr$r&r(pitch_predictorConv1dr) pitch_embr+r,r-energy_predictorr. energy_embr aligner)selfr[r\r^r_ __class__r?r@rfs~         zForwardTTS.__init__cCsd|_|jdur|js|jrtd|jdur|jj|_|jr7|j|_|jj|jjkr7t j |jj|jjd|_ |jrW|jsYt dt |j|jj|_t j|jjdddSdSdS)zjInit for multi-speaker training. Args: config (Coqpit): Model configuration. rNzq > SpeakerManager is not provided. You must provide the SpeakerManager before initializing a multi-speaker model.) in_features out_featuresz > Init speaker_embedding layer.gr')rnr_rMrK ValueErrorrJrNrir r Linearproj_gprintrlemb_ginituniform_weight)ryr[r?r?r@rhs    zForwardTTS.init_multispeakercCsp|dur|d}d||dk<tt|dd|j}t|dt|d}t||d|j}|S)zGenerate an attention mask from the durations. Shapes - dr: :math:`(B, T_{en})` - x_mask: :math:`(B, T_{en})` - y_mask: :math:`(B, T_{de})` Nr4r`) sumlongtorch unsqueezertodtypersqueeze)drx_masky_mask y_lengths attn_maskattnr?r?r@ generate_attn6s  zForwardTTS.generate_attncCsH||||}t|ddd|j|dddd}||fS)alGenerate attention alignment map from durations and expand encoder outputs Shapes: - en: :math:`(B, D_{en}, T_{en})` - dr: :math:`(B, T_{en})` - x_mask: :math:`(B, T_{en})` - y_mask: :math:`(B, T_{de})` Examples:: encoder output: [a,b,c,d] durations: [1, 3, 2, 1] expanded: [a, b, b, b, c, c, d] attention map: [[0, 0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 1, 1, 0], [0, 1, 1, 1, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0]] r4r`)rrmatmulr transposerr)ryenrrrro_en_exr?r?r@expand_encoder_outputsHs2z!ForwardTTS.expand_encoder_outputscCs2t|d||j}d||dk<t|}|S)aFormat predicted durations. 1. Convert to linear scale from log scale 2. Apply the length scale for speed adjustment 3. Apply masking. 4. Cast 0 durations to 1. 5. Round the duration values. Args: o_dr_log: Log scale durations. x_mask: Input text mask. Shapes: - o_dr_log: :math:`(B, T_{de})` - x_mask: :math:`(B, T_{en})` r4?)rexpr5round)ryo_dr_logro_drr?r?r@format_durationsas  zForwardTTS.format_durationsxrgreturncCst|dr|tj}||}|dur|d}||}|t|dd||}|durFt|drB| | |j ddd}||}||||fS)aPEncoding forward pass. 1. Embed speaker IDs if multi-speaker mode. 2. Embed character sequences. 3. Run the encoder network. 4. Sum encoder outputs and speaker embeddings Args: x (torch.LongTensor): Input sequence IDs. x_mask (torch.FloatTensor): Input squence mask. g (torch.FloatTensor, optional): Conditioning vectors. In general speaker embeddings. Defaults to None. Returns: Tuple[torch.tensor, torch.tensor, torch.tensor, torch.tensor, torch.tensor]: encoder output, encoder output for the duration predictor, input sequence mask, speaker embeddings, character embeddings Shapes: - x: :math:`(B, T_{en})` - x_mask: :math:`(B, 1, T_{en})` - g: :math:`(B, C)` rNrr4rr) hasattrtyper LongTensorrrrmrorrviewshape)ryrrrx_embo_enr?r?r@_forward_encodervs       zForwardTTS._forward_encoderrrrc Csltt|dd|j}|||||\}}t|dr"|||}|j|||d} | dd| ddfS)aDecoding forward pass. 1. Compute the decoder output mask 2. Expand encoder output with the durations. 3. Apply position encoding. 4. Add speaker embeddings if multi-speaker mode. 5. Run the decoder. Args: o_en (torch.FloatTensor): Encoder output. dr (torch.IntTensor): Ground truth durations or alignment network durations. x_mask (torch.IntTensor): Input sequence mask. y_lengths (torch.IntTensor): Output sequence lengths. g (torch.FloatTensor): Conditioning vectors. In general speaker embeddings. Returns: Tuple[torch.FloatTensor, torch.FloatTensor]: Decoder output, attention map from durations. Nr4rprr`) rrrrrrrrprqr) ryrrrrrrrro_der?r?r@_forward_decoders   zForwardTTS._forward_decoderpitchcCD|||}|durt||}||}|||fS||}||fS)aMPitch predictor forward pass. 1. Predict pitch from encoder outputs. 2. In training - Compute average pitch values for each input character from the ground truth pitch values. 3. Embed average pitch values. Args: o_en (torch.FloatTensor): Encoder output. x_mask (torch.IntTensor): Input sequence mask. pitch (torch.FloatTensor, optional): Ground truth pitch values. Defaults to None. dr (torch.IntTensor, optional): Ground truth durations. Defaults to None. Returns: Tuple[torch.FloatTensor, torch.FloatTensor]: Pitch embedding, pitch prediction. Shapes: - o_en: :math:`(B, C, T_{en})` - x_mask: :math:`(B, 1, T_{en})` - pitch: :math:`(B, 1, T_{de})` - dr: :math:`(B, T_{en})` N)rsrru)ryrrrro_pitch avg_pitch o_pitch_embr?r?r@_forward_pitch_predictor     z#ForwardTTS._forward_pitch_predictorenergycCr)aTEnergy predictor forward pass. 1. Predict energy from encoder outputs. 2. In training - Compute average pitch values for each input character from the ground truth pitch values. 3. Embed average energy values. Args: o_en (torch.FloatTensor): Encoder output. x_mask (torch.IntTensor): Input sequence mask. energy (torch.FloatTensor, optional): Ground truth energy values. Defaults to None. dr (torch.IntTensor, optional): Ground truth durations. Defaults to None. Returns: Tuple[torch.FloatTensor, torch.FloatTensor]: Energy embedding, energy prediction. Shapes: - o_en: :math:`(B, C, T_{en})` - x_mask: :math:`(B, 1, T_{en})` - pitch: :math:`(B, 1, T_{de})` - dr: :math:`(B, T_{en})` N)rvrrw)ryrrrro_energy avg_energy o_energy_embr?r?r@_forward_energy_predictorrz$ForwardTTS._forward_energy_predictoryrc Cst|dt|d}||dd|dd|d\}}t|ddd|d}t|d} |ddd}| |||fS)axAligner forward pass. 1. Compute a mask to apply to the attention map. 2. Run the alignment network. 3. Apply MAS to compute the hard alignment map. 4. Compute the durations from the hard alignment map. Args: x (torch.FloatTensor): Input sequence. y (torch.FloatTensor): Output sequence. x_mask (torch.IntTensor): Input sequence mask. y_mask (torch.IntTensor): Output sequence mask. Returns: Tuple[torch.IntTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]: Durations from the hard alignment map, soft alignment potentials, log scale alignment potentials, hard alignment map. Shapes: - x: :math:`[B, T_en, C_en]` - y: :math:`[B, T_de, C_de]` - x_mask: :math:`[B, 1, T_en]` - y_mask: :math:`[B, 1, T_de]` - o_alignment_dur: :math:`[B, T_en]` - alignment_soft: :math:`[B, T_en, T_de]` - alignment_logprob: :math:`[B, 1, T_de, T_en]` - alignment_mas: :math:`[B, T_en, T_de]` rr`r4N) rrrxrrr contiguousrrT) ryrrrrralignment_softalignment_logprob alignment_maso_alignment_durr?r?r@_forward_aligner s $  zForwardTTS._forward_aligner aux_inputcCsb|dd}|dd}|dur|durtd|dur%t|ds%td|dur-|}|S|}|S)N d_vectors speaker_idsz2[!] Cannot use d-vectors and speaker-ids together.rz>[!] Cannot use speaker-ids without enabling speaker embedding.)getr~r)ryrrrrr?r?r@_set_speaker_input6s   zForwardTTS._set_speaker_inputrr x_lengthsc Cs||} tt|dd} tt||jdd} ||| | \} } } } |jjr6| | | }n| | | }t t |dd|j }||d| }d}d}d}d}|jrw|| || | \}}}}|dd}|dd}|}d}d}|jjr|| | ||\}}}| |} d}d}|jjr|| | ||\}}}| |} |j| || |dd\}}||d|d||||||||||| | d}|S)aModel's forward pass. Args: x (torch.LongTensor): Input character sequences. x_lengths (torch.LongTensor): Input sequence lengths. y_lengths (torch.LongTensor): Output sequnce lengths. Defaults to None. y (torch.FloatTensor): Spectrogram frames. Only used when the alignment network is on. Defaults to None. dr (torch.IntTensor): Character durations over the spectrogram frames. Only used when the alignment network is off. Defaults to None. pitch (torch.FloatTensor): Pitch values for each spectrogram frame. Only used when the pitch predictor is on. Defaults to None. energy (torch.FloatTensor): energy values for each spectrogram frame. Only used when the energy predictor is on. Defaults to None. aux_input (Dict): Auxiliary model inputs for multi-speaker training. Defaults to `{"d_vectors": 0, "speaker_ids": None}`. Shapes: - x: :math:`[B, T_max]` - x_lengths: :math:`[B]` - y_lengths: :math:`[B]` - y: :math:`[B, T_max2]` - dr: :math:`[B, T_max]` - g: :math:`[B, C]` - pitch: :math:`[B, 1, T]` Nr4rr`r) model_outputs durations_log durationsattn_durations pitch_avg pitch_avg_gt energy_avg energy_avg_gt alignmentsrrrrrr)rrrrrWrrrirGrrdetachclamprrIrrr!rrr"rr*rr)ryrrrrrrrrrrrrrrro_attnrrrrrrrrrrrroutputsr?r?r@forwardCsd      zForwardTTS.forwardcCs||}t|jdd|j}tt||jdd|j }| |||\}}}}| | |}| || d} | d} d} |jjrX|||\} } || }d} |jjrj|||\}} ||}|j|| || dd\}}||| | |d}|S)a;Model's inference pass. Args: x (torch.LongTensor): Input character sequence. aux_input (Dict): Auxiliary model inputs. Defaults to `{"d_vectors": None, "speaker_ids": None}`. Shapes: - x: [B, T_max] - x_lengths: [B] - g: [B, C] r4r`Nr)rrrrr)rrtensorrrdevicerrrrWrrrrrrrir"rr*rr)ryrrrrrr_rrrrrrrrrrr?r?r@ inferences. $ zForwardTTS.inferencebatch criterioncCsp|d}|d}|d}|d}|jjr|dnd}|jjr"|dnd}|d} |d} |d } | | d } |j||||| ||| d } |jrI| d } td d^|| d||| d| |jr`| dnd|jrh| dnd|jrp| dnd|jrx| dnd||jr| dnd| d| d|jd}| d }t| | | }||d<Wd| |fS1swY| |fS)N text_input text_lengths mel_input mel_lengthsrrrrrr)rrrrrrF)enabledrrrrrrrrr)decoder_outputdecoder_targetdecoder_output_lens dur_output dur_target pitch_output pitch_target energy_output energy_target input_lensrralignment_hardrjduration_error) rir"r*rr!r rjrabsr)ryrrrrrrrrrrrrr loss_dictdurations_predrr?r?r@ train_steps\     zForwardTTS.train_stepcCs|d}|d}|d}|dj}|dj}|dj} t||ddt||ddt| ddd} |jjr|t|dd j} t|d d j} |j |d dj} t | | ddt | | ddd }| ||jj rt|d d j}t|dd j}|j |d dj} t || ddt || ddd}| |d|vr|ddj}t|jdd| d<||j}| d|ifS)zCreate common logger outputs.rrrrF) output_fig) prediction ground_truth alignmentr)rrrr)pitch_ground_truthpitch_avg_predictedrr)energy_ground_truthenergy_avg_predictedr alignment_hataudio)datacpunumpyrrrir"rr^decoderupdater*rTinv_melspectrogram)ryrrr\rrr pred_specgt_spec align_imgfiguresr pitch_avg_hatchars pitch_figuresrenergy_avg_hatenergy_figuresalignments_hat train_audior?r?r@ _create_logss>           zForwardTTS._create_logsrloggerLoggerassetsstepscC6||||j\}}|||||||jjdSN)r r\ train_figures train_audios sample_rateryrrrrrraudiosr?r?r@ train_log/s zForwardTTS.train_logcCs |||Sr)r)ryrrr?r?r@ eval_step6s zForwardTTS.eval_stepcCrr)r r\ eval_figures eval_audiosrrr?r?r@eval_log9s zForwardTTS.eval_logFcCs@t|td|d}||d|r||jrJdSdS)Nr) map_locationcachemodel)rrrload_state_dictevaltraining)ryr[checkpoint_pathr"rstater?r?r@load_checkpoint>s zForwardTTS.load_checkpointcCsddlm}||jS)Nr)ForwardTTSLoss)TTS.tts.layers.lossesr'r[)ryr'r?r?r@ get_criterionGs  zForwardTTS.get_criterioncCst|j|jjdd|_dS)zSchedule binary loss weight.rN)min epochs_doner[binary_loss_warmup_epochsrj)rytrainerr?r?r@on_train_step_startLszForwardTTS.on_train_step_startForwardTTSConfigsamplescCs>ddlm}||}t|\}}t||}t||||S)zInitiate model from config Args: config (ForwardTTSConfig): Model config. samples (Union[List[List], List[Dict]]): Training samples to parse speaker ids for training. Defaults to None. r)r])TTS.utils.audior]init_from_configrrrZ)r[r0r]r\r^ new_configr_r?r?r@r2Ps  zForwardTTS.init_from_config)NNNr)NN)FF)+rPrQrRrSrrrfrh staticmethodrrrrr FloatTensorrr IntTensorrrrrrrrno_gradrrYr Modulerr rTrrrr&r)r.rrr2 __classcell__r?r?rzr@rZsR  + ( ( $ )  ]+4/ "  .rZ), dataclassesrrtypingrrrrrcoqpitrr torch.cuda.amp.autocast_moder #TTS.tts.layers.feed_forward.decoderr #TTS.tts.layers.feed_forward.encoderr TTS.tts.layers.generic.alignerr #TTS.tts.layers.generic.pos_encodingr*TTS.tts.layers.glow_tts.duration_predictorrTTS.tts.models.base_ttsrTTS.tts.utils.helpersrrrrTTS.tts.utils.speakersrTTS.tts.utils.text.tokenizerrTTS.tts.utils.visualrrrr TTS.utils.iorrrZr?r?r?r@s*