o j[@sddlmZmZmZddlZddlmmZddl m Z ddlmZddl m Z ddl mZmZmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z m!Z!Gdddejj"Z#dS))CallableDictTupleN)Coqpit)nn) Conformer)PhonemeLevelProsodyEncoderUtteranceLevelProsodyEncoderget_mask_from_lengths) EnergyAdaptor)EmbeddingPaddedpositional_encoding)PhonemeProsodyPredictor) PitchAdaptor)VariancePredictor)AlignmentNetwork) generate_path maximum_path sequence_maskcseZdZ  d<     d=fdd Zdefd d Zed efd dZd efddZ d efddZ ddZ ddZ ed>ddZ dejdejdejdejfddZdejd ejdejd!ejd"ejd#eejejejejff d$d%Zd&ejd'ejd#ejfd(d)Z *  d?d+ejd,ejd-ejd.ejd/ejd0ejd"ejd1ed2ejd3ejd#eeejffd4d5Ze     d@d+ejd3ejd6ed7ed2ejd8ed9ed#ejfd:d;ZZS)A AcousticModelNargs ModelArgs tokenizer TTSTokenizerspeaker_managerSpeakerManagerc st||_||_||_||t|jjtr t |jjn|jj|_|j |_ t |jj |jj |jj|j|jj|jj|jjd|_t|jj |jjd|jj|jj|jj|jjd|_t|jj |jjd|jj|jj|jj|jjd|_t|jj|jj d|_t|jj |jjd|jj|jj|jjd|_ t!|jj"|jj#|jj$|jj%|jj&|jj |jj|jj'|jj(d |_)t*|jj |jj+|jj|jj'|jjd|_,t-|jj"|jj#|jj$|jj%|jj&|jj |jj|jj.|jjd |_/t*|jj |jj+|jj|jj.|jjd|_0t12|jj'|jj |_3t14|jj'|_5t12|jj.|jj |_6t14|jj.|_7t |jj8|jj9|jj:|j|jj;|jj<|jjd|_=|jj>j?}t@|jjA|jj |d |_Bt12|jj8|jj"|_CtDj1jEdd d dd |_F|jFGd dS)N)dimn_layersn_headsspeaker_embedding_dim p_dropoutkernel_size_conv_mod lrelu_slope)n_inputn_hiddenn_out kernel_sizeemb_kernel_sizer r") channels_inchannels_hidden channels_outr'r(dropoutr")in_query_channelsin_key_channels)r)channelsr+r'r r") num_melsref_enc_filters ref_enc_sizeref_enc_gru_sizeref_enc_stridesr%r,bottleneck_size_u token_num) hidden_sizer'r,bottleneck_sizer") r0r1r2r3r4r%r,bottleneck_size_pr) padding_idxFT)affinetrack_running_statsmomentum)Hsuper__init__rrrinit_multispeaker isinstance length_scaleintfloatn_hidden_conformer_encoderemb_dimrn_layers_conformer_encodern_heads_conformer_encoderembedded_speaker_dimdropout_conformer_encoder&kernel_size_conv_mod_conformer_encoderr"encoderrn_hidden_variance_adaptorkernel_size_variance_adaptor emb_kernel_size_variance_adaptordropout_variance_adaptor pitch_adaptorr energy_adaptorr out_channelsalignerrduration_predictorr r0!ref_enc_filters_reference_encoderref_enc_size_reference_encoder"ref_enc_gru_size_reference_encoder!ref_enc_strides_reference_encoder#bottleneck_size_u_reference_encodertoken_num_reference_encoderutterance_prosody_encoderr'predictor_kernel_size_reference_encoderutterance_prosody_predictorr#bottleneck_size_p_reference_encoderphoneme_prosody_encoderphoneme_prosody_predictorrLinear u_bottle_outInstanceNorm1du_norm p_bottle_outp_normn_hidden_conformer_decodern_layers_conformer_decodern_heads_conformer_decoderdropout_conformer_decoder&kernel_size_conv_mod_conformer_decoderdecoder characterspad_idr num_chars src_word_embto_meltorch BatchNorm1d energy_scalerrequires_grad_)selfrrrr: __class___/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/delightful_tts/acoustic_model.pyr?s          zAcousticModel.__init__cCsNd|_|jj|_d|_|jr|jj|_|jjr||jjr%|dSdS)z Init for multi-speaker training.rN) rIr num_speakersaudio_transformruse_speaker_embedding_init_speaker_embeddinguse_d_vector_file_init_d_vector)rwrrzrzr{r@s   zAcousticModel.init_multispeaker aux_inputcCsd\}}}}d|vr|ddur|d}|jdkr|d}d|vr6|ddur6t|d}|jdkr6|}d|vrD|ddurD|d}||||fS)zCSet the speaker conditioning input based on the multi-speaker mode.)NNNN speaker_idsNr d_vectors durations)ndim unsqueeze_F normalize)rsidglidrrzrzr{_set_cond_inputs     zAcousticModel._set_cond_inputcCs ||\}}}}|d||dS)N)r style_wavr language_ids)r)rwrrrr_rzrzr{ get_aux_inputszAcousticModel.get_aux_inputcCsb|dd}|dd}|dur|durtd|dur%t|ds%td|dur-|}|S|}|S)Nrrz2[!] Cannot use d-vectors and speaker-ids together.emb_gz>[!] Cannot use speaker-ids without enabling speaker embedding.)get ValueErrorhasattr)rwrrrrrzrzr{_set_speaker_inputs   z AcousticModel._set_speaker_inputcCs6|jdkrtd|jj|_t|j|j|_dSdS)Nrz. > initialization of speaker-embedding layers.)r|printrspeaker_embedding_channelsrIr Embeddingrrwrzrzr{rs  z%AcousticModel._init_speaker_embeddingcCs t|dr td|jj|_dS)NrzI[!] Speaker embedding layer already initialized before d_vector settings.)rrr d_vector_dimrIrrzrzr{rs zAcousticModel._init_d_vectorcCsp|dur|d}d||dk<tt|dd|j}t|dt|d}t||d|j}|S)aGenerate an attention mask from the linear scale durations. Args: dr (Tensor): Linear scale durations. x_mask (Tensor): Mask for the input (character) sequence. y_mask (Tensor): Mask for the output (spectrogram) sequence. Compute it from the predicted durations if None. Defaults to None. Shapes - dr: :math:`(B, T_{en})` - x_mask: :math:`(B, T_{en})` - y_mask: :math:`(B, T_{de})` Nr#r) sumlongrs unsqueezertodtypersqueeze)drx_masky_mask y_lengths attn_maskattnrzrzr{ generate_attns zAcousticModel.generate_attno_enrrrcCsNtt|dd|j}||||}td||g}|||ddfS)Nr#zkmn, kjm -> kjnr) rsrrrrreinsumrD transpose)rwrrrrrro_en_exrzrzr{_expand_encoder_with_durationssz,AcousticModel._expand_encoder_with_durationsxyr attn_priorsreturnc Cst|dt|d}||dd|dd||\}}t|ddd|d} t| d} |d}| dd} | ||| fS)aAligner forward pass. 1. Compute a mask to apply to the attention map. 2. Run the alignment network. 3. Apply MAS to compute the hard alignment map. 4. Compute the durations from the hard alignment map. Args: x (torch.FloatTensor): Input sequence. y (torch.FloatTensor): Output sequence. x_mask (torch.IntTensor): Input sequence mask. y_mask (torch.IntTensor): Output sequence mask. attn_priors (torch.FloatTensor): Prior for the aligner network map. Returns: Tuple[torch.IntTensor, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]: Durations from the hard alignment map, soft alignment potentials, log scale alignment potentials, hard alignment map. Shapes: - x: :math:`[B, T_en, C_en]` - y: :math:`[B, T_de, C_de]` - x_mask: :math:`[B, 1, T_en]` - y_mask: :math:`[B, 1, T_de]` - attn_priors: :math:`[B, T_de, T_en]` - aligner_durations: :math:`[B, T_en]` - aligner_soft: :math:`[B, T_de, T_en]` - aligner_logprob: :math:`[B, 1, T_de, T_en]` - aligner_mas: :math:`[B, T_de, T_en]` rrr#) rsrrTrrr contiguousrrC) rwrrrrrr aligner_softaligner_logprob aligner_masaligner_durationsrzrzr{_forward_aligner s'$    zAcousticModel._forward_aligneru_prosody_predsrc_maskcCs0|dd}|jddd|ddd}|S)N?r#T)keepdimr)rview)rwrrlengthsrzrzr{average_utterance_prosody;sz'AcousticModel.average_utterance_prosodyTtokenssrc_lensmelsmel_lenspitchesenergiesuse_ground_truthr speaker_idxc , Cs|| | d\} } } }t|}t|}||}||dd}|j||dd|dddf|dddf|d\}}}}|}d}| durM| }n | durYt| | }t |j t |j dt ||jd}|j||||d}||j||d }||j|j||d |d }|r|||}n|||}||j|||||d }||j||d }|r|||}n|||}|}|jj||||d \}} }!|jj||||d \}"}#}$|dd|!|$}|j||d }%|j||||dddfd\}&}'}(|j |'dd|||d})|!|)})t"#|d}t"$|%d}*|%|*|d|&}+id|)d|d| d|"d|#d|d|d|d|d|+d|(d|d|d|d|d|%&dd|&dd |iS)!Nrrrr#r)rrrrrdevicespeaker_embeddingencoding)rrrmaskrr)rrrrr)rtargetrrrrrr model_outputs pitch_pred pitch_target energy_pred energy_targetr u_prosody_refp_prosody_pred p_prosody_ref alignments_dp alignmentsrrrr dr_log_pred dr_log_targetspk_emb)'rr rq masked_fillrrrrrrr rFmaxshaperrLrer\rr^rcrgr`rarfrQget_pitch_embedding_trainrRget_energy_embedding_trainrUdetachrrmrrrslogexprr),rwrrrrrrrrrrrrrrrmel_masktoken_embeddingsrrrrrr pos_encodingencoder_outputsrrrrencoder_outputs_resravg_pitch_target pitch_embravg_energy_target energy_emblog_duration_prediction mel_pred_maskencoder_outputs_exrrdr_predrrzrzr{forwardBs                  zAcousticModel.forward p_control d_controlpitch_transformenergy_transformc"Csttj|jdgtj|jd}t|jdd|j} |||d\} } } } ||}| | dd}d}|durA| }n |durMt | | }t|j|jd|jd}|j||||d}||j|j||d |d }||||}||j||d }||||}|}|jj|||t|d r|jndt|d r|jndd \}}|jj|||d\}}| dd||}|j!|"|d }t#|d||j$}d||dk<t%|}|&d}|j'|||(d|dddfd\} }}ttj|jdgtj|jd}|jd|jdkr&t|j|jd|jd}|j)| dd|||d} |*| } | |||||d}!|!S)Nr#)rrrrrrrrrr pitch_mean pitch_std)rrrrr)rrrrr)rrrpitchenergyr)+r rstensorrint64rrrrqrrrrrr rFrLrerr^rc expand_asrgrarfrQget_pitch_embeddingrrrrRget_energy_embeddingrrUrrrBroundrrrrmrr)"rwrrrrrrrrrrrrrrrrrrrrpitch_emb_predrenergy_emb_predrlog_duration_pred duration_predrrrrrroutputsrzrzr{ inferences            zAcousticModel.inference)NN)rrrrrr)N)TNN)NNNNN)__name__ __module__ __qualname__r?rr@ staticmethodrrrrrrrrs FloatTensor IntTensorrrrTensorrboolstrrno_gradrDrr  __classcell__rzrzrxr{rs     1      r)$typingrrrrstorch.nn.functionalr functionalrcoqpitr'TTS.tts.layers.delightful_tts.conformerr&TTS.tts.layers.delightful_tts.encodersrr r ,TTS.tts.layers.delightful_tts.energy_adaptorr &TTS.tts.layers.delightful_tts.networksr r 7TTS.tts.layers.delightful_tts.phoneme_prosody_predictorr+TTS.tts.layers.delightful_tts.pitch_adaptorr0TTS.tts.layers.delightful_tts.variance_predictorrTTS.tts.layers.generic.alignerrTTS.tts.utils.helpersrrrModulerrzrzrzr{s