o j#@sddlmZmZmZddlZddlmZddlmmZ ddl m Z ddl m Z ddlmZdejdejfdd Zdd ejd edejfd dZGdddejZGdddejZGdddejZdS))ListTupleUnionN)!ConformerMultiHeadedSelfAttention) CoordConv1d)STLlengthsreturncCsR|jd}t|}tjd||jdd|d}||dd|k}|S)Nr)device)shapetorchmaxitemaranger unsqueezeexpand)r batch_sizemax_lenidsmaskrY/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/delightful_tts/encoders.pyget_mask_from_lengths s  rlensstridecCst||S)N)rceilint)rrrrr stride_lenssr cseZdZdZdedeeeeeeeefdedeeeeeeefdef fdd Zd ej d ej d e ej ej ej ffd d Z dededededed ef ddZ Z S)ReferenceEncodera Referance encoder for utterance and phoneme prosody encoders. Reference encoder made up of convolution and RNN layers. Args: num_mels (int): Number of mel frames to produce. ref_enc_filters (list[int]): List of channel sizes for encoder layers. ref_enc_size (int): Size of the kernel for the conv layers. ref_enc_strides (List[int]): List of strides to use for conv layers. ref_enc_gru_size (int): Number of hidden features for the gated recurrent unit. Inputs: inputs, mask - **inputs** (batch, dim, time): Tensor containing mel vector - **lengths** (batch): Tensor containing the mel lengths. Returns: - **outputs** (batch, time, dim): Tensor produced by Reference Encoder. num_melsref_enc_filters ref_enc_sizeref_enc_stridesref_enc_gru_sizec st|}||_t}|jgdg|tddddddg}fddtd|D} || t||_ tfddt|D|_ tj d |dd |_ dS) Nr rrT) in_channels out_channels kernel_sizerpaddingwith_rc s4g|]}tj||d|ddqS)r r)r'r(r)rr*)nnConv1d.0i)filtersr$stridesrr Es z-ReferenceEncoder.__init__..csg|] }tj|ddqS)T) num_featuresaffine)r,InstanceNorm1dr.)r#rrr3Rsr ) input_size hidden_size batch_first) super__init__n_mel_channelslenrrangeextendr, ModuleListconvsnormsGRUgru) selfr"r#r$r%r&r<KrAconvs2 __class__)r1r#r$r2rr;+s4      zReferenceEncoder.__init__xmel_lensr cCst|d}||d}t|j|jD]\}}||}t|d}||}qtdD]}t |}q+t|}||dd}| d}t j j jj||ddd}|j||\}}t j j jj|dd \}}|||fS) zR inputs --- [N, n_mels, timesteps] outputs --- [N, E//2] r rg333333?r)rrr TF)r9enforce_sorted)r9)rr masked_fillziprArBF leaky_relur>r permuterr,utilsrnnpack_padded_sequencecpurrDflatten_parameterspad_packed_sequence)rErJrK mel_masksconvnorm_memoryrrrforwardZs          zReferenceEncoder.forwardLr)rpadn_convscCs*t|D]}||d||d}q|S)Nrr )r>)rEr^r)rr_r`r[rrrcalculate_channelsvs z#ReferenceEncoder.calculate_channels)__name__ __module__ __qualname____doc__rrrr;rTensorrr]ra __classcell__rrrHrr!s6*/r!cseZdZdedeeeeeeeefdedeeeeeeefdedededed effd d Zd ej d ej dej fddZ Z S)UtteranceLevelProsodyEncoderr"r#r$r%r&dropoutn_hiddenbottleneck_size_u token_numc svt||_||_|_|} t|||||d|_t||jd|_ t || d|_ t|j| |_ t ||_dS)a* Encoder to extract prosody from utterance. it is made up of a reference encoder with a couple of linear layers and style token layer with dropout. Args: num_mels (int): Number of mel frames to produce. ref_enc_filters (list[int]): List of channel sizes for ref encoder layers. ref_enc_size (int): Size of the kernel for the ref encoder conv layers. ref_enc_strides (List[int]): List of strides to use for teh ref encoder conv layers. ref_enc_gru_size (int): Number of hidden features for the gated recurrent unit. dropout (float): Probability of dropout. n_hidden (int): Size of hidden layers. bottleneck_size_u (int): Size of the bottle neck layer. Inputs: inputs, mask - **inputs** (batch, dim, time): Tensor containing mel vector - **lengths** (batch): Tensor containing the mel lengths. Returns: - **outputs** (batch, 1, dim): Tensor produced by Utterance Level Prosody Encoder. r#r&r$r%r"r)rjrlN)r:r;Ed_qd_kr!encoderr,Linear encoder_prjrstlencoder_bottleneckDropoutri) rEr"r#r$r%r&rirjrkrlbottleneck_sizerHrrr;s  z%UtteranceLevelProsodyEncoder.__init__melsrKr cCsP|||\}}}||}|||}||}|dd|jdf}|S)z Shapes: mels: :math: `[B, C, T]` mel_lens: :math: `[B]` out --- [N, seq_len, E] r r )rqrsrurtriviewr )rErxrKr[embedded_prosodyoutrrrr]s   z$UtteranceLevelProsodyEncoder.forward rbrcrdrrrfloatr;rrfr]rgrrrHrrh~s*  $2rhcseZdZdedeeeeeeeefdedeeeeeeefdedededed effd d Zd ej d ej dej dej dej dej f ddZ Z S)PhonemeLevelProsodyEncoderr"r#r$r%r&rirjn_headsbottleneck_size_pc sdt||_||_|_| } t|||||d|_t|||_ t |||d|_ t|| |_ dS)Nrm)d_model num_heads dropout_p) r:r;rnrorpr!rqr,rrrsr attentionru) rEr"r#r$r%r&rirjrrrwrHrrr;s$  z#PhonemeLevelProsodyEncoder.__init__rJsrc_maskrxrKencodingr c Csl|||\}}}||}||jddddf} |j|||| |d\}}||}||dd}|S)z x --- [N, seq_len, encoder_embedding_dim] mels --- [N, Ty/r, n_mels*r], r=1 out --- [N, seq_len, bottleneck_size] attn --- [N, seq_len, ref_len], Ty/r = ref_len rr r )querykeyvaluerrg)rqrsrzr rrurMr) rErJrrxrKrr{r[rX attn_maskrrrr]s  z"PhonemeLevelProsodyEncoder.forwardr}rrrHrrsB  !r)r)typingrrrrtorch.nnr,torch.nn.functional functionalrO'TTS.tts.layers.delightful_tts.conformerr)TTS.tts.layers.delightful_tts.conv_layersr&TTS.tts.layers.delightful_tts.networksrrfrrr Moduler!rhrrrrrs    fH