o jE@sddlZddlmZddlZddlmZddlmmZddl m Z m Z m Z ddl mZdedeeeffddZGd d d ejZGd d d ejjZGd ddejZGdddejZGdddejZGdddejZGdddejZdS)N)Tuple) Conv1dGLUDepthWiseConv1dPointwiseConv1d) GLUActivation kernel_sizereturncCs|d}|||ddfS)N)rpadr r Z/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/delightful_tts/conformer.pycalc_same_padding srcsbeZdZdededededededeffdd Zd ejd ejd ejd ejdejf ddZZ S) Conformerdimn_layersn_headsspeaker_embedding_dim p_dropoutkernel_size_conv_mod lrelu_slopec sFttfddt|D|_dS)aI A Transformer variant that integrates both CNNs and Transformers components. Conformer proposes a novel combination of self-attention and convolution, in which self-attention learns the global interaction while the convolutions efficiently capture the local correlations. Args: dim (int): Number of the dimensions for the model. n_layers (int): Number of model layers. n_heads (int): The number of attention heads. speaker_embedding_dim (int): Number of speaker embedding dimensions. p_dropout (float): Probabilty of dropout. kernel_size_conv_mod (int): Size of kernels for convolution modules. Inputs: inputs, mask - **inputs** (batch, time, dim): Tensor containing input vector - **encoding** (batch, time, dim): Positional embedding tensor - **mask** (batch, 1, time2) or (batch, time1, time2): Tensor containing indices to be masked Returns: - **outputs** (batch, time, dim): Tensor produced by Conformer Encoder. c s$g|]}tdqS))rdropoutrr)ConformerBlock).0_d_kd_vrrrrrrr r 4s z&Conformer.__init__..N)super__init__nn ModuleListrange layer_stack)selfrrrrrrr __class__rr r s    zConformer.__init__xmaskspeaker_embeddingencodingrcCs@||jddd|jdf}|jD] }||||||d}q|S)z Shapes: - x: :math:`[B, T_src, C]` - mask: :math: `[B]` - speaker_embedding: :math: `[B, C]` - encoding: :math: `[B, T_max2, C]` rr )r) slf_attn_maskr*r+)viewshaper$)r%r(r)r*r+ attn_mask enc_layerr r r forwardCs zConformer.forward __name__ __module__ __qualname__intfloatr torchTensorr1 __classcell__r r r&r rs60rcspeZdZ ddededededededed effd d Zd ejd ejdejdejdejdejf ddZZ S)r333333?d_modeln_headrrrrrrc stt|trt|||d|d|_t||d|d|_t||||d|_ t ||_ t |||d|_t||||d|_dS)a A Conformer block is composed of four modules stacked together, A feed-forward module, a self-attention module, a convolution module, and a second feed-forward module in the end. The block starts with two Feed forward modules sandwiching the Multi-Headed Self-Attention module and the Conv module. Args: d_model (int): The dimension of model n_head (int): The number of attention heads. kernel_size_conv_mod (int): Size of kernels for convolution modules. speaker_embedding_dim (int): Number of speaker embedding dimensions. emotion_embedding_dim (int): Number of emotion embedding dimensions. dropout (float): Probabilty of dropout. Inputs: inputs, mask - **inputs** (batch, time, dim): Tensor containing input vector - **encoding** (batch, time, dim): Positional embedding tensor - **slf_attn_mask** (batch, 1, 1, time1): Tensor containing indices to be masked in self attention module - **mask** (batch, 1, time2) or (batch, time1, time2): Tensor containing indices to be masked Returns: - **outputs** (batch, time, dim): Tensor produced by the Conformer Block. r )r<rpadding embedding_dim)r<rrr)rrr)r< num_heads dropout_pN)rr isinstancer6r conditioning FeedForwardffConformerConvModuleconformer_conv_1r! LayerNormln!ConformerMultiHeadedSelfAttentionslf_attnconformer_conv_2) r%r<r=rrrrrrr&r r r _s" !   zConformerBlock.__init__r(r*r)r,r+rcCs|dur |j||d}|||}|||}|}||}|j|||||d\}}||}||dd}|||}|S)a* Shapes: - x: :math:`[B, T_src, C]` - mask: :math: `[B]` - slf_attn_mask: :math: `[B, 1, 1, T_src]` - speaker_embedding: :math: `[B, C]` - emotion_embedding: :math: `[B, C]` - encoding: :math: `[B, T_max2, C]` N) embeddings)querykeyvaluer)r+r)rDrFrHrJrL masked_fill unsqueezerM)r%r(r*r)r,r+resrr r r r1s zConformerBlock.forward)r;r2r r r&r r^s@  4rc sLeZdZ d dededededef fdd Zd ejd ejfd d ZZ S)rEr<rrrexpansion_factorcsdtt||_t||_tj|||||dd|_t ||_ tj|||dd|_ dS)aw Feed Forward module for conformer block. Args: d_model (int): The dimension of model. kernel_size (int): Size of the kernels for conv layers. dropout (float): probability of dropout. expansion_factor (int): The factor by which to project the number of channels. lrelu_slope (int): the negative slope factor for the leaky relu activation. Inputs: inputs - **inputs** (batch, time, dim): Tensor containing input vector Returns: - **outputs** (batch, time, dim): Tensor produced by the feed forward module. r rr>r )rN) rr r!DropoutrrIrJConv1dconv_1 LeakyReLUactconv_2)r%r<rrrrWr&r r r s    zFeedForward.__init__r(rcCsp||}|d}||}|d}||}||}|d}||}|d}||}d|}|S); Shapes: x: :math: `[B, T, C]` )rr r ?)rJpermuter[r]rr^r%r(r r r r1s          zFeedForward.forward)rVr2r r r&r rEs#rEc sReZdZ    ddedededed ef fd d Zd ejd ejfddZZ S)rGr 皙?r;r<rWrrrcst||}t||_t||d|_t|d|_t |||t |dd|_ t d||_ t||_t|||_t||_dS)a Convolution module for conformer. Starts with a gating machanism. a pointwise convolution and a gated linear unit (GLU). This is followed by a single 1-D depthwise convolution layer. Batchnorm is deployed just after the convolution to help with training. it also contains an expansion factor to project the number of channels. Args: d_model (int): The dimension of model. expansion_factor (int): The factor by which to project the number of channels. kernel_size (int): Size of kernels for convolution modules. dropout (float): Probabilty of dropout. lrelu_slope (float): The slope coefficient for leaky relu activation. Inputs: inputs - **inputs** (batch, time, dim): Tensor containing input vector Returns: - **outputs** (batch, time, dim): Tensor produced by the conv module. r )sloperrXr N)rr r!rIln_1rr[rconv_actrr depthwise GroupNormln_2r\ activationr^rYr)r%r<rWrrr inner_dimr&r r r s      zConformerConvModule.__init__r(rcCsp||}|ddd}||}||}||}||}||}||}|ddd}||}|S)r_rr r ) rfrar[rgrhrjrkr^rrbr r r r1s        zConformerConvModule.forward)r rcrdr;r2r r r&r rGs"+rGcsfeZdZdZdededeffdd Zdejdejd ejd ejd ejd e ejejff d dZ Z S)rKa Conformer employ multi-headed self-attention (MHSA) while integrating an important technique from Transformer-XL, the relative sinusoidal positional encoding scheme. The relative positional encoding allows the self-attention module to generalize better on different input length and the resulting encoder is more robust to the variance of the utterance length. Conformer use prenorm residual units with dropout which helps training and regularizing deeper models. Args: d_model (int): The dimension of model num_heads (int): The number of attention heads. dropout_p (float): probability of dropout Inputs: inputs, mask - **inputs** (batch, time, dim): Tensor containing input vector - **mask** (batch, 1, time2) or (batch, time1, time2): Tensor containing indices to be masked Returns: - **outputs** (batch, time, dim): Tensor produces by relative multi headed self attention module. r<rArBcs*tt||d|_tj|d|_dS)N)r<rA)p)rr RelativeMultiHeadAttention attentionr!rYr)r%r<rArBr&r r r 9s z*ConformerMultiHeadedSelfAttention.__init__rOrPrQr)r+rc Cs`|\}}}|ddd|jdf}||dd}|j|||||d\} } || } | | fS)Nr ) pos_embeddingr))sizer.repeatror) r%rOrPrQr)r+ batch_size seq_lengthroutputsattnr r r r1>s  z)ConformerMultiHeadedSelfAttention.forward) r3r4r5__doc__r6r7r r8r9rr1r:r r r&r rK's rKcs~eZdZdZ  ddedeffdd Zdejd ejd ejd ejd ejd eejejff ddZ dejd ejfddZ Z S)rna3 Multi-head attention with relative positional encoding. This concept was proposed in the "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" Args: d_model (int): The dimension of model num_heads (int): The number of attention heads. Inputs: query, key, value, pos_embedding, mask - **query** (batch, time, dim): Tensor containing query vector - **key** (batch, time, dim): Tensor containing key vector - **value** (batch, time, dim): Tensor containing value vector - **pos_embedding** (batch, time, dim): Positional embedding tensor - **mask** (batch, 1, time2) or (batch, time1, time2): Tensor containing indices to be masked Returns: - **outputs**: Tensor produces by relative multi head attention module. r<rAcst||dksJd||_t|||_||_t||_t |||_ t j ||dd|_ t j ||dd|_ t j ||dd|_t t|j|j|_t t|j|j|_tj j|jtj j|jt |||_dS)Nrz#d_model % num_heads should be zero.F)bias)rr r<r6d_headrAmathsqrtsqrt_dimr!Linear query_projkey_proj value_projpos_proj Parameterr8r9u_biasv_biasinitxavier_uniform_out_proj)r%r<rAr&r r r _s  z#RelativeMultiHeadAttention.__init__rOrPrQrpr)rcCsP|jd}|||d|j|j}|||d|j|jdddd}|||d|j|jdddd}|||d|j|j}|j |}|j |}|| dd} | | dd} || dd} | |dddd} | | } | | } | d|j} | |dt| d}|| dd}||d|j}|||fS)NrrRr r r@g?ge)r.rr-rAr{rrarrr expand_asr transpose_relative_shiftr~ masked_fill_Fsoftmax contiguousr<r)r%rOrPrQrpr)rsrra content_scoreb pos_scorescorervcontextr r r r1vs& &&     z"RelativeMultiHeadAttention.forwardrcCsr|\}}}}tj|||df|jd}tj||gdd}||||d|}|ddddddf|}|S)Nr )devicerRr)rqr8zerosrcatr-view_as)r%rrsrA seq_length1 seq_length2rpadded_pos_scorer r r rs  z*RelativeMultiHeadAttention._relative_shift)rxry) r3r4r5rwr6r r8r9rr1rr:r r r&r rnNs. !rncsNeZdZdZdedededeffdd Zdejd ejd ejfd d ZZ S) MultiHeadAttentionz input: query --- [N, T_q, query_dim] key --- [N, T_k, key_dim] output: out --- [N, T_q, num_units] query_dimkey_dim num_unitsrAcsVt||_||_||_tj||dd|_tj||dd|_tj||dd|_ dS)NF) in_features out_featuresrz) rr rrArr!rW_queryW_keyW_value)r%rrrrAr&r r r s zMultiHeadAttention.__init__rOrPrc Cs||}||}||}|j|j}tjtj||dddd}tjtj||dddd}tjtj||dddd}t|| dd}||j d}t j |dd}t||}tj tj|dddddd}|S)Nr rrr@r`r )rrrrrAr8stacksplitmatmulrrrrrsqueeze) r%rOrPqueryskeysvalues split_sizescoresoutr r r r1s      zMultiHeadAttention.forward) r3r4r5rwr6r r8r9r1r:r r r&r rs$ r)r|typingrr8torch.nnr!torch.nn.functional functionalr)TTS.tts.layers.delightful_tts.conv_layersrrr&TTS.tts.layers.delightful_tts.networksrr6rModulerrrErGrKrnrr r r r s   LT7>'R