o ´‹ j±Dã@srddlZddlZddlmZddlmZddlmZmZGdd„dej ƒZ Gdd„dej ƒZ Gd d „d ej ƒZ dS) éN)Únn)Ú functional)Ú LayerNormÚ LayerNorm2cs†eZdZdZ      d‡fdd„ Zddd „Zdd d „Zed d „ƒZedd„ƒZ dd„Z edd„ƒZ edd„ƒZ edd„ƒZ ‡ZS)Ú"RelativePositionMultiHeadAttentionaµMulti-head attention with Relative Positional embedding. https://arxiv.org/pdf/1809.04281.pdf It learns positional embeddings for a window of neighbours. For keys and values, it learns different set of embeddings. Key embeddings are agregated with the attention scores and value embeddings are aggregated with the output. Note: Example with relative attention window size 2 - input = [a, b, c, d, e] - rel_attn_embeddings = [e(t-2), e(t-1), e(t+1), e(t+2)] So it learns 4 embedding vectors (in total 8) separately for key and value vectors. Considering the input c - e(t-2) corresponds to c -> a - e(t-2) corresponds to c -> b - e(t-2) corresponds to c -> d - e(t-2) corresponds to c -> e These embeddings are shared among different time steps. So input a, b, d and e also uses the same embeddings. Embeddings are ignored when the relative window is out of limit for the first and the last n items. Args: channels (int): input and inner layer channels. out_channels (int): output channels. num_heads (int): number of attention heads. rel_attn_window_size (int, optional): relation attention window size. If 4, for each time step next and previous 4 time steps are attended. If default, relative encoding is disabled and it is a regular transformer. Defaults to None. heads_share (bool, optional): [description]. Defaults to True. dropout_p (float, optional): dropout rate. Defaults to 0.. input_length (int, optional): intput length for positional encoding. Defaults to None. proximal_bias (bool, optional): enable/disable proximal bias as in the paper. Defaults to False. proximal_init (bool, optional): enable/disable poximal init as in the paper. Init key and query layer weights the same. Defaults to False. NTçFc sˆtƒ ¡||dksJdƒ‚||_||_||_||_||_||_||_||_ d|_ |||_ t   ||d¡|_t   ||d¡|_t   ||d¡|_t   ||d¡|_t  |¡|_|dur’|r]dn|} |j d} t  t | |dd|j ¡| ¡} t  t | |dd|j ¡| ¡} | d| ¡| d| ¡t j |jj¡t j |jj¡| rº|jjj |jjj¡|jjj |jjj¡t j |jj¡dS)Nrz/ [!] channels should be divisible by num_heads.égà¿éÚ emb_rel_kÚ emb_rel_v)ÚsuperÚ__init__ÚchannelsÚ out_channelsÚ num_headsÚrel_attn_window_sizeÚ heads_shareÚ input_lengthÚ proximal_biasÚ dropout_pÚattnÚ k_channelsrÚConv1dÚconv_qÚconv_kÚconv_vÚconv_oÚDropoutÚdropoutÚ ParameterÚtorchÚrandnÚregister_parameterÚinitÚxavier_uniform_ÚweightÚdataÚcopy_Úbias)ÚselfrrrrrrrrÚ proximal_initÚ n_heads_relÚ rel_stddevr r ©Ú __class__©úV/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/glow_tts/transformer.pyr 7sD     ÿÿ  z+RelativePositionMultiHeadAttention.__init__cCsD| |¡}| |¡}| |¡}|j||||d\}|_| |¡}|S)z Shapes: - x: :math:`[B, C, T]` - c: :math:`[B, C, T]` - attn_mask: :math:`[B, 1, T, T]` )Úmask)rrrÚ attentionrr)r)ÚxÚcÚ attn_maskÚqÚkÚvr/r/r0Úforwardms    z*RelativePositionMultiHeadAttention.forwardcCsèg| ¡¢| d¡‘R\}}}}| ||j|j|¡ dd¡}| ||j|j|¡ dd¡}| ||j|j|¡ dd¡}t || dd¡¡t |j¡} |j duru||ksWJdƒ‚|  |j |¡} |  || ¡} |  | ¡} | t |j¡} | | } |jrŽ||ks€Jdƒ‚| | |¡j| j| jd} |dur¸|  |dkd ¡} |jdur¸t | ¡ d|j¡ |j¡} | | d d | } tj| dd }| |¡}t ||¡}|j durã| |¡}|  |j|¡}|| ||¡}| dd¡ ¡ |||¡}||fS) Nr ééþÿÿÿéÿÿÿÿz8Relative attention is only available for self-attention.z3Proximal bias is only available for self-attention.)ÚdeviceÚdtypergˆÃÀr)Údim)ÚsizeÚviewrrÚ transposer ÚmatmulÚmathÚsqrtrÚ_get_relative_embeddingsr Ú_matmul_with_relative_keysÚ'_relative_position_to_absolute_positionrÚ_attn_proximity_biasÚtor=r>Ú masked_fillrÚ ones_likeÚtriuÚtrilÚFÚsoftmaxrÚ'_absolute_position_to_relative_positionr Ú_matmul_with_relative_valuesÚ contiguous)r)ÚqueryÚkeyÚvaluer1ÚbÚdÚt_sÚt_tÚscoresÚkey_relative_embeddingsÚ rel_logitsÚ scores_localÚ block_maskÚp_attnÚoutputÚrelative_weightsÚvalue_relative_embeddingsr/r/r0r2{s:          z,RelativePositionMultiHeadAttention.attentioncCst || d¡¡}|S)a Args: p_attn (Tensor): attention weights. re (Tensor): relative value embedding vector. (a_(i,j)^V) Shapes: -p_attn: :math:`[B, H, T, V]` -re: :math:`[H or 1, V, D]` -logits: :math:`[B, H, T, D]` r)r rCÚ unsqueeze)r`ÚreÚlogitsr/r/r0rR¥s z?RelativePositionMultiHeadAttention._matmul_with_relative_valuescCst || d¡ dd¡¡}|S)a& Args: query (Tensor): batch of query vectors. (x*W^Q) re (Tensor): relative key embedding vector. (a_(i,j)^K) Shapes: - query: :math:`[B, H, T, D]` - re: :math:`[H or 1, V, D]` - logits: :math:`[B, H, T, V]` rr;r<)r rCrdrB)rTrerfr/r/r0rG´s z=RelativePositionMultiHeadAttention._matmul_with_relative_keysc Csvt||jddƒ}t|jd|dƒ}|d|d}|dkr-t |dd||ddg¡}n|}|dd…||…f}|S)z3Convert embedding vestors to a tensor of embeddingsrrr N)ÚmaxrrOÚpad)r)Úrelative_embeddingsÚlengthÚ pad_lengthÚslice_start_positionÚslice_end_positionÚpadded_relative_embeddingsÚused_relative_embeddingsr/r/r0rFÄsz;RelativePositionMultiHeadAttention._get_relative_embeddingsc Cs–| ¡\}}}}t |gd¢¡}| |||d|g¡}t |d|dddddg¡}| |||dd|dg¡dd…dd…d|…|dd…f}|S)zÐConverts tensor from relative to absolute indexing for local attention. Shapes: x: :math:`[B, C, T, 2 * T - 1]` Returns: A Tensor of shape :math:`[B, C, T, T]` )rrrrrrrrr rrN©r@rOrhrA©r3ÚbatchÚheadsrjÚ_Úx_flatÚx_finalr/r/r0rHÑs >zJRelativePositionMultiHeadAttention._relative_position_to_absolute_positionc Csž| ¡\}}}}t |d|dddddddg¡}| |||d||dg¡}t ||dddddg¡}| |||d|g¡dd…dd…dd…dd…f}|S)zk Shapes: - x: :math:`[B, C, T, T]` - ret: :math:`[B, C, T, 2*T-1]` rrr Nrprqr/r/r0rQãs   2zJRelativePositionMultiHeadAttention._absolute_position_to_relative_positioncCsJtj|tjd}t |d¡t |d¡}t t |¡¡ }| d¡ d¡S)zÖProduce an attention mask that discourages distant attention values. Args: length (int): an integer scalar. Returns: a Tensor with shape :math:`[1, 1, T, T]` )r>rr)r ÚarangeÚfloat32rdÚlog1pÚabs)rjÚrÚdiffr/r/r0rIós z7RelativePositionMultiHeadAttention._attn_proximity_bias)NTrNFF©N)Ú__name__Ú __module__Ú __qualname__Ú__doc__r r9r2Ú staticmethodrRrGrFrHrQrIÚ __classcell__r/r/r-r0r s,1ö 6 *    rcsFeZdZdZd‡fdd„ Zdd„Zdd „Zd d „Zed d „ƒZ ‡Z S)ÚFeedForwardNetworkaUFeed Forward Inner layers for Transformer. Args: in_channels (int): input tensor channels. out_channels (int): output tensor channels. hidden_channels (int): inner layers hidden channels. kernel_size (int): conv1d filter kernel size. dropout_p (float, optional): dropout rate. Defaults to 0. rFcsntƒ ¡||_||_||_||_||_|r|j|_n|j |_t   |||¡|_ t   |||¡|_ t  |¡|_dSr})r r Ú in_channelsrÚhidden_channelsÚ kernel_sizerÚ_causal_paddingÚpaddingÚ _same_paddingrrÚconv_1Úconv_2rr)r)r…rr†r‡rÚcausalr-r/r0r s  zFeedForwardNetwork.__init__cCsD| | ||¡¡}t |¡}| |¡}| | ||¡¡}||Sr})r‹r‰r ÚrelurrŒ)r)r3Úx_maskr/r/r0r9"s   zFeedForwardNetwork.forwardcCsH|jdkr|S|jd}d}ddgddg||gg}t || |¡¡}|S)Nrr©r‡rOrhÚ _pad_shape©r)r3Úpad_lÚpad_rr‰r/r/r0rˆ)s  z"FeedForwardNetwork._causal_paddingcCsR|jdkr|S|jdd}|jd}ddgddg||gg}t || |¡¡}|S)Nrr rrr’r/r/r0rŠ2s  z FeedForwardNetwork._same_paddingcCs |ddd…}dd„|Dƒ}|S)Nr<cSsg|] }|D]}|‘qqSr/r/)Ú.0ÚsublistÚitemr/r/r0Ú >sz1FeedForwardNetwork._pad_shape..r/)r‰ÚlÚ pad_shaper/r/r0r‘;szFeedForwardNetwork._pad_shape)rF) r~rr€rr r9rˆrŠr‚r‘rƒr/r/r-r0r„s   r„csZeZdZdZ     ddededed ed ed ed ed edef‡fdd„ Zdd„Z‡ZS)ÚRelativePositionTransformeraîTransformer with Relative Potional Encoding. https://arxiv.org/abs/1803.02155 Args: in_channels (int): number of channels of the input tensor. out_chanels (int): number of channels of the output tensor. hidden_channels (int): model hidden channels. hidden_channels_ffn (int): hidden channels of FeedForwardNetwork. num_heads (int): number of attention heads. num_layers (int): number of transformer layers. kernel_size (int, optional): kernel size of feed-forward inner layers. Defaults to 1. dropout_p (float, optional): dropout rate for self-attention and feed-forward inner layers_per_stack. Defaults to 0. rel_attn_window_size (int, optional): relation attention window size. If 4, for each time step next and previous 4 time steps are attended. If default, relative encoding is disabled and it is a regular transformer. Defaults to None. input_length (int, optional): input lenght to limit position encoding. Defaults to None. layer_norm_type (str, optional): type "1" uses torch tensor operations and type "2" uses torch layer_norm primitive. Use type "2", type "1: is for backward compat. Defaults to "1". rrNÚ1r…rr†Úhidden_channels_ffnrÚ num_layersrrÚlayer_norm_typec stƒ ¡||_||_||_||_||_||_| |_t   |¡|_ t   ¡|_ t   ¡|_t   ¡|_t   ¡|_t|jƒD]Œ} |j  t| dkrE|n|||| || d¡| dkr\|j t|ƒ¡n| dkri|j t|ƒ¡ntdƒ‚||kr€| d|jkr€t  ||d¡|_|j t|| d|jkrŽ|n||||d¡| dkr¬|j t| d|jkr§|n|ƒ¡q9| dkrÂ|j t| d|jkr½|n|ƒ¡q9tdƒ‚dS)Nr)rrrrœÚ2z [!] Unknown layer norm typer)r)r r r†rrržr‡rrrrrÚ ModuleListÚ attn_layersÚ norm_layers_1Ú ffn_layersÚ norm_layers_2ÚrangeÚappendrrrÚ ValueErrorrÚprojr„) r)r…rr†rrržr‡rrrrŸÚidxr-r/r0r Xs\      úÿ ûÿ $$Üz$RelativePositionTransformer.__init__cCs´| d¡| d¡}t|jƒD]D}||}|j||||ƒ}| |¡}|j|||ƒ}|j|||ƒ}| |¡}|d|jkrJt|dƒrJ| |¡}|j |||ƒ}q||}|S)zd Shapes: - x: :math:`[B, C, T]` - x_mask: :math:`[B, 1, T]` r r<rr©) rdr¦ržr¢rr£r¤Úhasattrr©r¥)r)r3rr5ÚiÚyr/r/r0r9›s   z#RelativePositionTransformer.forward)rrNNrœ) r~rr€rÚintÚstrr r9rƒr/r/r-r0r›Bs6ôþýüûúù ö õ ôCr›) rDr rÚtorch.nnrrOÚ$TTS.tts.layers.generic.normalizationrrÚModulerr„r›r/r/r/r0Ús  }<