o ji@s<ddlZddlZddlmZmZmZddlZddlZddl m Z ddl m m Z ddl mZddlmZmZmZmZmZmZmZmZmZeeZ    ddeeefd eejd ed ed ed edede dedej!fddZ"GdddZ#Gddde j$Z%Gddde j$Z&Gddde j$Z'Gddde j$Z(dS)N)ListOptionalTuple) LayerNorm) Fp32GroupNorm Fp32LayerNorm GLU_Linear GradMultiplyMultiheadAttentionSamePad TransposeLastget_activation_fninit_bert_paramsstaticFshape padding_mask mask_prob mask_length mask_type mask_other min_masks no_overlap min_spacereturnc  s|\} } t| | fd} t|| t|tj} t|| } g} t| D]}|durN| || }t||t|tj}t||}n| }| }|dkr]t||nA|dkrotjj ||dd|dn/|dkrtjj |||dd d Dn|d krtjj ||dd d Dntd |t dkrt||dd<|r gfdd}d|fg}t}tddD]>tfdd|Dtj}t |}|dkrn#|t |}tjjt||d}||\}}|||||qtn-t}|||kr||d}tjj|||ddtfdd ttD| t|kq&tdd | D}t| D]\}t|krhtjj|ddd| |f<qT| S)a Computes random mask spans for a given shape Args: shape: the the shape for which to compute masks. should be of size 2 where first element is batch size and 2nd is timesteps padding_mask: optional padding mask of the same size as shape, which will prevent masking padded elements mask_prob: probability for each token to be chosen as start of the span to be masked. this will be multiplied by number of timesteps divided by length of mask span to mask approximately this percentage of all elements. however due to overlaps, the actual number will be smaller (unless no_overlap is True) mask_type: how to compute mask lengths static = fixed size uniform = sample from uniform distribution [mask_other, mask_length*2] normal = sample from normal distribution with mean mask_length and stdev mask_other. mask is min 1 element poisson = sample from possion distribution with lambda = mask length min_masks: minimum number of masked spans no_overlap: if false, will switch to an alternative recursive algorithm that prevents spans from overlapping min_space: only used if no_overlap is True, this is how many elements to keep unmasked between spans FNruniform)sizenormalcSsg|] }tdtt|qS)r)maxintround.0xr&T/home/kuhnn/.local/lib/python3.10/site-packages/TTS/vc/modules/freevc/wavlm/wavlm.py asz(compute_mask_indices..poissoncSsg|]}tt|qSr&)r!r"r#r&r&r'r(dszunknown mask selection rcstj|||fddt|Dg}||kr,||df|||krA|||f|S)Nc3s|]}|VqdSNr&r$i span_startr&r' psz8compute_mask_indices..arrange..r)nprandomrandintextendrangeappend)selength keep_length new_parts)mask_idcrr-r'arrangensz%compute_mask_indices..arrangeT)reversec3s0|]\}}||kr||ndVqdS)rNr&)r$r6r7)r8rr&r'r/}s.z'compute_mask_indices..p)replacecs*g|]}t|D]}||q qSr&)r4)r$joffset)lengthsr;r&r'r(s*cSsg|]}t|qSr&)len)r$mr&r&r'r(s)r0fullr!floatr1randr r4longsumitemr2rr) ExceptionminsortedfromiterchoicerDpopr3asarrayr5unique enumerate)rrrrrrrrrbszall_szmask all_num_mask mask_idcsr,sznum_maskr<parts min_lengthlensl_sumprobscr6r7min_lenr&)r8rCr;rr'compute_mask_indices#s~        "rcc@s$eZdZdddZdefddZdS) WavLMConfigNcCsd|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ d |_ d |_ d |_ d |_d |_d |_d |_d |_d |_d|_d|_d|_d |_d |_d |_d|_d|_d|_d|_d|_d|_d|_ d|_!d|_"|durt|#|dSdS)Ndefault  geluFz0[(512,10,5)] + [(512,3,2)] * 4 + [(512,2,2)] * 2?皙?r g?rrri@i)$extractor_modeencoder_layersencoder_embed_dimencoder_ffn_embed_dimencoder_attention_heads activation_fnlayer_norm_firstconv_feature_layers conv_biasfeature_grad_mult normalizedropoutattention_dropoutactivation_dropoutencoder_layerdrop dropout_inputdropout_featuresrrmask_selectionrno_mask_overlapmask_min_spacemask_channel_lengthmask_channel_probmask_channel_selectionmask_channel_otherno_mask_channel_overlapmask_channel_min_spaceconv_posconv_pos_groupsrelative_position_embedding num_buckets max_distance gru_rel_posupdateselfcfgr&r&r'__init__sPzWavLMConfig.__init__rcCs|j|dSr*)__dict__rrr&r&r'rszWavLMConfig.updater*)__name__ __module__ __qualname__rdictrr&r&r&r'rds 9rdc seZdZdeddffdd ZddZdejd ejdejfd d Z   dd ejd e ejde de de e de f ddZ Z S)WavLMrrNcs(ttd|j||_t|j}|dd|_t |d|j |j d|_ |j|j kr6t|j|j nd|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_|j|_t|j|_t|j|_|j |_ t!t"#|j $|_%t&||_'t(|j|_)dS)NzWavLM Config: rr) conv_layersrzmoderw)*superrloggerinforrevalrvembedConvFeatureExtractionModelrorwfeature_extractorrqnnLinearpost_extract_projrrrrrrrrrrrrDropoutr~rrx Parametertorch FloatTensoruniform_mask_embTransformerEncoderencoderr layer_norm)rrfeature_enc_layers __class__r&r'rs>   zWavLM.__init__c Cs|j\}}}|jdkr/t||f||j|j|j|jd|j|jd }t | |j }|j ||<nd}|j dkr_t||fd|j |j|j|j|j|jd}t | |j dd|d}d||<||fS)Nrr)rrr)rrrr)rrrcrrrrrr from_numpytodevicerrrrrrr unsqueezeexpand)rr%rBTC mask_indicesmask_channel_indicesr&r&r' apply_mask s<    " zWavLM.apply_maskfeaturesrcCsZ|d|d}|dkr|ddd| f}||d|dd}|d}|S)Nrrr)rviewany)rrrextrar&r&r'forward_padding_mask,s  zWavLM.forward_padding_maskFsourcerWret_conv output_layerret_layer_resultsc Cs$|jdkr||}|jdkrt||j}nt ||}Wdn1s+wY|dd}||}|durE|||}|j durO| |}| |}|r_| ||\}} n|}|j |||durkdn|dd\}} |||| d} |r| dn| d} |r| | d f} | | d fS) Nrrjrr)rlayer)r%rr layer_resultsrr%rr) rxrr applyrno_grad transposerrrr~rr) rrrrWrrrrr%rrresfeaturer&r&r'extract_features9s4             zWavLM.extract_features)NFFNF)rrrrdrrrTensorrrboolr!r __classcell__r&r&rr'rs@.# rc sTeZdZ    ddeeeeefdedededef fd d Z dd d Z Z S)rrreFrrzrrw conv_typec st|dvs J   dfdd }||_|jdkr\d}t|_t|D]0\}} t| dks;Jdt| | \} } |j ||| | |d k|dkoR|d k|d }q)dS|jd krd}t|_t|D])\}} t| dksxJ| \} } |j t j || | |j t j }qldS|jd krd}d} t|_t|D]V\}} t| dksJ| \} } |j t jj || | dd|j t j | g|j t j }|ddd kr|j t jjddddtt| d} qdS dS)N>rerFc sfdd}|o |dksJd|r0t|tjdtttddttS|rFt|tjdtddtSt|tjdtS) Ncs&tjd}tj|j|S)N)stridebias)rConv1dinitkaiming_normal_weight)convrwkn_inn_outrr&r' make_convszEConvFeatureExtractionModel.__init__..block..make_convFz'layer norm and group norm are exclusiver>T)elementwise_affine)affine)r Sequentialrr rGELUr)rrrr is_layer_norm is_group_normrwrdimrzrr'blockxs*      z2ConvFeatureExtractionModel.__init__..blockrerzinvalid conv definition: rr)rrrwconv2dcustomP)paddingrT)r ceil_mode)FFF)rrrr ModuleListrrTrDstrr5rConv2dReLUr MaxPool2dr!mathceil) rrrzrrwrrin_dr,clrridimrrr'rlsj  %           z#ConvFeatureExtractionModel.__init__NcCs|d}|jdkr@|jD]}t|tjr$|dd}||dd}q ||}q |dd}|| dd| d}|S|jD]}||}qC|jdkrf| \}}}}|dd||||}|S)Nrrrrrrr) rrr isinstancerrr contiguousrr)rr%rWrbratfr&r&r'forwards        z"ConvFeatureExtractionModel.forward)rreFrer*) rrrrrr!rGrrrrrr&r&rr'rks"`rcs0eZdZfddZdddZdddZZS) rcs>tj_j_tjjjjjdjd_ d}t dd|jj}tj j j jd|dtj j jdtjjjj ddd_ tj tjt_ td rpj_j_j_n d _d_d_tfd d tjD_j_t j_!j"_#$t%dS) Nr) kernel_sizergroupsrrj)meanstdr)namerrFcsLg|]"}tjjjjjjjjj o|dkj j j d qS)r) embedding_dimffn_embedding_dimnum_attention_headsrzr{r|rtruhas_relative_attention_biasrrr) TransformerSentenceEncoderLayerrrrrsrzr{r|rtrurrrrr+argsrr&r'r(s" z/TransformerEncoder.__init__..)&rrrzrqrrrrrpos_convrsqrtrnormal_r constant_rutilsparametrizations weight_normrr rhasattrrrrrr4rplayersrurrr} layerdroprr)rr rzrrr r'rs@     zTransformerEncoder.__init__NcCs4|||||\}}|jr|dur||}||fSr*)rrur)rr%rstreaming_maskrrr&r&r'rs zTransformerEncoder.forwardc Cs|durd||<||dd}|dd}||7}|js#||}tj||j|jd}|dd}g}d}|durB|||fd}d} t|j D]0\} } t j } |jr\| |j krh| ||d|| d\}}} |durs|||f| |kr{|}nqK|dur|}|dd}||fS)Nrrr)r?trainingF)self_attn_padding_mask need_weightsself_attn_maskpos_bias) r rrurFrzrr5rTrr0r1r) rr%rr tgt_layerx_convrzrrr,rdropout_probabilityr&r&r'r"sF      z#TransformerEncoder.extract_features)NNN)rrrrrrrr&r&rr'rs  8rcseZdZdZ             d d ed ed ed ed edededededededededdffdd Z    d!de j de j de j defddZ Z S)"rz_ Implements a Transformer Encoder Layer used in BERT/XLM style pre-trained models. rgrhrkreluFrrrrrzr{r|rtrurrr rescale_initrrNc st||_||_||_||_t||_t|j||d| | | | | d |_ t ||_ t |j|_ t ||_||_t|j|_|jdkrOt|j|d|_nt |j||_t ||j|_t|j|_dS)NT)rzself_attentionrrrr#rgluswish)rrrrzr|activation_namer rtr self_attnrrdropout1dropout2dropout3rurself_attn_layer_normrfc1rfc2final_layer_norm)rrrrrzr{r|rtrurrrr#rrr&r'rWs6      z(TransformerSentenceEncoderLayer.__init__r%rrrc Cs>|}|jrP||}|j||||d||d\}}}||}||}|}||}|jdkr4||}n|||}||}| |}| |}||}nJ|j|||||||d\}}}||}||}||}|}|jdkrz||}n|||}||}| |}| |}||}||}|||fS)z LayerNorm is applied either before or after the self-attention/ffn modules similar to the original Transformer imlementation. F)querykeyvaluekey_padding_maskr attn_mask position_biasr%) rur,r(r)r/r'r-rtr*r.r+)rr%rrrrresidualattnr&r&r'rsZ                   z'TransformerSentenceEncoderLayer.forward) rgrhr!rkrkrkr"FFrrFF)NNFN) rrr__doc__rGrrr!rrrrrr&r&rr'rQsr     :r)rrrFr))loggingrtypingrrrnumpyr0rtorch.nnrtorch.nn.functional functionalrr#TTS.vc.modules.freevc.wavlm.modulesrrrr r r r r r getLoggerrrr!rrGrrndarrayrcrdModulerrrrr&r&r&r'sT   ,    y>vp