o j@s`ddlZddlmZGdddejZGdddejZGdddejZGd d d ejZdS) N)nncs&eZdZdfdd ZddZZS) LayerNorm-C6?csJt||_||_ttd|dd|_tt d|d|_ dS)aLayer norm for the 2nd dimension of the input. Args: channels (int): number of channels (2nd dimension) of the input. eps (float): to prevent 0 division Shapes: - input: (B, C, T) - output: (B, C, T) 皙?N super__init__channelsepsr Parametertorchonesgammazerosbetaselfr r  __class__W/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/generic/normalization.pyr s zLayerNorm.__init__cCsTtj|ddd}tj||dddd}||t||j}||j|j}|S)NrT)keepdim)r meanrsqrtr rr)rxrvariancerrrforwards zLayerNorm.forward)r)__name__ __module__ __qualname__r r __classcell__rrrrrsrcs*eZdZdZdfdd ZddZZS) LayerNorm2a Layer norm for the 2nd dimension of the input using torch primitive. Args: channels (int): number of channels (2nd dimension) of the input. eps (float): to prevent 0 division Shapes: - input: (B, C, T) - output: (B, C, T) h㈵>cs>t||_||_tt||_tt ||_ dSNrrrrrr *s zLayerNorm2.__init__cCs8|dd}tjj||jf|j|j|j}|ddS)Nr) transposer r functional layer_normr rrr rrrrrr2s   zLayerNorm2.forward)r$rr r!__doc__r rr"rrrrr#s r#cs.eZdZdZdfdd ZfddZZS) TemporalBatchNorm1dz6Normalize each channel separately over time and batch.Trcstj||||ddS)N)affinetrack_running_statsmomentum)rr )rr r.r/r0rrrr ;szTemporalBatchNorm1d.__init__cst|ddddS)Nrr)rrr'r*rrrr>szTemporalBatchNorm1d.forward)TTrr+rrrrr-8sr-csDeZdZdZdfdd ZdddZdd Zd d Zd d ZZ S)ActNormaoActivation Normalization bijector as an alternative to Batch Norm. It computes mean and std from a sample data in advance and it uses these values for normalization at training. Args: channels (int): input channels. ddi (False): data depended initialization flag. Shapes: - inputs: (B, C, T) - outputs: (B, C, T) Fc sHt||_| |_ttd|d|_ttd|d|_ dS)Nr) rr r initializedrr r rlogsbias)rr ddikwargsrrrr Ps zActNorm.__init__NcKs|durt|dd|dj|j|jd}t|ddg}|js,|||d|_|rB||j t |j |}d}||fS|j t |j ||}t|j |}||fS)Nrrr)devicedtypeT) r rsizetor7r8sumr2 initializer4expr3)rrx_maskreverser6x_lenzlogdetrrrrXs( zActNorm.forwardcCsdSr%r)rrrr store_inverseiszActNorm.store_inversecCs | |_dSr%)r2)rr5rrrset_ddils zActNorm.set_ddic Cstlt|ddg}t||ddg|}t|||ddg|}||d}dtt|d}| t| j|jjj |jj d}| j|j jj |j j d} |jj ||j j | WddS1sswYdS)Nrrg?gư>)r8)r no_gradr;log clamp_minr=viewr4shaper:r8r3datacopy_) rrr>denommm_sqvr3 bias_init logs_initrrrr<os  ("zActNorm.initialize)F)NF) rr r!r,r rrCrDr<r"rrrrr1Bs r1)r rModulerr# BatchNorm1dr-r1rrrrs