o j-@sddlmZmZddlZddlmmZddlmZddlm Z ddl m Z ddl m Z GdddejZGd d d ejZGd d d ejZGd ddZdS))ListTupleN)nn)tqdmLinear) ConvBNBlockcsPeZdZdZd fdd Zdejdejdeejejffd d Z d d Z Z S)EncoderanNeural HMM Encoder Same as Tacotron 2 encoder but increases the input length by states per phone Args: num_chars (int): Number of characters in the input. state_per_phone (int): Number of states per phone. in_out_channels (int): number of input and output channels. n_convolutions (int): number of convolutional layers. cst||_||_t|||_t|_t |D] }|j t ||ddqtj |t |d|ddddd|_d|_dS)NreluT) num_layers batch_firstbias bidirectional)super__init__state_per_phonein_out_channelsr Embeddingemb ModuleList convolutionsrangeappendrLSTMintlstm rnn_state)self num_charsrrn_convolutions_ __class__X/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/overflow/common_layers.pyrs    zEncoder.__init__xx_lenreturncCs|j\}}||dd}|jD]}||}q|dd}tjjj||dd}|j | |\}}tjjj |dd\}}| |||j |j}||j }||fS)aForward pass to the encoder. Args: x (torch.FloatTensor): input text indices. - shape: :math:`(b, T_{in})` x_len (torch.LongTensor): input text lengths. - shape: :math:`(b,)` Returns: Tuple[torch.FloatTensor, torch.LongTensor]: encoder outputs and output lengths. -shape: :math:`((b, T_{in} * states_per_phone, in_out_channels), (b,))` rrTr)shaper transposerrutilsrnnpack_padded_sequencecpur flatten_parameterspad_packed_sequencereshaperrr"r*r+bTolayerr%r(r(r)forward,s      zEncoder.forwardcCsr|j\}}||dd}|jD]}||}q|dd}||\}}||||j|j}||j}||fS)aInference to the encoder. Args: x (torch.FloatTensor): input text indices. - shape: :math:`(b, T_{in})` x_len (torch.LongTensor): input text lengths. - shape: :math:`(b,)` Returns: Tuple[torch.FloatTensor, torch.LongTensor]: encoder outputs and output lengths. -shape: :math:`((b, T_{in} * states_per_phone, in_out_channels), (b,))` rr)r.rr/rr r6rrr7r(r(r) inferenceFs     zEncoder.inference)r r ) __name__ __module__ __qualname____doc__rtorch FloatTensor LongTensorrr<r= __classcell__r(r(r&r)r s  &r c sJeZdZdZdeededededef fdd Zd d Zd d Z Z S) ParameterModelaMain neural network of the outputnet Note: Do not put dropout layers here, the model will not converge. Args: outputnet_size (List[int]): the architecture of the parameter model input_size (int): size of input for the first layer output_size (int): size of output i.e size of the feature dim frame_channels (int): feature dim to set the flat start bias flat_start_params (dict): flat start parameters to set the bias outputnet_size input_size output_sizeframe_channelsflat_start_paramscsjt||_tddt|g|dd|D|_t|d||_| |d|d|ddS)NcSsg|] \}}t||qSr(r).0inpoutr(r(r) xsz+ParameterModel.__init__..meanstd transition_p) rrrJrrziplayersr last_layerflat_start_output_layer)r"rGrHrIrJrKr&r(r)rls  zParameterModel.__init__cCsd|jjj||jjjd|j<t||jjj|jd|j<t||jjjd|jd<dS)Nrr) rVweightdatazero_rrJ OverflowUtilsinverse_softplusinverse_sigmod)r"rQrRrSr(r(r)rWs "z&ParameterModel.flat_start_output_layercCs(|jD] }t||}q||}|S)N)rUFr rV)r"r*r;r(r(r)r<s  zParameterModel.forward) r>r?r@rArrdictrrWr<rEr(r(r&r)rF_s rFc sReZdZdZ ddedededeededef fd d Zd d Z d dZ Z S) Outputnetz This network takes current state and previous observed values as input and returns its parameters, mean, standard deviation and probability of transition to the next state {Gz? encoder_dimmemory_rnn_dimrJrGrK std_floorc sHt||_||_||_||}d|d}t|||||d|_dS)Nrr)rGrHrIrKrJ)rrrJrKrdrFparametermodel) r"rbrcrJrGrKrdrHrIr&r(r)rs   zOutputnet.__init__c Cs|jd|jd}}|jd}|d|||}tj||fdd}||}|ddddd|jf|dddd|jd|jf|ddddd|jdfd}}}t |}| |}|||fS)aInputs observation and returns the means, stds and transition probability for the current state Args: ar_mel_inputs (torch.FloatTensor): shape (batch, prenet_dim) states (torch.FloatTensor): (batch, hidden_states, hidden_state_dim) Returns: means: means for the emission observation for each feature - shape: (B, hidden_states, feature_size) stds: standard deviations for the emission observation for each feature - shape: (batch, hidden_states, feature_size) transition_vectors: transition vector for the current hidden state - shape: (batch, hidden_states) rrrdimN) r. unsqueezeexpandrBcatrerJsqueezer^softplus _floor_std) r"ar_melsinputs batch_size prenet_dimNrQrRtransition_vectorr(r(r)r<s   $   zOutputnet.forwardcCs6|}tj||jd}t||krtd|S)a@ It clamps the standard deviation to not to go below some level This removes the problem when the model tries to cheat for higher likelihoods by converting one of the gaussians to a point mass. Args: std (float Tensor): tensor containing the standard deviation to be minzg[*] Standard deviation was floored! The model is preventing overfitting, nothing serious to worry about)clonedetachrBclamprdanyprint)r"rRoriginal_tensorr(r(r)rms zOutputnet._floor_std)ra) r>r?r@rArrr_floatrr<rmrEr(r(r&r)r`s$ r`c@s~eZdZedejjjdedefddZ ee ddZ edd d Z ed d Z ed dZeddZeddZdS)r[ data_loader out_channelsstates_per_phonecCsd}d}d}d}t|ddD].}|d}|d} |d} |t|7}|t| 7}|t| 7}|tt| d7}q|||} t|||t| d} |t|j} |t|j}|| }d|}| | ||fS) aGenerates data parameters for flat starting the HMM. Args: data_loader (torch.utils.data.Dataloader): _description_ out_channels (int): mel spectrogram channels states_per_phone (_type_): HMM states per phone rF)leavetoken_id_lengthsmel mel_lengthsrr)rrBsumpowsqrtlendataset)r}r~rtotal_state_len total_mel_len total_mel_sumtotal_mel_sq_sumbatch text_lengthsmelsr data_meandata_stdaverage_num_statesaverage_mel_lenaverage_duration_each_stateinit_transition_probr(r(r)"get_data_parameters_for_flat_starts&  z0OverflowUtils.get_data_parameters_for_flat_startcCs|jjjdd|dS)Ng?) neural_hmm output_netrerW)modelrSr(r(r)update_flat_start_transitionsz*OverflowUtils.update_flat_start_transition-C6?cCstj||d}t|S)z Avoids the log(0) problem Args: x (torch.tensor): input tensor eps (float, optional): lower bound. Defaults to 1e-04. Returns: torch.tensor: :math:`log(x)` rt)rBrxlog)r*eps clamped_xr(r(r) log_clamped s zOverflowUtils.log_clampedcCs&t|s t|}t|d|S)z1 Inverse of the sigmoid function r)rB is_tensortensorr[rr*r(r(r)r]s  zOverflowUtils.inverse_sigmodcCs(t|s t|}tt|dS)z2 Inverse of the softplus function r)rBrrr[rexprr(r(r)r\#s  zOverflowUtils.inverse_softpluscCsd|j|d\}}|td k}|||dj|dj|d}||d||td S)a Differentiable LogSumExp: Does not creates nan gradients when all the inputs are -inf yeilds 0 gradients. Args: x : torch.Tensor - The input tensor dim: int - The dimension on which the log sum exp has to be applied rfinfrr)maxr| masked_fill_rhrrr)r*rgmr%masksr(r(r) logsumexp,s $"zOverflowUtils.logsumexpcsDdddd|DD}t|fdd|D}tjjj|ddS)z: Pads the list of tensors in 2 dimensions cSsg|]}t|qSr()r)rLar(r(r)rO@z,OverflowUtils.double_pad..cSsg|]}|dqSrr()rLir(r(r)rO@rc s(g|]}t|dt|dfqSr)r^padr)rLr*second_dim_maxr(r)rOBs(Tr-)rrr0r1 pad_sequence)list_of_different_shape_tensorssecond_dim_lenspadded_xr(rr) double_pad;szOverflowUtils.double_padN)r)r>r?r@ staticmethodrBr0rY DataLoaderrrno_gradrrr]r\rrr(r(r(r)r[s, &     r[)typingrrrBtorch.nn.functionalr functionalr^ tqdm.autor%TTS.tts.layers.tacotron.common_layersr!TTS.tts.layers.tacotron.tacotron2rModuler rFr`r[r(r(r(r)s    S-R