o j`@sddlmZddlZddlmZddlmmZ ddlmZddl m Z ddl m Z mZddlmZddlmZGdd d ejZGd d d ejZGd d d ejZdS))ListN)nn) checkpoint) Outputnet OverflowUtils)Prenet) sequence_maskcseZdZdZ d/dedededededed ed ed ed ed eede dedeffdd Z ddZ e ddZ ddZddZe ddZe ddZdd Ze d0d"d#Zed$ejd%ejd&ed'ed(ef d)d*Zed+d,Ze d-d.ZZS)1 NeuralHMMu Autoregressive left to right HMM model primarily used in "Neural HMMs are all you need (for high-quality attention-free TTS)" Paper:: https://arxiv.org/abs/2108.13320 Paper abstract:: Neural sequence-to-sequence TTS has achieved significantly better output quality than statistical speech synthesis using HMMs. However, neural TTS is generally not probabilistic and uses non-monotonic attention. Attention failures increase training time and can make synthesis babble incoherently. This paper describes how the old and new paradigms can be combined to obtain the advantages of both worlds, by replacing attention in neural TTS with an autoregressive left-right no-skip hidden Markov model defined by a neural network. Based on this proposal, we modify Tacotron 2 to obtain an HMM-based neural TTS model with monotonic alignment, trained to maximise the full sequence likelihood without approximation. We also describe how to combine ideas from classical and contemporary TTS for best results. The resulting example system is smaller and simpler than Tacotron 2, and learns to speak with fewer iterations and less data, whilst achieving comparable naturalness prior to the post-net. Our approach also allows easy control over speaking rate. Args: frame_channels (int): Output dimension to generate. ar_order (int): Autoregressive order of the model. In ablations of Neural HMM it was found that more autoregression while giving more variation hurts naturalness of the synthesised audio. deterministic_transition (bool): deterministic duration generation based on duration quantiles as defiend in "S. Ronanki, O. Watts, S. King, and G. E. Henter, “Medianbased generation of synthetic speech durations using a nonparametric approach,” in Proc. SLT, 2016.". Defaults to True. encoder_dim (int): Channels of encoder input and character embedding tensors. Defaults to 512. prenet_type (str): `original` or `bn`. `original` sets the default Prenet and `bn` uses Batch Normalization version of the Prenet. prenet_dim (int): Dimension of the Prenet. prenet_n_layers (int): Number of layers in the Prenet. prenet_dropout (float): Dropout probability of the Prenet. prenet_dropout_at_inference (bool): If True, dropout is applied at inference time. memory_rnn_dim (int): Size of the memory RNN to process output of prenet. outputnet_size (List[int]): Size of the output network inside the neural HMM. flat_start_params (dict): Parameters for the flat start initialization of the neural HMM. std_floor (float): Floor value for the standard deviation of the neural HMM. Prevents model cheating by putting point mass and getting infinite likelihood at any datapoint. use_grad_checkpointing (bool, optional): Use gradient checkpointing to save memory. Defaults to True. Tframe_channelsar_orderdeterministic_transition encoder_dim prenet_type prenet_dimprenet_n_layersprenet_dropoutprenet_dropout_at_inferencememory_rnn_dimoutputnet_sizeflat_start_params std_flooruse_grad_checkpointingcst|_|_|_|_| _|_t_ t _ |dks*Jd||_t ||||| fddt |Ddd_tj|| d_t|| || | | _dt|d dS) Nrz)AR order must be greater than 0 provided csg|]}jqS)r).0_selfrU/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/layers/overflow/neural_hmm.py Usz&NeuralHMM.__init__..F) in_featuresrrdropout_at_inference out_featuresbias) input_size hidden_size go_tokens)super__init__r r r rrrTransitionModeltransition_model EmissionModelemission_modelrrangeprenetrLSTMCell memory_rnnr output_netregister_buffertorchzeros)rr r r r rrrrrrrrrr __class__rrr(0s0  zNeuralHMM.__init__c Cs|j\}}}t|}|ddd}||} |||\} } } } ||}|||j|\}}t |D]}| ||||\}}|j rQ|j rQt |j||\}}}n |||\}}}|dkro| ||dddf|||}n ||dd|f||||| dd|dddf||}tj|dd| dd|f<|| dd|fd| dd|ddf<|| dd|f<| |q4||| | \} } ||| || }tj| dd|}|| | | fS)aVHMM forward algorithm for training uses logarithmic version of Rabiner (1989) forward algorithm. Args: inputs (torch.FloatTensor): Encoder outputs inputs_len (torch.LongTensor): Encoder output lengths mels (torch.FloatTensor): Mel inputs mel_lens (torch.LongTensor): Length of mel inputs Shapes: - inputs: (B, T, D_out_enc) - inputs_len: (B) - mels: (B, D_mel, T_mel) - mel_lens: (B) Returns: log_prob (torch.FloatTensor): Log probability of the sequence rr&Ndim)shaper3maxpermute_initialize_log_state_priors'_initialize_forward_algorithm_variables _add_go_token_init_lstm_statesrr-_process_ar_timesteprtrainingrr1r,r* logsumexp unsqueezeappenddetach _mask_lengths#get_absorption_state_scaling_factorsum)rinputs inputs_lenmelsmel_lens batch_sizeNrT_maxlog_state_priorslog_clog_alpha_scaledtransition_matrixmeans ar_inputsh_memoryc_memorytmeanstdtransition_vectorlog_alpha_tempsum_final_log_c log_probsrrrforward^s6      ", zNeuralHMM.forwardcCs*t|}||}|d}||}||fS)a Mask the lengths of the forward variables so that the variable lenghts do not contribute in the loss calculation Args: mel_inputs (torch.FloatTensor): (batch, T, frame_channels) mel_inputs_lengths (torch.IntTensor): (batch) log_c (torch.FloatTensor): (batch, T) Returns: log_c (torch.FloatTensor) : scaled probabilities (batch, T) log_alpha_scaled (torch.FloatTensor): forward probabilities (batch, T, N) r7)rrD)rMrRrS mask_log_cmask_log_alpha_scaledrrrrGs   zNeuralHMM._mask_lengthscCsF|dd|||jfd}||}||||f\}}||fS)a Process autoregression in timestep 1. At a specific t timestep 2. Perform data dropout if applied (we did not use it) 3. Run the autoregressive frame through the prenet (has dropout) 4. Run the prenet output through the post prenet rnn Args: t (int): mel-spec timestep ar_inputs (torch.FloatTensor): go-token appended mel-spectrograms - shape: (b, D_out, T_out) h_post_prenet (torch.FloatTensor): previous timestep rnn hidden state - shape: (b, memory_rnn_dim) c_post_prenet (torch.FloatTensor): previous timestep rnn cell state - shape: (b, memory_rnn_dim) Returns: h_post_prenet (torch.FloatTensor): rnn hidden state of the current timestep c_post_prenet (torch.FloatTensor): rnn cell state of the current timestep Nr&)r flattenr.r0)rrYrVrWrX prenet_input memory_inputsrrrrAs  zNeuralHMM._process_ar_timestepcCsL|j\}}}|jd||j|j}tj||fddddd|f}|S)zAppend the go token to create the autoregressive input Args: mel_inputs (torch.FloatTensor): (batch_size, T, n_mel_channel) Returns: ar_inputs (torch.FloatTensor): (batch_size, T, n_mel_channel) rr&r8N)r:r%rDexpandr r r3cat)r mel_inputsrNTrr%rVrrrr?s "zNeuralHMM._add_go_tokenc CsH|j\}}}||||f}|||}||||f}g}||||fS)aoInitialize placeholders for forward algorithm variables, to use a stable version we will use log_alpha_scaled and the scaling constant Args: mel_inputs (torch.FloatTensor): (b, T_max, frame_channels) N (int): number of states Returns: log_c (torch.FloatTensor): Scaling constant (b, T_max) )r: new_zeros) rhrObrPrrSrRrTrUrrrr>s   z1NeuralHMM._initialize_forward_algorithm_variablescCs||||||fS)a Initialize Hidden and Cell states for LSTM Cell Args: batch_size (Int): batch size hidden_state_dim (Int): dimensions of the h and c device_tensor (torch.FloatTensor): useful for the device and type Returns: (torch.FloatTensor): shape (batch_size, hidden_state_dim) can be hidden state for LSTM (torch.FloatTensor): shape (batch_size, hidden_state_dim) can be the cell state for LSTM )rj)rNhidden_state_dim device_tensorrrrr@s  zNeuralHMM._init_lstm_statescCst|}|jd}t||d}|ddd|d}t|d|d} | |t d } t|d|d} t | } t | } | ||j} | | t d } | | }|jt|jjd}tj|dd}|S)aReturns the final scaling factor of absorption state Args: mels_len (torch.IntTensor): Input size of mels to get the last timestep of log_alpha_scaled log_alpha_scaled (torch.FloatTEnsor): State probabilities text_lengths (torch.IntTensor): length of the states to mask the values of states lengths ( Useful when the batch has very different lengths, when the length of an observation is less than the number of max states, then the log alpha after the state value is filled with -infs. So we mask those values so that it only consider the states which are needed for that length ) transition_vector (torch.FloatTensor): transtiion vector for each state per timestep Shapes: - mels_len: (batch_size) - log_alpha_scaled: (batch_size, N, T) - text_lengths: (batch_size) - transition_vector: (batch_size, N, T) Returns: sum_final_log_c (torch.FloatTensor): (batch_size) r7)max_lenr&inf)minr8)r3r;r:rrDrfgathersqueeze masked_fillfloatsigmoidr log_clampedget_mask_for_last_itemdeviceclampfinfodtyperqrC)rmels_lenrSrKr\rOmax_inputs_lenstate_lengths_masklast_log_alpha_scaled_indexlast_log_alpha_scaledlast_transition_vectorlast_transition_probability log_probability_of_transitioning!last_transition_probability_index final_log_cr^rrrrHs$      z-NeuralHMM.get_absorption_state_scaling_factorNcCsLt|}|durtjd||dntjd||d}||ddk}|S)aqReturns n-1 mask for the last item in the sequence. Args: lengths (torch.IntTensor): lengths in a batch device (str, optional): Defaults to "cpu". out_tensor (torch.Tensor, optional): uses the memory of a specific tensor. Defaults to None. Returns: - Shape: :math:`(b, max_len)` Nr)ry)outr&)r3r;itemarangerD)lengthsry out_tensorrnidsmaskrrrrx>s  &z NeuralHMM.get_mask_for_last_itemrJ input_lens sampling_tempmax_sampling_timeduration_thresholdc Cs|jd}gggggd}t|D]=}||||d|||||\} } } } |d| |d| jd|d| |d| |d| qtjjj|dd d |d<tj |d|j |j d |d<|S) aInference from autoregressive neural HMM Args: inputs (torch.FloatTensor): input states - shape: :math:`(b, T, d)` input_lens (torch.LongTensor): input state lengths - shape: :math:`(b)` sampling_temp (float): sampling temperature max_sampling_temp (int): max sampling temperature duration_threshold (float): duration threshold to switch to next state - Use this to change the spearking rate of the synthesised audio r) hmm_outputshmm_outputs_len alignmentsinput_parametersoutput_parametersr&rrrrrT) batch_first)r|ry) r:r-samplerErutilsrnn pad_sequencer3tensorr|ry) rrJrrrrrkoutputsineural_hmm_outputsstates_travelledrrrrr inferenceRs*    zNeuralHMM.inferencecCsggd}}}d} || |jdd|j|j} |d|j| \} } g} g}d} || dd}| | d| | f\} } |dd| fd}| | |\}}}t | }t | }| | | g||||g|jj|||d}t j| |fddddddf} || t ||f}||9}|js|dd}n||k}|r| d7} d}|| | |ks|r||dkrn|d7}q-t j|ddt||| |fS)aDSamples an output from the parameter models Args: inputs (torch.FloatTensor): input states - shape: :math:`(1, T, d)` input_lens (torch.LongTensor): input state lengths - shape: :math:`(1)` sampling_temp (float): sampling temperature max_sampling_time (int): max sampling time duration_threshold (float): duration threshold to switch to next state Returns: outputs (torch.FloatTensor): Output Observations - Shape: :math:`(T, output_dim)` states_travelled (list[int]): Hidden states travelled - Shape: :math:`(T)` input_parameters (list[torch.FloatTensor]): Input parameters output_parameters (list[torch.FloatTensor]): Output parameters rr&TN)rr8)rEr%rDrfr r r@rr.rcr0rsr1r3rvr,rrgr multinomialrstackFone_hot new_tensor)rrJrrrrrrrY current_staterdrWrXinput_parameter_valuesoutput_parameter_valuesquantile memory_inputz_trZr[r\transition_probabilitystaying_probabilityx_trTswitchrrrrsL "  )zNeuralHMM.samplecCs*|jd}||gtd }d|d<|S)zCreates the log pi in forward algorithm. Args: text_embeddings (torch.FloatTensor): used to create the log pi on current device Shapes: - text_embeddings: (B, T, D_out_enc) r&rpgr)r:new_fullru)text_embeddingsrOrQrrrr=s z&NeuralHMM._initialize_log_state_priors)TN)__name__ __module__ __qualname____doc__intboolstrrurdictr(r` staticmethodrGrAr?r>r@rHrxr3inference_mode FloatTensor LongTensorrrr= __classcell__rrr5rr sx0     .@    8  . Pr c@seZdZdZddZdS)r)zvTransition Model of the HMM, it represents the probability of transitioning form current state to all other statesc Cst|}t| }t|}t|}||}||} | jddd} td | dddf<t|} tjtj|| fdddd} | | td } | S)aS product of the past state with transitional probabilities in log space Args: log_alpha_scaled (torch.Tensor): Multiply previous timestep's alphas by transition matrix (in log domain) - shape: (batch size, N) transition_vector (torch.tensor): transition vector for each state - shape: (N) inputs_len (int tensor): Lengths of states in a batch - shape: (batch) Returns: out (torch.FloatTensor): log probability of transitioning to each state r&)dimsrpNrr7r8) r3rvrrwrollrurrCrrt) rrSr\rK transition_p staying_plog_staying_probabilitylog_transition_probabilitystayingleavinginputs_len_maskrrrrr`s    zTransitionModel.forwardN)rrrrr`rrrrr)s r)cs2eZdZdZd fdd ZddZdd ZZS) r+zrEmission Model of the HMM, it represents the probability of emitting an observation based on the current statereturnNcsttjj|_dSr)r'r(tdistnormalNormaldistribution_functionrr5rrr( s zEmissionModel.__init__cCs |dkr||||S|S)Nr)rr)rrUstdsrrrrrs zEmissionModel.samplecCs@|||}||d}t|d}tj||dd}|S)aCalculates the log probability of the the given data (x_t) being observed from states with given means and stds Args: x_t (float tensor) : observation at current time step - shape: (batch, feature_dim) means (float tensor): means of the distributions of hidden states - shape: (batch, hidden_state, feature_dim) stds (float tensor): standard deviations of the distributions of the hidden states - shape: (batch, hidden_state, feature_dim) state_lengths (int tensor): Lengths of states in a batch - shape: (batch) Returns: out (float tensor): observation log likelihoods, expressing the probability of an observation being generated from a state i shape: (batch, hidden_state) r&r7r8)rlog_probrDrr3rI)rrrUr state_lengthsemission_distsrrrrrr`s zEmissionModel.forward)rN)rrrrr(rr`rrrr5rr+s r+)typingrr3torch.distributions distributionsrtorch.nn.functionalr functionalrtorch.utils.checkpointr%TTS.tts.layers.overflow.common_layersrr%TTS.tts.layers.tacotron.common_layersrTTS.tts.utils.helpersrModuler r)r+rrrrs      X$