o jC@sddlZddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z m Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZGdddeZ Gdddej!Z"dS)N)DictListUnion)Coqpit)nn)TensorboardLogger)Encoder OverflowUtils) NeuralHMM)!get_spec_from_most_probable_state&plot_transition_probabilities_to_numpy)BaseTTS)SpeakerManager) TTSTokenizer)plot_alignmentplot_spectrogram)format_aux_input) load_fsspecc seZdZdZ   dBddddddd effd d Zd efd dZddZddZ ddZ ddZ e ddZ dedejfddZdedejfddZdefd d!Zeddddd"fd#ejfd$d%Ze d&d'Ze dCddd)eeeeeffd*d+Z ,dDded-ed.ed/efd0d1Zd2d3Z e!d4d5Z"ded6ed7d8d9ed:e#f d;d<Z$ded6ed7d8d9ed:e#f d=d>Z%d6ed7d8d9ed:e#d?df d@dAZ&Z'S)E NeuralhmmTTSao Neural HMM TTS model. Paper:: https://arxiv.org/abs/2108.13320 Paper abstract:: Neural sequence-to-sequence TTS has achieved significantly better output quality than statistical speech synthesis using HMMs.However, neural TTS is generally not probabilistic and uses non-monotonic attention. Attention failures increase training time and can make synthesis babble incoherently. This paper describes how the old and new paradigms can be combined to obtain the advantages of both worlds, by replacing attention in neural TTS with an autoregressive left-right no-skip hidden Markov model defined by a neural network. Based on this proposal, we modify Tacotron 2 to obtain an HMM-based neural TTS model with monotonic alignment, trained to maximise the full sequence likelihood without approximation. We also describe how to combine ideas from classical and contemporary TTS for best results. The resulting example system is smaller and simpler than Tacotron 2, and learns to speak with fewer iterations and less data, whilst achieving comparable naturalness prior to the post-net. Our approach also allows easy control over speaking rate. Audio examples and code are available at https://shivammehta25.github.io/Neural-HMM/ . Note: - This is a parameter efficient version of OverFlow (15.3M vs 28.6M). Since it has half the number of parameters as OverFlow the synthesis output quality is suboptimal (but comparable to Tacotron2 without Postnet), but it learns to speak with even lesser amount of data and is still significantly faster than other attention-based methods. - Neural HMMs uses flat start initialization i.e it computes the means and std and transition probabilities of the dataset and uses them to initialize the model. This benefits the model and helps with faster learning If you change the dataset or want to regenerate the parameters change the `force_generate_statistics` and `mel_statistics_parameter_path` accordingly. - To enable multi-GPU training, set the `use_grad_checkpointing=False` in config. This will significantly increase the memory usage. This is because to compute the actual data likelihood (not an approximation using MAS/Viterbi) we must use all the states at the previous time step during the forward pass to decide the probability distribution at the current step i.e the difference between the forward algorithm and viterbi approximation. Check :class:`TTS.tts.configs.neuralhmm_tts_config.NeuralhmmTTSConfig` for class arguments. NconfigNeuralhmmTTSConfigapAudioProcessor tokenizerrspeaker_managercst||||||_|D] }t||||qt|j|j|j|_t |j |j |j |j|j |j|j|j|j|j|j|j|j|jd|_|dtd|dtddS)N)frame_channelsar_orderdeterministic_transition encoder_dim prenet_type prenet_dimprenet_n_layersprenet_dropoutprenet_dropout_at_inferencememory_rnn_dimoutputnet_sizeflat_start_params std_flooruse_grad_checkpointingmeanrstd)super__init__rsetattrr num_charsstate_per_phoneencoder_in_out_featuresencoderr out_channelsrrrr r!r"r#r$r%r&r'r( neural_hmmregister_buffertorchtensor)selfrrrrkey __class__O/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/models/neuralhmm_tts.pyr-As.zNeuralhmmTTS.__init__statistics_dictcCs(t|d|j_t|d|j_dS)Nr)r*)r6r7r)datar*)r8r>r<r<r=update_mean_stdeszNeuralhmmTTS.update_mean_stdcCsH|jdks|jdkrt|j}||||}||||fS)Nrr+)r)itemr*r6loadmel_statistics_parameter_pathr@ normalize)r8texttext_lenmelsmel_lenr>r<r<r=preprocess_batchis     zNeuralhmmTTS.preprocess_batchcC||j|jSN)subr)divr*r8xr<r<r=rDqzNeuralhmmTTS.normalizecCrJrK)mulr*addr)rNr<r<r=inverse_normalizetrPzNeuralhmmTTS.inverse_normalizec CsZ|||||\}}}}|||\}}||||dd|\}}} } ||| | d} | S)a Forward pass for training and computing the log likelihood of a given batch. Shapes: Shapes: text: :math:`[B, T_in]` text_len: :math:`[B]` mels: :math:`[B, T_out, C]` mel_len: :math:`[B]` r+) log_probs alignmentstransition_vectorsmeans)rIr2r4 transpose) r8rErFrGrHencoder_outputsencoder_output_lenrUfwd_alignmentsrWrXoutputsr<r<r=forwardws  zNeuralhmmTTS.forwardcCsxi}|d|d<|d|d<|d|d|d<|d|d|d<|S)N text_lengthsavg_text_length mel_lengthsavg_spec_lengthavg_text_batch_occupancyavg_spec_batch_occupancy)floatr)max)batchstatsr<r<r=_training_statss $$zNeuralhmmTTS._training_statsrg criterionc Csf|d}|d}|d}|d}|j||||d}||d||}|||||fS)N text_inputr_ mel_inputra)rErFrGrHrU)r^sumupdateri) r8rgrjrkr_rlrar] loss_dictr<r<r= train_stepszNeuralhmmTTS.train_stepcCs |||SrK)rp)r8rgrjr<r<r= eval_steps zNeuralhmmTTS.eval_step aux_inputcCs2|}||j|j|jd|rt||S|S)zSet missing fields to their default value. Args: aux_inputs (Dict): Dictionary containing the auxiliary inputs.  sampling_tempmax_sampling_timeduration_threshold)copyrnrtrurvr)r8rrdefault_input_dictr<r<r=_format_aux_inputs zNeuralhmmTTS._format_aux_input) x_lengthsrtrurvrEc Csdtj|dkddi}|||}|j||d\}}|jj|||d|d|dd}|d |d }}||}|||d t |d |d <|S) aSampling from the model Args: text (torch.Tensor): :math:`[B, T_in]` aux_inputs (_type_, optional): _description_. Defaults to None. Returns: outputs: Dictionary containing the following - mel (torch.Tensor): :math:`[B, T_out, C]` - hmm_outputs_len (torch.Tensor): :math:`[B]` - state_travelled (List[List[int]]): List of lists containing the state travelled for each sample in the batch. - input_parameters (list[torch.FloatTensor]): Input parameters to the neural HMM. - output_parameters (list[torch.FloatTensor]): Output parameters to the neural HMM. rzrr+dimrtrurvrs hmm_outputshmm_outputs_len) model_outputsmodel_outputs_lenrV) r6rmryr2 inferencer4rSrnr double_pad) r8rErrrxrZr[r]rGmel_outputs_lenr<r<r=rs   zNeuralhmmTTS.inferencecCstSrK)NLLLossr<r<r<r= get_criterionszNeuralhmmTTS.get_criterionTsamplescCs@ddlm}|||}t|\}}t||}t||||S)a8Initiate model from config Args: config (VitsConfig): Model config. samples (Union[List[List], List[Dict]]): Training samples to parse speaker ids for training. Defaults to None. verbose (bool): If True, print init messages. Defaults to True. r)r)TTS.utils.audiorinit_from_configrrr)rrverboserrr new_configrr<r<r=rs  zNeuralhmmTTS.init_from_configFcheckpoint_pathevalstrictcCs>t|tdd}||d|r||jrJdSdS)Ncpu) map_locationmodel)rr6deviceload_state_dictrtraining)r8rrrrcachestater<r<r=load_checkpoints zNeuralhmmTTS.load_checkpointcCs0tj|jjr |jjrQ|jd|jdd}td|jjdt ||jj |jj \}}}td|jjd|||f| | | d}t||jjn)td |jjd t|jj}|d |d |d }}}td|||ft|r| n||jjd<t |j||j|dS)zIf the current dataset does not have normalisation statistics and initialisation transition_probability it computes them otherwise loads.NF)training_assetsrrz$ | > Data parameters not found for: z+. Computing mel normalization parameters...z | > Saving data parameters to: z : value: )r)r*init_transition_probz | > Data parameters found for: z). Loading mel normalization parameters...r)r*rz( | > Data parameters loaded with value: transition_p)ospathisfilerrCforce_generate_statisticsget_train_dataloader train_samplesprintr "get_data_parameters_for_flat_startr3r0rAr6saverB is_tensorr&update_flat_start_transitionrr@)r8trainer dataloader data_meandata_stdr statisticsr<r<r= on_init_starts>   zNeuralhmmTTS.on_init_startcCs|d|d}}tj|ddd}t|dddd t|dd d dd t|dd dd tt|d|dddt|ddddd}td|j|dddd|dddid}t|dddd|d<dd|ddD} dd|ddD} t t | ddD]} | d} | dd} t | | | | | | |d | <q| |ddj }|d!|ifS)"NrVrWrXr+r{rzForward alignment)r)titlefig_sizezForward log alignmentT)rplot_logrzTransition vectors) )rrl) alignment log_alignmentrWmel_from_most_probable_state mel_targetz) | > Synthesising audio from the model...rkrzr_)rrr synthesisedcSsg|]}|dqS)r+r<.0pr<r<r= Gsz-NeuralhmmTTS._create_logs..input_parameterscSsg|] }|dqS)rT)rnumpyrr<r<r=rHsoutput_parametersz%synthesised_transition_probabilities/audios)r6stackrexprr rr unsqueezerangelenr inv_melspectrogramTrr)r8rgr]rrVrWrXfiguresinference_outputstates#transition_probability_synthesisingistartendaudior<r<r= _create_logs/s6 $  zNeuralhmmTTS._create_logsr]loggerLoggerassetsstepscCs6||||j\}}|||||||jjdS)zLog training progress.N)rr train_figures train_audios sample_rate)r8rgr]rrrrrr<r<r= train_logTs zNeuralhmmTTS.train_logc Csxt|tr!|D]\}}|dd}|j||j|q | |||j \}} | ||| || |j j dS)z#Compute and log evaluation metrics../N) isinstancernamed_parametersreplacewriter add_histogramr?rrrr eval_figures eval_audiosr) r8rgr]rrrtagvaluerrr<r<r=eval_log\s   zNeuralhmmTTS.eval_logreturncCs*|||d|jj|||ddS)Nr+r) test_audiosrr test_figures)r8r]rrrr<r<r=test_logkszNeuralhmmTTS.test_log)NNN)NT)FTF)(__name__ __module__ __qualname____doc__rr-rr@rIrDrSr^ staticmethodridictrModulerprqryr6no_gradTensorrrrrrrstrboolrrinference_moderintrrr __classcell__r<r<r:r=rs,$   & &  ( $  rc@s$eZdZdZdejdefddZdS)rzNegative log likelihood loss.log_probrcCsi}| |d<|S)z{Compute the loss. Args: logits (Tensor): [B, T, D] Returns: Tensor: [1] loss)r))r8r return_dictr<r<r=r^us zNLLLoss.forwardN)rrrrr6rrr^r<r<r<r=rrsr)#rtypingrrrr6coqpitrr"trainer.logging.tensorboard_loggerr%TTS.tts.layers.overflow.common_layersrr "TTS.tts.layers.overflow.neural_hmmr &TTS.tts.layers.overflow.plotting_utilsr r TTS.tts.models.base_ttsr TTS.tts.utils.speakersrTTS.tts.utils.text.tokenizerrTTS.tts.utils.visualrrTTS.utils.generic_utilsr TTS.utils.iorrrrr<r<r<r=s&         ]