o jD@sddlZddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZmZddlmZddl m!Z!GdddeZ"Gdddej#Z$dS)N)DictListUnion)Coqpit)nn)TensorboardLogger)Encoder OverflowUtils)Decoder) NeuralHMM)!get_spec_from_most_probable_state&plot_transition_probabilities_to_numpy)BaseTTS)SpeakerManager) TTSTokenizer)plot_alignmentplot_spectrogram)format_aux_input) load_fsspecc seZdZdZ   dBddddddd effd d Zd efd dZddZddZ ddZ ddZ e ddZ dedejfddZdedejfddZdefd d!Zeddddd"fd#ejfd$d%Ze d&d'Ze dCddd)eeeeeffd*d+Z ,dDded-ed.ed/efd0d1Zd2d3Z e!d4d5Z"ded6ed7d8d9ed:e#f d;d<Z$ded6ed7d8d9ed:e#f d=d>Z%d6ed7d8d9ed:e#d?df d@dAZ&Z'S)EOverflowaOverFlow TTS model. Paper:: https://arxiv.org/abs/2211.06892 Paper abstract:: Neural HMMs are a type of neural transducer recently proposed for sequence-to-sequence modelling in text-to-speech. They combine the best features of classic statistical speech synthesis and modern neural TTS, requiring less data and fewer training updates, and are less prone to gibberish output caused by neural attention failures. In this paper, we combine neural HMM TTS with normalising flows for describing the highly non-Gaussian distribution of speech acoustics. The result is a powerful, fully probabilistic model of durations and acoustics that can be trained using exact maximum likelihood. Compared to dominant flow-based acoustic models, our approach integrates autoregression for improved modelling of long-range dependences such as utterance-level prosody. Experiments show that a system based on our proposal gives more accurate pronunciations and better subjective speech quality than comparable methods, whilst retaining the original advantages of neural HMMs. Audio examples and code are available at https://shivammehta25.github.io/OverFlow/. Note: - Neural HMMs uses flat start initialization i.e it computes the means and std and transition probabilities of the dataset and uses them to initialize the model. This benefits the model and helps with faster learning If you change the dataset or want to regenerate the parameters change the `force_generate_statistics` and `mel_statistics_parameter_path` accordingly. - To enable multi-GPU training, set the `use_grad_checkpointing=False` in config. This will significantly increase the memory usage. This is because to compute the actual data likelihood (not an approximation using MAS/Viterbi) we must use all the states at the previous time step during the forward pass to decide the probability distribution at the current step i.e the difference between the forward algorithm and viterbi approximation. Check :class:`TTS.tts.configs.overflow.OverFlowConfig` for class arguments. NconfigOverFlowConfigapAudioProcessor tokenizerrspeaker_managercst||||||_|D] }t||||q|j|_t|j|j|j |_ t |j|j |j |j |j|j|j|j|j|j|j|j|j|jd|_t|j|j|j|j|j|j|j|j |j!|j"|j#d |_$|%dt&'d|%dt&'ddS)N)frame_channelsar_orderdeterministic_transition encoder_dim prenet_type prenet_dimprenet_n_layersprenet_dropoutprenet_dropout_at_inferencememory_rnn_dimoutputnet_sizeflat_start_params std_flooruse_grad_checkpointing) dropout_p num_splits num_squeeze sigmoid_scale c_in_channelsmeanrstd)(super__init__rsetattr out_channelsdecoder_output_dimr num_charsstate_per_phoneencoder_in_out_featuresencoderr rrr r!r"r#r$r%r&r'r(r) neural_hmmr hidden_channels_deckernel_size_dec dilation_ratenum_flow_blocks_decnum_block_layers dropout_p_decr+r,r-r.decoderregister_buffertorchtensor)selfrrrrkey __class__J/home/kuhnn/.local/lib/python3.10/site-packages/TTS/tts/models/overflow.pyr3>sJzOverflow.__init__statistics_dictcCs(t|d|j_t|d|j_dS)Nr/r0)rDrEr/datar0)rFrLrJrJrKupdate_mean_stdrszOverflow.update_mean_stdcCsH|jdks|jdkrt|j}||||}||||fS)Nrr1)r/itemr0rDloadmel_statistics_parameter_pathrN normalize)rFtexttext_lenmelsmel_lenrLrJrJrKpreprocess_batchvs     zOverflow.preprocess_batchcC||j|jSN)subr/divr0rFxrJrJrKrR~zOverflow.normalizecCrXrY)mulr0addr/r\rJrJrKinverse_normalizer^zOverflow.inverse_normalizecCsp|||||\}}}}|||\}}||dd|\}}} |||||\} } } } | | | | | d}|S)a Forward pass for training and computing the log likelihood of a given batch. Shapes: Shapes: text: :math:`[B, T_in]` text_len: :math:`[B]` mels: :math:`[B, T_out, C]` mel_len: :math:`[B]` r1) log_probs alignmentstransition_vectorsmeans)rWr:rB transposer;)rFrSrTrUrVencoder_outputsencoder_output_lenz z_lengthslogdetrcfwd_alignmentsrerfoutputsrJrJrKforwards  zOverflow.forwardcCsxi}|d|d<|d|d<|d|d|d<|d|d|d<|S)N text_lengthsavg_text_length mel_lengthsavg_spec_lengthavg_text_batch_occupancyavg_spec_batch_occupancy)floatr/max)batchstatsrJrJrK_training_statss $$zOverflow._training_statsrx criterionc Csf|d}|d}|d}|d}|j||||d}||d||}|||||fS)N text_inputrp mel_inputrr)rSrTrUrVrc)rosumupdaterz) rFrxr{r|rpr}rrrn loss_dictrJrJrK train_stepszOverflow.train_stepcCs |||SrY)r)rFrxr{rJrJrK eval_steps zOverflow.eval_step aux_inputcCs2|}||j|j|jd|rt||S|S)zSet missing fields to their default value. Args: aux_inputs (Dict): Dictionary containing the auxiliary inputs.  sampling_tempmax_sampling_timeduration_threshold)copyrrrrr)rFrdefault_input_dictrJrJrK_format_aux_inputs zOverflow._format_aux_input) x_lengthsrrrrSc Csdtj|dkddi}|||}|j||d\}}|jj|||d|d|dd}|j|d dd |d d d \}}} ||dd }| ||dt |d|d<|S)aSampling from the model Args: text (torch.Tensor): :math:`[B, T_in]` aux_inputs (_type_, optional): _description_. Defaults to None. Returns: outputs: Dictionary containing the following - mel (torch.Tensor): :math:`[B, T_out, C]` - hmm_outputs_len (torch.Tensor): :math:`[B]` - state_travelled (List[List[int]]): List of lists containing the state travelled for each sample in the batch. - input_parameters (list[torch.FloatTensor]): Input parameters to the neural HMM. - output_parameters (list[torch.FloatTensor]): Output parameters to the neural HMM. rrr1dimrrrr hmm_outputsrbhmm_outputs_lenT)reverse) model_outputsmodel_outputs_lenrd) rDr~rr: inferencer;rBrgrarr double_pad) rFrSrrrhrirnrUmel_outputs_len_rJrJrKrs$  zOverflow.inferencecCstSrY)NLLLossrJrJrJrK get_criterionszOverflow.get_criterionTsamplescCs@ddlm}|||}t|\}}t||}t||||S)a8Initiate model from config Args: config (VitsConfig): Model config. samples (Union[List[List], List[Dict]]): Training samples to parse speaker ids for training. Defaults to None. verbose (bool): If True, print init messages. Defaults to True. r)r)TTS.utils.audiorinit_from_configrrr)rrverboserrr new_configrrJrJrKrs  zOverflow.init_from_configFcheckpoint_pathevalstrictcCsHt|tdd}||d|r ||j|jr"JdSdS)Ncpu) map_locationmodel)rrDdeviceload_state_dictrrB store_inversetraining)rFrrrrcachestaterJrJrKload_checkpoint s  zOverflow.load_checkpointcCs0tj|jjr |jjrQ|jd|jdd}td|jjdt ||jj |jj \}}}td|jjd|||f| | | d}t||jjn)td |jjd t|jj}|d |d |d }}}td|||ft|r| n||jjd<t |j||j|dS)zIf the current dataset does not have normalisation statistics and initialisation transition_probability it computes them otherwise loads.NF)training_assetsrrz$ | > Data parameters not found for: z+. Computing mel normalization parameters...z | > Saving data parameters to: z : value: )r/r0init_transition_probz | > Data parameters found for: z). Loading mel normalization parameters...r/r0rz( | > Data parameters loaded with value: transition_p)ospathisfilerrQforce_generate_statisticsget_train_dataloader train_samplesprintr "get_data_parameters_for_flat_startr5r8rOrDsaverP is_tensorr'update_flat_start_transitionrrN)rFtrainer dataloader data_meandata_stdr statisticsrJrJrK on_init_starts>   zOverflow.on_init_startcCs|d|d}}tj|ddd}t|dddd t|dd d dd t|dd dd tt|d|d|jddt|ddddd}td|j|dd dd|dd did}t|dddd|d<dd|ddD} dd|ddD} t t | ddD]} | d} | dd} t | | | | | | |d | <q| |ddj}|d!|ifS)"Nrdrerfr1rrzForward alignment)r)titlefig_sizezForward log alignmentT)rplot_logrzTransition vectors) )rr}) alignment log_alignmentremel_from_most_probable_state mel_targetz) | > Synthesising audio from the model...r|rrp)rr synthesisedcSsg|]}|dqS)r1rJ.0prJrJrK Wsz)Overflow._create_logs..input_parameterscSsg|] }|dqS)rb)rnumpyrrJrJrKrXsoutput_parametersz%synthesised_transition_probabilities/audios)rDstackrexprr rBrr unsqueezerangelenr inv_melspectrogramTrr)rFrxrnrrdrerffiguresinference_outputstates#transition_probability_synthesisingistartendaudiorJrJrK _create_logs?s6 $  zOverflow._create_logsrnloggerLoggerassetsstepscCs6||||j\}}|||||||jjdS)zLog training progress.N)rr train_figures train_audios sample_rate)rFrxrnrrrrrrJrJrK train_logds zOverflow.train_logc Csxt|tr!|D]\}}|dd}|j||j|q | |||j \}} | ||| || |j j dS)z#Compute and log evaluation metrics../N) isinstancernamed_parametersreplacewriter add_histogramrMrrrr eval_figures eval_audiosr) rFrxrnrrrtagvaluerrrJrJrKeval_logls   zOverflow.eval_logreturncCs*|||d|jj|||ddS)Nr1r) test_audiosrr test_figures)rFrnrrrrJrJrKtest_log{szOverflow.test_log)NNN)NT)FTF)(__name__ __module__ __qualname____doc__rr3rrNrWrRraro staticmethodrzdictrModulerrrrDno_gradTensorrrrrrrstrboolrrinference_moderintrrr __classcell__rJrJrHrKrs(4   ( &  ( $  rc@s$eZdZdZdejdefddZdS)rzNegative log likelihood loss.log_probrcCsi}| |d<|S)z{Compute the loss. Args: logits (Tensor): [B, T, D] Returns: Tensor: [1] loss)r/)rFr  return_dictrJrJrKros zNLLLoss.forwardN)rrrrrDrrrorJrJrJrKrsr)%rtypingrrrrDcoqpitrr"trainer.logging.tensorboard_loggerr%TTS.tts.layers.overflow.common_layersrr TTS.tts.layers.overflow.decoderr "TTS.tts.layers.overflow.neural_hmmr &TTS.tts.layers.overflow.plotting_utilsr r TTS.tts.models.base_ttsrTTS.tts.utils.speakersrTTS.tts.utils.text.tokenizerrTTS.tts.utils.visualrrTTS.utils.generic_utilsr TTS.utils.iorrrrrJrJrJrKs(          l