o jU@sddlmZmZmZmZmZddlZddlZddl Z ddl m Z ddl m Z ddl mZmZmZddl mZddlmZddlmZdd lmZddlmmmmZddlmmmmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&ddlm'Z'm(Z(ddl)m*Z*ddl+m,Z-ddl.m/Z/Gddde j0Z1Gddde j0Z2Gddde j j0Z3Gddde j j0Z4Gddde j j0Z5Gddde j j0Z6Gddde j j0Z,Gd d!d!e&Z7dS)")DictListOptionalTupleUnionN)Coqpit)nn)Conv1dConv2dConvTranspose1d) functional) spectral_norm) weight_norm)remove_parametrizations)SpeakerManager) load_fsspec) FreeVCConfig)BaseVC) get_padding init_weights)mel_spectrogram_torch)SpeakerEncoder) get_wavlmcs(eZdZd fdd Zd ddZZS) ResidualCouplingBlockrc st||_||_||_||_||_||_||_t |_ t |D]}|j tj||||||dd|j tq#dS)NT) gin_channels mean_only)super__init__channelshidden_channels kernel_size dilation_raten_layersn_flowsrr ModuleListflowsrangeappendmodulesResidualCouplingLayerFlip) selfrr r!r"r#r$ri __class__G/home/kuhnn/.local/lib/python3.10/site-packages/TTS/vc/models/freevc.pyrs.    zResidualCouplingBlock.__init__NFcCsL|s|jD] }|||||d\}}q|St|jD] }|||||d}q|S)Ngreverse)r&reversed)r,xx_maskr3r4flow_r0r0r1forward4s zResidualCouplingBlock.forward)rr)NF__name__ __module__ __qualname__rr: __classcell__r0r0r.r1rsrcs*eZdZ dfdd ZdddZZS) Encoderrcsrt||_||_||_||_||_||_||_t ||d|_ t j |||||d|_t ||dd|_dS)Nr)rr in_channels out_channelsr r!r"r#rrr prer)WNencproj)r,rDrEr r!r"r#rr.r0r1r?s zEncoder.__init__Nc Cstt||dd|j}|||}|j|||d}| ||}tj ||j dd\}}|t |t ||}||||fS)NrCrAr3)dim)torch unsqueezecommons sequence_masksizetodtyperFrHrIsplitrE randn_likeexp) r,r6 x_lengthsr3r7statsmlogszr0r0r1r:Os" zEncoder.forwardrNr;r0r0r.r1r@>sr@cs2eZdZ d fdd Zd ddZddZZS) Generatorrc  sJtt|t||_t||_t||dddd|_|dkr"tj ntj }t |_ tt||D]#\} \} } |j tt|d| |d| d| | | | ddq1t |_tt|j D]"} |d| d} tt||D]\} \} }|j|| | |qrqat| dddddd|_|j t|d krt ||d|_dSdS) NrApadding1rCF)rabiasr)rr]rlen num_kernels num_upsamplesr conv_prer) ResBlock1 ResBlock2rr%ups enumeratezipr(rr resblocksr' conv_postapplyrcond)r,initial_channelresblockresblock_kernel_sizesresblock_dilation_sizesupsample_ratesupsample_initial_channelupsample_kernel_sizesrr-ukchjdr.r0r1rZs:       zGenerator.__init__NcCs||}|dur|||}t|jD]>}t|tj}|j||}d}t|j D]!}|dur?|j ||j ||}q,||j ||j ||7}q,||j }qt|}| |}t |}|Sr\)rgrpr'rfF leaky_relur) LRELU_SLOPErjrermrnrLtanh)r,r6r3r-xsr{r0r0r1r:s     zGenerator.forwardcCs8td|jD]}t|dq|jD]}t|dqdS)NzRemoving weight norm...weight)printrjrrm)r,lr0r0r1remove_weight_norms     zGenerator.remove_weight_normr[r\)r<r=r>rr:rr?r0r0r.r1r]Ys   +r]cs&eZdZdfdd ZddZZS) DiscriminatorPr_Fcs tt|||_||_|dkrtnt}t|t dd|df|dft |ddfd|t dd|df|dft |ddfd|t dd|df|dft |ddfd|t dd|df|dft |ddfd|t dd|dfdt |ddfdg|_ |t ddd dd d|_ dS) NFrA rr`i)r_rA)rAr) rrrperioduse_spectral_normrr rr%r rconvsrn)r,rr!stridernorm_fr.r0r1rs&&&&" zDiscriminatorP.__init__cCsg}|j\}}}||jdkr$|j||j}t|d|fd}||}|||||j|j}|jD]}||}t|tj}| |q3| |}| |t |dd}||fS)NrreflectrA) shaperr}padviewrr~r)rr(rnrLflatten)r,r6fmapbctn_padrr0r0r1r:s     zDiscriminatorP.forward)rr_Fr;r0r0r.r1rsrc&eZdZdfdd ZddZZS)DiscriminatorSFcstt||dkr tnt}t|tdddddd|tdddd d d d |tdd dd dd d |td d dd dd d |td d dd d d d |td d ddddg|_|td ddddd|_ dS)NFrAr^r`@)r)groupsrarrrCr_) rrrrr rr%r rrn)r,rrr.r0r1rs zDiscriminatorS.__init__cCsZg}|jD]}||}t|tj}||q||}||t|dd}||fS)NrAr) rr}r~r)rr(rnrLr)r,r6rrr0r0r1r:s    zDiscriminatorS.forwardFr;r0r0r.r1rsrcr)MultiPeriodDiscriminatorFcsHtt|gd}tdg}|fdd|D}t||_dS)N)rCr_rr^ rcsg|]}t|dqS)r)r).0r-rr0r1 sz5MultiPeriodDiscriminator.__init__..)rrrrrr%discriminators)r,rperiodsdiscsr.rr1rs  z!MultiPeriodDiscriminator.__init__c Cspg}g}g}g}t|jD]$\}}||\} } ||\} } || || || || q ||||fSr\)rkrr() r,yy_haty_d_rsy_d_gsfmap_rsfmap_gsr-r|y_d_rfmap_ry_d_gfmap_gr0r0r1r:s       z MultiPeriodDiscriminator.forwardrr;r0r0r.r1rsrcs8eZdZdfdd ZddZdd Zdd d ZZS)rPr_rcs>tt|tj|||dd|_t|||_t|_ dS)NT) batch_first) rrrrLSTMlstmLinearlinearReLUrelu)r,mel_n_channelsmodel_num_layersmodel_hidden_sizemodel_embedding_sizer.r0r1rszSpeakerEncoder.__init__cCsD|j||\}\}}|||d}|tj|dddS)NrrAT)rKkeepdim)rflatten_parametersrrrLnorm)r,melsr9hidden embeds_rawr0r0r1r:s zSpeakerEncoder.forwardcCs8g}td|||D]}t|||}||q |S)Nr)r'rLaranger()r, total_framespartial_frames partial_hop mel_slicesr- mel_ranger0r0r1compute_partial_slices s  z%SpeakerEncoder.compute_partial_slicesrrc sd}dd| df}||krZ||||}tfdd|D}||tt|dd}t ||}Wdn1sIwYtj |dd d} | St ||} Wd| S1snwY| S)NrAc3s |] }dd|fVqdSr\r0)rsmelr0r1 sz1SpeakerEncoder.embed_utterance..r)axis) rPrlistr(rLstacktuplesqueezeno_gradmeanrM) r,rrrmel_lenlast_melrrpartial_embedsembedr0rr1embed_utterances$      zSpeakerEncoder.embed_utterance)rr_rr)rr)r<r=r>rr:rrr?r0r0r.r1rs rcs<eZdZdZd)dedeffdd ZeddZd d Z defd d Z    d*d e j de j de e j de e j de e j de e j dee j e j e j ee j e j e j e j e j e j fffddZe d+ddZddZddZe ddZddZed,ded!eeeeeffd"d#Zd-d%d&Zd'd(ZZS).FreeVCa Papaer:: https://arxiv.org/abs/2210.15418# Paper Abstract:: Voice conversion (VC) can be achieved by first extracting source content information and target speaker information, and then reconstructing waveform with these information. However, current approaches normally either extract dirty content information with speaker information leaked in, or demand a large amount of annotated data for training. Besides, the quality of reconstructed waveform can be degraded by the mismatch between conversion model and vocoder. In this paper, we adopt the end-to-end framework of VITS for high-quality waveform reconstruction, and propose strategies for clean content information extraction without text annotation. We disentangle content information by imposing an information bottleneck to WavLM features, and propose the spectrogram-resize based data augmentation to improve the purity of extracted content information. Experimental results show that the proposed method outperforms the latest VC models trained with annotated data and has greater robustness. Original Code:: https://github.com/OlaWod/FreeVC Examples: >>> from TTS.vc.configs.freevc_config import FreeVCConfig >>> from TTS.vc.models.freevc import FreeVC >>> config = FreeVCConfig() >>> model = FreeVC(config) Nconfigspeaker_managerc st|d|d|||jj|_|jj|_|jj|_|jj|_|jj|_|jj |_ |jj |_ |jj |_ |jj |_ |jj |_ |jj|_|jj|_|jj|_|jj|_|jj|_|jj|_|jj|_|jj|_t|jj|j|jddd|_t|j|j |j |j|j|j|j|jd|_t|j|j|jddd|jd|_t|j|jddd|jd|_|jst|j|jd|_n|t |_!dS)NrrArrBr)rr)"rrinit_multispeakerargs spec_channelsinter_channelsr filter_channelsn_headsr#r! p_dropoutrrrsrtrurvrw segment_sizerssl_dimuse_spkr@enc_pr]decenc_qrr8renc_spkload_pretrained_speaker_encoderrwavlm)r,rrr.r0r1rFsR                     zFreeVC.__init__cCst|jSr\)next parametersdevicer,r0r0r1rvsz FreeVC.devicecCstdtd|_dS)z@Load pretrained speaker encoder model as mentioned in the paper.z/ > Loading pretrained speaker encoder model ...zShttps://github.com/coqui-ai/TTS/releases/download/v0.13.0_models/speaker_encoder.ptN)rSpeakerEncoderEx enc_spk_exrr0r0r1rzs z&FreeVC.load_pretrained_speaker_encodercCs"|jj|_|jr|jj|_dSdS)aInitialize multi-speaker modules of a model. A model can be trained either with a speaker embedding layer or with external `d_vectors` computed from a speaker encoder model. You must provide a `speaker_manager` at initialization to set up the multi-speaker modules. Args: config (Coqpit): Model configuration. data (List, optional): Dataset items to infer number of speakers. Defaults to None. N)rnum_spksr)r,rr0r0r1rs zFreeVC.init_multispeakerrspecr3r c_lengths spec_lengthsreturnc Cs|durt|d|d|j}|dur*t|d|d|j}d}|js7||d}|||\}}} }|j | dd||d\} } } } |j | | |d}t | ||j\}}|j||d}||| | ||| | | ffS)au Forward pass of the model. Args: c: WavLM features. Shape: (batch_size, c_seq_len). spec: The input spectrogram. Shape: (batch_size, spec_seq_len, spec_dim). g: The speaker embedding. Shape: (batch_size, spk_emb_dim). mel: The input mel-spectrogram for the speaker encoder. Shape: (batch_size, mel_seq_len, mel_dim). c_lengths: The lengths of the WavLM features. Shape: (batch_size,). spec_lengths: The lengths of the spectrogram. Shape: (batch_size,). Returns: o: The output spectrogram. Shape: (batch_size, spec_seq_len, spec_dim). ids_slice: The slice indices. Shape: (batch_size, num_slices). spec_mask: The spectrogram mask. Shape: (batch_size, spec_seq_len). (z, z_p, m_p, logs_p, m_q, logs_q): A tuple of latent variables. NrrrArCrJ)rLonesrPrQrrrrMrr transposer8rNrand_slice_segmentsrr)r,rrr3rrrr9m_plogs_prZm_qlogs_q spec_maskz_pz_slice ids_sliceor0r0r1r:s!"" zFreeVC.forwardc Cs|dkrt|d|d|j}|js#|j|}|d}| ||\}}}}|j |||dd} |j | ||d} | S)a Inference pass of the model Args: c (torch.Tensor): Input tensor. Shape: (batch_size, c_seq_len). g (torch.Tensor): Speaker embedding tensor. Shape: (batch_size, spk_emb_dim). mel (torch.Tensor): Mel-spectrogram tensor. Shape: (batch_size, mel_seq_len, mel_dim). c_lengths (torch.Tensor): Lengths of the input tensor. Shape: (batch_size,). Returns: torch.Tensor: Output tensor. NrrTr2rJ) rLrrPrQrrrrrMrr8r) r,rr3rrrrrc_maskrZr r0r0r1 inferences"  zFreeVC.inferencecCsHt|j|d}Wdn1swY|dd}|S)zExtract WavLM features from an audio tensor. Args: y (torch.Tensor): Audio tensor. Shape: (batch_size, audio_seq_len). rNrArC)rLrrextract_featuresr)r,rrr0r0r1extract_wavlm_featuress  zFreeVC.extract_wavlm_featurescCst|trtj||jjjd\}}t|tjr t | |j }t|t j r,| |j }t|tr=t t| |j }|S)z Read and format the input audio.)sr) isinstancestrlibrosaloadraudioinput_sample_ratenpndarrayrL from_numpyrQrTensorrarrayfloat)r,wavr9r0r0r1 load_audios     zFreeVC.load_audioc Cs$||}tjj|dd\}}|jjjr/|j |}t |ddddf |j}n,t |d |j}t||jjj|jjj|jjj|jjj|jjj|jjj|jjj}||}||dddf}|jjjrx|j||d} n |j||ddd} | ddj} | S) z Voice conversion pass of the model. Args: src (str or torch.Tensor): Source utterance. tgt (str or torch.Tensor): Target utterance. Returns: torch.Tensor: Output tensor. r)top_dbNrrJrArCr)rcpunumpyreffectstrimr model_argsrrrrLrrQrrMrr filter_lengthn_mel_channelsr hop_length win_lengthmel_fminmel_fmaxr r rdatar) r,srctgtwav_tgtr9g_tgtmel_tgtwav_srcrrr0r0r1voice_conversions.   "  zFreeVC.voice_conversioncCdSr\r0r0r0r0r1 eval_step#zFreeVC.eval_stepTsamplescCs t|}|Sr\)r)rr4verbosemodelr0r0r1init_from_config&szFreeVC.init_from_configFcCs:t|td|d}|j|d|d|r|dSdS)Nr) map_locationcacher6)strict)rrLrload_state_dicteval)r,rcheckpoint_pathr<r:r9stater0r0r1load_checkpoint+s  zFreeVC.load_checkpointcCr1r\r0r0r0r0r1 train_step1r3zFreeVC.train_stepr\)NNNN)NNN)NT)FTF) r<r=r>__doc__rrrpropertyrrrrLrrrr:rr r rinference_moder0r2 staticmethodrrrrr7r?r@r?r0r0r.r1r*sV0  :   )& r)8typingrrrrrrrrrLcoqpitrrtorch.nnr r r r r}torch.nn.utilsr torch.nn.utils.parametrizationsrtorch.nn.utils.parametrizerTTS.vc.modules.freevc.commonsvcr)freevcrNTTS.vc.modules.freevc.modulesTTS.tts.utils.speakersr TTS.utils.iorTTS.vc.configs.freevc_configrTTS.vc.models.base_vcrrr$TTS.vc.modules.freevc.mel_processingr5TTS.vc.modules.freevc.speaker_encoder.speaker_encoderrrTTS.vc.modules.freevc.wavlmrModulerr@r]rrrrr0r0r0r1s:             $I'*