VITS voice conversion: Speaker Encoder gives 512 channels, posterior_encoder only accepts 256

[tom-huntington]

tts --model_name tts_models/en/vctk/vits --speaker_wav /mnt/c/Users/a/Downloads/target.m4b" \
                --encoder_path https://github.com/coqui-ai/TTS/releases/download/speaker_encoder_model/model_se.pth.tar \
                --encoder_config_path https://github.com/coqui-ai/TTS/releases/download/speaker_encoder_model/config_se.json \
                --reference_wav /mnt/c/Users/a/Downloads/source.m4a \

I am trying to take this path (although, I had to change the condition)

TTS/TTS/tts/models/vits.py

Lines 1218 to 1224 in dbf1a08

	elif not self.args.use_speaker_embedding and self.args.use_d_vector_file:
	g_src = F.normalize(speaker_cond_src).unsqueeze(-1)
	g_tgt = F.normalize(speaker_cond_tgt).unsqueeze(-1)
	else:
	raise RuntimeError(" [!] Voice conversion is only supported on multi-speaker models.")
	z, _, _, y_mask = self.posterior_encoder(y, y_lengths, g=g_src)

The problem is that

TTS/TTS/utils/synthesizer.py

Line 374 in dbf1a08

speaker_embedding = self.tts_model.speaker_manager.compute_embedding_from_clip(speaker_wav)

is producing a 512 channel embedding. But

z, _, _, y_mask = self.posterior_encoder(y, y_lengths, g=g_src)

only accepts a 256 channel embedding.

Am I using the wrong Encoder model? I tried using the your_tts model too.

Edit: Someone asked #2768 but I'm totally confused about what's going with embedding size.

[tom-huntington]

So I got it to work with the YourTTS model.

If you look at the config.json VITS has "speaker_embedding_channels": 256 and YourTTS has "speaker_embedding_channels": 512

I imagine the VITS inference_voice_conversion() and voice_conversion() functions are intended to be used with YourTTS (when using pretrained models).

[tom-huntington]

For my future reference #1480
Hmm, seems like you can actually use VITS for voice cloning #2507