Here I test models with audio purposes.
AI AUDIO
Automatic speech recognition (ASR) or STT speech to text
1. Use spellschecker on top of ASR model. Or use Seq2Seq (encoder decoder) type of model, in this case decoder will correct unlikly words. But if model is large and trained on big ammount of labeled data, it can learn word probabilities like LLM.
2. To resample audio bitrate:
from datasets import Audio
good_dataset = dataset_with_wrong_sample_rate.cast_column("audio", Audio(sampling_rate=sampling_rate_I_need))
Whisper
1. From OpenAI trained on 680k hours. Has from 38M to 1.5B parameters. Maps spectogram to text.
2. To load with Transformers >>> model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
3. Usage
pipe = pipeline("automatic-speech-recognition", model="openai/whisper-base", device="cuda")
pipe(my_audion_array, max_new_tokens=256)
4. Could be used for translation pipe(my_audion_array, max_new_tokens=256, generate_kwargs={"task": "translate"})
4.1. How to configure a model to translate to non-english language?
4.1.1. Try pipeline(generate_kwargs={“language”: “portuguese”}) // Doesn't work. Only first word is translated.
4.1.2. Try
forced_decoder_ids = processor.get_decoder_prompt_ids(language=portuguese, task=translate)
pipe(generate_kwargs={“forced_decoder_ids”: forced_decoder_ids})
// Doesn't work
4.1.3. Try
generate_kwargs = {"language":"<|pt|>","task": "transcribe"} // It works
4.1.4. Try:
model.generation_config.language = "portuguese"
model.generation_config.task = "translate"
model.generation_config.forced_decoder_ids = None
// Doesn't work
4.2. Model generates only 15 characters and then repeat characters.
4.2.1. generate_kwargs = {"temperature": 1} // Changing temperature parameter is helping somehow but translation is not very good.
4.2.2. generate_kwargs = {"repetition_penalty":1.5} // Helps
5. Attention mask is not needed.
MMS from Meta
0. Speech to text model.
1. Massively Multilingual Speech 1B parameters model trained on Wav2Vec2 dataset 500000 hours of speech on 1400 languages.
2. Raw model that should be fine-tuned for recognition? translation or classification.
3. I cannot load huggingface dataset from Google Colab with load_dataset.
3.0. Create Access Token in huggingface.co/settings/tokens
3.1 Add to Google Colab Secrets panel.
3.2. to access the key
from google.colab import userdata
my_key = userdata.get("My_key_name")
3.3. Then try to login from colab with !huggingface-cli login --token $my_key // To install $ pip install -U "huggingface_hub[cli]"
3.3.1. Or try add to load_dataset( use_auth_token="my_token_itself" ) // doesn't work
3.4. Get error on load_dataset doesn't have 'use_auth_key' // Try to use just 'token'
4. I cannot accomplish from transformers import AutoProcessor
4.1. try to update transformers // didn't helped
4.2. Change .py file name
4.3. Install from git // $ pip install git+https...
5. Got bag: RuntimeError: Decoding 'mp3' files requires system library 'libsndfile'>=1.1.0, You can try to update `soundfile` python library: `pip install "soundfile>=0.12.1"`. The problem this library is already installed and version is sufficient.
5.1. Reinstall libsndfile manually
$ git clone https://github.com/libsndfile/libsndfile.git
$ cd libsndfile/
$ autoreconf -vif # Get error
$ ./configure --enable-werror # Get error
$ make # Get error
make install
5.2. Try to downgrade $ pip install soundfile==0.12.0 // Doesn't help
5.3. I am tired of it. I will try to use some third party mp3 audiofile.
5.4. For me works only 1b-all model.
6. The model works quite good for different languages.
Audio generation
MusicGen
1. From Meta, trained on 20000 hours of music.
2. By default it generates audio of length 1500 units. To increase it try to change file generation_config.json max_length
It is possible also to generate a spectogram with an image diffuser models.
Text to Speech TTS
BARK
1. Transformer usage
pipe = pipeline("text-to-speech", model="suno/bark")
ou = pipe(text)
Audio(output["audio"], rate=24_000)
2. This model could even sign:
song = "♪ Let us go, children of the fatherland, Our day of glory has arrived. Against us the bloody flag of tyranny is raised; the bloody flag is raised.♪ "
output = pipe(song)
3. Set voice
processor = AutoProcessor.from_pretrained("suno/bark")
model = BarkModel.from_pretrained("suno/bark")
inp =processor("Some text here", voice_present="my_voice_from_available_in_bark_speaking_library")
audio = model.generate(**inp)
Fastspeech from Tensorflow
1. Trained on LJSpeech dataset.
2. To install TF TTS $ pip install TensorFlowTTS it includes MelGan, Tacotron-2, FastSpeech, Parallel WaveGAN, HiFi-GAN.
2.1. I cannot >>> import tensorflow_tts.inference get error No module named 'german_transliterate' // $ pip install git+https://github.com/repodiac/german_transliterate.git#egg=german_transliterate
2.2. Then get error TypeError: Descriptors cannot be created directly.
2.2.1. Try $ pip install protobuf==3.20.* // doesn't help
2.2.2. Try to install TensorFlowTTS from git. Download and from folder run $ pip install . // doesn't help
2.2.3. Try to create new venf for tts // doesn't help
2.2.4. Try previous release of tts grom github // doesn't work
Audio classification
1 Actually you can put spectogram to Image classification ResNet to get a good prediction.
Genre recognition
Emotion Recognition (ER)
voice conversion (VC)
speech enhancement (SE)
Language identification LID
Voice activity detection VAD
Speaker Diarization
1. github.com/pyannote/pyannote-audio
speech translation (ST)