My story of walktrough Unit 4 Audio Course hands on exercise from Hugging Face.
HuggingFace Audio course Hands-on UNIT4 certification completion
1. The task is to fine-tune some music genre classifiction model to 87% accuracy with a marsyas/gtzan dataset
1.1. Accuracy means all correct prediction of all classes divided on all predictions.
1.2. The main problem is the gtzan dataset is extremely small.
2. Choose model.
2.1. Find modern Leaderboard with best nowadays models available.
2.1.1. Where to find? // superbbenchmark.org/leaderboard, huggingface.co/spaces/hf-audio/open_asr_leaderboard, paperswithcode.com/sota/audio-classification-on-audioset
2.1.2. Most popular models trained on unsupervised data: Wav2Vec2, Wav2Vec2-BERT (trained on 4.5M hours), HuBERT, XLSR-Wav2Vec2, DiscreteBERT, SlimIPL, IPL, Noisy Student, DeCoAR 2.0, S2S, M-CTC-T (1B)
2.1.2.1. The biggest model is Hubert X-LARGE 1B params trained on 60k hours Libry light dataset. // But it is too large, I prefer to start with small one.
2.1.2.2. I start with ntu-spml/distilhubert model it is small version of Hubert-base "facebook/hubert-base-ls960" for my music classification task.
2.1.2.3. Large hubert gave results even worse then based hubert (only on small period of training like 8 epochs). Probably huge models require more training then small one (like 15 epochs maybe).
3.1. facebook/wav2vec2-base 94M
3.2. "facebook/wav2vec2-xls-r-300m" // Doesn't shows a good result even on 3 hours of training.
3.3. microsoft/wavlm-base 93M parameters is like Hubert Base. DistilHubert 23M still the best choice for blitz training. I have to look for other alternatives same size models close to 23M but there are no alternatives.
4. Small 23M model is gaining accuracy very fast. In 250 steps it get 70% accuracy but in recent 250 I achieved only 80%. Maybe those huge models would show best results on a big distance. After 5 epoch accuracy gained very poorly. On distlBert I get 85% accuracy on 6 epoch with training_batch 2, gradient_acc 1, leraning rate 5e-5, f16 false, split 0.1, sound length = 30s.
4.1. I'll try to increase training dataset (split=0.08) along with evaluation batch size = 5 and train on 7 epochs. // Best split is 0.9 and for optimal productivity batch size should be product of two (2,4,8,16 etc) bcs of GPU architecture.
4.2. During training, pay attention to Training loss value, while it is far from zero your model still has space to improve accuracy. If training loss is like 0.08 it is done.
3. During my first training I got a problem with accuracy metric evaluation. It simply doesn't change from epoch to epoch.
3.1. Increase per_device_eval_batch_size and maybe initial value of AdamW's learning rate:
training_args = TrainingArguments(
per_device_eval_batch_size=32,
learning_rate=5e-5,
) // It is better to use small batch like 4
3.1.1. Try to change eval_strategy for evaluation_strategy
TrainingArguments( eval_strategy="steps" ) // Helped
3.1.2. Actually zero accuracy could be at the beggining of the training if you still have a huge validation loss like 2.34 on batch size 32.
3.1.3. If you see that with each training step vaildation loss is increasing? probably you forget to shuffle test dataset while train_test_split.
3.1.4. But when I shuffle dataset gztan I loss 100 items somehow. Is should be 999.
3.1.4.1. Try to shuffle dataset before split shuffled_dataset = sorted_dataset.shuffle(seed=33) // helped
Improving accuracy
3.2. Try to increase lenght of truncated sound example in feature extractor. // inp=feature_extractor(aud_arr, sampling_rate=feature_extractor.sampling_rate, max_length=32000, truncation=True) // Increasing sample length more then 15 seconds doesn't show any improvement in accuracy but made training significantly slower.
3.2.1. Previous method with increasing increasing of audio lenght is definetly helpful despite small size of dataset.
3.2.2. Decreasing per_device_train_batch_size bring good results. Probably for small dataset batch decreasing is a good idea bcs weights are changed after whole batch has been done. Batch should be 2,4,8,16 etc 2**N because of GPU architecture. But decreasing per_device_eval_batch_size lead to increased variation of accuracy. Maybe accuracy seemingly to be improved because of imporoving metric's fluctuation.
training_args = TrainingArguments(
per_device_train_batch_size=4, #32,
per_device_eval_batch_size=4, #32,
)
3.3. Try to increase training dataset by changing train/test size proportion from 0.2 to 0.1. minds2 = minds.train_test_split(seed=33, shuffle=True, test_size=0.1) // dectreasing split to 0.05 made things worse probably on smal dataset 1000 items 50 training items are not enough or it should be 0.1, 0.2, 0.3 because we have 10 labels.
3.4. Decrease warm-up // Made accuracy worse and increasing warm-up to 0.2 also made it worse. So 0.1 is seems the best options.
3.5. Remove quantization f16=False // Helps
3.6. Try TrainingArguments(bf16=True)
3.7. Try different optimizers. In Training_arguments param optim. What options are available:
"adamw_hf" # Seemingly better then ordinar adamw
"adamw_torch" # Default option and good enough
"adamw_torch_fused"
"adamw_torch_xla"
"adamw_torch_npu_fused"
"adamw_apex_fused"
"adafactor" # It actually looks faster then adamW and allows hit 88% on 9 epoch during 2 hours. But this good result doesn't reproduce yourself, probably it is good idea to use seed in TrainingArgs.
"adamw_anyprecision"
"adamw_torch_4bit"
"ademamix" # doesn't work
"sgd" # doesn't works for me
"adagrad" # very bad optimizer
"adamw_bnb_8bit"
"adamw_8bit" # just an alias for adamw_bnb_8bit
"ademamix_8bit"
"lion_8bit"
"lion_32bit" # I have to install something additional
"paged_adamw_32bit" # I have to install something additional
"paged_adamw_8bit"
"paged_ademamix_32bit"
"paged_ademamix_8bit"
"paged_lion_32bit"
"paged_lion_8bit"
"rmsprop" # Fast and good for small epoch but on long run (more then 5 epoch) it is less accurate.
"rmsprop_bnb"
"rmsprop_bnb_8bit"
"rmsprop_bnb_32bit"
"galore_adamw" # I have to install something additional
"galore_adamw_8bit"
"galore_adafactor"
"galore_adamw_layerwise"
"galore_adamw_8bit_layerwise"
"galore_adafactor_layerwise"
"lomo" # I have to install something additional
"adalomo"
"grokadamw"
"schedule_free_adamw"
"schedule_free_sgd"
3.7.1. Try to decrease beta. adam_beta1 and adam_beta2 parameters in TrainingArguments. // Decreasing beta1 to 0.8 doesn't helped (81%) but decrease beta2=0.9 not so bad 85%. Decreasing betas makes learning rate decrese slower.
Decreasing beta1=0.7 and beta2=0.8 achived 87% accuracy on 6 epoch! I had distliHubert model, adamW_torch, train_batch=2, eval_batch=4, fp16=False, split=0.1, lr=5e-5, logging_steps=100, grad_acc_steps=1, warmup_ratio=0.1, dataset 30 seconds.
But increasing beta1=0.95 also brought surprisingly good results. Accuracy improved sustainable and very fast. I've reached 86% on 6 epoch and my free limit expired.
3.7.2. Do not normalize dataset. // Doesn't help. Reach 80% on 8 epochs.
3.7.3. Try regularization L1, L2. // actually L2 is a weight_decay parameter. Set it to 1e-4 (get overfitting very quick). Setting to 1.0 also just made things worse.
3.7.4. Change adam_epsilon. If you increse epsilon weights will be updated on smaller value. // adam_epsilon=1e-07 brings good results 86% but it is unstable.
3.8. If you are using base model try to find large one for example facebook/hubert-large-ll60k or facebook/hubert-xlarge-ll60k // big models are too slow, on 8 epochs I could reach only 70% accuracy
3.9. Increasing logging_steps doesn't affect accuracy but significantly increase speed. // It actually could help to catch good result if you run evaluations often.
3.10 Increase padding max_length for feature_extractor like inputs = feature_extractor(max_length=16_000*15) // Good idea
4. Incrementation of gradient_accumulation_steps seems to be decrease accuracy. // I set it to 1
5. Learning rate decrease from 5e-5 to 1e-5, 4e-5 makes things worse. Increasing Lr also doesn't help.
6. Maybe I have to wait two days to collect time in Google Colab, then run training on 50 evalutaion logging steps. // I've reached 84% on 9 epoch ith 3 hours of training. While 84 was reached accuray just stoped to improve training_loss is close to zero and training dataset was overfitted.
6.1. Basically the more often you do evaluation the better you chance to catch good result. So 50 eval_steps better then 100 but slower.
7. I believe it is hard af to achieve 87% on 23M param model and that small dataset on stable basis but its possible sometimes. And its probably depends of seed. You definetely need a bigger model.
7.1. I will try to increase training dataset to 0.95% and train it for 3 hours with eval batch 2. If it doesn't help I'll change my disthubert model for regular hubert and will train it as much as possible. // On that small test dataset it actually reach 78% on 9 epochs and stops to improve. So it is not an option. Increasing test set to 20% also doesn't imrove result, model fast reach 80% on 7 epoch and overfited.
7.2. Train Wav2vec2 base. // For 5 hours of training and 10 epochs I've reached only 85% but the big model improved accuracy more sustainable and seemingly had potential to increase and it was not that big differencs between evaluation and training loss. My mistake was usage of a small batch. On a big model you could increase batch to 8 and sample length decrease to 15s it made training fast.
7.3. 25 evaluation steps are optimal (evaluation batch = 4).
8. For big model try to increase batch, use quantization or decrease audio length or use LoRA.
8.1. On Wav2Vec2 base 93M you can actually achieve 88% on stable basis on 12 epochs with batch=4 and dataset 15 seconds, default AdamW optimizer, f16=False, gradient_accumulation=1.
9. Change lr_scheduler_type:
“linear” = get_linear_schedule_with_warmup # default
“cosine” = get_cosine_schedule_with_warmup // 83%
“cosine_with_restarts” = get_cosine_with_hard_restarts_schedule_with_warmup // 81%
“polynomial” = get_polynomial_decay_schedule_with_warmup // 80%
“constant” = get_constant_schedule
“constant_with_warmup” = get_constant_schedule_with_warmup
“inverse_sqrt” = get_inverse_sqrt_schedule
“reduce_lr_on_plateau” = get_reduce_on_plateau_schedule
“cosine_with_min_lr” = get_cosine_with_min_lr_schedule_with_warmup
“warmup_stable_decay” = get_wsd_schedule
10. Augmenting audio dataset somehow. Maybe made faster or slower, or without normalization. Or apply some sound effects.
11. Try LoRa optimization technique. From PEFT.