DEEPFAKE AUDIO DETECTION USING YOLOV8 WITH MEL-SPECTROGRAM ANALYSIS: A CROSS-DATASET EVALUATION

Abstract

Context. The problem of detecting deepfake audio has become increasingly critical with the rapid advancement of voice synthesis technologies and their potential for misuse. Traditional audio processing methods face significant challenges in distinguishing sophisticated deepfakes, particularly when tested across different types of audio manipulations and datasets. The object of study is
developing a deepfake audio detection model that leverages mel-spectrograms as input to computer vision techniques, focusing on improving cross-dataset generalization capabilities.
Objective. The goal of the work is to improve the generalization capabilities of deepfake audio detection models by employing
mel-spectrograms and leveraging computer vision techniques. This is achieved by adapting YOLOv8, a state-of-the-art object detection model, for audio analysis and investigating the effectiveness of different mel-spectrogram representations across diverse datasets.
Method. A novel approach is proposed using YOLOv8 for deepfake audio detection through the analysis of two types of melspectrograms:
traditional and concatenated representations formed from SincConv filters. The method transforms audio signals into visual representations that can be processed by computer vision algorithms, enabling the detection of subtle patterns indicative of
synthetic speech. The proposed approach includes several key components: BCE loss optimization for binary classification, SGD with momentum (0.937) for efficient training, and comprehensive data augmentation techniques including random flips, translations, and HSV color augmentations. The SincConv filters cover a frequency range from 0 Hz to 8000 Hz, with a step size of approximately
533.33 Hz per filter, providing detailed frequency analysis capabilities. The effectiveness is evaluated using the EER metric across multiple datasets: ASVspoof 2021 LA (25,380 genuine and 121,461 spoofed utterances) for training, and ASVspoof 2021 DF,
Fake-or-Real (111,000 real and 87,000 synthetic utterances), In-the-Wild (17.2 hours fake, 20.7 hours real), and WaveFake (117,985
fake files) datasets for testing cross-dataset generalization.
Results. The experiments demonstrate varying effectiveness of different mel-spectrogram representations across datasets. Concatenated
mel-spectrograms showed superior performance on diverse, real-world datasets (In-the-Wild: 34.55% EER, Fake-or-Real:
35.3% EER), while simple mel-spectrograms performed better on more homogeneous datasets (ASVspoof DF: 28.99% EER, Wave-
Fake: 34.55% EER). Feature map visualizations reveal that the model’s attention patterns differ significantly between input types, with concatenated spectrograms showing more distributed focus across relevant regions for complex datasets. The training process, conducted over 50 epochs with a learning rate of 0.01 and warm-up strategy, demonstrated stable convergence and consistent performance
across multiple runs.
Conclusions. The experimental results confirm the viability of using YOLOv8 for deepfake audio detection and demonstrate that
the effectiveness of mel-spectrogram representations depends significantly on dataset characteristics. The findings suggest that input
representation should be selected based on the specific properties of the target audio data, with concatenated spectrograms being
more suitable for diverse, real-world scenarios and simple spectrograms for more controlled, homogeneous datasets. The study provides
a foundation for future research in adaptive representation selection and model optimization for deepfake audio detection.