将节拍器与移动设备上的音频输入同步

Question

我正在开发一个针对移动设备的 Unity 项目，我需要将节拍器与麦克风传入的音频同步。目标是实时检测鼓的敲击声并将其与节拍器的滴答声对齐。以下是我当前 DrumInput 脚本的简化版本：

using UnityEngine;
using System.Collections;

public class DrumInput : MonoBehaviour
{
    private AudioClip microphoneClip; // Audio clip to capture microphone input
    private int sampleRate = 44100; // Sample rate for audio capture
    public float threshold = 0.1f; // Threshold to detect drum hits
    public int smoothingWindow = 10; // Window size for smoothing amplitude
    public float checkInterval = 0.01f; // Interval for checking audio input
    private float[] prevSamples; // Array to store previous samples for smoothing

    void Start()
    {
        StartMicrophone();
        prevSamples = new float[smoothingWindow];
        StartCoroutine(CheckAudio());
    }

    // Start capturing audio from the microphone
    void StartMicrophone()
    {
        string device = Microphone.devices[0];
        microphoneClip = Microphone.Start(device, true, 1, sampleRate);
    }

    // Coroutine to periodically check audio input
    IEnumerator CheckAudio()
    {
        while (true)
        {
            int micPosition = Microphone.GetPosition(null);
            float[] samples = new float[microphoneClip.samples * microphoneClip.channels];
            microphoneClip.GetData(samples, 0);
            ProcessAudio(samples, micPosition);
            yield return new WaitForSeconds(checkInterval);
        }
    }

    // Process audio samples to detect drum hits
    void ProcessAudio(float[] samples, int micPosition)
    {
        float smoothedAmplitude = 0.0f;
        for (int i = 0; i < micPosition; i++)
        {
            smoothedAmplitude = (smoothedAmplitude * (smoothingWindow - 1) + Mathf.Abs(samples[i])) / smoothingWindow;
            prevSamples[i % smoothingWindow] = Mathf.Abs(samples[i]);

            if (smoothedAmplitude > threshold)
            {
                double currentTime = AudioSettings.dspTime;
                double inputLatency = (micPosition / (double)sampleRate);
                double adjustedHitTime = currentTime - inputLatency;
                
                Debug.Log($"Drum hit detected at: {currentTime:F1}, Adjusted Hit Time: {adjustedHitTime:F1}, Amplitude: {smoothedAmplitude}, Threshold exceeded by: {smoothedAmplitude - threshold}, Input Latency: {inputLatency * 1000:F1}ms (Compensation Time: {inputLatency:F4}s)");

                // Handle drum hit detection
                break;
            }
        }
    }
}

该脚本捕获音频输入，对其进行处理以检测鼓击，并尝试补偿输入延迟。但是，我很难在检测到的鼓击声和节拍器滴答声之间实现精确同步。是否有人对如何更好地在 Unity 中将实时音频输入与节拍器同步（特别是对于移动设备）有经验或建议？任何有关提高延迟补偿的准确性或处理时序差异的提示将不胜感激。 谢谢！

该脚本捕获音频输入，对其进行处理以检测鼓击，并尝试补偿输入延迟。但是，我很难在检测到的鼓声和节拍器滴答声之间实现精确同步。

Answer 1

使用协程处理音频永远不会产生好的结果。 您可以做的是在任何具有 AudioSource 的 MonoBehavior 上使用 OnAudioFilterRead 回调。

您可以丢弃传入缓冲区，并将其用于传入音频处理，重要的是它将与音频线程（而不是 Unity 渲染线程）同步，并且将被同步。

请记住，您将不被允许接触该线程中的大部分 Unity API，并且您需要存储结果并读取它们（通过协程或其他方式）

https://docs.unity3d.com/ScriptReference/MonoBehaviour.OnAudioFilterRead.html