Adding spatial sound to our Unity model in HoloLens Part 1

As mentioned last time, upgrading to Windows 10 has opened the door to spatial sound in our HoloLens model.

The foundational step is to set our Unity project’s audio settings to have the Spatializer Plugin as the “MS HRTF Spatializer”: if you don’t see this option then you may need to upgrade your OS (as I did).

At this point we need to think a little about how best to implement spatial sound in this project, particularly as it literally contains a number of moving parts. I see three main alternatives:

A single sound is assigned to our robot
- When the robot stops completely, so does the sound
The same sound is assigned to each of the robot’s parts
- When each part stops moving, so does the sound for that part
A different sound is assigned to each of the robot’s parts
- When each part stops moving, so does the sound for that part

In some ways option 1 is simplest – we only need to attach a single sound to the root of the robot – but in others it’s the most complex, as we have to keep track of which parts are moving. But we’ll start with that and see how it goes: I have a sneaking suspicion that adding a sound for each part (i.e. options 2 & 3) add unnecessary (and unwelcome) processing overhead for the device.

The first order of business is to add the chosen sound as an AudioSource to our root object (by dragging and dropping the project asset) and set its properties according to the Holograms 101 tutorial:

I’ve added a Buzz.cs script to manage the “buzz” of the robot, based on the movements of the various parts:

using UnityEngine;
using System;
 
public class Buzz : MonoBehaviour
{
 
  public int parts = 6;
 
  private int stopped = 0;
  private int max = 0;
  private AudioSource audioSource;
 
  void Start()
  {
    // Calculate the sum of all possible part flags
 
    max = 0;
    for (int i = 0; i < parts; i++)
    {
      max += (int)Math.Pow(2, i);
    }
 
    // Store the audio source for later access
 
    audioSource = this.gameObject.GetComponent<AudioSource>();
  }
 
  public void StartPart(int part)
  {
    // If our part is on the stopped list, remove it
 
    int flag = (int)Math.Pow(2, part);
    if ((stopped & flag) > 0)
    {
      stopped -= flag;
    }
 
    // Start the audio if it isn't already playing
 
    if (audioSource && !audioSource.isPlaying)
    {
      audioSource.Play();
    }
  }
 
  public void StopPart(int part)
  {
    // If our part isn't on the stopped list, add it
 
    int flag = (int)Math.Pow(2, part);
    if ((stopped & flag) == 0)
    {
      stopped += flag;
    }
 
    // If all parts are stopped, stop the audio
 
    if (audioSource && (stopped == max))
    {
      audioSource.Stop();
    }
  }
}
 

We do this by having a “number of parts” setting that tells us how to populate a “powers of 2” value (much as you might do with enumeration flags you need to keep track of). So if the value is 1, we know that part 0 is stopped, while if it’s 23, we know parts 0, 1, 2 & 4 are stopped (23 = 1 + 2 + 4 + 16 = 2⁰ + 2¹ + 2² + 2⁴). We can pre-compute the maximum value: all we need to do to check whether all parts are stopped is to see whether the stopped value has reached this maximum. At which point we stop the sound.

Then we need to update our Rotate.cs script to allow for a unique number assigned to each part – something we do in the Unity UI – as well as providing a simpler Start/Reverse/Stop/TogglePart interface:

using UnityEngine;
 
public class Rotate : MonoBehaviour
{
  // Parameters provided by Unity that will vary per object
 
  public int partNumber = 0;          // Part number to help identify when all are stopped
  public float speed = 50f;           // Speed of the rotation
  public Vector3 axis = Vector3.up;   // Axis of rotation
  public float maxRot = 170f;         // Minimum angle of rotation (to contstrain movement)
  public float minRot = -170f;        // Maximim angle of rotation (if == min then unconstrained)
  public bool isFast = false;         // Flag to allow speed-up on selection
  public bool isStopped = false;      // Flag to allow stopping
 
  // Internal variable to track overall rotation (if constrained)
 
  private float rot = 0f;
  private Buzz buzz;
 
  void Start()
  {
    buzz = this.gameObject.GetComponentInParent<Buzz>();
  }
 
  public void StartPart()
  {
    isStopped = false;
    if (buzz)
      buzz.StartPart(partNumber);
  }
 
  public void ReversePart()
  {
    speed = -speed;
  }
 
  public void StopPart()
  {
    isStopped = true;
    if (buzz)
      buzz.StopPart(partNumber);
  }
 
  public void TogglePart()
  {
    isStopped = !isStopped;
    if (isStopped)
    {
      buzz.StopPart(partNumber);
    }
    else
    {
      ReversePart();
      buzz.StartPart(partNumber);
    }
  }
 
  void Update()
  {
    if (isStopped)
      return;
 
    // Calculate the rotation amount as speed x time
    // (may get reduced to a smaller amount if near the angle limits)
 
    var locRot = speed * Time.deltaTime * (isFast ? 2f : 1f);
 
    // If we're constraining movement (via min & max angles)...
 
    if (minRot != maxRot)
    {
      // Then track the overall rotation
 
      if (locRot + rot < minRot)
      {
        // Don't go below the minimum angle
 
        locRot = minRot - rot;
      }
      else if (locRot + rot > maxRot)
      {
        // Don't go above the maximum angle
 
        locRot = maxRot - rot;
      }
 
      rot += locRot;
 
      // And reverse the direction if we're at a limit
 
      if (rot <= minRot || rot >= maxRot)
      {
        speed = -speed;
      }
    }
 
    // Perform the rotation itself
 
    transform.Rotate(axis, locRot);
  }
}
 

The PartCommands.cs script then needs to be updated to call through to this new interface, rather than modifying the part directly. Which actually makes it a bit simpler:

using UnityEngine;
using System;
 
public class PartCommands : MonoBehaviour
{
  // Called by GazeGestureManager when the user performs a Select gesture
 
  public void OnSelect()
  {
    CallOnParent(r => r.TogglePart());
  }
 
  public void OnStart()
  {
    CallOnParent(r => r.StartPart());
  }
 
  public void OnStop()
  {
    CallOnParent(r => r.StopPart());
  }
 
  public void OnQuick()
  {
    CallOnParent(r => r.isFast = true);
  }
 
  public void OnSlow()
  {
    CallOnParent(r => r.isFast = false);
  }
 
  public void OnReverse()
  {
    CallOnParent(r => r.ReversePart());
  }
 
  private void CallOnParent(Action<Rotate> f)
  {
    var rot = this.gameObject.GetComponentInParent<Rotate>();
    if (rot)
    {
      f(rot);
    }
  }
}
 

Now let’s take a look at this approach in action. This video shows the robot buzzing fairly discreetly – you only really notice that it’s not background noise when the movement stops completely. You don’t really get a sense for the spatial nature of the sound: if you turn your head away from the hologram, you definitely hear the sound behind you. Which is both really cool and apparently an important part of designing a compelling holographic experience.