Integrating Hyperion sprites into Dasher

As mentioned on Monday, I’ve been having lots of fun getting Hyperion to work inside Dasher, this week. All good preparation for the AU class I hope will be accepted for this year’s conference.

Right now I have sprites working, which is an important first step (and doesn’t actually involve any of the more gnarly, data-related issues I expect to have when w tackle surface shading/heatmaps). In this post we’re going to step through the main steps taken to make this happen, but also to describe some of the important changes that were needed to make it all work well.

Dasher’s codebase contains primarily TypeScript, which means that it’s ideal – for API discoverability and for increased code quality – to have type definitions for any new platform capabilities. Inside Dasher we had our own set of types for both the Forge viewer and THREE.js. When I asked about the Data Visualization types, I was pointed to this NPM package which is built from this Definitely Typed repo. I’d been aware of this work – and had even supported it with some code – in the past, but hadn’t quite gotten around to integrating it into Dasher. This was the push that I needed.

The types themselves were well structured and reasonably complete, but Dasher makes really heavy use of the Forge viewer and THREE.js, so there were still several gaps in the current implementation. I ended up forking the repo and submitting my changes as a pull request. Many thanks to Jan Liska for helping me through that process promptly. I’m looking forward to integrating the next version of the @types/forge-viewer package into Dasher!

So now we have functioning DataViz types inside Dasher, and have scratched the longstanding itch caused by having non-standard type definitions in the codebase. All goodness. Here’s the section of these DataVisualization namespace that’s relevant to the use of sprites:

  namespace DataVisualization {
    namespace Core {
      const MOUSE_CLICK = 'DATAVIZ_OBJECT_CLICK';
      const MOUSE_CLICK_OUT = 'DATAVIZ_CLICK_OUT';
      const MOUSE_HOVERING = 'DATAVIZ_OBJECT_HOVERING';

      enum ViewableType {
        SPRITE = 1,
        GEOMETRY = 2
      }

      class CustomViewable {
        constructor(position: THREE.Vector3, style: ViewableStyle, dbId: number);

        get dbId(): number;
        get position(): THREE.Vector3;
        get style(): ViewableStyle;
      }

      class SpriteViewable extends CustomViewable {
        get color(): THREE.Color;
        get highlightedColor(): THREE.Color;
        get type(): ViewableType;
      }

      class ViewableData {
        spriteSize: number;

        constructor(options?: { atlasWidth: number, atlasHeight: number });

        get spriteAtlas(): any;
        get viewables(): CustomViewable[];

        addViewable(viewable: CustomViewable): void;
        getViewableColor(dbId: string, highlighted: boolean): THREE.Color;
        getViewableUV(dbId: string, highlighted: boolean): object;
        finish(): Promise<void>;
      }

      class ViewableStyle {
        color: THREE.Color;
        highlightedColor: THREE.Color;
        highlightedUrl: string;
        spriteUrls: Set<string>;
        type: ViewableType;
        url: string;

        constructor(type?: ViewableType, color?: THREE.Color, url?: string, highlightedColor?: THREE.Color, highlightedUrl?: string, animatedUrls?: string[]);

        get preloadedSprites(): string[];

        preloadSprite(spriteUrl: string): void;
      }
    }
  }

Now for what was needed to integrate sprites into Dasher. This section of the Hyperion documentation was very much my friend as I worked on this, incidentally.

Inside Dasher we have a NodeManager class that can be used to display “sensor dots” or a variety of other sprites. This is part of the package that we shared with the Hyperion team when they first started building out their feature-set.

Way back when – when we first started rethinking Dasher using Forge – we used SVG elements in the DOM to represent sensors. This worked fine for a handful (or even two) of sensors, but over time it made more sense to swap this mechanism out for a stylized point cloud (which is the way Hyperion works, too). We evolved the implementation to perform hit-testing – to find out when you’re hovering over sensors – via a reference image rather than raycasting. Simon Breslav and I have both talked about this at past Autodesk Universities, in case you’re interested.

Anyway, the NodeManager has been a very helpful abstraction inside Dasher: we’ve used it to display sensors, equipment and even building occupants. Each of these had their own point cloud, but worked well together for hovering/tooltips/etc.

Hyperion currently maintains a global list of viewables, which means that the last feature that comes in and initializes their viewables wins. This is something that is likely to change, but for now it’s possible to make it work by having each NodeManager contribute viewables to a static list and pass this to Hyperion when needed. A little gymnastics were needed to make this happen, but it works well enough.

Here’s the rough approach needed to add sprites to your Forge application:

Load the Data Visualization Extension
Create one or more ViewableStyle objects for your various icons

Colour can be changed later, so you can use the same style for different colours (and probably, although this is untested by me, icons)

Create a ViewableData object that will act as a collection of your viewables
Determine a globally unique (across your various viewables) dbID

We use ranges for sensors, equipment and occupants as 1000-1999, 2000-2999 & 3000-3999 respectively

Create a SpriteViewable with the appropriate style and a unique dbId

You can set the position, but this can also be a placeholder that gets set later

Call ViewableData.addViewable() to add your newly created viewable to the list
Repeat steps 5 and 6 until you’ve added them all
Call ViewableData.finish() and be sure to wait for this to finish (haha) before continuing
Call addViewables() on the extension object to add your ViewableData to the scene
Optionally call showHideViewables() on the extension to globally turn off the viewable display or have them occluded by the scene geometry

You can then manipulate the properties of these sprites during the functioning of your application by calling invalidateViewables() on the extension, passing in the dbIds of the sprites to change along with one or more of these properties:

color

We change the colour of nodes when they’re hovered over or selected

position

We move occupants around by changing their position

scale

We can turn individual sprites on or off by setting their scale to 1 or 0

We haven’t yet needed to change a sprite’s icon, but this is presumably how you’d do it

highlighted

We don’t use the standard highlighting in Dasher, so haven’t used this

We attach handlers to watch for MOUSE_HOVERING and MOUSE_CLICK events, so that we can show tooltips and plots respectively (using our standard mechanism) and also change the colour of the sprite. The default highlighting behaviour of the selection event is for it to change the scale of the sprite to be 1.5. In Dasher we want use colour rather than size to denote selection – which we do using invalidateViewables(), as described above – so we turn off the default scale behaviour by calling event.hasStopped in the callback.

Here’s an animation of the hovering, which I’m reasonably happy with.

Bear in mind that while implementing sprites I tweaked the display to make the green and orange colours pop a little more, so this is really an apples to oranges comparison (haha) with the old design.

Of course it took some incremental steps (and even mis-steps) to get to this point, as usual. Here’s one of the more interesting ones – a bug or a feature?

A big test of this implementation was the classic occupancy demo we developed a few years ago with Schneider Electric to display the data captured using their (then) new sensors called MIRTIC (which presumably standards for MIcro Retinal Thermal Infrared Camera?). This is still our best test of dynamic sprites inside Dasher (aside from skeleton animation, I suppose), and the Hyperion implementation worked wonderfully (the heatmaps are still based on our existing Dasher implementation, of course):

The sharp-eyed among you will notice that occupant display is now also respecting (for better or worse) the global occlusion setting. Which is actually pretty cool.

So far I’m very happy with the progress we’ve been able to make with Hyperion. Sprites was always going to be the “low hanging fruit” of the integration project, so we’ll see how things progress as we start to focus on surface shading.