Over the last month or so we’ve been focused on the benefits brought to Project Dasher by combining 2D heatmaps with our traditionally 3D view of the world.
We started by creating 2D heatmaps, making them resizable, adding lots of them inside Dasher, and then making them pinnable per-level, before introducing the idea of overlaying building systems information, too. Most recently we took a sneak peek at the topic of today’s post, a new UI that simplifies access to these 2D (and 3D) heatmaps.
The idea came up during a team meeting: as we realised we had this combinatorial explosion between levels of a building and the sensor types available for surface shading – and that it was cumbersome to find the combination you wanted via multiple dropdowns – the suggestion was made to visualise these combinations graphically via some kind of stacked card UI. My friend and colleague Mike Lee had done some prototyping for a different project that used THREE.js to display a series of images as stacked cards: he kindly shared his code with me, which certainly made building out this UI much simpler.
Here’s a screenshot of Mike’s prototype:
I had to find a way to render heatmaps for each sensor type/level combination to an image that could then be navigated through as a stack. A few things are worth mentioning about this implementation:
- For navigating the stack I had to find and integrate the appropriate version of the OrbitControls component for THREE.js r71.
- I wanted to have the textures display building information, too.
- Two sets of textures might therefore be needed (if the user setting to display these layers gets toggled on).
- When I started this journey I was creating WebGLRenderer objects for each panel.
- Chrome only has 16 WebGL context’s available, so it’s really easy to hit this limit.
- I ended up refactoring Dasher to share a single WebGLRenderer for all our off-screen rendering.
- To support double-click (to open a heatmap) as well as to integrate tooltips, quite a lot of work was needed.
- The biggest challenge I hit related to transparent sections of the textures (or which we have quite a lot).
- Clicking on a transparent area inevitably brought up the wrong heatmap from a user’s perspective.
- I started down the path of raycasting into the scene.
- Once you get the intersection point of an individual texture, you should be able to look up into the source image.
- The intersection code in r71 doesn’t provide the UV of an intersection with a texture!
- I ported a more recent version of the code into Dasher, which was quite a lot of work.
- Everything was working properly, but I then decided to drop this approach…
- Next I implemented a reference image for the Types By Level panel.
- This is the approach we use for sensor dot hovering.
- We render a data texture with the level and type information for each pixel.
- Rather than raycasting on each mouse movement, we can do a simple X-Y pixel lookup.
- We just need to render the texture on zoom, orbit or resize of the panel.
I’m very happy with how the feature has turned out, but it certainly wasn’t an easy one to implement.
Here’s how the overall navigation – which includes zoom and pan, as well as on-hover tooltips – works.
This panel appears (assuming it’s enabled in the user settings), when you launch the Surface Shading feature.
When you double-click the sensor type/level combination that interests you, the level will be navigated to in the 3D view, and the 2D heatmap for that combination will be displayed/pinned.
You can double-click a number of different options – to display them at the same time – and then select the final 3D heatmap to display by double-clicking the 2D heatmap that’s of interest. So this creates a simple (and hopefully compelling) way to choose between the various options available.
It also works for “level-less” models, although it’s at this point only really compelling if you have a certain number of sensor types available for surface shading:
I think this UI has a lot of potential for simplifying navigation and simultaneous display of various heatmaps, which is of great interest when correlating multiple data streams.
I’ll be talking about this more during my upcoming AU 2020 class (entitled “Lessons from Project Dasher: Building a Digital Twin Using Forge”), in case you’re interested in learning more about it.
