26 June 2017

Building a floating HoloLens 'info screen' - 1

Intro

Those who have seen my HoloLens apps (most notably Walk the World) have noticed I tend to use floating "info screens", especially for help screens. My apps are mostly voice command driven as I don't like to have floating controls that are in view all of the time. They stress the fact that you are in an virtual environment, and that degrades the actual immersive experience, IMHO. So I go as much for gestures and voice as possible.

But where there are no visual clues for functionality, there's also lack of discoverability. So I tend to include a voice command "help" or "show help" that brings up a simple floating screen that shows what the app can do.

A few important things that you might not see right away:

  • The screen follows your gaze
  • The screen tries to move away from you to be readable, but will stop moving if it get's pushed against an obstacle. So it won't disappear into another hologram or a physical object, like the wall or a floor. Or at least it tries to. I must admit it does not always works perfectly.
  • Notice that at first it will appears like 1 meter before you and move into view, next time it will appears where you last left it and then move into view.

In a two-part post I will describe how I have created such a screen.

  • The first part will handle building the actual visual structure in Unity (and one little behaviour)
  • The second part describes the code for all other Unity behaviours.

I am going to assume you know a bit about Unity but not too much, so there's going to be lot of images.

Setting up a base project

imageThat's easy, as I described that in my previous blog post. Just make sure you download a HoloToolkit from June 16, 2017, or later. This includes this pull request by yours truly that we will need in this app. And while you are importing stuff, also import LeanTween from the Unity Asset Store (hit CTRL-9 to open it immediately without having to hunt it down in the menu). When doing so, make sure you deselect everything but the Plugisn checkbox.

The basic setup of an info screen

My info screens basically exist out of three simple components:

So let's build those!

Background plane

Inside the HologramCollection that we inherited from my previous post we will first make an empty game object that I called "HelpHolder" as this will be a help screen, but you can call it anything you like. To make designing a little easier, set it's Z position to 1, else it will be sitting over the camera, which is always on 0,0,0 in a HoloLens app. That kind of obscures the view.

image

Inside that HelpHolder we first make a common Plane. This gives the standard, way too big 10x10m horizontal square. Change it's rotation to 270 and change X and Z scale to 0.07m (changing the Y scale makes no sense as a Plane essentially has no Y dimension).

image

Double click the Plane in the HelpHolder - this will make your scene zoom in. Now use that hand button and left top the scene screen and the CTRL key to rotate around to you get to see the white side of the Plane (the other side is translucent). Notice the HoloLens cursor ;)

image

Now a white background for a text doesn't look good for me, I find it too bright. So we are going to make a material to make it look better.

To keep things organized, we first create an "App" folder in "Assets", and within that a "Materials" folder. In that Materials folder we create a HelpScreenMaterial Material

image

Setting some color and reflection

Now over on the right side:

  • Set "Shader" to "HoloToolkit/StandardFast"
  • Set "Rendering Mode" to "Transparent"
  • Set "Color" to a background color you fancy, I took a kind of sky blue (#0080FFFF)
  • Move the "Smoothness" slider all the way to the left - we don't want any reflections or stuff from this 'screen'

image

Now you only have to drag the material on your plane and it will turn blueish.

image

Rather standard Unity stuff this, but I thought it nice to point it out for beginners.

Changing the collider

A Collider is a component that determines how a game object collides with other objects. When you use primitive objects, those are of the same type as actual shape you use, although the names are not always the same. So a Sphere typically has a Sphere Collider, but a Cube has a Box Collider. There is no Plane collider, as a Plane is a generic Mesh - so it uses a Mesh Collider. And here we run into an issue, because a Plane typically has one side and it looks the Mesh Collider has that as well - and if not, it does not prevent the help window from getting pushed though the floor or a wall. As I found out  making this demo :D.

So select the Plane, hit the Add Component button at the bottom and add a Box Collider.

Then

  • Unselect the checkmark next to "Mesh Collider". This will disable the old Mesh Collider. A game object may have only one active Collider so we want to get rid of this. You can also delete it if you want using the dropdown that appears if you click the gear icon all the way to the right.
  • Put "0.02" in the Y filed in the "Size" Section. This will make the Collider as big as the blue plane, and 2 cm thick.

imageWhat may seem confusing is that the Collider claims it's 10 x10 meters in size. That is true, but it is also scaled to 0.07 in X direction, and 0.050535134 in Z direction. If you remember the default size of a Plane is 10x10, this makes the screen about 70 cm wide and 50 cm height, which looks like the size you saw on the video. A Plane has no thickness, so if the scale of Y is set to 1, the colliders width will be the actual size in the Y field.










imageIf you look at the screen on edge, you can see the green indicator lines showing the outline of the Collider:





Adding text

Find the 3DTextPrefab and drag it onto the HelpHolder:

image

It should end up under the Plane. Zoom a little in on the plane to see the text clearly.

Now change the text into the help text you want (I took some Lorum Ipsum) and change some of the text settings:

image

  • Change Y to 0.133 (this will move the text towards the top of the 'screen', making room for the button later on).
  • Change Z to -0.02 (this will move the text to 2cm before the 'screen', this will prevent rendering issues later on
  • Change "Alignment" to left

I wish to stress the Y value hugely depends on the size of your text, the the font size, and the size of your  'help screen'. To get it right, it requires some fiddling around (as we will see later on).

Building the button - part 1

imageRight click the HelpHolder and add a Sphere. This will - like almost everything is initially - way too large. To change it's scale to 0.08 in all three dimensions. Then change it's Z-value to -0.045 (this will put the button in front of the 'screen' and also change the Z value to -0.01

This results in the following and now you can see where the fiddling starts, because that screen is too big for the text and the button it not quite where we want it

image

Some in-between fiddling around

imageWith these two buttons you can very easily move (left) or resize (right) objects. Select the Plane, then select the desired function.


imageimage

By dragging the blue block in the left image you can change the screen size in vertical direction, the red block will do so in horizontal. With the yellow arrow (right image) you can move the plane upward until it is where you like it.

In my Inspector pane on the right it said this when I was done:

image

But... now all of or stuff is quite off center as far as the HelpHolder, the root object, is concerned. It's center point is pretty low on our screen, which means the screen it too high.

image

This can be fixed by selecting all three denizens of HelpHolder (using the CTRL button), select the Move button again, grab the yellow arrow and move the whole combination downward until the read arrow is more or less on the horizon.

image

It does not have to be a super precise hit, as long as it's not so much off center as it first was.

Building the button - part 2

A white sphere is not a button, so add some that makes sure you can click it. I think a real designer might have to say something about it, but I have found that a red texture with a large OK text on it works - in the sense that I never had to explain to anyone that it's something you can air tap and that will act like something of a button. So I created this awesome :D picture, created a "Texture" folder under app and put it there

imageimage

It's odd shape will become clear soon.

First, create a "DoneMaterial" in Materials. Then:

  • Set "Shader" once again to "HoloToolkit/StandardFast"
  • Set "Rending Mode" to "Fade" (not Transparent)
  • Select the "Textures" folder in the App/Assets folder, and drag the "Done" texture on top of the little square left of the "Albedo" text
  • Change the X value of "Emissions/Tiling" to 3. This will morph three images on the Sphere, repeating them horizontally.

image

If you have done everything correct, you will see the material looks like above.

Now drag the DoneMaterial from the Materials folder onto the Sphere in the Hierarchy

image

And the button on the screen looks already familiar :). I left the default Shader settings as it's a bit smooth, so it looks like it reflects a little light adding to it's 3D appearance.

Turn off shadows

This is a thing trick I learned from Dennis Vroegop, long time Microsoft MVP, my unofficial MVP 'mentor' who learned me to deal with suddenly being a Microsoft MVP too way back in 201,1 and long time expert on Natural User Interface. The trick is this: unless your app really really uses them for some reasons, turn off "receive shadows" and "cast shadows" for the renderers of all three objects. As you can see the actual real light sources in your room through HoloLens, shadows will appear on the wrong places anyway and only give cause for confusion at best - at worse they will 'break the spell'. As a bonus, the Unity engine won't need to calculate the shadows so you will save some performance as well.

So select all three objects (use the CTRL key for that like in any other normal Windows program) and turn this off in one go:

image

A little code for dessert

This is part of my HoloToolkitExtensions libary, that I one day will publish in full, but, well, time. It is called HorizontalSpinner, and basically is the grandchild of the SimpleRotator that briefly passed by (without explaination) in my post about a generic toggle component for HoloLens. It uses LeanTween and looks like this:

using UnityEngine;

namespace HoloToolkitExtensions.Animation
{
    public class HorizontalSpinner: MonoBehaviour
    {
        public float SpinTime = 2.5f;

        public bool AbsoluteAxis = false;

        private void Start()
        {
            if (AbsoluteAxis)
            {
                LeanTween.rotateAround(gameObject, Vector3.up, 360 / 3.0f, 
                   SpinTime / 3.0f).setLoopClamp();
            }
            else
            {
                var rotation = Quaternion.AngleAxis(360f / 3.0f, Vector3.up).eulerAngles;
                LeanTween.rotateLocal(gameObject, rotation, SpinTime / 3.0f).setLoopClamp();
            }
        }

        private void OnEnable()
        {
            LeanTween.resume(gameObject);
        }

        private void OnDisable()
        {
            LeanTween.pause(gameObject);
        }
    }
}


Default this spins an object around for 120° in local space. Since our the tiling of our 'button' is 3 in X direction, this will look like the button spins around in 2.5 seconds, while in reality, it will jump back to it's original position. But as you can see if you press the Unity Play button, it looks like the button is rotating around endlessly. LeanTween is doing most of the work: it rotates the game object around and the setLoopClamp at the end makes it a loop. No need for coroutines, lerps, and heaven knows what.

I am using local space because I want the button to perpendicular to the user's view at any given moment, but since the screen moves around with the user's gaze and gaze angle, it needs to be in the local space of the HelpHolder.

The HorizontalSpinner is in HoloToolkitExtensions/Scripts/Animations. Simply drag it on top of the Sphere, then change values as you like, although I would recommend not changing the Absolute Axis setting.

Conclusion

We have built the visual parts of a help screen but with very little functionality or interaction. It's actually not hard to do, if you know what you are doing. I hope I have helped you getting a bit more feeling for that.

In the next installment, which will hardly contain images, we will see WAY more code.

The project so far can be found at GitHub, as always.

14 June 2017

Setting up a HoloLens project with the HoloToolkit - June 2017 edition

Preface with lame apologies

My apologies for the longest blogging hiatus in the history of this blog, I was very busy with things both HoloLens and not-HoloLens related. Chief culprit though was my HoloLens app "Walk the World", whose development completely captured and absorbed me. Passion for tech sometimes does that with me ;). Then came some work-related issues, but now I finally am up to speed again.

Intro

A consequence of the hiatus is a lot of change (the only thing that changes faster than your average JavaScript-framework-of-the-month is the HoloToolkit) so any HoloLens how-to blog post I'd start with I would have to start with 'how to set up a project'. So I make a little separate post of that subject, thus I can refer to it often. Until the HoloToolkit changes again ;)

File/new project and Getting what you need

First, start a new Unity Project. I have called mine June2017HoloLensSetup.

Then, you open a browser and go to Rafael Rivera's  HoloToolkit download site and you click the top HoloToolkit-somenumber-unitypackage. At the time of this writing the top looks like this:

image

Downlaod and double click the resulting .unitypackage file - Unity will automatically import it into your project. This will give you a window with a lot of checkboxes:

image

This looks a bit intimidating, but just collapse the HoloToolkit node in the list, that will reveal another node "HoloToolkit-test" that you don't need, and you can de-select itimage

And then hit the Import-button. Then very soon, Unity will show a popup:

image

Select "Force Text Serialization". This is very important if you want to be able to easily upgrade the HoloToolkit and be able to check it in into your source control of choice.

The HoloToolkit will now be imported. Well, that was not so hard, wasn't it? This is time to get some coffee as Unity will churn a little on this.

Setting up the camera

HoloLens apps need some settings applied to the camera, and the HoloToolkit folks have taken care of that. So first of all, you delete the Main Camera from your Hierarchy - just select it and delete it.

image

Then, find the HoloLensCamera by entering "Camera" into the search box indicated with a red ellipse, and drag the new Camera into your hierarchy

image 

This is also a good moment to save your scene. Hit CTRL+S (or click File/Save Scenes), enter an name for your scene (I usually take "Main") and you are done with the this part.

Setting a whole lot of properties - the easy way

Now you may not have noticed it, but after importing the HoloToolkit you got an extra extra menu options.

image

This was added because you have to set a whole lot of properties necessary for or beneficial to a HoloLens projects -  scattered all over Unity, and good luck to you determining what you forgot if something doesn't work and where you have to correct that. Those were the bad old days (like a year ago ;) ). Now it's easy.

Clicking "Apply HoloLens Scene Settings" gives you this popup. Make sure everything is selected, then hit Apply. Always hit apply!

image

Next up is "Apply HoloLens Project Settings"

image

Same procedure: make sure everything is selected, hit apply. Unity asks if you want to reload the project - click Yes and wait for the project to reload. Finally, "Apply HoloLens Capability Settings"

image

This is what I always select - my HoloLens apps all tend to use voice commands, all use Spatial Perception (that is, after all, HoloLens' biggest USP) and most of them download some data from the internet. I never use the camera. Make your selections according to your necessary capabilities and hit apply.

Sound should be Spatial!

For some reason this is not part of these nice HoloToolkit setup screens - but if you use spatial sound in your apps (and you should!) there is an extra step. You select "Edit/Project Settings/Audio" like this

image

"Project settings is almost all the way down in a long Edit menu. Now on the right side of the editor, all the way on the top, a new menu has appeared:

image

The Spatializer Plugin is default set to "none", which will give you the default Unity stereo sound - nice, but that's sub-optimal usage of a $3K (or $5K) HoloLens - we want real spatial sound. So set it to MS HRTF Spatializer. This is, by the way, a very important and often overlooked part of immersive experiences. Human beings are 3D aware not only by vision, but also by sound. Think for instance of how you move around in a city, you hear cars coming to you even if they are outside of your vision (and that has more than once saved my life). In your house, there is the hum of the air conditioner, the bubbling of the fish tank filter, the voice of your partner - you don't have to see any one of those, but still you can tell where they are. It all contributes to your 3D awareness - and so does spatial sound contribute heavily to an immersive holographic experience.

So far for this soapbox moment ;).

Some organizing

It tend to create two empty gameobjects - HologramCollection and Managers. The HologramCollection will actually hold Holograms that are part of the application, the Managers object is like a container for global functions, object and standard stuff.

image

Some standard stuff to always add

What (I think) you will always need is an InputManager (the thing that acts as a kind of hub for all input, be it gestures, speech or whatever), SpatialMapping (duh) and some kind of cursor. I usually take the object that is just called "Cursor", there's also "DefaultCursor" and "CursorWithFeedback".

image

The easiest way to find them is once again entering their name in the search box. Now this can sometimes give cause for some confusion. For instance, if you enter "Cursor" in the search box you will get a lot of hits.

image

Only choose the so-called prefabs, the things that get the simple sky blue cube next to it. The same goes, by the way, for InputManager and SpatialMapping. Always take the prefabs.

When you are done adding SpatialMapping, find the property "Surface Material" and change it to "Occlusion" by clicking the little circle button behind it. Then, select it from the "Select Material" window that pops up

image

Finally some stubbornness

The current settings make that stuff is clipped when you are within 85cm of a Hologram. There are good technical and performance reasons for it, yet I am a stubborn *** ** * ***** so I tend to make this 15 cm. image

Simply because I think it's cool to get that close to a Hologram.

A final word on source control

Assuming you use Git, making a good .gitignore is pretty hard. Especially the HoloToolkit contains stuff you would expect not to reside in a source repository, there's all kinds of dlls in it and folders called x86 and x64, typically folders that are part of a build path and should usually be ignored. I've been bitten so many times by just that little piece of the HoloToolkit, especially when upgrading. Therefore a hard-won piece of advice: if you value your sanity, put a .gitignore file in Assets\HoloToolkit containing just these lines:

!*.*
!*

That should always add everything, I think.

You can download the (empty) project for reference, with my .gitignore files (there are no less than three) here.

12 April 2017

Using a messenger to communicate between objects in HoloLens apps

Intro

I know I am past the ‘this has to work one way or the other’ stage of a new development environment when I start spinning off reusable pieces of software. I know that I am really getting comfortable when I am start thinking about architecture and build architectural components.

Yes my friends, I am going to talk architecture today.

Unity3D-spaghetti – I need a messenger

Coming from the clean and well-fleshed out of UWP XAML development using MVVM (more specifically MVVMLight), Unity3D can be a bit overwhelming. Apart from the obvious – the 3D stuff itself - there is no such thing as data binding, there is no templating (not sure how this would translate to a 3D environment anyway) and in samples (including some of my own) components communicate by either getting references to each other by looking in parent or child objects and calling methods in those components. This is a method that breaks as soon as 3D object hierarchies change and it’s very easy to make spaghetti code of epic proportions. Plus, it hard links classes. Especially speech commands come in just ‘somewhere’ and need to go ‘somewhere else’. How lovely it would be to have a kind of messenger. Like the one in MVVMLight. There is a kind of messaging in Unity, but in involves sending messages up or down the 3D object hierarchy. No way to other branches in that big tree of objects without a lot of hoopla. And to make things worse, you need to call methods by (string) name. A very brittle arrangement.

Good artist steal…

I will be honest up front – most of the code in the Messenger class that I show here is stolen. From here, to be precisely. But although it solves one problem – it creates a generic messenger – it still uses strings for event names. So I adapted it quite heavily to use typed parameters, and now – in usage – it very much feels like the MVVMLight messenger. I also made it a HoloToolkit Singleton. I am not going to type out all the details – have a look in the code if you feel inclined to do so. This article concentrates on using it.

So basically, you simply drag this thing anywhere in your object hierarchy – I tend to have a special empty 3D object “Managers” for that in the scene – and then you have the following simple interface:

  • To subscribe to a message of MyMessageType, simply write code like this
Messenger.Instance.AddListener<MyMessage>(ProcessMyMessage);

private void ProcessMyMessage(MyMessage msg)
{
    //Do something
}
  • To be notified to a message of MyMessageType, simply call
 Messenger.Instance.Broadcast(new MyMessage());
  • To stop being notified of MyMessageType, call
Messenger.Instance.RemoveListener<MyMessage>(ProcessMyMessage);

Example setup usage

imageI have revisited my good old CubeBouncer, the very first HoloLens app I ever made and wrote about (although I never published it as such) that basically uses everything a HoloLens can do: it uses gaze, gestures, speech recognition, spatial awareness, interaction of Holograms with reality, occlusion, and spatial sound. Looking back at it now it looks a bit clumsy, which is partially because of my vastly increased experience with Unity3D and HoloLens, but also because of the progress of the HoloToolkit. But anyway, I rewrote it using the new HoloToolkit and using the Messenger class as a working demo of the Messenger.

In the Managers object that I use to group, well, manager-like scripts and objects, I have placed the a number of components that basically control the whole app. You see the messenger, a ‘Speech Command Handler’ and a standard HoloToolkit Keyword manager. This is a enormous improvement over building keyword recognizing script manually, as I did in part 4 of the original CubeBouncer series. In case you need info on how the Keyword Manager works, see this post on moving objects by gestures where it plays a supporting role.

Note, by the way, that I also assigned a keyboard key to all speech commands. This enables to test quickly within the Unity3D editor without actually speaking, thus preventing distracting (or getting funny looks and/or remarks) from your colleagues ;).

 

 

The SpeechCommandHandler class is really simple

using CubeBouncer.Messages;
using UnityEngine;
using HoloToolkitExtensions.Messaging;

namespace CubeBouncer
{
    public class SpeechCommandHandler : MonoBehaviour
    {
        public void CreateNewGrid()
        {
            Messenger.Instance.Broadcast(new CreateNewGridMessage());
        }

        public void Drop(bool all)
        {
            Messenger.Instance.Broadcast(new DropMessage { All = all });
        }

        public void Revert(bool all)
        {
            Messenger.Instance.Broadcast(new RevertMessage { All = all });
        }
    }
}

It basically forwards all speech commands as messages, for anyone who is interested. Notice now, as well, that in the Keyword Manager both “drop” and “drop all” call the same method, but if you you look at the image above you will see a checkbox that is only selected for ‘drop all’. This is pretty neat, the editor that goes with this component automatically generates UI components for target method parameters.

Indeed, very similar to how it's done in MVVMLight

Example of consuming messages

image

Now the CubeManager, the thing that creates and manages cubes (it was called “MainStarter” in the original CubeBouncer) is sitting in the HologramCollection object. This is for no other reason than to prove the point that the location of the consumer in the object hierarchy doesn’t matter. This is (now) the only consumer of messages. It's start method goes like this.

void Start()
{
    _distanceMeasured = false;
    _lastInitTime = Time.time;
    _audioSource = GetComponent<AudioSource>();
    Messenger.Instance.AddListener<CreateNewGridMessage>(p=> CreateNewGrid());
    Messenger.Instance.AddListener<DropMessage>( ProcessDropMessage);
    Messenger.Instance.AddListener<RevertMessage>(ProcessRevertMessage);
}

It subscribes to three types of messages. To process those messages, you can either used a Lambda expression or just a regular method, as shown above.

The processing of the message is like this:

public void CreateNewGrid()
{
    foreach (var c in _cubes)
    {
        Destroy(c);
    }
    _cubes.Clear();

    _distanceMeasured = false;
    _lastInitTime = Time.time;
}
	
private void ProcessDropMessage(DropMessage msg)
{
    if(msg.All)
    {
        DropAll();
    }
    else
    {
        var lookedAt = GetLookedAtObject();
        if( lookedAt != null)
        {
            lookedAt.Drop();
        }
    }
}

private void ProcessRevertMessage(RevertMessage msg)
{
    if (msg.All)
    {
        RevertAll();
    }
    else
    {
        var lookedAt = GetLookedAtObject();
        if (lookedAt != null)
        {
            lookedAt.Revert(true);
        }
    }
}

For Drop and Revert, if the “All” property of the message is set, all cubes are dropped (or reverted) and that’s it, the rest works as before. Well kind of – for the actual revert method I now used two LeanTween calls to move the Cube back to it’s original location – the actual code shrank from two methods of about 42 lines together to one 17 line method – that actually has an extra check in it. So as an aside – please use iTween, LeanTween or whatever for animation. Don’t write them yourself. Laziness is a virtue ;).

Conclusion

I will admit it’s a bit contrived example, but the speech recognition is now a thing on it’s own and it’s up to any listener to act on it – or not. My newest application “Walk the World” uses the Messenger quite a bit more extensively and components all over the app communicate via that Messenger to receive voice commands, show help screen, and detect the fact the user has moved too far from the center and the map should be reloaded. These components do not need to have hard links to each other, they just put their observations on the Messenger and other components can choose to act. This makes re-using components for application assembly a lot easier. Kind of like in the UWP world.

01 April 2017

A ‘roller blind’ animation component for HoloLens applications

Intro

This is a cool little tidbit that I wrote in the cause of a project that required 2D images to be shown in a 3D context. Not everyone has 3D models of everything, and sometimes you just have a schematic drawing, a picture, or whatever 2D thing you want to see on a ‘screen’. That does not excuse you from making a good user experience, and I made this little tidbit to give just that little extra pizzazz to a boring ole’ 2D image in a HoloLens. So you click on a 3D device, out comes a schematic drawing. So it’s 2D in a 3D context, not 2D per se.

Say what ?

It basically pulls an image ‘down’ like a roller blind is being expanded. Typically you ‘hang’ this below a ceiling or the object the image is connected to/needs to clarify. Without much further ado, let’s just show what I mean

Nice, eh? I have the feeling my study is becoming quite a household scene by now for the regular readers of this blog ;).

Setting the stage

Being the lazy b*st*rd that I am, I just made a branch of my previous post, deleted the floating 3D objects, implemented the roller blind as a Togglable, and used the 3D ‘button’ already present in the project as a Toggler. So now I have something to click on and start the animation. I also reused my DynamicTextureDownloader from the post before that to show this image of daffodils in my front garden because that what less work than actually making a texture. Did I mention already I can be pretty lazy at times?

Unity setup

imageWhat we have, initially, is just the button and a floating plane. There are some important settings to its rotation and it’s scaling. The rotation is because we want use see the image head-on. This important, as a Plane has only one side – if you look from it from behind you look right trough it.

The default position of a plane is horizontal, like a flat area. So in order so see it head-on, we first need to rotate in 90⁰ over x (that will put it upright) and then 180⁰ over z to see the ‘front’ side (that used to be the top). Don’t try to be a clever geometrist and say “Hey, I can just rotate it 270⁰ and then I will look at the front side as well”. Although you are technically right, the picture will appear upside down. So unless you are prepared to edit all your textures to compensate for that, follow the easy path I’d say. The picture left shows the result, and the picture below it how it’s done.

 

image

So to the Plane, called RollerBlind, we add two components. First the DynamicTextureDownLoader. Set it’s Image Url to http://www.schaikweb.net/dotnetbyexample/daffodils.jpg, which is a nice 1920x1080 picture of dwarf daffodils on the edge of my front garden

(yeah, I was a bit pessimistic about the ‘return rate’ and the buggers turned out to be multi headed too – so I am aware it’s a bit overdone for the space). Important is not to check the “Resize Plane” checkbox here as that will totally mess up the animation. You have to manually make sure the x and z(!!) sizes match up. So as the image is 1920*1080, horizontal size = 1.78 x vertical size. As the horizontal size is 0.15, so vertical size should be 0.15 / 1.78 = 0.084375. Be aware that a standard plane’s size – at scale 1 = is 10x10m, so this make the resulting picture appear as about 150 by 84 cm. I will never understand why standard shapes in Unity3D have different default sizes at scale 1 – for instance a cube =1x1x1m, a capsule roughly 1x2x1m, and a plane 10x10m – but I am sure there’s logic in that. Although I still fail to see it. But I digress.

I stuck the RollerBlind into a “Holder” and used that to position the whole thing around. I place it 1.5 meters from the camera (same distance as the rotating button) and 70cm below it. Go convert that to feet if you must ;)

image

The only thing missing is the RollerBlindAnimator itself

Code! Finally!

We start with the first part - basically all state date and the stuff that collects the initial data

using HoloToolkit.Unity.InputModule;
using UnityEngine;

namespace HoloToolkitExtensions
{
    public class RollerBlindAnimatior : Togglable
    {
        public float PlayTime = 1.5f;

        private AudioSource _selectSound;

        private float _foldedOutScale;
        private float _size;

        private bool _isBusy = false;

        public bool IsOpen { get; private set; }

        public void Start()
        {
            _selectSound = GetComponent<AudioSource>();
            _foldedOutScale = gameObject.transform.localScale.z;
            var startBounds = gameObject.GetComponent<Renderer>().bounds;
            _size = startBounds.size.y;
            AnimateObject(gameObject, 0, 0);
        }
    }
}

_selectSound is a placeholder for a sound to be played when this thing is being toggled, but the button is already taking care of that, so that’s not used here. Now the roller blind is going to be animated over what appears to be y, but since it’s rotated over x, that should now be the z-axis. So we collect the initial z scale. We also collect the objects apparent y–size. That we get from the bounds of the renderer, that apparently gives back it’s value in absolute values, not taking rotation into account. And then it quickly ‘closes’ the blind so it’s primed for use.

Why do we need to know this size and center stuff?

The issue, my friends, is that a Plane’s origin is at it’s center. So if you start shrinking the scale of the z-axis, the plane does not collapse to the top or bottom, but indeed - to it’s center. So rather than a roller blind going up, we get the effect of an old tube CRT TV being turned off (who is old enough to remember that? I am) – the picture collapses to a line in the middle. In order to compensate for that, for every decrease of scale by n, we need to move the whole thing 0.5*n up.

And that is exactly what AnimateObject does:

private void AnimateObject(GameObject objectModel, float targetScale, float timeSpan)
{
    _isBusy = true;

    var moveDelta = (targetScale == 0.0f ? _size : -_size) / 2f;
    LeanTween.moveLocal(objectModel,
            new Vector3(objectModel.transform.localPosition.x,
                objectModel.transform.localPosition.y + moveDelta,
                objectModel.transform.localPosition.z), timeSpan);

    LeanTween.scale(objectModel,
               new Vector3(objectModel.transform.localScale.x,
                   objectModel.transform.localScale.y,
                   targetScale), timeSpan).setOnComplete(() => _isBusy = false);
}

As you can see I have taken a liking to LeanTween over iTween, as I find it a bit easier to use – no hashes but method chaining, that supports IntelliSense so I don’t have to remember that much names (did I mention I was lazy already?).

The last thing missing is the Toggle methods that you can override in Togglable. That’s not very special and only mentioned here for the sake of completeness

public override void Toggle()
{
    if (_isBusy)
    {
        return;
    }

    AnimateObject(gameObject, !IsOpen ? _foldedOutScale : 0, PlayTime);

    if (_selectSound != null)
    {
        _selectSound.Play();
    }

    IsOpen = !IsOpen;
}

Two final things

We need to tell the toggle button that it needs to toggle the roller blind when it’s tapped.So we set it’s Togglers Size value to 1 and drag the RollerBlind object from the hierachy to the Element 0 field.

image

And the very final thing: this app accesses the internet. It downloads the daffodil image after all. Do not forget to set the ‘internet client’ capability. I did. And spent an interesting time cursing my computer before the penny dropped. Sigh.

Concluding words

I hope I have added once again a fun new tool to your toolbox to make HoloLens experiences just a bit better. I notice I get a bit of a feeling for this – past the ‘OMG how am I ever going to make this work’, now into spinning of reusable components and pieces of architecture. As I said, I was too lazy to set up a proper repo, so I’ve put this in a branch of the code belonging to previous blog post. Enjoy!

21 February 2017

A generic toggle component for HoloLens apps

Intro

The following scenario is one I have seen a lot of times – the user taps on a UI element, and then it and/or a couple of elements need to fade out, disappear, whatever. I suppose every developer has felt this itch that occurs when you basically make something the same the second time around, and you feel there will be a third and a fourth time coming up. Time for spinning up a new reusable component. Meet Toggler and it’s friend, Togglable.

The Toggler

This is a simple script that you can attach to any object that will function as a toggle – a ‘button’ if you like. It’s so simple and concise I just write the whole thing in one go:

using System;
using System.Collections.Generic;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

namespace HoloToolkitExtensions
{
    public class Toggler : MonoBehaviour, IInputClickHandler
    {
        private AudioSource _selectSound;

        public List<Togglable> Toggles = new List<Togglable>();

        public virtual void Start()
        {
            _selectSound = GetComponent<AudioSource>();
        }

        public virtual void OnInputClicked(InputClickedEventData eventData)
        {
            foreach (var toggle in Toggles)
            {
                toggle.Toggle();
            }
            if (_selectSound != null)
            {
                _selectSound.Play();
            }
        }
    }
}

This thing has a list of Togglable. When it’s clicked, it calls the method “Toggle” on all Togglable objects in the list, and optionally plays a feedback sound to confirm the toggle has been clicked.

The Togglable

This is almost embarrassingly simple.

using UnityEngine;

namespace HoloToolkitExtensions
{
    public abstract class Togglable : MonoBehaviour
    {
        public abstract void Toggle();
    }
}

and in itself completely uninteresting. What is interesting though is that you can use this base class to implement behaviours that actually do something useful (which is the point of bas classes, usually. D’oh). I will give a few examples.

A toggleable that ‘just disappears’

Also not very complicated, although there’s a bit more to it than you would think

namespace HoloToolkitExtensions
{
    public class ActiveTogglable : Togglable
    {
        public bool IsActive = true;
        public virtual void Start()
        {
            gameObject.SetActive(IsActive);
        }

        public override void Toggle()
        {
            IsActive = !IsActive;
            gameObject.SetActive(IsActive);
        }

        public virtual void Update()
        {
            // This code to make sure the logic still works in someone
            // set the IsActive field directly
            if (IsActive != gameObject.activeSelf)
            {
                gameObject.SetActive(IsActive);
            }
        }
    }
}

To if Toggle is called, SetActive is called with either true or false and it will make the gameobject that it’s attached to flash in and out of existence.

A toggleable that fades in or out

This is a bit more work, but with the use of LeanTween animating opacity is pretty easy:

using UnityEngine;

namespace HoloToolkitExtensions
{
    public class FadeTogglable : Togglable
    {
        public bool IsActive = true;
        public float RunningTime = 1.5f;
        private bool _isBusy = false;
        private Material _gameObjectMaterial;

        public virtual void Start()
        {
            Animate(0.0f);
            _gameObjectMaterial = gameObject.GetComponent<Renderer>().material;
        }

        public override void Toggle()
        {
            IsActive = !IsActive;
            Animate(RunningTime);
        }

        public virtual void Update()
        {

            // This code to make sure the logic still works in someone
            // set the IsActive field directly
            if (_isBusy)
            {
                return;
            }
            if (IsActive != (_gameObjectMaterial.color.a == 1.0f))
            {
                Animate(RunningTime);
            }
        }

        private void Animate(float timeSpan)
        {
            _isBusy = true;
            LeanTween.alpha(gameObject, 
                IsActive ? 1f : 0f, timeSpan).setOnComplete(() => _isBusy = false);
        }
    }
}

Initially it animates to the initial state in 0 seconds (i.e. instantly), and when the Toggle is called it animates in the normal running time from totally opaque to transparent – or the other way around.

There is a little caveat here – the object that needs to fade out then needs to use a material that actually supports transparency. So, for instance:

image

So what is the point of all this?

I have created a little sample application to demonstrate the point. There is one ‘button’ – a rotating blue sphere with red ellipses on it, and four elements that need to be toggled when the button is clicked – two cubes that simply need to wink out, and two capsules that need to fade in and out:

image

You drag the ActiveTogglable on both cubes, and FadeTogglable on both capsules. In fact, I did it a little bit different: I made prefab of both cube and capsule and dragged two instances on the scene. Force of habit. But in the end it does not matter. What does matter is that, once you have dragged a Toggle script on top of the sphere, you can now simply connect the Toggle and the Toggleables in the Unity editor, like this:

image

Which makes it pretty darn powerful and reusable I’d say – and extendable, since nothing keeps you from implementing your own Toggleables.

The result in action looks like this:

Why not an interface in stead of a superclass?

Yeah, that’s what I thought too. But you just try – components that can me dragged on top of each other need to be just that – components. So everything you drag needs to be a component at minimum, but you want the concrete class to be behaviours. So – you have to use a base class that’s a behaviour too. Welcome to the wondrous world of Unity, where nothing is what it seems – or what you think it is supposed to be ;)

Concluding remarks and some thoughts about 3D interfaces

Remember how Apple designed skeuomorphic user interfaces, that for instance required you to take a book out of a bookshelf? For young people, who never may have held much physical books, that’s about as absurd as the floppy disk icon for save – that is still widely used. But it worked in the real world, so we took that to the digital 2D world, even when it did no longer make sense. Microsoft took the lead with what was then called ‘Metro’ for the ‘digital native’ float design. Now buttons no longer mimic 3D (radio buttons) and heaven knows what.

We are now in the 2007 of 3D UI design. No-one has any idea how to implement true 3D ‘user interfaces’, and there is no standard at all. So we tend to fall back on what worked – 2D design elements or 3D design elements that resemble 3D objects – like 3D ‘light switch buttons’ attached to some ‘wall’. Guilty as changed – my HoloLens app for Schiphol has a 2D ‘help screen’  complete with button.

With my little rotation globe I am trying to find a way to ‘3D digital native design’, although I am not a designer at all. But I am convinced the future is somewhere in that direction. We need a ‘digital design language’ for Mixed Reality. Maybe it’s rotating globes. Maybe it’s something else. But I am sure as hell about what it’s not – and that is floating 2D or 3D buttons or ‘devices’ resembling physical machinery.

Code, as per my trademark, can be found here.