Summary
This project is the (by)product of a bachelor thesis dealing with the development of an "AR-optimized Manufacturing-Assistant" system for the HoloLens 2.
This system was developed taking into account current findings and results from works that deals with the topic of AR in training,
assembly and maintenance tasks.
The main purpose of this system is to guide an inexperienced user (e.g. a trainee of some kind) through a manual assembly or maintenance task.
Additionally the system allows the HoloLens 2 sensors to record the user's hand and head movements. These recordings can later be visualized, for example,
with this provided program.
Feature Overview
Prerequisites
Setup
Assembly Setup (optional)
Recording and Replay (optional)
Additional assembly hints
Digital Twin:
In its core-function ARoMA uses a digital twin to show the current assembly step and highlights step-relevant parts by animating them in such a way that mounting
position and direction are shown. Additionally the user can always grab the twin and manipulate its position, orientation and size.
Part Amount Indicator:
It is possible, during the setup process, to designate part-bins, which are then, during assembly, highlighted and accompanied by numbers to indicate to the user what and how many parts are required
for a given step.
Proccess Indicator:
During assembly the current assembly-step and the total amount of steps are shown close to the user's right hand.
Voice commands:
Via the voice command "Next" the subsequent assembly step can be initiated.
Recording and Replay:
ARoMA can record hand and head movement of the user. A program to replay the data is also provided.
Software:
Unity 2019 or higher (https://unity.com)
MixedRealityToolkit for Unity (https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/?view=mrtkunity-2021-05)
Visual Studio 2019 or higher (https://visualstudio.microsoft.com)
3D modeling program that can export .obj-files. (this project did use TinkerCad (https://www.tinkercad.com))
Microsoft Excel
Hardware:
HoloLens 2 Microsoft
Step 1: Download the Project
Download the project ZIP file and unpack it to a desired location (e.g. Desktop).
Step 2: Importing the 3D-Data
To import the data for the 3D twin, move your obj. files into "ARoMA-main\Assets\HD\3D Objects"
You will find that there are already some obj. files. These are some 3D models you might want to use later.
This project also provides some 3D files for testing purposes under ARoMA/Extra Files/Demo/3D Objects.
Step 3: Start Unity
To start the project in Unity go to "ARoMA-main\Assets" and click on "Assembly.unity".
After this, Unity will prepare the environment. This step might take a moment.
When Unity opens a dialogue window "MRTK Project Configurator" opens. Click on "Skip This Step" then on "Next" and after that on "Done".
To move in the scene go with the mouse courser in the Scene window. Here you can look around while holding the right mouse button. To move, while holding the right mouse button, use WASD.
Step 4: Setup the Projekt
Prepare the 3D model
In the project hierachy navigate to "Assets > HD > 3D Objects.
Mark all 3D objects (hold SHIFT + click on first object in list -> click on last object)
and click on "Model" in the "Inspector" window. Now set the "Scale Factor" to 0.01 and click on "Apply"
(Depending on the size of your 3D model you might need to adjust the Scale Factor).
To color the 3D parts click on them individually and select in the Inspector window under "Materials"->"Remapped Materials"
a material of your choice. A small selection of colors is already provided.
In case you need more colors or textures for your parts you might want to refer to
this and this.
Place the 3D model
Now mark all 3D objects and drop them into the "3D_Model" game object in the scene.
You can now position and rotate your 3D model onto the build-platform.
After positioning all parts you need to add the "AnimationSlave" script to all the 3D parts. Every game object under "3D_Model" should have one "AnimationSlave" script attached to it (exept "AssemblyBase" if you want to use it, if not you can delete it). Save the scene.
Step 4: Determine build and animation sequence
Sequence
Now we want to tell the program at which step the different parts should be displayed and animated.
To do that, we need the "Order Of Activity.xlsx" file. Under "Name" you enter the names of your 3D models of your individual components.
These names should be the same as in "3D_Model" in your Unity hierarchy.
Now under "Step" you enter a "1" for every step at which the component should be displayed.
Animation
The first time a component appears during an assembly it will be animated according to the two parameters you can enter
under "Direction" and "Distance".
"Direction" is the direction from where the part moves to its final assembly position.
Valid directions are "Down", "Up", "Back", "Forward", "Left" and "Right"
"Distance" is the distance the part starts the animation from to its final assembly positions( "0.1" is a good start) .
If you don't want a part to be animated set "Direction" to any of the valid directions and "Distance" to "0.0".
In the end your "Order Of Activity.xlsx" file should lock something like this:
Now navigate to the "3D_Model" game object in the Hierarchy-window in Unity. Go to the Inspector-window and now copy the content of cell B1 of the "Order Of Activity.xlsx" into the "Order Of Activity" field.
Step 5: Define order of component bins
Open the "Zone Activity.xlsx" file. Here you can enter at what step, how many and from which Box components should be picked.
For example: You need "Item 1" for step-1 two times, for step-5 two times. Also, you need one "Item 2" for step-1 , for step-8 five times.
Then your "Zone Activity.xlsx" should look like this:
Note: If you do not intent to use this feature just fill in "1" for the first box for the amount of assembly steps you have.
You do not need to fill the "Note" fields. They are just for convenience to keep track which box contains what part.
Now navigate to the "AssemblyDirector" game object in the Hierachy-window in Unity. Go to the Inspector-window and now copy the content of cell B1 of the "Order Of Activity.xlsx" into the "Zone Activity" field.
Step 6: Test in Editor
At this Point you should be able to test the assembly in Unity.
For this click on "Play". By clicking "Next" you can go through the assembly process. At all times you should be
able to press the "Restart" button to restart the assembly process.
If everything looks good you can now deploy the program to the HoloLens 2.
Step 7: Deployment to HoloLens 2 via Wifi
Note: This is just one of multiple methods to deploy a program to the HoloLens 2 but only this way was tested.
For alternatives see https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/build-and-deploy-to-hololens
First you go to "File" (in Unity), then click on "Build settings...". Under "Platform select "Universal Windows Platform" and make sure your settings look like this:
Click "Switch Platform" and then "Build". Unity should ask you where to save the build, you can create a folder "Build" in your project folder and save it there. When Unity is done, navigate to your Build folder and open the "Fittslaw.sln". Visual Studio should start now. In Visual Studio go to "Project" then "Properties". For Configuration, select "Release" and for Platform select "ARM64". Now under "Configuration Properties" -> "Debugging" enter the IP of your HoloLens 2 under "Machine Name" and then click on "OK". To get the IP of your HoloLens 2, just say, "What is my IP" while wearing it.
Now also set "Release" and "ARM64" like this:
At this point make sure the HoloLens 2 is on and connected to the same WiFi as your computer. Click on "Remote Machine". Visual Studio should now transfer the program to your HoloLens 2. This might take some time. After Visual Studio is done the program should start on the HoloLens 2. On first start the program will ask for permissions which you should grand.
Note: This step is optional
Setup of zones
After starting the application on the HoloLens you have the option to set up the bin-zones at their respective locations.
To do that say "Interface" and a widow over your right hand will appear. By pressing "Next" you can summon a Zone (starting with "Zone_1) which you can now place.
To place the new zone just grab it with your left hand and move it to its location. You can close the placement-mode by again saying "Interface".
During and after an assembly you call also call the interface window to restart the assembly process. The Zones do not need to be placed again when you restart an assembly.
Adding more Zones
If you need more zones you can add them in the ARoMA program in Unity. In the Hierarchy go to HandUI->HandFollow->ZoneMagazine and copy an existing zone and then change the number to the next highest
value. For example, if you have 18 Zones and want to add two more, just copy Zone_1 two times and change the names of the new objects to "Zone_19" and "Zone_20".
If you need less zones just delete the Zone-objects, starting with the highest values.
Note: This step is optional
ARoMA can record the hand and head movement of the user for all build-steps(except step 0 and the last step(build-complete)). If you are not interested you can disable this function in the ARoMA program in Unity. In the Hierarchy go to AssemblyDirector->CursorFocus and uncheck the "World Cursor" Script in the Inspector window.
To get the recorded data and replay them follow the next steps:
Step 0: Prerequisites
To download the data from the HoloLens you need to get Spyder(Anaconda) and the Python scripts from the folder ARoMA/DownloadData
Step 1: Download Data
Start Anaconda/Spyder as admin. Next, you enter these commands one at a time:
cd *Path to the DownloadData folder*
pip install opencv
import cv2
run recorder_console_app.py --dev_portal_address *ip address of the HoloLens* --workspace_path "*Path to the DownloadData folder*" --dev_portal_username "*your Username*"--dev_portal_password "*your password*"
Here an example, what is can look like:
cd C:\Users\JohnSmith\Desktop\ARoMa Git\ARoMA\DownloadData
pip install opencv
import cv2
run recorder_console_app.py --dev_portal_address 192.168.1.1 --workspace_path "C:\Users\JohnSmith\Desktop\ARoMa Git\ARoMA\DownloadData" --dev_portal_username "JohnSmith"--dev_portal_password "ABCD123456789"
When all worked, you have now the option to download the data do the "DownloadData" folder.
Step 2: Unpack and start the visualizer
Unpack the "AssemblyTaskVisualization.zip" file from "ARoMA\Extra Files\AssemblyTaskVisualization".
Now open "AssemblyTaskVisualization\AssemblyTaskVisualization\Assets\AssemblyTaskVisualization.unity" with Unity.
Step 3: Replay the Recording with the visualizer
Demo video:
Now we can load the data from the visualizer. To do that, first cilck on "Select Folder" and navigate to the recordings you previously downloaded. Select the folder of the recording
you want to replay.
Then select the step you want to visualize by entering it in the field next to "Set Page" and the click "Set Page".
Now you can Play/Pause the recording or single-step through the recording by pressing "Step -" or "Step +" while the replay is paused.
It is possible to not only have the digital-twin as assembly guit, but also use supplementary text or image instructions.
Text
For test hints simply use the provided "Hint1" object under "3D_Model" in the Hierarchy window. Under Hint1->group_0_15277357->Canvas->Text you can enter a instruction under the "Text" field in the Inspector window.
Then you use the Hint1 object like another component in your assembly. Add it to the "Order Of Activity Demo.xlsx". Copy and repeat as required.
Images
For images delete the Canvas object. Then right-click on the group_0_15277357 object and under UI you can add a Image.