The Setup

After joining the team, the first task was to decide on the hardware for the head-mounted display. The lab had access to various options, which we quickly narrowed down to either a Meta Quest Pro or a phone-based headset such as HoloKit. The former would provide excellent tracking for both head and hands via controllers, at the cost of being less accessible to the public and having slightly worse passthrough. The latter was very accessible and would open up the possiblity of computer vision, though it left the problem of tracking a welding gun unsolved.

In the end, we chose the Quest Pro for its controllers, as well as its ability to track motion when vision from the front-facing cameras was poor (due to the auto-darkening screen in the welding helmet). Lack of computer vision was fine- none of our goals required it (except for the headset's own tracking), and the passthrough quality wasn't noticeably worse than normal vision once an auto-darkener was involved.

As for replicability by others, we knew a cheaper headset solution with decent passthrough would soon be available in the Quest 3 (which we used in later designs).

Our first helmet design had 3D-printed clips for the Quest Pro inside the front of a standard welding helmet, as well as holes cut in the side of the helmet for USB-C access, and to allow side cameras to track motion, We found that front cameras could see pretty well through an auto-darkener, but they would not support tracking by themselves.

To track the welding gun, we designed an attachment for the right controller with a cover to block sparks. The entire attachment was designed to be fastened using zip ties to keep it as portable as possible, but still secure.

There was also a small button placed on the trigger of the gun, which was wired to a board inside the attachment. This board could connect to a board attached to the Quest in the helmet using Bluetooth, allowing us to transmit whether the gun was active without needing excessive wiring.

Our later helmet designs were similar, but were custom designed and 3D printed to fit a Quest 3 instead, as well as being considerably lighter than our original.

Software

We decided to use Unity for a couple of reasons: First, it supported building for VR and Quest without requiring excessive setup. Second, I had experience building apps for Quest in Unity and would be able to more quickly prototype designs compared to a different engine or custom solution.

Setting up the basic app was pretty simple. Passthrough was easy to enable, and once I knew how to orient the virtual welding gun relative to the controller based on the attachment, I could begin prototyping overlays.

In my opinion, the main technical challenge at this point was working with the serial port on the headset. As mentioned, we needed to determine whether the welding gun's trigger was pressed, and we also wanted the possibility of collecting additional sensor data (to measure breathing, for example). We ended up having a Seeed board connected to the Quest using USB-C, which would repeatedly send serial data.

I found a convenient Unity plugin that supported reading serial data on Android devices, which should therefore work for the Quest Pro (which runs on Android). However, I ran into a couple of issues.

First, use of the port required requesting permission from the user. This is handled by editing the app's AndroidManifest.xml file, normally to present a prompt when the app is launched. This didn't seem to work well on the Quest- the permission prompt would render behind regular Quest interfaces and was not interactable. I sideloaded a normal Android serial reader app to test the port in general, and it seemed to display the permission request correctly, so either I messed something up or Quest just really doesn't support this type of permission request.

Switching to using intent filters in order to prompt the user to launch the app when a USB device is connected seemed to work, though, so I opted to use that. It's worth noting that this solution still doesn't seem to be completely supported on the Quest, evidenced by the fact that having more than one app with such an intent filter specified caused the actual device USB service to crash, which is pretty funny.

The second problem involved the app's device_filter.xml file. In order for a USB device to be supported by an Android app, its product and vendor IDs must be specified in this file. This file was included in the plugin I was using, but it didn't specify the board we were trying to work with, so I had to add its product and vendor IDs myself.

Once that was done, we were able to read data from the Seeed board, including whether the gun's trigger was pressed.

App Design

For this project, the lab partnered with a local welding workshop for training and demo sessions with feedback. This was helpful as most of the team was inexperienced with welding, so we needed input from others and practice in order to know what was most important to display. As we were designing for an in-situ use case, our main consideration for the app was what information needed to be displayed, and how.

When I joined, the team had an idea of what they wanted to have implemented. We had to first localize and indicate what the welding surface was, and then we should be able to indicate the welder's speed, angles relative to path and surface, and standoff distance. Our UX designer gave me a specification for these, so I began working quickly.

Setting the surface was easy as we assumed the weld would occur on a flat plane, so we only needed to mark two points for the weld line.

For the angles and standoff distance, I implemented 2D circles and line renderers which changed color as the user lined up with the target, which was pretty satisfying.

For standoff distance, though, I felt that it was difficult to see the dotted line indicator on top of the welding gun, so I opted to include another indicator to the side of the gun that was sort of like an altitude indicator in various games.

For speed, we had discussed the possibility of having a point move at the same speed that the user would match, but I felt that it would be difficult to know when to start and stop its movement, so to begin I just used a dashed line that scrolled at the desired speed according to our resources.

Design Flaws

Although we felt happy with the visual quality of the indicators, we found that they were difficult for users to get value from in practice. Some were just difficult to concentrate on when welding. There was a lot of information around the gun, and welding is already a stressful activity. Looking away from the bead to look at my altitude indicator was pretty much impossible, and the scroling line was too hard to see as it moved very slowly.

Another problem we discovered was that a technique-based activity like welding can have a lot of variance in what is considered "correct" depending on the person. Speed may be pretty consistent assuming settings are the same, but we found that programming a target "travel" or "work" angle was not the correct way to go about things, as the experienced welders testing our app found themselves constantly "incorrect" despite using their honed abilities.

To reduce visual clutter and make the app more intuitive, we intend to forgo indicators for specific values such as angle, speed, and so on in favor of showing just one object moving correctly, which users can try to follow. We have yet to conduct studies but we plan on testing a single dot on the weld surface and a wireframe gun model for this purpose.

Evaluation

Currently, the app is able to record and playback welds in 3D, letting users or instructors view welds from different angles and pause to see values at any point. These recordings are also saved on the device, which is useful for research purposes.

We've recently been focusing on post-weld feedback and a scoring system, integrating algorithms used in the drawing tutoring system SketchTivity. Having a consistent performance measuring system will allow us to conduct stronger user studies to compare the effects of different types of training, visualizations, and such.