You see a lot of coverage about spatial computing in business applications like construction, mapping, and maintenance. But you won’t hear too much about manufacturing, an extremely controlled environment ideal for testing and developing new kinds of 3D technologies.
That’s why sought out Valentin Heun, VP of Innovation Engineering at PTC, one of the most technologically adventurous firms in the manufacturing space. Heun heads up PTC’s Reality Lab, where he leads development of a platform and GUI (graphical user interface) that will give users intuitive control of real, three-dimensional objects in the real world.
Highlights:
- What a computing interface for interacting with reality looks like
- How PTC’s Reality Lab is already applying this interface in factories
- An example: controlling a robot
- Developing the “cut/paste” functions for 3D space
- How far off is this technology, really?
- Why we can be optimistic about the future of spatial computing
Sean Higgins: What does the term spatial computing mean to you? It seems simple, but everyone I ask gives a very different answer.
Valentin Heun: This is a really interesting and loaded question, and it’s a particularly interesting time. There are two revolutions going on right now. One is the web3, a distributed-accounting wave on the internet, enabling NFTs, digital currency, smart contracts and DAOs. The other is spatial computing technology. For the first time, this allows us to go beyond the desktop and make computing part of our daily physical life. This will make computing indistinguishable from reality and therefore enable a better human experience in the here and now.
From my perspective, spatial computing is a revolution that is similar to the personal computing revolution. It is really a new paradigm for interacting with a computer.
Before there were desktop computers, experts would use computers to do a lot of really complicated calculations that they couldn’t do otherwise. But the normal, general public wasn’t able to understand what the computer would do for them.
And then, with personal computing, and the desktop paradigm, suddenly there were killer applications like the spreadsheet and Word and a calendar. Suddenly, we’re able to do all the things that you would do in paper on a desk. But much better and much faster. Eventually, personal computing allowed us to connect our desktops and increase shared productivity.
What does spatial computing bring to the mix? How is it going to change the way we compute?
One problem of desktop computing is that it entirely misses the notion of space. [Points at the Zoom video chat on his screen]. Right now, I have no idea where you are sitting. I see a kitchen in the background, but that’s all I get from this video.
This chat is a 2D thing. It fits well into the desktop paradigm. I have a little window here that I can move around with my mouse to put you somewhere else. I can put my calendar next to it, or a Word document. It’s perfect multitasking, it’s amazing.
But I don’t know how your kitchen looks, I don’t know where you are, I don’t know how difficult it is for you to make a coffee while you’re having this video call. This chat is missing 3D information, and that’s clear even in a simple video call for an interview.
So now think about that problem, but translated into a factory. I have robots that move around in space, I have work processes that unfold through thousands of square feet, and an entire factory running complex 3D spatial processes. But the only way I can manage them is through the desktop paradigm. Through a personal computer, in 2D.
And managing 3D space through a 2D interface takes a lot of brain power.
Yeah. As an expert in a factory, I’m not training on how to move a robot in 3D space. I’m training on how to operate a desktop interface that has no notion of space and using that interface to move a thing in 3D. My mind is the translator. It translates from the 2D interface into the 3D space in the factory. That mind juggling is what an expert has to learn over the years.
It’s hard, if you fly a drone, to do the simple translation between your joystick and how the drone is positioned in space. It’s turned around, so what was forward is now backward, and what is left is now right. This is very complicated, and it takes a long time to learn.
And I haven’t even started talking about connected shared productivity in this context.
Let’s talk about what work you’re doing to solve this problem. You’re developing an OS that will make it easier and more intuitive to interact with 3D spaces?
My team has worked for the last 10 years to build a platform for exactly this moment that we have right now — to do spatial computing, to give you the next version of a desktop, to allow you and everyone you know to dive into virtual spaces, or into the real world. It’s all one from our perspective.
With this platform, you can operate a physical robot. You can literally take the robot in your virtual hand and move it around. Instead of having a 2D video call, you can use our platform to do spatial teleportation that allows you do interface with these robots from anywhere in the world — you can just “fly in” and do it.
You can meet with experts in the field while sitting in your living room. Or, define an action plan in a physical space and then send in a team with an AR interface. And they’ll already know what to do because the work has already laid out in 3D for them. That’s just fascinating.
We get to actually go into a 3D space and build an operation environment for the physical space, one that allows us to gain the productivity leap with spatial problems that we did before with desktop problems.
To try to paraphrase, spatial computing is the technology that will allow us to collapse the virtual world, which exists in the computer, and the real world, as we see in our daily lives. It blends them together seamlessly.
From our perspective, you won’t even say that there’s “virtual” or “real,” because virtual is the construction of a desktop interface that tries to depict something in a way that could look real.
But if spatial computing reaches its full extent, it comes really close what Mark Weiser calls the computer for 21st century. He has this cool idea that the most profound technology blends into the fabric of the world, it’s just indistinguishable from the real world.
Can you give an example to illustrate?
If you drive a car, there are many little motors everywhere and little computers all around you. When you turn the wheel, it drives by wire. [When you turn the steering wheel, it doesn’t turn the car’s wheels using a mechanical coupling, instead it digitally activates motors that turn the car’s wheels for you.]
There’s nothing real in this car, it’s all computers, the whole thing is virtual. But for you, the car is real, it’s still a real thing. And so your car is already part of a spatial computing world, a complete amalgam of the digital and the physical. And that’s how I think we will see the real world in the future, it will all melt together. Hopefully, you will never think about virtual and physical again.
Maybe this is a good time to talk about your work at PTC with the Reality Lab. How did this project get started, and how are you working to make computing blend into the real world in a factory setting?
Around 2013, I was a student at the MIT Media Lab, and I saw another researcher experimenting with augmented reality with a little robot. I was fascinated by the seamless connection between the digital and the physical. It all felt so intuitive and meaningful. It was around that time the vision for the Reality Editor and the Reality Lab was formed. We dreamed up a world that now in 2022 is a reality.
I knew from the research and papers that I was reading at the time that the world would become saturated with technology like the Internet of Things. Technology would become smaller, more integrated, more connected. It would be ubiquitous, all around you. And so the idea was: How do we actually manage all of that?
At the time, there was this big idea that you would have touchscreens everywhere around you. Beyond the touchscreen, there’s some computer agent that works on your behalf. There’s some magical thing that will solve your problems with machine learning. Whatever you could want, this agent would do for you, on your behalf, using that Internet of Things.
From my perspective, the reason why we would think this way is because our channel for controlling the Internet of Things was very limited, for all the reasons that I just tried to explain.
So imagine you try to talk with somebody. But all you can do is push a button, and you need somebody on the other side to interpret what you might want. For that to work, you need machine learning to learn everything about you. “Okay, he pushed the button three times, he might want spaghetti, he usually likes spaghetti at this time.”
The original vision for my group was to build a better pipeline from you to that Internet of Things. To give you better control of the world. To externalize your ideas into the world around you, so you can actually control the world the way you want. Not the way an agent might think you want to control the world.
And that’s where your Reality Editor came in?
We built a tool that we call the Reality Editor in 2013. And then in 2015, we open-sourced it at the MIT Media Lab. In 2017, we built Spatial Programming. This is like an advanced automation tool that allows you to deploy logic into the world. You can use it to tell the world your ideas and what you want to do.
In 2017, we ran into Jim Heppelmann, CEO of PTC and we realized we have the same vision. We were exploring it from the consumer perspective, and he was exploring the same vision from an industrial perspective.
That made sense for me. The biggest use of this technology is when you have more things you need to control in space, because then it becomes very difficult to do using a desktop. The consumer space wasn’t there at the time, but the factory is full of these problems.
Together with Jim Heppelmann, Steve Dertien and Ben Reynolds we started this lab and continued working on the original Reality Lab vision. But from an industrial perspective. And we called it the Reality Lab, because we had the Reality Editor. In 2020, we open sourced that work and we call it the Spatial Toolbox.
We’ve continuously worked on this project ever since 2013. And I think the ultimate vision is still the same after 10 years: We want to give you a tool that allows you to control and understand the connected physical world.
That’s spatial computing, the paradigm shift from the desktop that will make the metaverse possible. We have built the operation environment, the graphical user interface, the thing that allows you to control the physical world.
I had planned to ask why a factory is a good space to work out the problem of a new type of computing, but I think it’s clear: The factory presents a bunch of spatial problems. On top of that, all of the systems of the factory, and the way they interact, are already well defined. That solves a basic problem for you.
Yeah those modules are already connected. Think about that car again. That car is the closest entity to a factory that everyone already knows.
There are so many highly integrated subsystems that are all interconnected. From the steering wheel to the windshield wipers that go up and down, the handle that opens the door, the buttons for the lock — there’s a whole little internet in your car that’s holding it all together.
That is a lot like the complexity of a factory. And that’s why the factory is the playground to make this real, you know, like this is this is the place where we can explore all of this.
Let’s get back to the graphical user interface for interacting with 3D space. How do you even start building this GUI? What kind of problems do you need to solve?
I think the big change is that you need to understand the surface layer of the physical world, how its parts interact, and how we interact with it. So if I have a mug, then I put it on my table, these two things are modules that just fit together.
The table can interact with the affordances of this mug. It holds the mug tight, right? There’s a handle here that I can hold, so it connects with me.
Everything, every object, every grain of sand in this world, has affordances to interact with anything else in this world. So the world is this gigantic modular thing.
And you need to understand how these modules connect with each other, and then how you can create a user interface that also works this way. The desktop won’t work. It uses the idea that I have a piece of paper. I can write on a piece of paper, I can cut something out of this piece of paper, I can glue it into another piece of paper.
But if you want to build a paradigm for the real world, you have to actually understand how we interact with the world and bring that into a computing paradigm. So we’ve been looking at how to generate an interconnected virtual world that represents the ideas of the physical world.
How did you solve this problem?
From our perspective, the virtual interface should be identical to the physical interface, and they should be modular in the same way. So we created a spatial user interface that allows you to interact with objects that are digital and physical at the same time. You can think about using the platform as browsing through an ocean of digital interfaces and information tied to the physical objects in space. So each object around you would have a number of digital interfaces tied to it, as well as a variety of data.
To help illustrate how this works from the user’s point of view, can you give me an example in a factory context?
One example of an interface that we’re building is what we call a spatial envelope. These spatial envelopes help you to group interfaces for spatial purposes.
You might want to use an envelope to daisy-chain events in space, so that you can walk through them as a linear, temporal sequence. Or you might want to track events in time and learn about events via a timeline.
So for example, if I have the ability to analyze a person’s physical work task in a factory, then I can put that on such a timeline. That will show the work across time, give in-depth understanding of how the work is performed in space, and reveal how to optimize it. It’s very powerful.
Another such envelope enables path planning. This is really handy when you want to program robots in space. Here we have a robot that can drive around. And we can take one of these path planning envelopes, and we drag and drop it on the robot to connect the envelope. The envelope is now in charge of the robot.
So whatever you do within the envelope, when it’s open, will control the robot.
We can add mission points to this path in space. And the robot understands these path points, because the robot is connected to the envelope.
There is a profound detail about this work: Each of these mission points is an entirely unique web application. Each of them holds the potential to represent an entire web page connecting to the physical world, each of them could be a modified version of something if that works better for you. Through this modification and remixing, you could tell your robot to turn right normally — but turn left if the Google forecast expects rain.
Once you deploy all your mission points and you like how the robot will move in space, you can tap the start button, and now the robot is deployed.
Looking at these points, that’s one, two, three, four applications plus the button, which makes five. And they all have meaning together. That’s an example of how we’re looking at how to generate that interconnected, modular virtual world that represents the ideas of the physical world.
For your 3D spatial operating system, these functions seem like the copy and paste functions in a desktop environment. They’re the basic functions you can perform, and they have direct analogs in the ways we interact with the physical world already.
Yeah, we’re really trying to find the essential shapes for the operating system, then combine the shapes into more complexity but in a way that it gives you a simple interface.
It looks pretty well developed. How have people reacted to this in a factory setting? Or are we still some way off from seeing this kind of thing implemented?
We work with a selection of universities and customers that use our work for research.
Everything that we build is open source, everyone can download it, deploy it, try to understand it, and experiment with it. We’re talking with a lot of other people who want to make this real.
Let’s say I run a factory and I’m interested in deploying this GUI and the underlying platform. Should I wait until it’s “fully ready,” or is there value in diving in now?
If you have an innovation team, or a research team that also looks into the future, they are capable of downloading our source code. They can deploy it today and experiment with it. Anyone else will need to wait a bit longer.
Everything you do is open source?
Yes, all the research that we run in our Reality Lab is open source. We do that because it allows us to understand our problems better.
What I really love is that we have students who build spatial applications for our spatial computing platform, the same way you can build applications for a desktop. They extend this whole platform in a way that we never thought about. And it’s just so fun to see all of these things working out.
But it’s only the beginning. It’s far from being extremely general-public user friendly, and we are working to make it more simple, more digestible for everyone you know.
We’ve talked a lot so far about what you could call the minimum viable product of spatial computing. But where do you imagine this technology might be in 10 years? In the factory, and maybe in consumer applications as well?
In the factory it’s really easy to imagine, because everything is about time and energy. I have material, I put it into the factory, it requires some time and energy — human energy or electric energy — to make that material into the product. So wherever you can cut energy or time, you have a value proposition and you have a clear product.
In the general public, it’s a little bit more difficult. I think the best analogy is the video chat we’re having right now, like it’s the most normal thing in the world. Why would we not, it’s fantastic? But before Apple put that camera into the lid of their laptops, it was an awkward thing. You had these super clunky cameras you put on top of your computer screen, and by doing that you would signal somebody else that you’re the biggest geek in the world. And why would you want to do that when you can just have a phone call?
But then there came a moment when the camera ended up in the lid. Suddenly, there was one button to push and the video’s there. So it became a commodity. It just worked.
And I have the same feeling for spatial computing and the metaverse, because I see utility when it hits the real world. It will travel from that niche into everyone’s world and become a commodity, the same way that this camera in your screen is a commodity.
What do you see as the potential risks of this kind of technology? I like to ask this question in every interview, and I have gotten a lot of different answers.
Look at the metaverse from the perspective of the car, that is already a full amalgam of physical and virtual. I don’t see any problems there. I don’t see our world getting lost in the digital.
I only see how it will take us out of that digital-only box. We have so many people staring at their phones all the time, because they are trying to see the world through a desktop paradigm. And that’s what’s eerie to us, because we want to say that there’s the real world out there.
But if the real world is also the digital world, then then these problems are not there. And that’s actually a really nice perspective of the future.
What I also really support are virtual meetups like Spatial or VRchat. I think this is amazing for teenagers today. They have this amazing playground to reinvent themselves without the pressures of a permanent camera-controlled world and they can be together anywhere in the world.
They can experiment in this new world and they can reinvent themselves in a way they never could before. You could sit in the middle of Idaho on a farm, and you turn on the VR headset and suddenly you are in the middle of a buzzing new world.
Most people I ask are much more wary about the effects of this technology, but you sound like very optimistic.
We have to be mindful, but also not see the negative side too much. I think it’s a fascinating time that we’re in right now. We’re just ahead of a revolution that’s unfolding. We have the technology at hand, we just don’t know yet exactly how to put it together.
But once we know how to put it together, we are going to experience computing the way we’ve never seen it before. The internet is a small precursor to what is about to come for us.