Category Archives: AR/VR

The Vision Pro’s R1 Chip is Real-time, not “Fast”

Almost all coverage of the R1 in the press is just a reprint of Apple’s PR, so it just says that the

[…] R1 chip processes input from 12 cameras, five sensors, and six microphones to ensure that content feels like it is appearing right in front of the user’s eyes, in real time. R1 streams new images to the displays within 12 milliseconds

Or some simple variation on that. Obviously, 12 milliseconds is “fast”, but that’s not the point of the R1.

If “fast” was all that was required, Apple probably could have gotten a process on the M2 to average 12 millisecond display times, maybe plus or minus a few milliseconds. The point of the R1 is that it guarantees to do that processing in an exact amount of time. That’s what “real-time” in real time chips and operating systems means. It lets you express your requirements to run tasks in a given amount of real time and can guarantee it before runtime.

This is not the first time Apple tried to get “real-time” working for a single element of a device. In the original iPhone, scrolling with your finger was given such high priority that displaying the view was partially delayed. John Gruber described it well in his review from 2007:

Update: Real-time dragging is such a priority that if the iPhone can’t keep up and render what you’re dragging in real-time, it won’t even try, and you get a checkerboard pattern reminiscent of a transparent Photoshop layer until it catches up (typically, an instant later). I.e. iPhone prioritizes drag animation over the rendering of the contents; feel over appearance.

This was the compromise they had to make in 2007, but in 2023, with the same requirement to prioritize “feel”, they decided to keep appearance this time and trade-off price, complexity, and battery-life.

Approaching Infinity

Moore’s Law predicts that the number of transistors on a chip doubles every eighteen months. But, it has always been understood to be a statement about system capability as well. Speed, memory—we’re even getting advancements in power consumption now with Apple Silicon.

The doubling results in an exponential curve, but at the start, doubling a tiny number doesn’t get you much. My first computer had 4k of memory, but it was already an old model when I got it. By the next year, I had a Commodore 64 with 64k, then a Commodore 128(k) a few years later. My C64 was 1MHz in 1984. In 1992, my first work computer was a 16MHz 386 with 1MB of memory. Nice growth, but from a very low base, so still very underpowered in absolute numbers.

But, just like in personal finance, compounding eventually has enormous impact. It’s not just speed and power. We’re feeling it across all industries. Ubiquitous Software Copilots, Vision Pro, new vaccines, technology-enabled sports analytics, pervasive remote-work—all enabled by the last few doublings.

A doubling means that you have the equivalent impact of the entire industry back to the UNIVAC compressed into eighteen months. And the next 18 months doubles that.

I know this is nothing new. Ray Kurzweil’s Singularity described this in 2005. I’m more pointing out that here we are, and it seems like an inflection is happening where we’re doubling big numbers.

In my 30+ year career as a developer, I experienced a steady stream of big industry shifts. In the 90’s, it was web, then, in the 2000’s, it was web 2.0 and the advent of smart phones. The 2010’s were driven by XaaS (platform, infrastructure, etc) technologies. I could learn these as they happened. There wasn’t instantaneous adoption—you could keep up.

Now these waves are coming very fast, and I wonder if this is what it feels like when you start to approach infinity.

On The Vision Pro’s Price

Apple has clearly decided that low-price or even affordability was not important at all for the Vision Pro.

They have a history of this, and it has been a disaster. The Lisa was almost $30k in current dollars (listed at $10k in 1983). They tried to do better with the Mac, but it launched at the equivalent of $7k the next year. The cost for making this tradeoff in 1984 was the loss of almost the entire PC market to Microsoft and Intel and was an existential problem for Apple until they brought Steve Jobs back.

Apple could never shake the perception that they were overpriced. In 2007, Steve Jobs tried to frame Apple prices as competitive with comparable products and said that they don’t ship junk to compete with the low-end.

Our goal is to make products we are proud to sell and would recommend to our families and friends. And we want to do that at the lowest prices we can. […] What you’ll find is our products are usually not premium priced. […] The difference is we don’t offer stripped down, lousy products.

2007 was the same year that the iPhone came out at prices many times that as most cell phones. Even now, the iPhone competes well in a market with a big low-end (of arguably junk). The effect on Apple was quite different from the Lisa and Mac. The iPhone built Apple into a $3T company.

So, is Vision Pro like the Lisa and way over what the market will bear for the category, or is it like an iPhone that redefines the category around a high end?

My gut is that it’s like the watch, iPad, or AirPods. A great, multi-billion dollar business that will lead the category, but not something that drives the entire business.

Vision Pro Accessory Ideas

Since the failure mode of the Vision Pro is blindness, which can happen if the magnetic battery cable detaches or if you run out of power, it would have been nice if the headset had some onboard battery for a grace period.

There will be 3rd party batteries with more power. It would be good if they can also support swapping charged batteries in and out without losing power.

Here’s an idea for an accessory: A thin, disk-shaped battery that attaches to the headset magnetically (perhaps also with a strap) that you attach the included battery to. It has enough power for a 1 to 2 minute grace period, and keeps itself charged from the main battery. This is meant to help you swap batteries or just in case the cable detaches.

It’s important that it be light because it will be on the headset all the time. It should also look good.

The strap alone might be a good accessory. If there’s a way to make the magnetic cable more secure, I think I would want that.

The Failure Mode of the Vision Pro is Blindness

If the Vision Pro crashes, runs out of battery, or its magnetic battery cable detaches, you will be immediately plunged into darkness.

This means that the Vision Pro is really unsuited to be worn while moving. Walking around your house or job will probably be ok, but walking around outside isn’t. Luckily the headset looks too goofy to attempt that.

I had really hoped that this could work as a fitness device, but even anything above a jog on a treadmill seems dangerous. A slow walk would be fine. My initial reaction was that I would like to wear it on a rower. That would be ok too, because if it turns off, you aren’t going to fall down.

I am really afraid that someone will attempt to drive with this on. I hope that Apple adds a way to detect this and warn against it (or disable itself with a warning). It should certainly not approve apps that are meant to be used while driving.

If the Vision Pro is just a really good monitor, then this is not really a problem. But it does feel like pass-through displays that block your vision without power aren’t the future of AR (unless they can become transparent).

Is Vision Pro Just a Really Good Monitor?

I just read Ben Thompson’s take on the Vision Pro, which is admittedly a gushing, glowing, overly optimistic take. But …. he’s actually tried one, so I am taking it seriously. One worry I had was whether the displays actually matched the demo, and it does seem that they do.

His conclusion is that the Vision Pro might be in the same product category as a Mac and if that’s true, the $3499 price isn’t that bad. I absolutely could see a world where you use this instead of a laptop, but probably not on day one because it won’t have the apps I need as a developer.

Even so, compared to a laptop, the biggest downside is travel—I value how thin and light my MacBook Air is, and this is certainly not thin. I can’t easily stick it in a backpack. I also can’t see using this in a café or shared work space.

But, that had me thinking that maybe it’s not a laptop replacement, but an external monitor replacement. I have been eyeing the Studio Display at $1599 and also the new Dell 6K displays at $3200. If a Vision Pro is a better display than those, I don’t need it do much more.

It does make me think I should definitely not just upgrade my monitor yet.

WWDC 2023 Reaction

I watched the WWDC 2023 keynote, and here’s what I think as compared to my wishlist.

The new 15″ MacBook Air looks great, but it’s not for me. My main requirement is weight, and this is a half pound heavier. If I wanted to go up in size and weight, I’d consider the Pros. I skipped the M2 Air, and so I’ll just wait for the M3 and see how I feel about my M1 when that comes out. Right now, it seems fine.

There didn’t seem to be any more anti-theft help in iOS. I do appreciate the improvements to auto-correct.

I never even installed macOS Ventura. There’s nothing in Sonoma I care about, but I will probably install Ventura soon and consider upgrading in the Winter. My macOS upgrades are dictated by Xcode requirements. Surprisingly it hasn’t forced me to upgrade this year.

They said that tvOS would be better at knowing which audio device you want to be connected to, which would be great. As I mentioned in the wishlist, this is shockingly bad right now considering that literally every piece of hardware I use with my TV comes from Apple. If this works, it will be the only thing I unequivocally got from my wishlist.

Both Apple and I seem to agree that watchOS doesn’t need any more work. Not sure how the new design language affects Sprint-o-Mat—I’ll have to see as I use it. My favorite new feature is putting a waypoint on a hiking map for the last place you had cell coverage—I have needed that.

Ok, the headset.

I was expecting goggles, but (even though I think it was heavily implied in the rumors), I really thought they would be see-through with a projection. Instead, they are displays that show how cameras see the outside world, and external displays that show your eyes. I had discounted this rumor because it sounded insane, but it actually looks pretty good.

The problem is that the failure mode for the Vision Pro is blindness. Even when it’s working, it looks like it would randomly obstruct vision. Apps are completely opaque rectangles from what I could see. I didn’t see any demo of an application annotating reality. 3D objects could be placed in your scene, and I assume that apps will be able to pass through the camera feed, but that’s for games, not as an always-on feature.

This makes it impossible to use as a fitness device. I would not feel safe running with these on (and just forget about biking). I had hopes that I could make Sprint-o-Mat into a racing simulator (with pace runners on the track with you), but that feels unsafe to me. Maybe for track use only.

Also, my ideas for AR Apps that make the world into a playable game are not going to work with this device, and I really think it will not be used to navigate the real world. This is a stay-at-home entertainment device. It’s a very good one, but I was hoping for something that would be ok to use in real life.

If it stays at home, it does help alleviate the problem of always on cameras being creepy. They did address some of the issue by not letting 3rd party apps get this always-on feed. They also don’t let 3rd party apps get the eye-tracking data, which is also great.

The price being higher than the rumor was a surprise. I would love to try one, but it’s hard to justify $3499 for basically an awesome 1-person TV. For me, the giant workspace, immersive video conferencing, and cinema experience are very compelling. I don’t play games, but I bet there are going to be fitness games that I would enjoy (like a rowing simulator). I would only buy one if I think I’d develop an app for it.

WWDC 2023 Wishlist

WWDC23 is next week, so I put together a wishlist. I last did this in 2021, where I broke it down to watchOS, iOS, and developer tools. Whenever I write these wishlists, they are very centered on the work I am doing in the moment and what I need to help me. This year, I am doing less Apple device development, but I use the devices a lot and here are the things I am thinking about.

Headset

There are a lot of rumors that Apple will release an AR/VR headset. It seems like it will cost about $3,000, have an external battery pack, and come with a new framework.

This rumor has been around a while. For the Fall 2021 Apple event (when we really thought a headset would be coming), I wrote:

So, the main thing I’d hope for is something in AR. I’ve written about how I think AR could make apps more like games, and I do think that there’s space for a workout AR device. I would love to extend Sprint-o-Mat to make it feel like you’re in a race against the pace-runner. It would also be a good addition to Fitness+, which could extend to outdoor activities.

So, while I do have development ideas for an AR headset and would love to try one while running, it’s not worth $3,000 for me. If it’s a gaming device, I am not interested.

If the headset could somehow help me in my work (make me a more productive software engineer), then I would be more interested. GitHub Copilot seems to do a good enough job just in VSCode’s interface, but I could imagine being immersed in a VR world with even more heads up information. It would be interesting if there is some kind of meeting space VR, but since I mostly work alone, it would not be worth it to me.

I continue to be worried about headsets that have cameras. I think that it’s inherently creepy out in the real world and dangerous if camera access is extended to apps. I wrote about some ideas for Socially acceptable cameras in AR that I hope are in this headset if they are meant to be worn in public.

New Mac Hardware

If they release new hardware, I am in the market for a new MacBook Air. I love mine, but it’s an M1, so it only has 16GB. I wouldn’t mind expanding on that. I am holding out for a better camera. This seems impossible in the razor thin lid of the MacBook Air. I would be ok with some kind of camera array and a notch, if that’s what it took.

watchOS

My watch needs are driven by my app, Sprint-o-Mat. Aside from the AR features I mentioned above, I am pretty happy with where it is right now and don’t think there’s anything more I need in watchOS for it.

iOS/iPadOS

I hope that Apple adds more safeguards against device theft. One thing they could do is autolock the device if it moves out of connection with the watch. And they obviously need to do something about the fact that the device password gives too much access to iCloud and the Apple ID.

As for a system-wide feature, the biggest thing I miss on iOS is a clipboard manager. Even if they just kept a clipboard history and exposed an API, so that apps could fill the gap, I would be satisfied.

tvOS

I have an Apple TV, HomePods, and my wife and I both have AirPods (all made by Apple). But for some reason, the Apple TV insists on being connected to the TV audio by default. There seems to be no way to get to stay on the HomePods.

Socially acceptable cameras in AR

At some point, there’s going to be an AR device that looks exactly like glasses. I also hope that they don’t have cameras. If they have a camera, then they also need an indicator that it’s on. Walking around, for example, NYC with people wearing the glasses equivalent of 2000’s style iPod white earbuds is probably creepy. There’s no way around it if they also have lights and cameras.

But AR needs reality to augment, and a lot of reality is percieved visually. So, without a camera, these glasses will be a lot less useful. If we look at Apple’s AR features as a guide to how it feels about cameras, you can see that they are pro-camera. At WWDC 2021, they demoed a feature where you can orient yourself in a city by pointing your camera at the surrounding buildings. This would be very useful in a heads up display.

So, we will probably have the front-end of a camera to take in visual information. But, AR could still be very useful without ever producing a visual from the hardware.

One obvious (and currently available) representation is a depth map. Apple is testing out LIDAR on iPads and use a depth sensor on iPhones for face detection. The representation is a mesh, not an image, so it might be acceptable and is useful for a lot of AR.

Another thing they could do is pre-process the video feed into a given set of layers in a neural network. The first few layers in an image processor usually do some down-sampling and feature extraction. I’m not an expert, but if these things cannot be reversed back into the original image, they might be acceptable. In current feature detectors, you can retrieve some bits of images (see this), so there’s some work to do. But even if this is ok, it’s a big public education project to get it to be socially acceptable.

But Apple has shown some willingness to talk about its privacy protecting “provably cryptographically safe” technologies and open them up to third-parties for verification, so maybe they’d be willing to go this route to get a “camera” into their AR glasses.

AR Opens up Playability Possibilities

Yesterday, I pointed out that Pokémon Go was a playable workout app, where playable means that the game design ideas are driving the app, as opposed to gamification, where it’s slapped on.

I was thinking more about this and realized how many apps may be turned into games via AR. Again, not with badges, but by making playing the point of the app.

Not just for workout apps, which I think will drive lots of AR games.

But for something really different: consider an app that wanted to help you to eat healthier by guiding you while grocery shopping.

We all know what that app would look like: a list of grocery items, maybe color coded with “healthiness”. You tap tap tap when you buy eggplant, spinach, and blueberries (“You got an anti-oxidant badge!”). You lose points when you scan that box of mini-donuts.

The AR version has zombies in the cookie aisle.

Interestingly, the produce section seems to have no zombies—better scavenge there. Cookie boxes emit a piercing sound when they are in your cart, drawing the dead towards you. Leave them behind to draw them away.

An art appreciation app could help me get more out of a museum by telling me a little about what I might see and then making a ad-hoc quiz show as I take in the art. Or putting me in a pub quiz later based on what I looked at.

AR could turn a tour guide app into a spy hunt game. Follow that lady in the black trench coat and see what she’s up to—she’s boarding the boat to the Statue of Liberty! It’s practically the plot of North by Northwest.

These would be games, which are fun, not gamification.