Android XR to Enable Phone-like Camera Access

Passthrough camera access is stirring up quite a conversation in the XR community these days. We already know the stances of big players like Meta, Apple, and Pico, which naturally leads to curiosity about how Google intends to handle this with Android XR. After having a chat directly with Google, I can confirm they’ll be rolling out a solution reminiscent of what’s available on phones. Stick around to find out more!

The Camera Access Dilemma

First, let’s backtrack a bit for those who might not be fully in the loop. As you might already know, the latest wave of standalone VR headsets actually operate as MR headsets, providing users with an RGB passthrough view of their surroundings thanks to front cameras. This functionality is critical for experiencing those fantastic mixed reality apps like Cubism, Starship Home, and Pencil.

For these headsets, the OS displays passthrough by utilizing the camera frames. Developers like me are eager to access these frames, enabling us to leverage AI and computer vision to enhance user experiences in breathtaking ways. I’ve always championed camera access as a gateway to true mixed reality by making apps contextually aware of their environment. I managed to prototype an interior design app using a workaround on Quest, which wouldn’t have been possible without camera access.

However, there’s a significant counterpoint: privacy concerns. If camera access falls into the wrong hands, a rogue developer might exploit it to capture sensitive images without the user’s consent, scraping personal data like ID documents and credit card details lying on a table. And that’s not even touching on potential misuse of images of faces or bodies.

This situation creates a tricky balancing act: safeguarding user privacy while unlocking the vast potential of mixed reality.

The Behavior of the XR Companies

Initially, companies offered unrestricted camera access. If you’ve been following my journey, you might recall the experiments my team and I conducted with camera textures on the Vive Focus around 2019, dabbling in diminished reality and marker tracking, among other things.

However, as mixed reality gained traction, firms grew cautious and began restricting camera access due to privacy worries. Players like Meta, Pico, HTC, and Apple all took steps to lock down this feature.

The standardization of such restrictions led the XR developer community to advocate for camera access, emphasizing its potential while maintaining transparency with users. Trailblazers like Cix Liv, Michael Gschwandtner, and myself have been vocal about requesting access in a transparent manner that aligns with user consent.

This push resulted in companies like Meta announcing a "Passthrough API" rollout earlier this year. But the question remains—what will Google do with Android XR?

Android XR to Treat the Headset Like a Phone

Android powers most smartphones globally and allows app developers to request camera access upon user approval. By specifying a camera ID—usually 0 for the rear camera—developers can obtain the necessary frames to work with.

Google aims for Android XR to seamlessly integrate with Android apps, which it achieves by adhering to similar mechanics on its new operating system. Despite many rumors, I got an official statement from Google confirming this. To clear things up, here’s an excerpt from their response about camera access on Android XR:

Similar to any Android app, a developer can use existing camera frames with user permission for XR…

In summary, just as with phones, developers can request main world-facing or selfie-camera streams through standard API calls, adapting functionalities across platforms effectively.

A highlight here is that while the standard front-facing and rear camera streams are supported, the inward-facing camera collects data used by avatar services—mirroring how Vision Pro operates.

Android developers can rejoice since standard camera management classes like CameraX are functional on XR-based headsets, heralding more opportunities to harness these streams for innovation.

Yet, while the selfie-camera offers genuine access, the rear camera stream represents a virtual avatar akin to Vision Pro. This mimicry ensures Android XR’s consistency with phone-like behavior across devices.

A Little Caveat About Android XR

It’s worth noting that Android XR is still under development, with no official headset releases yet. So, some details might evolve before its final launch. Though unlikely, keeping this in mind adds an extra layer of consideration.

The Opening Up of Camera Access

With Google and Meta both embracing camera access, other industry leaders might soon follow. As we look towards 2025, the year promises to unlock exciting new possibilities in mixed reality—and I can’t wait to see what the developer community will innovate!

(Header image courtesy of Samsung)

Please note that this blog contains advertisements and affiliate links that help keep this ship sailing. Clicking an affiliate link earns me a small commission without costing you a dime. For full transparency, feel free to check out the detailed disclosure.

Share this with fellow innovators!