ARKit visionOS Developer#

Description and Goals#

This skill provides comprehensive guidance for implementing ARKit-powered features on visionOS. ARKit on visionOS uses ARKitSession with data providers to access world tracking, hand tracking, plane detection, scene reconstruction, and other spatial data, which can then be bridged into RealityKit content.

Goals#

• Enable developers to set up and manage ARKitSession on visionOS
• Guide proper authorization handling for ARKit data providers
• Help developers choose and configure appropriate data providers
• Support anchor processing and RealityKit integration
• Ensure proper lifecycle management of ARKit sessions

What This Skill Should Do#

When implementing ARKit features on visionOS, this skill should:

1. Guide ARKitSession setup - Help you create and manage long-lived ARKitSession instances
2. Handle authorization - Show how to request and check authorization for required data types
3. Select data providers - Help you choose the right providers (world tracking, hand tracking, plane detection, etc.)
4. Process anchors - Demonstrate how to consume anchor updates and map them to RealityKit entities
5. Manage lifecycle - Ensure proper session start/stop and task cancellation
6. Bridge to RealityKit - Show how to integrate ARKit anchors with RealityKit content

Load the appropriate reference file from the tables below for detailed usage, code examples, and best practices.

Quick Start Workflow#

1. Add NSWorldSensingUsageDescription, NSHandsTrackingUsageDescription, and NSMainCameraUsageDescription to Info.plist as needed for the providers you use.
2. Use the presentation style required by the selected providers. Some providers require immersive space, while others have more specific rules.
3. Create a long-lived ARKitSession and the data providers you need.
4. Request authorization for provider-required data types before running the session.
5. Run the session with your providers and observe anchorUpdates streams.
6. Map anchors to RealityKit entities and keep state in a model layer.
7. Observe ARKitSession.events for authorization changes and errors.
8. Stop the session and cancel tasks when leaving the immersive space.

Information About the Skill#

Core Concepts#

ARKitSession Lifecycle

• Keep a strong reference to the session; call run(_:) with providers, stop on teardown.
• Sessions stop automatically on deinit, so maintain references throughout the immersive experience.

Authorization

• Use requestAuthorization(for:) or queryAuthorization(for:) and handle denied states gracefully.
• Request authorization before running the session with providers that require it.

Data Providers

• Choose providers for world tracking, plane detection, scene reconstruction, and hand tracking based on the feature set.
• Providers expose anchorUpdates streams that you consume to process anchors.

Anchors and Updates

• Consume provider anchorUpdates and reconcile added, updated, and removed anchors.
• Normalize anchor IDs to your own state model for reliable entity updates.

RealityKit Bridge

• Use ARKitAnchorComponent to inspect backing ARKit data on entities when needed.
• Treat ARKit streams as authoritative and keep rendering logic in RealityKit.

Implementation Patterns#

• Prefer one session per immersive experience and reuse providers when possible.
• Normalize anchor IDs to your own state model for reliable entity updates.
• Treat ARKit streams as authoritative and keep rendering logic in RealityKit.

Provider References#

ProviderWhen to UseWorldTrackingProviderWhen tracking device position and orientation in 3D space.HandTrackingProviderWhen tracking hand poses and gestures for interaction.PlaneDetectionProviderWhen detecting horizontal and vertical surfaces (floors, walls, tables).SceneReconstructionProviderWhen creating detailed 3D mesh reconstructions of the environment.ImageTrackingProviderWhen tracking known 2D images in the environment.ObjectTrackingProviderWhen tracking 3D objects in the environment.RoomTrackingProviderWhen tracking room boundaries and room-scale experiences.AccessoryTrackingProviderWhen tracking Apple Vision Pro accessories.BarcodeDetectionProviderWhen detecting and reading barcodes in the environment.CameraFrameProviderWhen accessing raw camera frames for custom processing.CameraRegionProviderWhen accessing camera frames from specific regions.EnvironmentLightEstimationProviderWhen estimating ambient lighting conditions.SharedCoordinateSpaceProviderWhen sharing coordinate spaces across multiple sessions.StereoPropertiesProviderWhen accessing stereo camera properties.

General ARKit Patterns#

ReferenceWhen to UseREFERENCE.mdWhen implementing ARKit session setup, authorization, and general provider patterns.

Pitfalls and Checks#

• In SwiftUI-first visionOS apps, prefer RealityView for presentation and ARKitSession for tracking data; use ARView only when you specifically need its UIKit/AppKit-style view APIs.
• Do not assume every ARKit provider has the same presentation requirements; check the provider-specific guidance before choosing Shared Space, a volumetric window, or an immersive space.
• Do not block the main actor while awaiting provider updates.
• Do not drop session references; ARKit stops sessions on deinit.