RealityKit visionOS Developer#

Description and Goals#

This skill provides comprehensive guidance for implementing RealityKit-based spatial experiences on visionOS. RealityKit uses an Entity Component System (ECS) architecture where entities are lightweight containers, behavior comes from components, and systems drive per-frame updates.

Goals#

• Enable developers to build immersive 3D experiences on visionOS using RealityKit
• Provide clear guidance on when to use each component and system
• Help developers understand ECS patterns and best practices
• Support debugging and optimization of RealityKit scenes
• Ensure proper integration with SwiftUI via RealityView

What This Skill Should Do#

When implementing RealityKit features on visionOS, this skill should:

1. Guide component selection - Help you choose the right components for rendering, interaction, physics, audio, and animation needs
2. Provide system implementation patterns - Show how to create custom systems for continuous behavior
3. Offer code examples - Demonstrate common patterns like async asset loading, interactive entities, and custom systems
4. Highlight best practices - Emphasize proper async loading, component registration, and performance considerations
5. Warn about pitfalls - Identify common mistakes like using ARView on visionOS or blocking the main actor

Load the appropriate component or system reference file from the tables below for detailed usage, code examples, and best practices.

Information About the Skill#

Core Concepts#

Entities and Components

• Entities are lightweight containers; behavior comes from components.
• Prefer composition over inheritance and use custom Component + Codable when you need per-entity state.
• Register custom components once with Component.registerComponent() before use.
• Mutate entities from documented RealityKit entry points such as RealityView closures, event handlers, and systems.

RealityView and Attachments

• Use RealityView to bridge SwiftUI and RealityKit.
• Load assets with Entity(named:) or Entity(contentsOf:) asynchronously and handle errors.
• Use the RealityView attachments closure when you want SwiftUI attachment entities defined alongside the view, and use ViewAttachmentComponent when a component-based attachment is the better fit.

Systems and Queries

• Use a custom System for continuous, per-frame behavior.
• Query entities with EntityQuery + QueryPredicate and process them in update(context:).
• Use SystemDependency to control update order when multiple systems interact.

Components Reference#

Use this table to decide which component reference file to load when implementing RealityKit features:

Rendering and Appearance

ComponentWhen to UseModelComponentWhen rendering 3D geometry with meshes and materials on entities.ModelSortGroupComponentWhen experiencing depth fighting (z-fighting) issues with overlapping geometry or need to control draw order.OpacityComponentWhen creating fade effects, making entities semi-transparent, or implementing visibility transitions.AdaptiveResolutionComponentWhen optimizing performance in large scenes by reducing render quality for distant objects.ModelDebugOptionsComponentWhen debugging rendering issues, visualizing model geometry, or inspecting bounding boxes during development.MeshInstancesComponentWhen rendering many copies of the same mesh efficiently (trees, crowds, particle-like objects).BlendShapeWeightsComponentWhen implementing facial animation, character expressions, or morphing mesh deformations.

User Interaction

ComponentWhen to UseInputTargetComponentWhen making entities interactive (tappable, draggable) or handling user input events.ManipulationComponentWhen implementing built-in drag, rotate, and scale interactions with hand gestures or trackpad.GestureComponentWhen implementing custom gesture recognition beyond what ManipulationComponent provides.HoverEffectComponentWhen providing visual feedback when users look at or hover over interactive entities.AccessibilityComponentWhen making entities accessible to screen readers, VoiceOver, or other assistive technologies.BillboardComponentWhen creating 2D sprites, text labels, or UI elements that should always face the viewer.

Anchoring and Spatial

ComponentWhen to UseAnchoringComponentWhen anchoring virtual content to detected planes, tracked images, hand locations, or world targets.ARKitAnchorComponentWhen accessing the underlying ARKit anchor data for an anchored entity.SceneUnderstandingComponentWhen accessing scene understanding data like detected objects or room reconstruction.DockingRegionComponentWhen defining regions where content can automatically dock or snap into place.ReferenceComponentWhen implementing lazy loading of external entity assets or referencing entities in other files.AttachedTransformComponentWhen attaching an entity's transform to another entity for hierarchical positioning.

Cameras

ComponentWhen to UsePerspectiveCameraComponentWhen configuring a perspective camera with depth and field of view for 3D scenes.OrthographicCameraComponentWhen configuring an orthographic camera without perspective distortion for 2D-like views.ProjectiveTransformCameraComponentWhen implementing custom camera projection transforms for specialized rendering needs.

Lighting and Shadows

ComponentWhen to UsePointLightComponentWhen adding an omnidirectional point light that radiates in all directions from a position.DirectionalLightComponentWhen adding a directional light with parallel rays (like sunlight) for consistent scene lighting.SpotLightComponentWhen adding a cone-shaped spotlight for focused, directional lighting effects.ImageBasedLightComponentWhen applying environment lighting from HDR textures for realistic reflections and ambient lighting.ImageBasedLightReceiverComponentWhen enabling entities to receive and respond to image-based lighting in the scene.GroundingShadowComponentWhen adding grounding shadows to visually anchor floating content to surfaces.DynamicLightShadowComponentWhen enabling real-time dynamic shadows cast by light sources onto entities.EnvironmentLightingConfigurationComponentWhen configuring environment lighting behavior, intensity, or blending modes.VirtualEnvironmentProbeComponentWhen implementing reflection probes for accurate reflections in virtual environments.

Audio

ComponentWhen to UseSpatialAudioComponentWhen playing 3D positioned audio that changes based on listener position and orientation.AmbientAudioComponentWhen playing non-directional ambient audio that doesn't change with listener position.ChannelAudioComponentWhen playing channel-based audio content (stereo, surround, etc.) without spatialization.AudioLibraryComponentWhen storing and managing multiple audio resources for reuse across entities.ReverbComponentWhen applying reverb effects to an entity's audio for spatial acoustic simulation.AudioMixGroupsComponentWhen grouping audio sources for centralized mixing control and volume management.

Animation and Character

ComponentWhen to UseAnimationLibraryComponentWhen storing multiple animations (idle, walk, run) on a single entity for character animation.CharacterControllerComponentWhen implementing character movement with physics, collision, and ground detection.CharacterControllerStateComponentWhen storing runtime state (velocity, grounded status) for a character controller.SkeletalPosesComponentWhen providing skeletal pose data for skeletal animation and bone transformations.IKComponentWhen implementing inverse kinematics for procedural animation (e.g., reaching, pointing).BodyTrackingComponentWhen integrating ARKit body tracking data to animate entities based on real-world body poses.

Physics and Collision

ComponentWhen to UseCollisionComponentWhen defining collision shapes for hit testing, raycasting, or physics interactions.PhysicsBodyComponentWhen adding physical behavior (mass, gravity, forces) to entities for physics simulation.PhysicsMotionComponentWhen controlling linear and angular velocity of physics bodies programmatically.PhysicsSimulationComponentWhen configuring global physics simulation parameters like gravity or timestep.ParticleEmitterComponentWhen emitting particle effects (smoke, sparks, debris) from an entity position.ForceEffectComponentWhen applying force fields (gravity wells, explosions) that affect multiple physics bodies.PhysicsJointsComponentWhen creating joints (hinges, springs) between physics bodies for articulated structures.GeometricPinsComponentWhen defining geometric attachment points for connecting entities at specific locations.

Portals and Environments

ComponentWhen to UsePortalComponentWhen creating portals that render a separate world or scene through an opening.WorldComponentWhen designating an entity hierarchy as a separate renderable world for portal rendering.PortalCrossingComponentWhen controlling behavior (teleportation, scene switching) when entities cross portal boundaries.EnvironmentBlendingComponentWhen blending virtual content with the real environment for mixed reality experiences.

Presentation and UI

ComponentWhen to UseViewAttachmentComponentWhen embedding SwiftUI views into 3D space for interactive UI elements or labels.PresentationComponentWhen presenting SwiftUI modals, sheets, or system UI from an entity interaction.TextComponentWhen rendering 3D text directly on entities without using SwiftUI views.ImagePresentationComponentWhen displaying images or textures on entities in 3D space.VideoPlayerComponentWhen playing video content on entity surfaces using AVPlayer.

Networking and Sync

ComponentWhen to UseSynchronizationComponentWhen synchronizing entity state, transforms, and components across networked multiplayer sessions.TransientComponentWhen marking entities as temporary, non-persistent, and excluded from network synchronization.

Systems Reference#

Use this reference when implementing custom ECS behavior:

System/APIWhen to UseSystem and Component CreationWhen creating custom systems for continuous, per-frame behavior or custom components for per-entity state.

Implementation Patterns#

RealityView Async Load

swift

RealityView { content in
    do {
        let entity = try await Entity(named: "Scene")
        content.add(entity)
    } catch {
        print("Failed to load entity: \(error)")
    }
}

Interactive Entity Setup

swift

let entity = ModelEntity(mesh: .generateBox(size: 0.1))
entity.components.set(CollisionComponent(shapes: [.generateBox(size: [0.1, 0.1, 0.1])]))
entity.components.set(InputTargetComponent())
entity.components.set(ManipulationComponent())

Custom System Skeleton

swift

import RealityKit

struct SpinComponent: Component, Codable {
    var speed: Float
}

struct SpinSystem: System {
    static let query = EntityQuery(where: .has(SpinComponent.self))

    init(scene: Scene) {}

    func update(context: SceneUpdateContext) {
        for entity in context.entities(matching: Self.query, updatingSystemWhen: .rendering) {
            guard let spin = entity.components[SpinComponent.self] else { continue }
            entity.transform.rotation *= simd_quatf(angle: spin.speed * Float(context.deltaTime), axis: [0, 1, 0])
        }
    }
}

SpinSystem.registerSystem()

Pitfalls and Checks#

• Always load assets asynchronously; avoid blocking the main actor.
• Avoid ARView on visionOS; use RealityView.
• Add CollisionComponent + InputTargetComponent for draggable or tappable entities.
• Use the RealityView update closure for state-driven content updates, and prefer a custom System for continuous per-frame behavior that spans many entities.
• Built-in mesh generation supports more than the basic primitives, including text and custom mesh content through MeshResource APIs.