Pick Event

In 3D applications, it is often necessary to click on objects in the scene. The engine supports ray bounding-box picking and frame buffer picking.

The supported pick event types:

Name	Explanation
PICK_OVER	Triggered once when the touch point enters the range of the collider
PICK_OUT	Triggered once when the touch point leaves the range of the collider
PICK_CLICK	Triggered once on release when the touch point is pressed and released within the range of the collider
PICK_MOVE	Triggered when the touch point moves within the range of the collider
PICK_UP	Triggered once when the touch point is released within the range of the collider
PICK_DOWN	Triggered once when the touch point is pressed within the range of the collider

Pick Detection

The engine uniformly listens to mouse events, performs pick detection on all clickable Object3D in the scene, and triggers the corresponding events. Users can add a ColliderComponent component to an object to mark it as clickable, and listen to the corresponding PointerEvent3D event.
The engine provides a unified wrapper for both pick types, which can be switched through simple configuration.

// Picking and the pick type need to be enabled in the configuration before the engine starts
let engine = await Engine3D.init({
    setting: {
        pick: {
            enable: true,
            // bound: bounding-box picking, pixel: frame buffer picking
            mode: `bound`, // or 'pixel'
        }
    }
})
// Pick detection depends on the Collider component
let obj = Object3D();
obj.addComponent(ColliderComponent);

// Add a PickEvent listener on the node; the corresponding event can be obtained in the callback function
obj.addEventListener(PointerEvent3D.PICK_CLICK, onPick, this);

// Or listen to click events of all objects through view.pickFire
view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, onPick, this);

// Get the event information in the callback function
function onPick(e: PointerEvent3D) {
    e.target // the clicked Object
    e.data.worldPos // the clicked position in world coordinates
    e.data.worldNormal // the normal at the clicked position in world coordinates
    ...
}

Bounding-Box Picking

Ray bounding-box is a commonly used CPU-based picking method. It needs to calculate the intersection between the ColliderShape of the ColliderComponent component and the mouse ray. It performs well in scenes with a small number of objects, but its precision is poor, because a bounding box often cannot accurately represent the real shape of an object.

Currently, the basic ColliderShape provided by the engine includes BoxColliderShape, SphereColliderShape, and CapsuleColliderShape. You can also construct a MeshColliderShape from the object's own Mesh shape.

import {Object3D, Collider, BoxColliderShape, Vector3} from '@orillusion/core';

let box = new Object3D();
let mr = box.addComponent(MeshRenderer);
// Set the box geometry
mr.geometry = new BoxGeometry(1,1,1);
// Add collider detection
let collider = box.addComponent(ColliderComponent);
// In bound mode, the collider shape and size need to be set manually
// The picking precision depends on how well box.geometry matches collider.shape
collider.shape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(1, 1, 1));

• The box on the left uses a BoxColliderShape of the same shape for detection, with good precision
• The sphere in the middle uses a BoxColliderShape, but the detection area is larger than the actual model, resulting in lower precision
• The sphere on the right uses a MeshColliderShape, which can perfectly fit all vertices of the model with the highest precision, but consumes more performance for collision detection. It is not recommended for complex objects

WebGPU is not supported in your browser
Please upgrade to latest Chrome/Edge

<

import { Engine3D, Scene3D, Vector3, Object3D, AtmosphericComponent, Camera3D, View3D, LitMaterial, MeshRenderer, BoxColliderShape, ColliderComponent, BoxGeometry, Color, PointerEvent3D, SphereGeometry, DirectLight, BoundingBox, MeshColliderShape } from '@orillusion/core';
import { Graphic3D } from '@orillusion/graphic';

class TouchDemo {
    scene: Scene3D;
    cameraObj: Object3D;
    camera: Camera3D;
    graphic3D: Graphic3D;

    constructor() {}

    async run() {
        console.log('start demo');
        // enable pick and use bound mode
        let engine = await Engine3D.init({
            setting: {
                pick: {
                    enable: true,
                    mode: `bound`
                }
            }
        });

        this.scene = new Scene3D();
        this.scene.addComponent(AtmosphericComponent);
        this.cameraObj = new Object3D();
        this.camera = this.cameraObj.addComponent(Camera3D);
        this.scene.addChild(this.cameraObj);
        this.camera.lookAt(new Vector3(0, 0, 10), new Vector3(0, 0, 0));
        this.camera.perspective(60, engine.aspect, 1, 10000.0);

        // add a base light
        let lightObj = new Object3D();
        lightObj.addComponent(DirectLight);
        this.scene.addChild(lightObj);

        let box = this.createBox(-4, 0, 0);
        let sphere = this.createSphere(0, 0, 0);
        let sphere2 = this.createSphere2(4, 0, 0);

        this.graphic3D = new Graphic3D();
        this.scene.addChild(this.graphic3D);
        this.graphic3D.drawBoundingBox(box.instanceID, box.bound as BoundingBox, Color.COLOR_GREEN);
        this.graphic3D.drawBoundingBox(sphere.instanceID, sphere.bound as BoundingBox, Color.COLOR_GREEN);
        this.graphic3D.drawMeshWireframe(sphere2.instanceID, new SphereGeometry(1.01, 8, 8), sphere2.transform, Color.COLOR_GREEN)
        
        let view = new View3D();
        view.scene = this.scene;
        view.camera = this.camera;
        // start render
        engine.startRenderView(view);

        // listen all pick_click events
        view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, this.onPick, this);
    }

    createBox(x: number, y: number, z: number) {
        let boxObj = new Object3D();
        boxObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: BoxColliderShape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(size, size, size));
        // add a box collider
        let collider = boxObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = boxObj.addComponent(MeshRenderer);
        mr.geometry = new BoxGeometry(size, size, size);
        mr.material = new LitMaterial();
        this.scene.addChild(boxObj);
        return boxObj;
    }

    createSphere(x: number, y: number, z: number) {
        let sphereObj = new Object3D();
        sphereObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: BoxColliderShape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(size, size, size));
        // add a box collider
        let collider = sphereObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = sphereObj.addComponent(MeshRenderer);
        mr.geometry = new SphereGeometry(size / 2, 8, 8);
        mr.material = new LitMaterial();
        this.scene.addChild(sphereObj);
        return sphereObj;
    }

    createSphere2(x: number, y: number, z: number) {
        let sphereObj = new Object3D();
        sphereObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: MeshColliderShape = new MeshColliderShape()
        
        // add a box collider
        let collider = sphereObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = sphereObj.addComponent(MeshRenderer);
        mr.geometry = shape.mesh = new SphereGeometry(size / 2, 8, 8);
        mr.material = new LitMaterial();
        this.scene.addChild(sphereObj);
        return sphereObj;
    }

    onPick(e: PointerEvent3D) {
        console.log('onClick:', e);
        let mr: MeshRenderer = e.target.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }
}
new TouchDemo().run();

Frame Buffer Picking

Unlike the bound mode, frame buffer picking in pixel mode utilizes the pixel detection of the GPU, consuming almost no CPU performance. It can ignore the number and shape complexity of interactive objects in the scene and supports all touch events. When the scene models have complex shapes or there are a large number of objects, we recommend using the pixel mode for pick detection.

WebGPU is not supported in your browser
Please upgrade to latest Chrome/Edge

<

import { AtmosphericComponent, BoxColliderShape, Camera3D, CameraUtil, ColliderComponent, Color, View3D, DirectLight, Engine3D, LitMaterial, HoverCameraController, KelvinUtil, MeshRenderer, Object3D, PointerEvent3D, Scene3D, SphereGeometry, Vector3 } from '@orillusion/core';

class Sample_MousePick {
    lightObj: Object3D;
    cameraObj: Camera3D;
    scene: Scene3D;
    hover: HoverCameraController;
    engine: Engine3D;

    constructor() {}

    async run() {
        // enable pick and use pixel mode
        this.engine = await Engine3D.init({
            setting: {
                pick: {
                    enable: true,
                    mode: `pixel`,
                },
            },
        });

        this.scene = new Scene3D();
        this.scene.addComponent(AtmosphericComponent);
        let camera = CameraUtil.createCamera3DObject(this.scene);
        camera.perspective(60, this.engine.aspect, 1, 5000.0);

        this.hover = camera.object3D.addComponent(HoverCameraController);
        this.hover.setCamera(-30, -15, 120);

        let wukong = await this.engine.res.loadGltf('https://cdn.orillusion.com/gltfs/wukong/wukong.gltf');
        wukong.transform.y = 30;
        wukong.transform.scaleX = 20;
        wukong.transform.scaleY = 20;
        wukong.transform.scaleZ = 20;
        wukong.forChild((node) => {
            if (node.hasComponent(MeshRenderer)) {
                node.addComponent(ColliderComponent);
            }
        });
        this.scene.addChild(wukong);

        this.initPickObject(this.scene);

        let view = new View3D();
        view.scene = this.scene;
        view.camera = camera;
        // start render
        this.engine.startRenderView(view);

        // listen all mouse events
        view.pickFire.addEventListener(PointerEvent3D.PICK_UP, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_DOWN, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_OVER, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_OUT, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_MOVE, this.onPick, this);
    }

    private initPickObject(scene: Scene3D): void {
        /******** light *******/
        {
            this.lightObj = new Object3D();
            this.lightObj.rotationX = 125;
            this.lightObj.rotationY = 0;
            this.lightObj.rotationZ = 40;
            let lc = this.lightObj.addComponent(DirectLight);
            lc.lightColor = KelvinUtil.color_temperature_to_rgb(5355);
            lc.castShadow = true;
            lc.intensity = 5;
            scene.addChild(this.lightObj);
        }

        let size: number = 9;
        let shape = new BoxColliderShape();
        shape.setFromCenterAndSize(new Vector3(), new Vector3(size, size, size));

        let geometry = new SphereGeometry(size / 2, 20, 20);
        for (let i = 0; i < 10; i++) {
            let obj = new Object3D();
            obj.name = 'sphere ' + i;
            scene.addChild(obj);
            obj.x = (i - 5) * 10;

            let mat = new LitMaterial();
            mat.emissiveMap = this.engine.res.grayTexture;
            mat.emissiveIntensity = 0.0;

            let renderer = obj.addComponent(MeshRenderer);
            renderer.geometry = geometry;
            renderer.material = mat;
            obj.addComponent(ColliderComponent);
        }
    }

    private onPick(e: PointerEvent3D) {
        console.log(e.type, e.target.name, e.data);
        if(e.type !== 'onPickMove'){
            let obj = e.target as Object3D;
            let mr = obj.getComponent(MeshRenderer);
            mr.material.baseColor = Color.random();
        }
    }
}
new Sample_MousePick().run();