Pick Event

In 3D applications, it is often necessary to click on objects in the scene. The engine supports ray box picking and frame buffer picking.

The supported pick events are:

Name	Explanation
PICK_OVER	Triggered once when the touch point enters the collision body range
PICK_OUT	Triggered once when the touch point leaves the collision body range
PICK_CLICK	Triggered once when the touch point is pressed and released in the collision body range
PICK_MOVE	Triggered when the touch point moves within the collision body range
PICK_UP	Triggered once when the touch point is released within the collision body range
PICK_DOWN	Triggered once when the touch point is pressed within the collision body range

Listening to Events

Picking events depend on the ColliderComponent component, and we can directly listen for PointerEvent3D events on the Object3D . The engine has unified two types of picking methods that can be switched through simple configuration.

//Pick and pick type need to be configured before the engine starts
Engine3D.setting.pick.enable = true;
// Bound: ray box picking, pixel: frame buffer picking
Engine3D.setting.pick.mode = `bound`; // or 'pixel'

await Engine3D.init()
// Picking detection depends on the Collider component
let obj = Object3D();
obj.addComponent(ColliderComponent);

// Add a PickEvent event listener to the node, where the corresponding event can be obtained in the callback function
obj.addEventListener(PointerEvent3D.PICK_CLICK, onPick, this);

// Or listen to all object click events through view.pickFire
view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, onPick, this);

//Get event information in the callback function
function onPick(e: PointerEvent3D) {
    ...
}

Ray Box Picking

Ray box picking is a commonly used CPU-based picking method. It needs to calculate the intersection of the ColliderComponent component's shape and the mouse ray. It performs well in scenes with few objects, but has poor accuracy because the bounding box often cannot accurately represent the true shape of the object.

In order to maintain cpu performance, currently, ray box picking only supports active pick click picking and does not support over/hover state picking.

import {Object3D, Collider, BoxColliderShape, Vector3} from '@orillusion/core';

let box = new Object3D();
let mr = box.addComponent(MeshRenderer);
// Set the box geometry
mr.geometry = new BoxGeometry(1,1,1);
// Add collision box detection
let collider = box.addComponent(ColliderComponent);
// For the bound mode, the style and size of the collision box need to be set manually
// The picking accuracy depends on the match between box.geometry and collider.shape
collider.shape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(1, 1, 1));

The box on the left uses BoxColliderShape with the same shape for detection, which has better accuracy.
The sphere on the right also uses BoxColliderShape，, but the clickable area is larger than the actual model, resulting in lower accuracy.

WebGPU is not supported in your browser
Please upgrade to latest Chrome/Edge

import { Engine3D, Scene3D, Vector3, Object3D, AtmosphericComponent, Camera3D, View3D, LitMaterial, MeshRenderer, BoxColliderShape, ColliderComponent, BoxGeometry, ComponentBase, Color, PointerEvent3D, SphereGeometry, DirectLight } from '@orillusion/core';

class TouchDemo {
    scene: Scene3D;

    cameraObj: Object3D;

    camera: Camera3D;

    constructor() {}

    async run() {
        console.log('start demo');
        // enable pick and use bound mode
        Engine3D.setting.pick.enable = true;
        Engine3D.setting.pick.mode = `bound`;

        await Engine3D.init();

        this.scene = new Scene3D();
        this.scene.addComponent(AtmosphericComponent);
        this.cameraObj = new Object3D();
        this.camera = this.cameraObj.addComponent(Camera3D);
        this.scene.addChild(this.cameraObj);
        this.camera.lookAt(new Vector3(0, 0, 10), new Vector3(0, 0, 0));
        this.camera.perspective(60, Engine3D.aspect, 1, 10000.0);

        // add a base light
        let lightObj = new Object3D();
        lightObj.addComponent(DirectLight);
        this.scene.addChild(lightObj);

        let box = this.createBox(-2, 0, 0);
        let sphere = this.createSphere(2, 0, 0);

        let view = new View3D();
        view.scene = this.scene;
        view.camera = this.camera;
        // start render
        Engine3D.startRenderView(view);

        // listen all pick_click events
        view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, this.onPick, this);
    }

    createBox(x: number, y: number, z: number) {
        let boxObj = new Object3D();
        boxObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: BoxColliderShape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(size, size, size));
        // add a box collider
        let collider = boxObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = boxObj.addComponent(MeshRenderer);
        mr.geometry = new BoxGeometry(size, size, size);
        mr.material = new LitMaterial();
        this.scene.addChild(boxObj);
        return boxObj;
    }

    createSphere(x: number, y: number, z: number) {
        let sphereObj = new Object3D();
        sphereObj.transform.localPosition = new Vector3(x, y, z);

        let size: number = 2;
        let shape: BoxColliderShape = new BoxColliderShape().setFromCenterAndSize(new Vector3(0, 0, 0), new Vector3(size, size, size));
        // add a box collider
        let collider = sphereObj.addComponent(ColliderComponent);
        collider.shape = shape;
        let mr: MeshRenderer = sphereObj.addComponent(MeshRenderer);
        mr.geometry = new SphereGeometry(size / 2, 20, 20);
        mr.material = new LitMaterial();
        this.scene.addChild(sphereObj);
        return sphereObj;
    }

    onPick(e: PointerEvent3D) {
        console.log('onClick:', e);
        let mr: MeshRenderer = e.target.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }
}
new TouchDemo().run();

Frame Buffer Picking

Unlike the pixel mode and the bound mode, Frame Buffer Picking utilizes the pixel detection of the GPU , which consumes almost no CPU performance and can ignore the number and complexity of interactive objects in the scene, supporting all touch events. When the shape of the scene model is complex or there are a large number of objects, we recommend using the pixel mode for picking detection.

WebGPU is not supported in your browser
Please upgrade to latest Chrome/Edge

import { AtmosphericComponent, BoxColliderShape, Camera3D, CameraUtil, ColliderComponent, Color, View3D, DirectLight, Engine3D, LitMaterial, HoverCameraController, KelvinUtil, MeshRenderer, Object3D, PointerEvent3D, Scene3D, SphereGeometry, Vector3 } from '@orillusion/core';

class Sample_MousePick {
    lightObj: Object3D;
    cameraObj: Camera3D;
    scene: Scene3D;
    hover: HoverCameraController;

    constructor() {}

    async run() {
        // enable pick and use pixel mode
        Engine3D.setting.pick.enable = true;
        Engine3D.setting.pick.mode = `pixel`;

        await Engine3D.init({});

        this.scene = new Scene3D();
        this.scene.addComponent(AtmosphericComponent);
        let camera = CameraUtil.createCamera3DObject(this.scene);
        camera.perspective(60, Engine3D.aspect, 1, 5000.0);

        this.hover = camera.object3D.addComponent(HoverCameraController);
        this.hover.setCamera(-30, -15, 120);

        let wukong = await Engine3D.res.loadGltf('https://cdn.orillusion.com/gltfs/wukong/wukong.gltf');
        wukong.transform.y = 30;
        wukong.transform.scaleX = 20;
        wukong.transform.scaleY = 20;
        wukong.transform.scaleZ = 20;
        wukong.forChild((node) => {
            if (node.hasComponent(MeshRenderer)) {
                node.addComponent(ColliderComponent);
            }
        });
        this.scene.addChild(wukong);

        this.initPickObject(this.scene);

        let view = new View3D();
        view.scene = this.scene;
        view.camera = camera;
        // start render
        Engine3D.startRenderView(view);

        // listen all mouse events
        view.pickFire.addEventListener(PointerEvent3D.PICK_UP, this.onUp, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_DOWN, this.onDow, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_CLICK, this.onPick, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_OVER, this.onOver, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_OUT, this.onOut, this);
        view.pickFire.addEventListener(PointerEvent3D.PICK_MOVE, this.onMove, this);
    }

    private initPickObject(scene: Scene3D): void {
        /******** light *******/
        {
            this.lightObj = new Object3D();
            this.lightObj.rotationX = 125;
            this.lightObj.rotationY = 0;
            this.lightObj.rotationZ = 40;
            let lc = this.lightObj.addComponent(DirectLight);
            lc.lightColor = KelvinUtil.color_temperature_to_rgb(5355);
            lc.castShadow = true;
            lc.intensity = 20;
            scene.addChild(this.lightObj);
        }

        let size: number = 9;
        let shape = new BoxColliderShape();
        shape.setFromCenterAndSize(new Vector3(), new Vector3(size, size, size));

        let geometry = new SphereGeometry(size / 2, 20, 20);
        for (let i = 0; i < 10; i++) {
            let obj = new Object3D();
            obj.name = 'sphere ' + i;
            scene.addChild(obj);
            obj.x = (i - 5) * 10;

            let mat = new LitMaterial();
            mat.emissiveMap = Engine3D.res.grayTexture;
            mat.emissiveIntensity = 0.0;

            let renderer = obj.addComponent(MeshRenderer);
            renderer.geometry = geometry;
            renderer.material = mat;
            obj.addComponent(ColliderComponent);
        }
    }

    private onUp(e: PointerEvent3D) {
        console.log('onUp', e.target.name, e.data.pickInfo);
        let obj = e.target as Object3D;
        let mr = obj.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }

    private onDow(e: PointerEvent3D) {
        console.log('onDown', e.target.name, e.data.pickInfo);
        let obj = e.target as Object3D;
        let mr = obj.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }

    private onPick(e: PointerEvent3D) {
        console.log('onPick', e.target.name, e.data.pickInfo);
        let obj = e.target as Object3D;
        let mr = obj.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }

    private onOver(e: PointerEvent3D) {
        console.log('onOver', e.target.name, e.data.pickInfo);
        let obj = e.target as Object3D;
        let mr = obj.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }

    private onOut(e: PointerEvent3D) {
        console.log('onOut', e.target.name, e.data.pickInfo);
        let obj = e.target as Object3D;
        let mr = obj.getComponent(MeshRenderer);
        mr.material.baseColor = Color.random();
    }

    private onMove(e: PointerEvent3D) {
        console.log('onMove', e.target.name, e.data.pickInfo);
    }
}
new Sample_MousePick().run();

WebGPU Shader

Post Effects

Pick Event

Listening to Events

Ray Box Picking

Frame Buffer Picking

Pick Event ​

Listening to Events ​

Ray Box Picking ​

Frame Buffer Picking ​

Pick Event

Listening to Events

Ray Box Picking

Frame Buffer Picking