tapwithus/tap-web-sdk/tap-vision-models

Vision models and air gestures

This SDK delivers raw gesture detections. Python, Web, iOS, and Android all receive those codes. iOS and Android add a processed click / drag / drop layer. This Web SDK does not. Never say swipes or pinches are iOS/Android-only.

TapXR developer firmware and TapBand use the same BLE API and the full set: taps, thumb swipes up/down/left/right, thumb-to-finger pinches with hold, fist with hold. TapBand is coming soon (waitlist). Do not invent a firmware version number.

`TapSDKWeb2` (framed firmware) for model switching. TapSDKWeb air gestures are at the end of this page.

The framed-firmware device runs one vision model at a time:

Enable and switch

import {
    DeviceFeatures,
    ModelTypes,
    TapSDKWeb2,
    UnifiedAirGestures,
    VisionSensorOpModes,
} from '@tapwithus/tapsdk';
async function useAirGestures(sdk: TapSDKWeb2): Promise<void> {
    await sdk.setFeature(DeviceFeatures.MODEL_DETECTION, true);
    await sdk.setVisionSensorModel(ModelTypes.AIR_GESTURE);
    await sdk.setVisionSensorOpMode(VisionSensorOpModes.STREAM);
}
async function useTapping(sdk: TapSDKWeb2): Promise<void> {
    await sdk.setFeature(DeviceFeatures.MODEL_DETECTION, true);
    await sdk.setVisionSensorModel(ModelTypes.TAPPING);
    await sdk.setVisionSensorOpMode(VisionSensorOpModes.TRIGGER);
}

Call these after connect() returns and after you register callbacks. connect() already started notifications. There is no start(). You can switch at runtime, as often as needed. Read the current state with await sdk.getVisionSensorModel() and await sdk.getVisionSensorOpMode(). Each call times out after 2 s if the device does not answer.

To start from a clean state, turn each feature off. Do not loop Object.values(DeviceFeatures) — a numeric enum also lists names.

await sdk.setFeature(DeviceFeatures.RAW_IMU_DATA, false);
await sdk.setFeature(DeviceFeatures.MODEL_DETECTION, false);
await sdk.setFeature(DeviceFeatures.IMU_MOTION_DATA, false);
await sdk.setFeature(DeviceFeatures.TRIGGER_DETECTIONS, false);
await sdk.setFeature(DeviceFeatures.STANDBY_GESTURE_DETECTION, false);

UnifiedAirGestures codes

Letters: A = thumb, B = index, C = middle, D = ring, E = pinky. "AB" = thumb touches index (pinch).

Verified: tapIncMsg parses MODEL_DETECTION + AIR_GESTURE into { type: 'air_gesture', data: [code] }. TapSDKWeb2 calls registerAirGestureEvents with that array.

function onAirGesture(_identifier: string, data: number[]): void {
    if (data[0] === UnifiedAirGestures.COMBINED_GESTURE_LEFT) {
        goBack();
    }
}

The device sends gesture packets continuously in STREAM mode, so the same code repeats. To act once per gesture, act when the code changes, or ignore repeats within about 40 ms. Holds repeat until the hand relaxes and NONE (100) arrives. Wait for a few NONE packets in a row before you treat a hold as released (see tap-dpad).

tap-knob and tap-dpad are app recipes that build click / drag / drop in the page from these raw codes. That is not an SDK layer.

Example: switch models with a gesture

Fist-hold switches to tapping; all-five-finger tap (31) switches back:

function onAirGesture(_identifier: string, data: number[]): void {
    if (data[0] === UnifiedAirGestures.COMBINED_GESTURE_FIST_HOLD) {
        void useTapping(sdk);
    }
}
function onTap(_identifier: string, data: number[]): void {
    if (data[0] === 31) {
        void useAirGestures(sdk);
    }
}

Standby

DeviceFeatures.STANDBY_GESTURE_DETECTION lets the user put the device in standby with a gesture. registerStandbyStateEvents(cb(id, isStandby)) reports changes. await sdk.setStandbyState(false) wakes it. await sdk.getStandbyState() reads it.

TapSDKWeb air gestures (classic firmware)

TapSDKWeb has no model switching. In InputModeController(), registerAirGestureEvents gives cb(id, gesture: number) that matches AirGestures (for example UP_ONE_FINGER, LEFT_TWO_FINGERS, PINCH, THUMB_FINGER, STATE_FIST). registerAirGestureStateEvents gives cb(id, MouseModes) when the air-mouse state changes.

Docs

• Enumerations: src/enumerations.ts in this repository
• Demo: https://tapwithus.github.io/tap-web-sdk/