🚨 This project useable but not officially released yet
This Handsfree.js powered browser extension is a designed to help you browse the web handsfree through face and/or hand gestures. The aspiration is to develop a "User Script Manager" like Tampermonkey and Greasemonkey, but for handsfree-ifying web pages, web games, web apps, and any other type of content found online.
This browser extension also includes handsfree Dev Tools, primarily to assist in handsfree WebXR development. In short, this extension is designed to help you use your entire browser and web totally handsfree!
By default, each page will get a "Face Pointer" or a set of "Palm Pointers" for you to browse pages handsfree.
However, in addition to the pointers you can add custom handsfree interactions. For example, here's a demonstration of handsfree-ifying Jason Webb's Reaction Diffusion Playgound where each individual finger pinch does something different:
This project is a started as a fork of the Mozilla WebXR Emulator Extension. This section of the browser extension was initiated from conversations with Alex Ionkov.
Currently the DevTools helps you emulate WebXR headsets with basic controller support.
Each of the files are located in their respective context folders in /src/. Handsfree.js specific scripts are named handsfree.js and the Mozilla WebXR specific ones are labeled webxr.js. Other than organizing the files the WebXR code is mostly untouched.
- When you first install the extension,
/src/background/handsfree.jschecks if you've approved the webcam. If not, then it'll open the options page insrc/options/stream-capture.html - The popup panel has a "Start/Stop Webcam" button that communicates with the background script to start the webcam:
/src/popup/index.html - The background page is where the models are stored and run. This keeps everything isolated and only asks for webcam permission once (vs on every domain):
/src/background/handsfree.js - The background page also uses the "Picture in Picture" API to "pop the webcam" out of the browser. It renders the webcam feed and debug canvases into a single canvas, and uses that as the
srcObjectto a separate video element which is the PiP'ed
- Chrome: Install as an unpacked chrome extension. Visit
chrome://extensionsand enable Developer Mode on the top right, then click Load unpacked and select this project's root folder
- Firefox: Coming soon, I switched computers and didn't realize I continued developing on Chrome 😆
- Alex Ionkov for the original handsfree WebXR Emulator vision, research, and our planning and discussions
- Mozilla WebXR Emulator Extension team from which the WebXR Emulator Dev Tools was forked from