I agree the keyboard will remain as the best text input device but I'm pretty confident that eye tracking will be categorically better than a mouse/trackpad. I'll expect that eye tracking solutions for normal monitors will also become more mainstream once people get used to it in the Vision Pro and want to use it elsewhere.
Eye tracking will probably lose the productivity metric, though. When you remove the ability to look at one thing and interact with another, you get farther from the way that humans tend to interact with their immediate physical environment.
I can cut vegetables without looking at them. I can use that dynamic to offset the planning and acting phases of my thought process. Falling short of that efficiency will feel limiting.
I often am looking at another element before interacting with it with my mouse. With eye tracking enabled menus hovering over an element immediately because I'm looking at it becomes an annoyance.
Yea, I don't think a 1-to-1 translation of mouse movements with eye tracking is the correct answer here. I do think it's probable that eye tracking + {X} can be a 1-to-1 translation for clicking for 80%+ clicks.
Maybe X is a button on the keyboard. Maybe X is a gesture.
I can think of some Portal puzzles in particular where timing is important, and you need to hold your aim but wait to click until something happens somewhere else on the screen (so the place you're clicking is not the same as the place you're looking).
I think the same thing applies to e.g. recording network activity in Chrome dev tools. My eyes are on the page to see when the thing I'm interested in finishes loading; my mouse cursor is on the button to stop recording.
It's not a super common pattern, but probably common enough that it would be annoying not to be able to do it.
Yea, I agree with this for gaming. Eye tracking as a gaming interface probably requires rethinking a lot about games. In VR for example the movespeed is much slower. In normal video games the character movement is constantly super-human speed, and this is jarring while in VR.
I am speaking mostly about the desktop interactions. In your Chrome Dev situation, I would look at the cursor before clicking on the stop recording button. I think I might be able to trust the MBP trackpad to do a primed click without looking at the cursor, but I wouldn't trust a traditional desktop mouse to have stayed steady enough.
Interesting, I just tried to use my pointer with my peripheral vision, and while possible, it was indeed more difficult and harder to focus on. Something I hadn't tried before.
I'm not sure what the SOTA is currently but the Vision Pro has much better eye tracking that other systems I've used in the past. Idk if the sensors are actually better or if it's just the native integration into the OS and apps, but it feels more seamless than a mouse and I've never "misclicked" on the wrong thing with my eyes. It kinda feels like if the mouse cursor instantly jumped to the right place every time with no travel time or physical movements required.
I've used a 3D mouse for CAD but am not sure where else it would be helpful?
Tobii https://gaming.tobii.com/product/eye-tracker-5/ is standard in consumer eye tracking, they even have integration with some popular video games! There's just no value in it. I tried an older version in a store, and it was basically perfect, tracking my eye's movements exactly, with zero delay.
Nobody wants it for day to day computer interaction. Most people using eye tracking for computer interaction are disabled, because it's a terrible experience.
I primarily use i3wm with 4-40 terminal windows + firefox open. I'd LOVE it if the terminal that I'm looking at was the `active` one. I can't count the number of times I'd lookup some error code, type in more commands, hit enter; only to discover that I had been typing away in the wrong window.
This will be no problem at all in the sdk. Hell I saw someone do this as some point on the mac using the macbook camera popping windows to the front based on eye-dwell... I think they started with cursor control.. essentially eye-mouse.
As we use an iPhone so close to our face, I would love to see an eye tracking system work for them. But maybe it really need that incredible close proximity, two cameras per eye, and IR illumination in the Vision Pro to work.
I definitely want to see a monitor with Vision Pro-style eye tracking and hand tracking. I'd love for eyesight and subtle gestures to fully replace pointer devices.
The pinch gesture on Vision Pro is pretty smooth and easy, about as much physical exertion as a mouse click. Right click is harder, I'd expect most app developers to do something similar to iOS and treat long clicks as right clicks but that's pure speculation.
I'd love to have an eye tracking setup though on a normal desktop computer where I could devote a keyboard key to clicking and I'd never have my hands leave the keyboard.
There's also apparently other gestures and voice commands that can be selected in the settings, though I haven't seen any real detail on it other than "they're there".
