[palosanto]

I've updated to 3.8.19.
Has anyone tried the face gestures feature?   It doesn't seem to work at all in my device, a Samsung Galaxy running Android 11.
I've enabled face gestures and configured a couple of gestures to turn pages with no response at all.
BTW, is it also available in Windows? The latest windows version doesn't seem to mention it, or is it only an Android feature?
Thanks

[Zubersoft]

The first thing you should do is go to Settings->Face Gesture Settings->Configure Gesture Actions, and tap Test Gestures. Do you see your face show up with blue points around your mouth, eyes, eyebrows and face? If so, that means MobileSheets is able to detect your face through the MLkit library. You can test out various gestures like smiling to see if MobileSheets is detecting things properly. You can adjust the gesture size slider for a particular gesture if needed. A larger gesture size means you need to smile bigger, raise your eyebrows more, etc, to trigger the gesture, whereas a smaller gesture size will trigger with less motion. This can lead to more false positives though. You can also record a gesture if none of the built-in ones are working for a particular gesture which just compares the recorded face points against the current ones with an allowed deviation percentage.

I've been able to get the face gestures to work on all of my devices except for the Samsung Galaxy Note Pro 12.2 (which is why the feature requires Android 6.0 or higher). You said it's not working - please let me know if any points show up in the video capture on your device.

The feature is only available on Android at the moment, but I'm implementing it on Windows and iPadOS next. The iPad version should be much easier as Apple has support built into their framework for all this. The Windows version has been a monumental pain so far. I'm still working through it, but I will have to see how well it works.

Thanks,
Mike

[palosanto]

Thanks Mike,
I already configured the settings and actions.  However, I've reset them and tried again.
Smile and turn head right/left seem to be the only ones responding consistently.
Others are either not responding or going crazy, for example raising eyebrows to go to next page results on multiple pages being turned or showing and hiding the audio player multiple times if it's the action assigned to it.
Blink either right or left eyes won't respond at all.
Turning head up/down is registered when I test it but it won't perform the action when using it.  It also seem to get confused when raising eyebrows, so for example tilting the head up it often registers raising eyebrows and tilting head down it also registers raising eyebrows immediately after probably because raising the head back also raises the eyebrows points.
Anyway, I'll keep experimenting and hopefully some can be actually used reliably.

[Akki]

I have also played with this feature for quite some time and left it alone... It just didn't work for me.

[Rey_a]

The 'Test Gestures' dialog can also be used to learn how to perform the movements for best results. You may have to make the gestures a bit different than you normally would for best results.

One mistake I made at first, is not to hold the movements long enough. Once I realised that, things improved quite a bit.
Also try making the intended movement, while keeping the rest of your face muscles as immobile as possible.
Finally, how well your face is lit and how well the camera performes in darker situations obviously has an impact as well ....

For me, the head movements (left/right, up/down) are now 100% reliable (on my Samsung Galaxy S7 FE) and I can keep my eyes on the score while doing them. So I can perfectly use those in stead of my 2 pedal footswitch for turning pages (and more)
Most other gestures are at 85-90 % reliability for the moment, but I will practice a bit more using the 'Test Gestures' dialog and see if I can still improve on that.

So for me, this is a very usefull addition to MobileSheets !

Thanks Mike!

Rudy

[Akki]

After all, this may not work for me, as I will usually gig in "dark-er" environment.
Looks like I will still have to figure out best but smallest Bluetooth button/switch (for page turns) that I still didn't find (maybe not available))

[art]

Hi everyone a very useful feature Mike, but I L also am having problems I tried a few gestures earlier the right turn forward worked sort of ok but that doesn't work for me as it takes me away from reading the parts, tried the winking didn't work will play with it tomorrow , thanks

[Zubersoft]

When reporting issues with this feature, I need to know whether you are able to trigger the gestures in the test dialog that shows the points on your face. If your head is rotated too much in any direction, it's going to stop the gesture from working, because the feature relies on the face being mostly pointed towards the camera (it allows around 14-18 degrees rotation in any axis). It is incredibly complex to take a video frame of a person's face, use facial recognition libraries to capture the contours of the face, then track the contours changing over time to determine if the person is making a particular gesture. Some gestures like "smiling" should be incredibly reliable as I'm using Google's own image recognition code to check the probabiliy that a smile is occurring. If that is not working for you, then something is wrong like your head is rotated too much, or something is interfering with Google's ability to capture the contours of your face (like an environment that is too dark or poor image quality due to a low resolution camera). Some gestures like "left wink", "right wink" or moving the lips left/right are some of the hardest to detect due to the inaccurate nature of Google's facial landmark detection. Regardless, if I place my face around 12-24 inches (30-60 cm) in front of my tablet my face centered and pointing towards the camera, I can trigger all of the gestures accurately with no errors. 

One thing I want to mention is that there is a "Gesture Duration" setting. That determines how long you have to hold the gesture for it to be recognized. If you drop that to 0.1 seconds, then the library will fire the gesture far more quickly, but it may also be more likely to trigger false positives.

Mike

[Oz Cello]

After all, this may not work for me, as I will usually gig in "dark-er" environment.
Looks like I will still have to figure out best but smallest Bluetooth button/switch (for page turns) that I still didn't find (maybe not available))

I've been using this one recently. Tiny, cheap, works well. Back page is double tap by default.

[Akki]

After all, this may not work for me, as I will usually gig in "dark-er" environment.
Looks like I will still have to figure out best but smallest Bluetooth button/switch (for page turns) that I still didn't find (maybe not available))

I've been using this one recently. Tiny, cheap, works well. Back page is double tap by default.

Interesting enough I noticed it earlier on Amazon.ca ??

[Lawrie]

While I can't use this feature (my TabX doesn't have a camera   , a point I missed when I researched it) I am curious how it works for wind players.

E.G. I predominantly play both Trombone and Tuba - the instruments obscure different parts of my face, depending on which instrument.  Can I assume this feature is unlikely to work for me anyway?

[Zubersoft]

I'm not sure how well Google's library will handle that... at best, you'd only be able to use a few gestures such as winking, raising your eyebrows, or turning your head.

Mike

[Lawrie]

Interesting, I'm a little surprised it would work that well.

Thanks mate.

[Cody94549]

OK, first off, I love this concept.  I've been using a BT Pedal (Piano) for a year with this software.  I've struggled on fast page turns with that (basically searching for the pedal with my foot), so the idea of FACE Gestures really appeals to me.  So my 1st day of testing results on a SAMUSUNG Tablet 8 Ultra :

- Eye winks/blinks just won't work at all. Even in the TEST GESTURE .. It "sees" my eyes and all other face parts, just doesn't work.
- I tried the Head Up/Down -- I get way too many false actions (the pages turn when I don't want. Probably because I just can't keep my head still while reading & playing ... e.g., I have to look at the keys from time to time resulting in my head movement .. think LONG jumps)
- I have it set now to Mouth Open.  Again unpredictable resulting in false page turns. I tried the various settings of how much/wide and duration and can't seem to find the sweet spot. 

I'll keep playing with this. It's a great concept once the bugs get sorted out. Thank you !!

[Zubersoft]

Google's library does a poor job of reporting eye coordinates. There is almost no difference between an eye closed versus open. So I had to rely on one of their features where they provide a probability that each eye is open. A wink, in this case, is when there is a large probability that one eye is open and the other isn't. If this isn't working for you, it means Google's library cannot clearly identify when your eyes are open or closed. I'm not sure there is much I can do about that I'm afraid.

The head tilt detection is just based on a rotation amount. The gesture starts when the head is tilted less than 8 degrees up or down (so mostly center) and triggers when the head has tilted more than 16 degrees. It then waits until the head is less than 8 degrees again before starting the next gesture. If you are triggering this, it means your head is moving up or down more than 8 degrees at a time which is a lot. The gesture size increases or decreases the amount of rotation required. If you set the gesture size to the largest, MobileSheets will require 28.8 degree rotation. I'm not sure how you could possibly generate false positives with that... So if you haven't tried increasing the gesture size, start by increasing one at a time and then testing.

Open mouth is one of the more simple ones, but determining the amount the mouth has to be open to trigger is somewhat complicated. Increase the gesture size will require the mouth to be open more, but this is not one where false positives should be generated as the mouth has to be mostly closed to even start the gesture. If you are leaving your mouth open the entire time while playing, then this is probably not the best gesture to use unless you use a really large gesture size. If the largest gesture size isn't large enough, I can always increase the offset a little more.

Mike