Mouse.move() - move by more than 127 units

Projects Programming
1

Hi everybody!

I'm trying to program an Arduino Leonardo to automate grinding a silly Android game (not for getting onto a leaderboard, don't worry).

The Leonardo is acting as a HID Mouse. So the word "pointer" will signify the Mouse-Pointer-arrow-thingy, and not a C++ *pointer.

My goal is being able to point the cursor to multiple places on the screen and having the Leonardo repeatedly double click in all those places.

The problem I'm facing is Mouse accelereation. Because of Mouse Acceleration, it is very difficult to do precise, repeatable movements between points on the screen.

One possible solution I can think of would be to get from point to point in one single HID-Instruction (I don't know how this is really called), so that the Mouse driver only sees one input and doesn't trigger Acceleration. The problem is that as it is, you can only move a maximum of 127 units per instruction. I thought it may be possible to expand that.
According to the HID-Specification, the values for Logical_Minimum and Logical_Maximum can be freely chosen, as long as they fit in the given Report_Size.

I tried changing the corresponding lines in Mouse.cpp to

    0x16, 0x00, 0x80,              //     LOGICAL_MINIMUM (-32768)
    0x26, 0xff, 0x7f,              //     LOGICAL_MAXIMUM (32767)
    0x75, 0x16,                    //     REPORT_SIZE (16)

and the definition of

void Mouse_::move(signed char x, signed char y, signed char wheel)

to

void Mouse_::move(int16_t x, int16_t y, signed char wheel)

in both Mouse.cpp and Mouse.h

and while changing the minima and maxima had no effect whatsoever, changing the report_size to 16bit broke the functionality entirely, eg. no pointer movement at all. Bummer. But I feel this should theoretically be possible, no?

Now my question:
Is it possible to do larger pointer "leaps" than +/- 127 units? Or is there a better way to do what I need?

Any help is greatly appreciated! this whole HID-thing is new to me, and I'm finding it very interesting, but also incredibly overwhelming.

Best,
Bin

2

You're braver than I am :wink: No experience with this, so take it for what it is.

Just passing integers will not be the only thing to change. The variable m in ::move will also need to be changed to int, else you will still truncate the integers to signed chars.

And I suspect that a changing the report size to 0x16 (22 decimal) might not be supported; maybe 0x10 will work better.

3

I did a project some years ago to use an Arduino for automated bidding on eBay. I had the same need to reliably and repeatably click specific locations on the screen, and had the same problem with mouse acceleration. I took a very caveman approach to working around this issue, which was to use a series of small mouse movements, and published it as an Arduino library:

I'm certain the HID approach is the right way to do it, and I share some projects that use other approaches in the readme:

So even if my code is not useful to you, perhaps one of those links will be.

The library did work perfectly for my needs, and a friend was also successful using it for an Instagram-related project. It also seems to have had some popularity with the "USB Rubber Ducky" crowd (to be clear, I don't support that usage unless it's used for legitimate pen testing).

4

Thank you sterretje for your kind words and advice!

sterretje:
And I suspect that a changing the report size to 0x16 (22 decimal) might not be supported; maybe 0x10 will work better.

You're right, big blunder on my side! But sadly, even 0x10 still stops all mouse functionality. The TX led flickers though, for what it's worth.

sterretje:
Just passing integers will not be the only thing to change. The variable m in ::move will also need to be changed to int, else you will still truncate the integers to signed chars.

Good catch! What I don't understand here is that m is normally of type uint8_t, so there is a cast from signed to unsigned which I don't understand the purpose of...
But since it's not working at all rather than truncated, I suspect the main issue lies somewhere else. (The HID Report I think).

pert:
I did a project some years ago to use an Arduino for automated bidding on eBay. I had the same need to reliably and repeatably click specific locations on the screen, and had the same problem with mouse acceleration. I took a very caveman approach to working around this issue, which was to use a series of small mouse movements, and published it as an Arduino library:
GitHub - per1234/MouseTo: Arduino library for moving mouse pointer to absolute screen coordinates

Thank you for those ressources, pert! I thought about doing something like this as well. My main concern with this approach is speed, since you need to delay between the mouse movements so that acceleration doesn't trigger. Or have you had differing experiences?

pert:
I'm certain the HID approach is the right way to do it, and I share some projects that use other approaches in the readme:
MouseTo/README.md at master · per1234/MouseTo · GitHub

So even if my code is not useful to you, perhaps one of those links will be.

Great, I'll look further into it, thanks.

pert:
The library did work perfectly for my needs, and a friend was also successful using it for an Instagram-related project. It also seems to have had some popularity with the "USB Rubber Ducky" crowd (to be clear, I don't support that usage unless it's used for legitimate pen testing).

Yeah, once you put your code out there, there will be people willing to abuse it.

I'll keep looking into this, and if I find something that works, I'll let you know.

Best,
Bin

5

Binom:
My main concern with this approach is speed, since you need to delay between the mouse movements so that acceleration doesn't trigger. Or have you had differing experiences?

For my use, the auction listing pages load time was the limiting factor, so a very different situation than your project. I suspect that my solution would not work well for you.

6

So, update:
It turns out the whole HID thing isn't as standardized as I thought when reading the documentation.

I got it to work with Android, and - with some caveats - with Linux. But I can't for the dear life of me get any luck with Windows.
But since I only plan to use this with Android, that's the best possible outcome for me.

After some digging, here's what I found: the Mouse library sends the Reports using HID.SendReport(), which in turn uses USB_Send() from USBCore.cpp, which only accepts data in unsigned 8 bit int arrays.
So in the Mouse library, m needs to be an uint8_t array.

I altered the .move function so that it accepts 16bit signed ints, and splits these into 2 8bit array fields, which are then sent out.
I'll attach the modified files, maybe they are useful for someone else.
Note that I capped the pointer movement at +/- 2047 units. this can be increased by changing the values in Mouse.cpp to

    0x16, 0x00, 0x80,              //     LOGICAL_MINIMUM (-32768)
    0x26, 0xff, 0x7f,              //     LOGICAL_MAXIMUM (32767)

On Ubuntu, mouse acceleration behaves unpredictably, but can be turned off entirely, so this modified Mouse.h will also work there.

On windows, I get nothing. But since this is my first HID Project, I'm pretty happy with the result!

But if anyone would like to teach me, I have three questions:
Is the code I modified OK or could it be improved?
How does windows parse HID report descriptors? Why doesn't this work?

Thank you guys for your help! I always love learning new stuff!

Mouse.cpp (3.64 KB)

Mouse.h (1.7 KB)