Accessibility: A New Perspective?

According to the American Foundation for the Blind, nearly 27 million adult Americans (~10%) are blind. So, society has done amazing things to ensure these people can still use a computer! A similar idea applies to those who are deaf.

Now, here’s the key insight. A computer program can control the mouse and keyboard, but current state-of-the-art methods have a hard time interpreting audio and video inputs. One way to think about this is that computers are partially blind - although they can “see” perfectly, they can’t interpret it easily, so assume that it can’t see at all. Then a whole suite of tools designed for visually impaired computer usage becomes available!

Let’s start with this: For the Experienced Computer User with a New Visual Impairment (afb.org).

Windows GUI Automation: pywinauto

Microsoft also provides an API for screen readers and other accessibility devices to use: UI Automation Overview - .NET Framework | Microsoft Learn.

Initial Attempt

But I tried using the py_inspect.py tool from pywinauto - which is meant to determine whether certain GUI elements are visible in both UI Automation and an older Microsoft API for the same purpose - but it didn’t find anything. It’s probably because Discord’s desktop app is actually just a browser in the backend.

Since Discord is considered a relatively simple modern web app to “automate” (relative to something like League), I doubt this UI Automation would work for most purposes. So rip Windows-specific UI Automation I guess :(

Wait…

So I tried using Inspect.exe on Discord and it gave me this…

I clicked on a couple of the buttons and got this huge thing:

And yes, I just got Gyazo and am realizing how powerful it is. ;)

So we could probably use UIA or other types to do this, but it wouldn’t generalize to other apps like League, so I’ll stick with image recognition UI automation.

SikuliX and Lackey

“Don’t reinvent the wheel” is my motivation for trying to find new ways to do this whole UI automation thing (I’m also kinda lazy xd). This has gotta be something that tons of people need to do, right?

Well, I found a pair of automation libraries that allow you to write scripts to automate graphical tasks. Or really, SikuliX, Lackey is just a port (ish) of that over to Python. Let’s see if it works - and if it’s easy enough to use.

Here’s a tutorial for the SikuliX IDE (the exact type of script writing tool I was looking for!).

Oh my god. This is literally exactly what I was looking for. Look at it go… it even saves the script in a nice little folder, for use with other apps… Love at first sight <3

Well then. Now we just need to see if I can port it to Python…

I looked around for a better option, but it looks like Lackey is the way to go - even though it hasn’t been maintained for 3 years now. Welp, I guess I’ll fork it and make it not deprecated.

Last updated: 09 April 2023

Back to previous page