Using Computer Vision to Scan A Voting Ballot

March 16, 2019 No Comments yet Posted in: Uncategorised

I acted as the chief electoral officer for the Cardston & District Historical Society in 2018 and 2019. I decided it’d be fun and useful to build a ballot scanner.

The Problem

The most time consuming election to get the results for is when we elect board members. In 2018, I think we needed to elect eight people to the board of directors and had a dozen nominees. All authorized voters (or their proxies) on which to write down up to eight names on for the people they are voting for. [If they voted for nine people, it’d be a spoiled ballot and not count; they could choose to vote for fewer people, and that might be strategic.]

So, how do you count it? On a sheet of paper, write down the names of all the people running, and then, for each ballot, check that it isn’t spoiled, and put a tally mark down for every person indicated. Finally, figure out which 8 people got the most votes.

The Theory

Having recently worked with ArUco markers, I knew that they (magically) are identified in an image, providing the location of the marker and the numeric code the marker represents.

If I built a ballot that had ArUco markers on it, and circles to fill in (such as those used in provincial exams), I could identify where the markers were, and use that to identify where the circles were. Then, it should be a simple matter to look at whether a circle is filled in or not, and attribute the vote to the correct candidate. Piece of cake, right?

Have you met my friend incidental complexity?

Building The Ballot

This is what the final ballot looks like (a blank one, anyway):

Voting card, with ArUco codes in the corners, and 20 lines to fill in with names

voting-card-spare Download

I wrote a python program using the ArUco and FPDF libraries to generate a PDF with four voting cards on it. (I initially made one card for use on a 3×5 card, but I could only print them one at a time and my printer kept jamming. This was so much simpler.)

The code also spat out the locations of the AR markers and circles (in JSON format).

The Scanner

The scanner application is built in Unity game engine using C# and the OpenCV for Unity asset.

It starts processing incoming frames from the webcam, looking for ArUco markers. If it finds four of them, it checks if they are the codes expected for the four corners and if so, uses that information to figure out which corner is which (ie. which corner is the upper-left corner on the card).

Then it creates a perspective transform and uses it to make a straightened-out version of the image. It saves the color image, and then makes a black-and-white version of it.

Raw photograph and straightened binary image of the ballot, showing whether circles are thought to be filled in or not — The binary image (‘X-Ray View’); note the card in the photo is upside-down

Then, for every circle in the image:

it creates a submatrix (a small image) for the smallest square that fits around the circle
it blacks out the area outside the circle
it counts all the black pixels in the square
depending upon the count, it reports that the circle is filled in, or that it is blank

The calling code knows which circle it is and which candidate it corresponds to. The user it shown the colour image with circles drawn around each circle in blue or red, indicating if it is filled in or not (so far as the computer can tell). The user then hits a button to accept the result (or not).

Finally, the user can click on a ‘totals’ button and see what the overall totals were like.

Raw photograph and straightened colour image of the ballot — A colour image, showing the straightened voting ballot

Was It Worth It?

I put many hours into building the ballot scanner, and it could use a fair bit of love to be a better application (say, something I could sell on an app store). Having used it for an election, I was surprised how useful the structured ballots are for quickly counting up winners — this part of the process was a bigger win than the app itself, especially for elections with a small number of candidates.

So, I’m chalking this one up to a learning experience, but it was really neat to do. I’ve got some ideas for future improvements should the mood hit me — such as using a newer version of the ArUco library [which is supposed to be a fair bit faster], detecting corners without using ArUco fiducial markers, using a static threshold instead of a adaptive one, and supporting more advanced forms of elections.