Raise the Hammer
this blog entry has been updated
The City of Hamilton's election results page is human-readable (if inelegant) but not particularly machine-readable.
Because it's just stuff on a web page, the structure of the data is difficult to extract in such a way that it can be processed and analyzed. If you want the low-level poll-by-poll data, you need to open up 207 separate web pages and copy out the results. That's just crazy.
I asked Tony Fallis, the city's Manager of Elections, if he could provide the data in a more structured format. He explained that it is only available as HTML on the city's website or, alternately, in hard copy.
I know some people in the community have been copying numbers off the city's web site and doing manual calculations. (See, for example, several comments by "Fred Street" in this article.)
That's admirable, but it entails tedious, manual busywork and I'm a lazy programmer. To save time, I wrote a script to do the manual work of crawling all 207 web pages, extracting the line-by-line results out of them and formatting it all into a database table.
(Aside: the script is ugly and hacky even by my standards, and currently breaks a bit when it reaches Ward 14, in which the sole candidate was acclaimed. I intend to fix the bug and clean up the code as soon as I have a chance.)
Now that the data is in a database, we can start to do interesting things with it. As a first step, I've added the results to the RTH Election site. You can view a ward-by-ward summary, a more detailed poll-by-poll summary, a list of polls, a list of wards, or the raw data.
Even if you're not a programmer, the benefit of having tabular data in clean HTML tables is that you can copy it en masse, paste it into Microsoft Excel or OpenOffice Spreadsheet and start messing with it.
(Note: if you notice any errors or discrepancies in the data, please let us know so we can fix them.)
More important, the same data is also available in JSON format (a format for transferring structured data over a network) in the RTH Elections API. (You can jump straight to the Results section of the API documentation.)
Because you can access the data programmatically, anyone can easily grab the data from the RTH Election site and analyze it further or combine it with other data and services to produce more valuable insights into the results.
All the data on the RTH Election site is licenced under the Creative Commons Attribution-ShareAlike 3.0 Unported License. That means anyone can access, recombine, and redistribute the data as long as they a) attribute the source, and b) use an equivalent licence for any resulting works they distribute.
This is the direction in which public data is moving in progressive cities around the world.
It's the direction in which Hamilton needs to start moving if we are to encourage the kind of value-add citizen engagement with public data that cities like Ottawa, Vancouver and Edmonton are already enjoying.
Update: The City just changed the structure of their election results pages, and so the script I wrote to scrape the results has stopped working. I'll update this page when I get it working again.
Awesome. I'm not someone with the skills to use the open source data and make anything of it. But I certainly appreciate that it's there for reference - and that a lot of people who do have those skills will.
By Hopeful (registered) | Posted October 28, 2010 at 00:21:51
Morelli swept the nursing homes!
By jasonaallen (registered) - website | Posted October 28, 2010 at 08:42:23
When I clicked on the "results section" hyperlink, I got an internal server error. Other than that, the concept of having our data easily used and interpreted by anyone is a huge step...but it just reinforces why the Federal Govt is so terrified of the long census form. Knowledge is power.
Comment edited by jasonaallen on 2010-10-28 07:42:48
By Pxtl (registered) - website | Posted October 28, 2010 at 09:39:36
Haven't had a chance to properly break down the data yet, but in Ward 02 it looks like the big difference happened in the advanced polls.
By PseudonymousCoward (registered) | Posted October 28, 2010 at 13:30:38
^Boring troll is boring.
By simonge (registered) | Posted October 28, 2010 at 14:11:33
The City really should be able to make this data more accessible to us. Thanks for taking your time to do this Ryan.
turbo, you can't see the benefit of opening one page of data instead of 207 pages and sorting through the to find the information you're looking for?
For example, let's say we want to know which 10 polls had the best and worst voter turnout across the whole city, or which poll had the most votes for a certain candidate.
To do that, I don't have to open a bunch of pages, copy the data and paste it. It's all there in a format I can access easily. It doesn't take me an hour to access, it takes a few minutes.
And then I can either search all that data as-is or throw it into Excel and use one of their really basic "sort by" functions to sort it to find out information, not spend a big chunk of time opening pages, copying, and searching.
This is the exact same thing the article explained, but since I'm doing only the crudest of data manipulation, things that most internet users do, it's something that even someone like you would find relevant.
By allantaylor97 (registered) | Posted October 28, 2010 at 21:32:39
Comment edited by turbo on 2010-10-28 20:34:30
So if someone doesn't have the time to sort through that kind of information, or has a desire or need for more information than you do, too bad for them?
By your logic, unless someone has unlimited time they shouldn't expect to be well-informed, or should rely on others to compile and disseminate that information. That's pretty scary.
Anything that makes it easier to access this information means that more people will be informed on a wider variety of topics - and how that's a bad thing is beyond me. You can't seriously be advocating for holding back information and making it harder to access... and there I will stop attempting to argue these points.
By allantaylor97 (registered) | Posted October 28, 2010 at 21:57:47
Comment edited by turbo on 2010-10-28 20:59:33
By Gardien de zoo (anonymous) | Posted October 29, 2010 at 08:40:58
Ne nourrisez pas le troll, s'il vous plait.
By nobrainer (registered) | Posted October 29, 2010 at 09:50:32
"A troll is contrary just for the sake of starting a fight." You mean, like saying, 'I don't personally need this therefore no one needs it and it's a waste of time'?
By PseudonymousCoward (registered) | Posted October 29, 2010 at 11:13:37
Turbo has demonstrated abundantly, consistently, and above all persistently that he is a troll seeking only to disrupt this forum. Don't waste your time trying to prove to him that he is a troll, he already knows what he is doing and abject denial is part of his game. Please refrain from further encouragement of his behaviour and stop responding to his comments.
Comment edited by PseudonymousCoward on 2010-10-29 10:16:27
By allantaylor97 (registered) | Posted October 29, 2010 at 15:02:30
I am not a troll. I am presenting the opposite case because frankly thats what I believe in. I also believe in honest open debate, a concept that seems to be foreign to all who insist I'm a troll. In fact its the ones throwing out disparaging names that have issues with behavioral issues. You will never convince the people who don't see eye to eye with you if you continue to try to bully them into submission. I have news for you, you'll only harden their position and you'll never achieve very much of your agenda There are reasons RTH isn't taken seriously and your treatment of anyone who you disagree with is the biggest one
Comment edited by turbo on 2010-10-29 14:02:55
By adrian (registered) | Posted October 29, 2010 at 21:25:23
Ne nourrisez pas le troll, s'il vous plait.
Je suis d'accord.
By allantaylor97 (registered) | Posted October 29, 2010 at 22:29:10
you do realize the majority of folks with a highschool education can read French at the grade 1 level you are using. What a bunch of brilliant guys you are
Comment edited by turbo on 2010-10-29 21:29:31
By TnT (registered) | Posted October 30, 2010 at 10:10:47
This open city concept should be embraced by one of the pioneers of open source: Bob Young.
Now if I could only find a program that allows me to download the proof that the WH won't work for them....
By allantaylor97 (registered) | Posted October 30, 2010 at 20:08:45
See I don't regurgitate everything Bob Young says. He's wrong on this one. I have a mind of my own as do all Ticat fans
You must be logged in to comment.
There are no upcoming events right now.
Why not post one?