Understanding Open Public Data

Ryan McGreal, Raise the Hammer

CIPS - Golden Horseshoe Monthly Event

Wednesday, March 23, 2011

What is Open Public Data?

Open Public Data Is:

Information created or collected by the government that is:

Open Public Data Is: Publicly Available

Information created or collected by the government that is:

Open Public Data Is: Publicly Available

Municipal Freedom of Information and Protection of Privacy Act (MFIPPA):

4. (1) Every person has a right of access to a record or a part of a record in the custody or under the control of an institution unless,

(a) the record or the part of the record falls within one of the exemptions ... or

(b) the head is of the opinion on reasonable grounds that the request for access is frivolous or vexatious.

Open Public Data Is: Publicly Available

Two Purposes of MFIPPA:

(a) to provide a right of access to information under the control of institutions in accordance with the principles that

(i) information should be available to the public,

(ii) necessary exemptions ... should be limited and specific, and

(iii) decisions on the disclosure of information should be reviewed independently of the institution controlling the information;

Open Public Data Is: Publicly Available

Two Purposes of MFIPPA:

(b) to protect the privacy of individuals with respect to personal information about themselves held by institutions and to provide individuals with a right of access to that information.

Open Public Data Is: Publicly Available

Note on exemptions:

An exemption from disclosure of a record ... does not apply if a compelling public interest in the disclosure of the record clearly outweighs the purpose of the exemption.

Open Public Data Is: Findable

Information created or collected by the government that is:

Open Public Data Is: Findable

Search results for '150 main street west'

Open Public Data Is: Findable

Thank goodness for Google

Open Public Data Is: Findable

Findable means:

Open Public Data Is: Findable

Not findable:

http://www.hamilton.ca/Hamilton.Portal/Templates/Generic5.aspx?NRMODE=Published&NRORIGINALURL=%2fCityDepartments%2fPublicWorks%2f­Environment_Sustainable_Infrastructure%2f­StrategicPlanning%2fStrategicEnvironmentalPlanningProjects%2f­GRIDS%2fTransportation%2bMaster%2bPlan%2ehtm­&NRNODEGUID=%7b0153AB57-4228-4E6E-8092-B5B6FF291204%7d&NRCACHEHINT=Guest#TMP%20-%20Policy%20Papers

Open Public Data Is: Accessible

Information created or collected by the government that is:

  • Publicly available
  • Findable
  • Accessible
  • Machine-readable
  • Open licence

Open Public Data Is: Accessible

Accessible means:

  • Clean, semantic HTML for documents
  • No read-only text formats (PDF, scans)
  • Open standard formats for structured data
  • Can be linked directly, i.e. not behind:
    • Form POST events
    • Javascript redirects
    • Session querystrings
    • Proprietary UI, e.g. Java
  • Easy for others to share
  • People of all abilities and disabilities can access it

Open Public Data Is: Machine-Readable

Information created or collected by the government that is:

  • Publicly available
  • Findable
  • Accessible
  • Machine-readable
  • Open licence

Open Public Data Is: Machine-Readable

Standard formats for structured data:

Open Public Data Is: Machine-Readable

Web service API:

  • REST - Create, read, update and delete resources via HTTP methods, request and response codes
  • SOAP - remote procedure calls defined in WSDL, delivered via HTTP/HTTPS POST requests and XML messages

Open Public Data Is: Open Licenced

Information created or collected by the government that is:

  • Publicly available
  • Findable
  • Accessible
  • Machine-readable
  • Open licenced

Open Public Data Is: Open Licenced

Rights:

  • Copy, publish, distribute and transmit
  • Adapt and combine
  • Use commercially in a product or service

Responsibilities:

  • Attribute data sources
  • Disclaim endorsement
  • Lawful use

Open Public Data Is: Open Licenced

The British Open Government Licence for public sector information:

  • 950 words
  • Clear language
  • Permissive terms
  • Standard across all British public bodies

Open Public Data Is: Open Licenced

Open Data Commons ODC Public Domain Dedication and Licence (PDDL):

Open Public Data Is: Open Licenced

U.S. Data.gov Data Policy:

  • 652 words
  • U.S. public data is public domain
  • No copyright restriction

Open Public Data Is: Open Licenced

Government of Canada Open Data Licence Agreement for Unrestricted Use of Canada's Data:

  • Published March 17, 2011
  • Royalty-free, non-exclusive, worldwide, non-assignable
  • Use, reproduce, extract, modify, translate, develop and distribute
  • Attribution: "Reproduced and distributed with the permission of the Government of Canada"
  • No identifying individuals, families or households X

Open Public Data Is: Open Licenced

Government of Canada Open Data Licence Agreement for Unrestricted Use of Canada's Data:

5.3 You shall not use the data made available through the GC Open Data Portal in any way which, in the opinion of Canada, may bring disrepute to or prejudice the reputation of Canada.

Open Public Data Is: Open Licenced

Government of Canada Open Data Licence Agreement for Unrestricted Use of Canada's Data:

5.3 You shall not use the data made available through the GC Open Data Portal in any way which, in the opinion of Canada, may bring disrepute to or prejudice the reputation of Canada.

Removed on March 18, 2011.

Open Public Data Is: Open Licenced

City of Vancouver Open Licence:

  • World-wide, royalty-free, non-exclusive licence
  • Use, modify, and distribute for any lawful purpose
  • Attribution not required
  • Share-alike required X
  • No right to sue City for how it provides data
  • City not liable if you abuse data
  • Disclaim endorsement / association

Why Open Public Data?

Why Open Public Data?

Open public data encourages:

  • Transparency
  • Engagement
  • Serendipity
  • Economy

Why Open Public Data? Transparency

Open public data encourages:

  • Transparency
  • Engagement
  • Serendipity
  • Economy

Why Open Public Data? Transparency

Monthly Building Permits:

  • Media article tied rising permit values to lower commercial tax rates
  • Building permit data only available in PDF summaries
  • Manually built database of permit results by month, type
  • Most permits are for residential development

Why Open Public Data? Transparency

Monthly Activity Stacked Bar Chart

Why Open Public Data? Transparency

Truck traffic:

  • City studied truck route
  • Citizens advocated removing residential streets near schools, parks
  • Staff denied requests on basis of traffic flow
  • Citizens asked city to share data
  • Volunteers had to count trucks over 24 hours
  • Council agreed to remove route

Why Open Public Data? Transparency

Crime data:

  • Tried to access crime stats by neighbourhood
  • Police said, "information you are looking for is not available"
  • No way for crime database to export data
  • Stats must be compiled manually, event by event
  • NICHE RMS will allow possibility for export
  • Will still require programming to add functionality

Why Open Public Data? Engagement

Open public data encourages:

  • Transparency
  • Engagement
  • Serendipity
  • Economy

Why Open Public Data? Engagement

Raise the Hammer Elections web page

Why Open Public Data? Engagement

Raise the Hammer Elections web page

Why Open Public Data? Engagement

Ward 2 candidate response table

Why Open Public Data? Engagement

Let your vote be your voice

Why Open Public Data? Engagement

Gender of Candidates

Why Open Public Data? Serendipity

Open public data encourages:

  • Transparency
  • Engagement
  • Serendipity
  • Economy

Why Open Public Data? Serendipity

Steven Johnson, Where Good Ideas Come From

  • An idea is a network
  • Ideas are cobbled together from parts already lying around
  • You have half an idea, somebody else has the other half, and if you're in the right environment, they turn into something larger than the sum of their parts
  • Chance favours the connected mind

Why Open Public Data? Serendipity

Boston Subway Map

Why Open Public Data? Serendipity

School crossing guard locations

Why Open Public Data? Economy

Open public data encourages:

  • Transparency
  • Engagement
  • Serendipity
  • Economy

Why Open Public Data? Economy

Efficiency multiplied thousands of times:

  • Create a resource using public data
  • Share the resource
  • Use it again and again and again

Why Open Public Data? Economy

Efficiency multiplied thousands of times:

That's the beauty of the open source model: One person has an itch to scratch, everyone benefits.

-- Where is my Street Car developers

Why Open Public Data? Economy

Where is my Street Car?

Objections:

Objections:

Arguments against open public data include:

  • Cost - making public data accessible costs money
  • Attacks - public data can be used for political attacks
  • Confusion - raw data could confuse people

Objections: Cost

Arguments against open public data include:

  • Cost - making public data accessible costs money
  • Attacks - public data can be used for political attacks
  • Confusion - raw data could confuse people

Objections: Cost

The status quo is costly:

  • Silo mentality is extremely inefficient
  • City staff frustrated trying to obtain data
  • Read-only formats cost time and money to access

Objections: Cost

3 ways open public data boosts productivity:

  • Less converting data from accessible formats to inaccessible formats
  • Less trying to obtain data from other departments
  • Less converting data from inaccessible formats back into accessible formats

Objections: Cost

Proprietary service provisioning:

  • Consultants are expensive
  • Proprietary technologies
  • Restrictive licences
  • Changes / enhancements slow, expensive

Objections: Cost

Community service provisioning:

  • Free
  • Open standard technologies
  • Generous licences
  • Anyone can contribute or fork
  • Does not prevent hiring consultants

Objections: Attacks

Arguments against open public data include:

  • Cost - making public data accessible costs money
  • Attacks - public data can be used for political attacks
  • Confusion - raw data could confuse people

Objections: Attacks

Concerns about using public data to attack politicians:

Accountability is very good, but political witch-hunting is not.

-- Larry Di Ianni

Objections: Attacks

Open public data can reduce effectiveness of attacks:

  • Absence of open public data has not prevented attacks
  • Good data counters emotion with information
  • Good data counters opinion with facts
  • Peer review keeps data users honest

Objections: Confusion

Arguments against open public data include:

  • Cost - making public data accessible costs money
  • Attacks - public data can be used for political attacks
  • Confusion - raw data could confuse people

Objections: Confusion

Open public data can reduce confusion:

  • Citizens use information together, not atomically
  • Platform building is cumulative
  • Expanded opportunities for visualization

Objections: Confusion

Japan Quake Map

Objections: Confusion

Percent of votes for Rob Ford in 2010 election

Open Public Data in Other Cities

Open Public Data in Other Cities

Vancouver

Open Public Data in Other Cities

Nanaimo

Open Public Data in Other Cities

Calgary

Open Public Data in Other Cities

Edmonton

Open Public Data in Other Cities

Toronto

Open Public Data in Other Cities

Ottawa

Open Public Data in Hamilton

Open Public Data in Hamilton

We're already an open data city!

Open data in Hamilton

http://www.hamilton.ca/ProjectsInitiatives/OpenData/

Transit data already sent to Google Transit was shared with Mohawk College students to create mobile apps

Open Public Data in Hamilton

City of Hamilton Website Acceptable Use Agreement:

Unless prior written permission is obtained from the City, the Content's owner and the City's licensors, You may not reproduce, publish, copy, link to, frame, tag, embed, merge, modify, recompile, license, distribute, sell, store in an electronic retrieval system, download (except by the browser of a single user) or transmit, in while or in part, in any form or by any means whatsoever, be they physical, electronic or otherwise, the Portal and/or the Content.

Open Public Data in Hamilton

Signs of change:

Open data is part of a transformation agenda. ... Information is traditionally seen as power, and opening it up means sending a message within the City administration that this is the public's information and not ours to control. Opening data is just one means to demonstrate and strengthen value for the citizens' tax dollars.

-- Chris Murray, City Manager

Open Public Data in Hamilton

Signs of Change:

Open Hamilton

Open Public Data in Hamilton

Open Hamilton Draft Motion: Open Data in Hamilton

  • Open and Accessible Data - the City of Hamilton will freely share with citizens, businesses and other jurisdictions the greatest amount of data possible while respecting privacy and security concerns;
  • Open Standards - the City of Hamilton will move as quickly as possible to adopt prevailing open standards for data, documents, maps, and other formats of media
  • Open Source Software - the City of Hamilton, when replacing existing software or considering new applications, will place open source software on an equal footing with commercial systems during procurement cycles

Open Public Data in Hamilton

Hamilton Spectator supports open public data

Conclusion: Government as Platform

Conclusion: Government as Platform

  1. Embrace open standards that encourage innovation
  2. Start simple and evolve iteratively
  3. Design for wide participation
  4. Learn and take leads from users
  5. Lower barriers to experimentation
  6. Build a culture of measurement
  7. Celebrate developers instead of merely tolerating them
  8. Re-use existing open source solutions

http://www.slideshare.net/timoreilly/government-as-platform