Understanding Open Public Data
Ryan McGreal, Raise the Hammer
CIPS - Golden Horseshoe Monthly Event
Wednesday, March 23, 2011
Wednesday, March 23, 2011
What is Open Public Data?
Open Public Data Is:
Information created or collected by the government that is:
- Publicly available
- Findable
- Accessible
- Machine-readable
- Open licence
Open Public Data Is: Publicly Available
Information created or collected by the government that is:
- Publicly available
- Findable
- Accessible
- Machine-readable
- Open licence
Open Public Data Is: Publicly Available
Municipal Freedom of Information and Protection of Privacy Act (MFIPPA):
4. (1) Every person has a right of access to a record or a part of a record in the custody or under the control of an institution unless,
(a) the record or the part of the record falls within one of the exemptions ... or
(b) the head is of the opinion on reasonable grounds that the request for access is frivolous or vexatious.
Open Public Data Is: Publicly Available
Two Purposes of MFIPPA:
(a) to provide a right of access to information under the control of institutions in accordance with the principles that
(i) information should be available to the public,
(ii) necessary exemptions ... should be limited and specific, and
(iii) decisions on the disclosure of information should be reviewed independently of the institution controlling the information;
Open Public Data Is: Publicly Available
Two Purposes of MFIPPA:
(b) to protect the privacy of individuals with respect to personal information about themselves held by institutions and to provide individuals with a right of access to that information.
Open Public Data Is: Publicly Available
Note on exemptions:
An exemption from disclosure of a record ... does not apply if a compelling public interest in the disclosure of the record clearly outweighs the purpose of the exemption.
Open Public Data Is: Findable
Information created or collected by the government that is:
- Publicly available
- Findable
- Accessible
- Machine-readable
- Open licence
Open Public Data Is: Findable
Open Public Data Is: Findable
Open Public Data Is: Findable
Findable means:
- Clear titles, descriptions written in plain language
- Distinct resources (orthogonality)
- Search engine friendly - people use search, not hierarchy
- Clean, 'hackable' URLs
Open Public Data Is: Findable
Not findable:
Open Public Data Is: Accessible
Information created or collected by the government that is:
- Publicly available
- Findable
- Accessible
- Machine-readable
- Open licence
Open Public Data Is: Accessible
Accessible means:
- Clean, semantic HTML for documents
- No read-only text formats (PDF, scans)
- Open standard formats for structured data
- Can be linked directly, i.e. not behind:
- Form POST events
- Javascript redirects
- Session querystrings
- Proprietary UI, e.g. Java
- Easy for others to share
- People of all abilities and disabilities can access it
Open Public Data Is: Machine-Readable
Information created or collected by the government that is:
- Publicly available
- Findable
- Accessible
- Machine-readable
- Open licence
Open Public Data Is: Machine-Readable
Standard formats for structured data:
Open Public Data Is: Machine-Readable
Web service API:
Open Public Data Is: Open Licenced
Information created or collected by the government that is:
- Publicly available
- Findable
- Accessible
- Machine-readable
- Open licenced
Open Public Data Is: Open Licenced
Rights:
- Copy, publish, distribute and transmit
- Adapt and combine
- Use commercially in a product or service
Responsibilities:
- Attribute data sources
- Disclaim endorsement
- Lawful use
Open Public Data Is: Open Licenced
The British Open Government Licence for public sector information:
- 950 words
- Clear language
- Permissive terms
- Standard across all British public bodies
Open Public Data Is: Open Licenced
Open Data Commons ODC Public Domain Dedication and Licence (PDDL):
- Permissive terms: share, create, adapt
- Includes a plain language summary (with pictures!)
Open Public Data Is: Open Licenced
U.S. Data.gov Data Policy:
- 652 words
- U.S. public data is public domain
- No copyright restriction
Open Public Data Is: Open Licenced
Government of Canada Open Data Licence Agreement for Unrestricted Use of Canada's Data:
- Published March 17, 2011
- Royalty-free, non-exclusive, worldwide, non-assignable
- Use, reproduce, extract, modify, translate, develop and distribute
- Attribution: "Reproduced and distributed with the permission of the Government of Canada"
- No identifying individuals, families or households X
Open Public Data Is: Open Licenced
Government of Canada Open Data Licence Agreement for Unrestricted Use of Canada's Data:
5.3 You shall not use the data made available through the GC Open Data Portal in any way which, in the opinion of Canada, may bring disrepute to or prejudice the reputation of Canada.
Open Public Data Is: Open Licenced
Government of Canada Open Data Licence Agreement for Unrestricted Use of Canada's Data:
5.3 You shall not use the data made available through the GC Open Data Portal in any way which, in the opinion of Canada, may bring disrepute to or prejudice the reputation of Canada.
Removed on March 18, 2011.
Open Public Data Is: Open Licenced
City of Vancouver Open Licence:
- World-wide, royalty-free, non-exclusive licence
- Use, modify, and distribute for any lawful purpose
- Attribution not required
- Share-alike required X
- No right to sue City for how it provides data
- City not liable if you abuse data
- Disclaim endorsement / association
Why Open Public Data?
Why Open Public Data?
Open public data encourages:
- Transparency
- Engagement
- Serendipity
- Economy
Why Open Public Data? Transparency
Open public data encourages:
- Transparency
- Engagement
- Serendipity
- Economy
Why Open Public Data? Transparency
Monthly Building Permits:
- Media article tied rising permit values to lower commercial tax rates
- Building permit data only available in PDF summaries
- Manually built database of permit results by month, type
- Most permits are for residential development
Why Open Public Data? Transparency
Why Open Public Data? Transparency
Truck traffic:
- City studied truck route
- Citizens advocated removing residential streets near schools, parks
- Staff denied requests on basis of traffic flow
- Citizens asked city to share data
- Volunteers had to count trucks over 24 hours
- Council agreed to remove route
Why Open Public Data? Transparency
Crime data:
- Tried to access crime stats by neighbourhood
- Police said, "information you are looking for is not available"
- No way for crime database to export data
- Stats must be compiled manually, event by event
- NICHE RMS will allow possibility for export
- Will still require programming to add functionality
Why Open Public Data? Engagement
Open public data encourages:
- Transparency
- Engagement
- Serendipity
- Economy
Why Open Public Data? Engagement
Why Open Public Data? Engagement
Why Open Public Data? Engagement
Why Open Public Data? Engagement
Why Open Public Data? Engagement
Why Open Public Data? Serendipity
Open public data encourages:
- Transparency
- Engagement
- Serendipity
- Economy
Why Open Public Data? Serendipity
Steven Johnson, Where Good Ideas Come From
- An idea is a network
- Ideas are cobbled together from parts already lying around
- You have half an idea, somebody else has the other half, and if you're in the right environment, they turn into something larger than the sum of their parts
- Chance favours the connected mind
Why Open Public Data? Serendipity
Why Open Public Data? Serendipity
Why Open Public Data? Economy
Open public data encourages:
- Transparency
- Engagement
- Serendipity
- Economy
Why Open Public Data? Economy
Efficiency multiplied thousands of times:
- Create a resource using public data
- Share the resource
- Use it again and again and again
Why Open Public Data? Economy
Efficiency multiplied thousands of times:
That's the beauty of the open source model: One person has an itch to scratch, everyone benefits.
-- Where is my Street Car developers
Why Open Public Data? Economy
Objections:
Objections:
Arguments against open public data include:
- Cost - making public data accessible costs money
- Attacks - public data can be used for political attacks
- Confusion - raw data could confuse people
Objections: Cost
Arguments against open public data include:
- Cost - making public data accessible costs money
- Attacks - public data can be used for political attacks
- Confusion - raw data could confuse people
Objections: Cost
The status quo is costly:
- Silo mentality is extremely inefficient
- City staff frustrated trying to obtain data
- Read-only formats cost time and money to access
Objections: Cost
3 ways open public data boosts productivity:
- Less converting data from accessible formats to inaccessible formats
- Less trying to obtain data from other departments
- Less converting data from inaccessible formats back into accessible formats
Objections: Cost
Proprietary service provisioning:
- Consultants are expensive
- Proprietary technologies
- Restrictive licences
- Changes / enhancements slow, expensive
Objections: Cost
Community service provisioning:
- Free
- Open standard technologies
- Generous licences
- Anyone can contribute or fork
- Does not prevent hiring consultants
Objections: Attacks
Arguments against open public data include:
- Cost - making public data accessible costs money
- Attacks - public data can be used for political attacks
- Confusion - raw data could confuse people
Objections: Attacks
Concerns about using public data to attack politicians:
Accountability is very good, but political witch-hunting is not.
-- Larry Di Ianni
Objections: Attacks
Open public data can reduce effectiveness of attacks:
- Absence of open public data has not prevented attacks
- Good data counters emotion with information
- Good data counters opinion with facts
- Peer review keeps data users honest
Objections: Confusion
Arguments against open public data include:
- Cost - making public data accessible costs money
- Attacks - public data can be used for political attacks
- Confusion - raw data could confuse people
Objections: Confusion
Open public data can reduce confusion:
- Citizens use information together, not atomically
- Platform building is cumulative
- Expanded opportunities for visualization
Objections: Confusion
Objections: Confusion
Open Public Data in Other Cities
Open Public Data in Other Cities
Open Public Data in Other Cities
Open Public Data in Other Cities
Open Public Data in Other Cities
Open Public Data in Other Cities
Open Public Data in Other Cities
Open Public Data in Hamilton
Open Public Data in Hamilton
We're already an open data city!
http://www.hamilton.ca/ProjectsInitiatives/OpenData/
Transit data already sent to Google Transit was shared with Mohawk College students to create mobile apps
Open Public Data in Hamilton
City of Hamilton Website Acceptable Use Agreement:
Unless prior written permission is obtained from the City, the Content's owner and the City's licensors, You may not reproduce, publish, copy, link to, frame, tag, embed, merge, modify, recompile, license, distribute, sell, store in an electronic retrieval system, download (except by the browser of a single user) or transmit, in while or in part, in any form or by any means whatsoever, be they physical, electronic or otherwise, the Portal and/or the Content.
Open Public Data in Hamilton
Signs of change:
Open data is part of a transformation agenda. ... Information is traditionally seen as power, and opening it up means sending a message within the City administration that this is the public's information and not ours to control. Opening data is just one means to demonstrate and strengthen value for the citizens' tax dollars.
-- Chris Murray, City Manager
Open Public Data in Hamilton
Signs of Change:
Open Public Data in Hamilton
Open Hamilton Draft Motion: Open Data in Hamilton
- Open and Accessible Data - the City of Hamilton will freely share with citizens, businesses and other jurisdictions the greatest amount of data possible while respecting privacy and security concerns;
- Open Standards - the City of Hamilton will move as quickly as possible to adopt prevailing open standards for data, documents, maps, and other formats of media
- Open Source Software - the City of Hamilton, when replacing existing software or considering new applications, will place open source software on an equal footing with commercial systems during procurement cycles
Open Public Data in Hamilton
Conclusion: Government as Platform
Conclusion: Government as Platform
- Embrace open standards that encourage innovation
- Start simple and evolve iteratively
- Design for wide participation
- Learn and take leads from users
- Lower barriers to experimentation
- Build a culture of measurement
- Celebrate developers instead of merely tolerating them
- Re-use existing open source solutions