Package dictionary .. how it varies - ckan/ckan GitHub Wiki
Form to database
Here we follow the package in its various formats, from form submission to storing it in the database.
Form submission (new dataset)
- Parameters are serialized
add-resource-urlis only in the CKAN 1.7.1 form because you first say what sort of resource it is
UnicodeMultiDict(
[('title', u'OpenStreetMap'),
('name', u'openstreetmap'),
('url', u''),
('license_id', u'odc-pddl'),
('notes', u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.'),
('tag_string', u' 'map, geo spatial'),
('add-resource-url', u''),
('add-resource-url', u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'),
('resources__0__name', u'Planet dump'),
('resources__0__description', u'Full dump of the planet'),
('resources__0__url', u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'),
('resources__0__format', u'OSM'),
('resources__0__resource_type', u'file'),
('resources__0__webstore_url', u''),
('resources__0__last_modified', u''),
('resources__0__size', u'234555'),
('resources__0__mimetype', u''),
('resources__0__mimetype_inner', u''),
('resources__0__id', u''),
('resources__0__hash', u''),
('resources__0__KEY', u'VALUE'),
('author', u''),
('author_email', u''),
('maintainer', u''),
('maintainer_email', u''),
('version', u''),
('extras__0__key', u'location'),
('extras__0__value', u'uk'),
('extras__1__key', u''),
('extras__1__value', u''),
('extras__2__key', u''),
('extras__2__value', u''),
('extras__3__key', u''),
('extras__3__value', u''),
('log_message', u'Created new dataset.'),
('save', u'Add Dataset')
])
Tuplized (package controller)
- Dictionary
- Keyed by a tuple containing the index and whether it is key or value, to make it easier to group them in the next 'unflatten' stage.
{
('title',): u'OpenStreetMap',
('name',): u'openstreetmap',
('url',): u'',
('license_id',): u'odc-pddl',
('notes',): u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
('tag_string',): u'map, geo spatial',
('add-resource-url',): [u'',
u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'],
('resources', 0, 'KEY'): u'VALUE',
('resources', 0, 'description'): u'Full dump of the planet',
('resources', 0, 'format'): u'OSM',
('resources', 0, 'hash'): u'',
('resources', 0, 'id'): u'',
('resources', 0, 'last_modified'): u'',
('resources', 0, 'mimetype'): u'',
('resources', 0, 'mimetype_inner'): u'',
('resources', 0, 'name'): u'Planet dump',
('resources', 0, 'resource_type'): u'file',
('resources', 0, 'size'): u'234555',
('resources', 0, 'url'): u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
('resources', 0, 'webstore_url'): u'',
('author',): u'',
('author_email',): u'',
('maintainer',): u'',
('maintainer_email',): u'',
('version',): u''
('extras', 0, 'key'): u'location',
('extras', 0, 'value'): u'uk',
('extras', 1, 'key'): u'',
('extras', 1, 'value'): u'',
('extras', 2, 'key'): u'',
('extras', 2, 'value'): u'',
('extras', 3, 'key'): u'',
('extras', 3, 'value'): u'',
('log_message',): u'Created new dataset.',
('save',): u'Add Dataset',
}
unflattened (package controller)
- converts to lists of dicts: extras, resources
{
'title': u'OpenStreetMap',
'name': u'openstreetmap',
'url': u'',
'license_id': u'odc-pddl',
'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
'tag_string': u'map, geo spatial',
'add-resource-url': [u'',
u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'],
'resources': [{'KEY': u'VALUE',
'description': u'Full dump of the planet',
'format': u'OSM',
'hash': u'',
'id': u'',
'last_modified': u'',
'mimetype': u'',
'mimetype_inner': u'',
'name': u'Planet dump',
'resource_type': u'file',
'size': u'234555',
'url': u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
'webstore_url': u''}],
'author': u'',
'author_email': u'',
'maintainer': u'',
'maintainer_email': u'',
'version': u''
'extras': [{'key': u'location', 'value': u'uk'},
{'key': u'', 'value': u''},
{'key': u'', 'value': u''},
{'key': u'', 'value': u''}],
'log_message': u'Created new dataset.',
'save': u'Add Dataset',
}
Cleaned (package controller)
- Removes blank values
- This is the format of the package returned by package_show if
context{'validate': False}which is the one used byckan.lib.search:SynchronousSearchPluginfor SOLR indexing.
{
'title': u'OpenStreetMap',
'name': u'openstreetmap',
'url': u'',
'license_id': u'odc-pddl',
'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
'tag_string': u'map, geo spatial',
'add-resource-url': [u'',
u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'],
'resources': [{'KEY': u'VALUE',
'description': u'Full dump of the planet',
'format': u'OSM',
'hash': u'',
'id': u'',
'last_modified': u'',
'mimetype': u'',
'mimetype_inner': u'',
'name': u'Planet dump',
'resource_type': u'file',
'size': u'234555',
'url': u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
'webstore_url': u''}],
'author': u'',
'author_email': u'',
'maintainer': u'',
'maintainer_email': u'',
'version': u''
'extras': [{'key': u'location', 'value': u'uk'}],
'log_message': u'Created new dataset.',
'save': u'Add Dataset',
}
And 'type': None (implies default - 'package') is inserted (for example) before sending to package_create.
Validated (package_create)
- 'validate' also does various little transforms, like converting
tag_stringtotags. These are all defined in the schema. Default schema is ckan.logic.schema.default_create_package_schema add-resource-urland 'save' are removed finally
{
'title': u'OpenStreetMap',
'name': u'openstreetmap',
'url': u'',
'license_id': u'odc-pddl',
'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
'tag_string': u'map, geo spatial',
'tags': [{'name': u'map'}, {'name': u'geo spatial'}],
'resources': [{'KEY': u'VALUE',
'description': u'Full dump of the planet',
'format': u'OSM',
'hash': u'',
'last_modified': None,
'mimetype': u'',
'mimetype_inner': u'',
'name': u'Planet dump',
'resource_type': u'file',
'size': 234555,
'url': u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
'webstore_url': u''}],
'author': u'',
'author_email': u'',
'maintainer': u'',
'maintainer_email': u'',
'extras': [{'key': u'location', 'value': u'uk'}],
'version': u''
'log_message': u'Created new dataset.',
}
Database
package:
id: 'e3f0339e-e548-42e3-a194-2f5b09fe3c2d'
name: 'openstreetmap'
title: 'OpenStreetMap'
version: ''
url: ''
notes: 'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.'
license_id: 'odc-pddl'
revision_id: 'bc088d17-7826-4d75-afb7-eea464704daf'
author: '' author_email: '' maintainer: '' maintainer_email: ''
state: 'active'
type: ''
resource:
id: 'd5a3ca4b-10b7-401d-8152-f7e69a818582'
resource_group_id: '3ea44f6e-ac7d-4fcf-8a82-0da1e72a6898'
name: 'Planet dump'
url: 'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'
format: 'OSM'
description: 'Full dump of the planet'
extras: {"KEY": "VALUE"}
position: 0
revision_id: 'c2b583aa-1ab6-4e33-80f6-da29a78ffe63'
hash: ''
state: 'active'
resource_type: 'file'
size: 234555
mimetype: '' mimetype_inner: '' last_modified: Null cache_url: '' cache_last_updated: Null webstore_url: '' webstore_last_updated: ''
created: 2013-08-02 20:57:06.359635
package_extra:
id: '43f3cbb8-f7f2-428d-b998-11a51aafd985'
package_id: 'e3f0339e-e548-42e3-a194-2f5b09fe3c2d'
key: 'location'
value: "uk"
revision_id: 'c2b583aa-1ab6-4e33-80f6-da29a78ffe63'
state: 'active'
Non-validating form
- If the validation stage has errors, then the form is to be redrawn with the previously submitted values in the boxes. This is the package dict which is passed to the template, which is the same as the Cleaned Dict above.
{
'title': u'OpenStreetMap',
'name': u'openstreetmap',
'url': u'',
'license_id': u'odc-pddl',
'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
'tag_string': u'map, geo spatial',
'add-resource-url': [u'', u''],
'resources': [{'description': u'Full dump of the planet',
'format': u'OSM',
'hash': u'',
'id': u'',
'last_modified': u'',
'mimetype': u'',
'mimetype_inner': u'',
'name': u'Planet dump',
'resource_type': u'api',
'size': 234555,
'url': u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
'webstore_url': u''}],
'author': u'',
'author_email': u'',
'maintainer': u'',
'maintainer_email': u'',
'version': u''
'extras': [{'key': u'location', 'value': u'uk'}],
'log_message': u'Created new dataset.',
'save': u'Add Dataset',
'type': None,
}
package_show
If you call package_show on the API (/api/action/package_show?id=openstreetmap) then you get:
- usual API fields 'success' and 'help' with the package in 'result'
- 'license_title', 'license_url' and 'isopen' added, based on license_id
- resources now have 'position', 'tracking_summary'
- 'relationships_as_object', 'relationships_as_subject', 'id', 'state', 'tracking_summary', 'revision_timestamp' added
{
"success" : true,
"help" : null,
"result" : {
"title" : "OpenStreetMap",
"name" : "openstreetmap",
"url" : "",
"license_id" : "odc-pddl",
"license_url" : "http://www.opendefinition.org/licenses/odc-pddl"
"license_title" : "Open Data Commons Public Domain Dedication and Licence (PDDL)",
"notes" : "OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.",
"tags" : [
{
"name" : "geo spatial",
"display_name" : "geo spatial",
"id" : "5bbb3684-e8a2-47a2-b61c-3991239d1927",
"revision_timestamp" : "2013-08-02T20:44:41.421427",
"vocabulary_id" : null,
"state" : "active"
},
{
"name" : "map",
"display_name" : "map",
"id" : "5b97bbba-02bb-4416-aa7f-453336e96218",
"revision_timestamp" : "2013-08-02T20:44:41.421427",
"vocabulary_id" : null,
"state" : "active"
}
],
"resources" : [
{
"position" : 0,
"state" : "active",
"size" : 234555,
"cache_last_updated" : null,
"url" : "http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2",
"id" : "c83244dd-2089-4f2b-8a8e-50b0c04ff7bd",
"resource_type" : "api",
"revision_timestamp" : "2013-08-02T20:36:35.453823",
"tracking_summary" : {
"recent" : 0,
"total" : 0
},
"resource_group_id" : "aa82bbce-e84b-462b-b366-cbda6682c57c",
"webstore_last_updated" : null,
"cache_url" : null,
"last_modified" : null,
"name" : "Planet dump",
"description" : "Full dump of the planet",
"revision_id" : "6ca47e2d-04e0-4a95-b472-45e999cdc716",
"created" : "2013-08-02T21:36:35.491343",
"hash" : "",
"format" : "OSM",
"webstore_url" : "",
"mimetype_inner" : "",
"mimetype" : ""
}
],
"author" : "",
"author_email" : "",
"maintainer" : "",
"maintainer_email" : "",
"version" : "",
"extras" : [
{
"key" : "location",
"value" : "\"uk\"",
"id" : "dff97ec1-66eb-4167-9fef-b30296b0fa7e",
"revision_id" : "9833f60b-5038-42ea-b20f-14976c0e64f9",
"revision_timestamp" : "2013-08-02T20:44:41.421427",
"package_id" : "57323b25-af96-484c-b5ba-2714257042e8",
"state" : "active"
}
],
"isopen" : true,
"groups" : [],
"id" : "57323b25-af96-484c-b5ba-2714257042e8",
"tracking_summary" : {
"recent" : 0,
"total" : 0
},
"revision_timestamp" : "2013-08-02T20:36:35.453823",
"revision_id" : "6ca47e2d-04e0-4a95-b472-45e999cdc716",
"relationships_as_subject" : [],
"relationships_as_object" : [],
"metadata_modified" : "2013-08-02T20:36:35.453823",
"type" : null,
"metadata_created" : "2013-08-02T20:36:35.453823",
"state" : "active",
}
}
- If you do
context['for_edit'] = Truethen you don't get tracking_summary. - If you do
context['for_view'] = Truethen it will call IPackageController.before_view