DRW: Add "CKAN" as an additional category for this page

2023-11-21T18:22:26Z

Add "CKAN" as an additional category for this page

← Older revision		Revision as of 18:22, 21 November 2023
Line 20:		Line 20:
	To avoid keeping local databases about datasets, store such information (such as the last time an ETL job was run on a given package) in the 'extras' metadata field of the CKAN package, as much as possible. This stores information in a centralized location so ETL jobs can be run from multiple computers without any other coordination. The extras metadata fields are currently cataloged on the [[CKAN_Metadata]] page.		To avoid keeping local databases about datasets, store such information (such as the last time an ETL job was run on a given package) in the 'extras' metadata field of the CKAN package, as much as possible. This stores information in a centralized location so ETL jobs can be run from multiple computers without any other coordination. The extras metadata fields are currently cataloged on the [[CKAN_Metadata]] page.

	[[Category:Onboarding]]		[[Category:Onboarding]] [[Category:CKAN]]

DRW: Add CKAN Tricks page

2022-04-28T17:06:59Z

Add CKAN Tricks page

New page

== Manage datasets ==
=== Undelete deleted datasets ===
If you know the URL of the deleted dataset AND you are logged in as an administrator, you can go to that URL in your web browser and see the deleted dataset. "[Deleted]" will have been appended to the title of the dataset, and the value of the metadata field <code>state</code> will be equal to "deleted".

To undelete such a dataset, just use the CKAN API or use the
[https://github.com/WPRDC/utility-belt/blob/master/gadgets.py#L767 set_package_parameters_to_values() function] to set the package's <code>state</code> metadata value to "active". For the latter option, invoke the function like this:
<code>set_package_parameters_to_values("https://data.wprdc.org", deleted_package_id, ['state'], ['active'], API_key)</code>

== Queries ==
=== Make queries faster ===
[https://ckan.org/2017/08/10/faster-datastore-in-ckan-2-7/ Speed up datastore_search queries] by including the <code>include_total=False</code> parameter to skip calculation of the total number of rows (which can reduce response time by a factor of 2). The [https://docs.ckan.org/en/ckan-2.7.3/maintaining/datastore.html#ckanext.datastore.logic.action.datastore_search datastore_search API call] lets you search a given datastore by column values and return subsets of the records. There's more on benchmarking CKAN performance [http://urbanopus.net/benchmarking-the-ckan-datastore-api/ here].

Another way to speed up datastore search queries is to index fields used in the filtering. Note that (at least when the primary key is a combination of fields), if you don't list each primary key field as a separate field to index, those fields don't get indexed and queries take way longer.

=== Avoiding stale query caches ===
Queries/API responses can be cached based upon nginx settings. If you find that your SQL query is getting a stale response, try changing your query slightly. For instance, instead of `SELECT MAX(some_field) AS biggest FROM <resource-id>`, you could change the assigned variable name (`SELECT MAX(some_field) AS biggest0413 FROM <resource-id>`) or add another field that you ignore (`SELECT MAX(some_field) AS biggest, MAX(some_field) AS whatever FROM <resource-id>`).

== Scripts that interact with CKAN through the API ==
=== Run those CKAN-monitoring/modifying scripts from multiple servers by centralizing data ===
To avoid keeping local databases about datasets, store such information (such as the last time an ETL job was run on a given package) in the 'extras' metadata field of the CKAN package, as much as possible. This stores information in a centralized location so ETL jobs can be run from multiple computers without any other coordination. The extras metadata fields are currently cataloged on the [[CKAN_Metadata]] page.

[[Category:Onboarding]]

CKAN Tricks - Revision history

DRW: Add "CKAN" as an additional category for this page

DRW: Add CKAN Tricks page