The NUMA API
Posted: 2010.04.11 (14:32)
Recently, I've been asked by a couple of individuals for programmatic access to NUMA's data. With that in mind, I've created a simple API, which should facilitate this and other uses. The operation of the API is detailed below.
The API is broken up into two parts, the query API and the firehose API. Both return information in a common format, which is documented last.
The API is a JSON-based service, and is read-only. A GET request to the appropriate URL will return JSON data in the response.
Query API
The query API allows you to fetch detailed information about a single map. To do so, make a request to '/{map_id}/mapdata' for map data, or /{map_id}/commentdata' for comment data. The mapdata endpoint will return a JSON dictionary with information about the map, in the format detailed below for map data. The commentdata endpoint will return a JSON list of dictionaries with information about every comment made on the map, or 1000 comments, whichever is smaller, in the format detailed below for comment data.
Firehose API
The firehose API allows you to create a dump of all the map and comment data in NUMA. Firehose API endpoints take two query string parameters, both optional. 'count' specifies the number of entries to return, and defaults to 100. 'start' specifies a start point, and defaults to starting at the oldest entry.
All firehose API endpoints return a dict with three keys: 'count', being the actual number of results returned, 'data', being a list of dictionaries in the format described below for the appropriate type of data, and 'next', which is an opaque string that can be used to fetch the next set of results.
To fetch all the map or comment data from NUMA, use the following procedure:
1) Make a request to the appropriate endpoint with count=x and no start parameter (Eg, http://www.nmaps.net/api/maps?count=100), to return the first x results. It will return a JSON dict of the form {'count': 100, 'data': [...], 'start': 'somestring'}.
2) Make a request to the same URL, with the 'start' parameter set to the value returned by the previous call (Eg, http://www.nmaps.net/api/maps?count=100 ... somestring) to return the next x results. The returned data will be in the same format as above.
3) Repeat step 2 until fewer than the requested number of results are returned. When a request returns fewer than the expected number of results, store the returned 'start' string and try again later, after some reasonable interval (Eg, once a day) to get new data.
The current endpoints for the firehose API are /api/maps and /api/comments. Returned entries are ordered by their 'last updated' date. For a comment, this is always when the comment was first submitted. The last updated date on a map is changed when the map is first submitted and when it is edited by the author. As such, you will see the same map occur multiple times in the feed if it is edited by the user. It is not changed if the map is rated, featured, or delisted, so fetching this information will require getting the map data from the query API. In future, the API may be updated to reinclude the map in the firehose feed on more criteria - eg, featuring, delisting, and possibly rating - or to include only the modified data in these events.
Result format
Both APIs use the same result format.
Maps are represented as a dictionary with the following keys:
map_id: The ID of the map
user: The username of the user who submitted the map
float_num: The 'float number' of the map, used for sorting in the hot maps page
name: The name or title of the map
description: The description of the map, in BBCode format.
mapdata: The map's N mapdata
rating: The map's current rating, as a floating point number
votes: The number of votes the map has received so far
disableratings: Boolean: True if the owner has disabled ratings on this map
unlisted: Boolean: True if this map has been delisted
comment_count: Current number of comments on the map
featured_by: If this map is featured, the username of the user who featured it
featured_text: The text of the map's feature review
tags: A list of tags for this map, including 'special' tags such as the author tag.
lastupdated: The datetime this map was last updated, in ISO timestamp format.
created: The datetime this map was created, in ISO timestamp format.
featured_date: The datetime this map was featured, in ISO timestamp format.
Comments are represented as a dictionary with the following keys:
map_id: The ID of the map this comment was made on
comment_id: The per-map comment ID of this comment
title: The title of the comment.
text: The comment's text, in BBCode format.
demodata: The comment's demodata, if any.
lastupdated: The datetime this comment was posted.
author: The username of the comment's author.
Usage
If you intend to use the firehose API, or to poll the Query API or otherwise use it heavily, please contact me here or on the forums. To reduce resource consumption, I would rather make sure all heavy users are using the API efficiently, and modify the API if necessary, than sustain a heavy polling load.
The API is broken up into two parts, the query API and the firehose API. Both return information in a common format, which is documented last.
The API is a JSON-based service, and is read-only. A GET request to the appropriate URL will return JSON data in the response.
Query API
The query API allows you to fetch detailed information about a single map. To do so, make a request to '/{map_id}/mapdata' for map data, or /{map_id}/commentdata' for comment data. The mapdata endpoint will return a JSON dictionary with information about the map, in the format detailed below for map data. The commentdata endpoint will return a JSON list of dictionaries with information about every comment made on the map, or 1000 comments, whichever is smaller, in the format detailed below for comment data.
Firehose API
The firehose API allows you to create a dump of all the map and comment data in NUMA. Firehose API endpoints take two query string parameters, both optional. 'count' specifies the number of entries to return, and defaults to 100. 'start' specifies a start point, and defaults to starting at the oldest entry.
All firehose API endpoints return a dict with three keys: 'count', being the actual number of results returned, 'data', being a list of dictionaries in the format described below for the appropriate type of data, and 'next', which is an opaque string that can be used to fetch the next set of results.
To fetch all the map or comment data from NUMA, use the following procedure:
1) Make a request to the appropriate endpoint with count=x and no start parameter (Eg, http://www.nmaps.net/api/maps?count=100), to return the first x results. It will return a JSON dict of the form {'count': 100, 'data': [...], 'start': 'somestring'}.
2) Make a request to the same URL, with the 'start' parameter set to the value returned by the previous call (Eg, http://www.nmaps.net/api/maps?count=100 ... somestring) to return the next x results. The returned data will be in the same format as above.
3) Repeat step 2 until fewer than the requested number of results are returned. When a request returns fewer than the expected number of results, store the returned 'start' string and try again later, after some reasonable interval (Eg, once a day) to get new data.
The current endpoints for the firehose API are /api/maps and /api/comments. Returned entries are ordered by their 'last updated' date. For a comment, this is always when the comment was first submitted. The last updated date on a map is changed when the map is first submitted and when it is edited by the author. As such, you will see the same map occur multiple times in the feed if it is edited by the user. It is not changed if the map is rated, featured, or delisted, so fetching this information will require getting the map data from the query API. In future, the API may be updated to reinclude the map in the firehose feed on more criteria - eg, featuring, delisting, and possibly rating - or to include only the modified data in these events.
Result format
Both APIs use the same result format.
Maps are represented as a dictionary with the following keys:
map_id: The ID of the map
user: The username of the user who submitted the map
float_num: The 'float number' of the map, used for sorting in the hot maps page
name: The name or title of the map
description: The description of the map, in BBCode format.
mapdata: The map's N mapdata
rating: The map's current rating, as a floating point number
votes: The number of votes the map has received so far
disableratings: Boolean: True if the owner has disabled ratings on this map
unlisted: Boolean: True if this map has been delisted
comment_count: Current number of comments on the map
featured_by: If this map is featured, the username of the user who featured it
featured_text: The text of the map's feature review
tags: A list of tags for this map, including 'special' tags such as the author tag.
lastupdated: The datetime this map was last updated, in ISO timestamp format.
created: The datetime this map was created, in ISO timestamp format.
featured_date: The datetime this map was featured, in ISO timestamp format.
Comments are represented as a dictionary with the following keys:
map_id: The ID of the map this comment was made on
comment_id: The per-map comment ID of this comment
title: The title of the comment.
text: The comment's text, in BBCode format.
demodata: The comment's demodata, if any.
lastupdated: The datetime this comment was posted.
author: The username of the comment's author.
Usage
If you intend to use the firehose API, or to poll the Query API or otherwise use it heavily, please contact me here or on the forums. To reduce resource consumption, I would rather make sure all heavy users are using the API efficiently, and modify the API if necessary, than sustain a heavy polling load.