The NUMA API

Talk about the Nmaps.net website.

Moderators: Rose, Sunset

Plus (Size) Member
Posts: 47
Joined: 2008.09.30 (12:52)

Postby Arachnid » 2010.04.11 (14:32)

Recently, I've been asked by a couple of individuals for programmatic access to NUMA's data. With that in mind, I've created a simple API, which should facilitate this and other uses. The operation of the API is detailed below.

The API is broken up into two parts, the query API and the firehose API. Both return information in a common format, which is documented last.

The API is a JSON-based service, and is read-only. A GET request to the appropriate URL will return JSON data in the response.

Query API

The query API allows you to fetch detailed information about a single map. To do so, make a request to '/{map_id}/mapdata' for map data, or /{map_id}/commentdata' for comment data. The mapdata endpoint will return a JSON dictionary with information about the map, in the format detailed below for map data. The commentdata endpoint will return a JSON list of dictionaries with information about every comment made on the map, or 1000 comments, whichever is smaller, in the format detailed below for comment data.

Firehose API

The firehose API allows you to create a dump of all the map and comment data in NUMA. Firehose API endpoints take two query string parameters, both optional. 'count' specifies the number of entries to return, and defaults to 100. 'start' specifies a start point, and defaults to starting at the oldest entry.

All firehose API endpoints return a dict with three keys: 'count', being the actual number of results returned, 'data', being a list of dictionaries in the format described below for the appropriate type of data, and 'next', which is an opaque string that can be used to fetch the next set of results.

To fetch all the map or comment data from NUMA, use the following procedure:
1) Make a request to the appropriate endpoint with count=x and no start parameter (Eg, http://www.nmaps.net/api/maps?count=100), to return the first x results. It will return a JSON dict of the form {'count': 100, 'data': [...], 'start': 'somestring'}.
2) Make a request to the same URL, with the 'start' parameter set to the value returned by the previous call (Eg, http://www.nmaps.net/api/maps?count=100 ... somestring) to return the next x results. The returned data will be in the same format as above.
3) Repeat step 2 until fewer than the requested number of results are returned. When a request returns fewer than the expected number of results, store the returned 'start' string and try again later, after some reasonable interval (Eg, once a day) to get new data.

The current endpoints for the firehose API are /api/maps and /api/comments. Returned entries are ordered by their 'last updated' date. For a comment, this is always when the comment was first submitted. The last updated date on a map is changed when the map is first submitted and when it is edited by the author. As such, you will see the same map occur multiple times in the feed if it is edited by the user. It is not changed if the map is rated, featured, or delisted, so fetching this information will require getting the map data from the query API. In future, the API may be updated to reinclude the map in the firehose feed on more criteria - eg, featuring, delisting, and possibly rating - or to include only the modified data in these events.

Result format

Both APIs use the same result format.

Maps are represented as a dictionary with the following keys:

map_id: The ID of the map
user: The username of the user who submitted the map
float_num: The 'float number' of the map, used for sorting in the hot maps page
name: The name or title of the map
description: The description of the map, in BBCode format.
mapdata: The map's N mapdata
rating: The map's current rating, as a floating point number
votes: The number of votes the map has received so far
disableratings: Boolean: True if the owner has disabled ratings on this map
unlisted: Boolean: True if this map has been delisted
comment_count: Current number of comments on the map
featured_by: If this map is featured, the username of the user who featured it
featured_text: The text of the map's feature review
tags: A list of tags for this map, including 'special' tags such as the author tag.
lastupdated: The datetime this map was last updated, in ISO timestamp format.
created: The datetime this map was created, in ISO timestamp format.
featured_date: The datetime this map was featured, in ISO timestamp format.

Comments are represented as a dictionary with the following keys:

map_id: The ID of the map this comment was made on
comment_id: The per-map comment ID of this comment
title: The title of the comment.
text: The comment's text, in BBCode format.
demodata: The comment's demodata, if any.
lastupdated: The datetime this comment was posted.
author: The username of the comment's author.

Usage

If you intend to use the firehose API, or to poll the Query API or otherwise use it heavily, please contact me here or on the forums. To reduce resource consumption, I would rather make sure all heavy users are using the API efficiently, and modify the API if necessary, than sustain a heavy polling load.

Global Mod
Global Mod
Posts: 715
Joined: 2009.11.07 (19:20)
NUMA Profile: www.nmaps.net/user/Sunset
Steam: www.steamcommunity.com/id/
MBTI Type: INTP
Location: Iowa City, IA

Postby Sunset » 2010.05.19 (00:39)

i don't understand that much, and i'm gonna sway off topic right now...

so, ever since early 2009 the rated maps queue has been broken. this hasn't been much of a problem, but it has distracted many from the actual maps rated that have been found, and after a while of ignoring it you kind of get the sensation that something has to be done about it. so, can you please fix it?

User avatar
Phei Phei Pho Phum
Posts: 1456
Joined: 2008.09.26 (12:28)
NUMA Profile: http://nmaps.net/user/Pheidippides
MBTI Type: ISFJ
Location: New Jersey

Postby Pheidippides » 2010.05.19 (01:30)

Hero wrote:i don't understand that much, and i'm gonna sway off topic right now...

so, ever since early 2009 the rated maps queue has been broken. this hasn't been much of a problem, but it has distracted many from the actual maps rated that have been found, and after a while of ignoring it you kind of get the sensation that something has to be done about it. so, can you please fix it?
We are quite aware of this problem. Unfortunately, Arachnid has, for quite some time (years) now, been too busy to do any coding for NUMA. Just as unfortunately, no one has stepped up to do much coding in his place. This predicament is pretty much common knowledge at this point. So there's your answer.

Also, you revived this topic solely to make an off-topic post, which is pretty bad etiquette and thus really not the way to get the attention you seek.

Edit: With the name change, I actually thought you were a newbie at first glance. Sunset, you've been here long enough to where you should know all of this.
Image
Follow me! @ninjarobotfeidi #nmaps

http://www.starstore.com/acatalog/Planet_Apes-lawgiver-statue.jpg
Posts: 514
Joined: 2008.09.26 (19:13)
MBTI Type: ISTJ

Postby Pikman » 2010.05.19 (02:47)

Pheidippides wrote:Edit: With the name change, I actually thought you were a newbie at first glance. Sunset, you've been here long enough to where you should know all of this.
It's like a name change changes the entire person. I'm no longer as cute and cuddly as I was with my old name, or so people have told me. I think it's just the switch to generally darker forum skins.

Remotely on-topic: Is LittleViking doing anything for NUMA at the moment, or are we sitting on our hands waiting for nothing to happen? It's obvious the community wants change, but only a select group of people are actually willing to enact it - and that's why they're the administration here. But there's no one brave enough to step up and assume responsibility? I would want to fully learn Python just to get things done (if nothing is truly being done) and let me get my hands out from under my rear end. They're starting to hurt a bit.

Raigan and the Horse-Woman
Raigan and the Horse-Woman
Posts: 182
Joined: 2008.09.27 (02:14)
NUMA Profile: www.nmaps.net/user/sidke
Steam: www.steamcommunity.com/id/shagdish
Location: ⑨ 
Contact:

Postby sidke » 2010.05.19 (06:40)

Pikman wrote:
funny how you think you're just the ones waiting.
pikman, you've been saying you'll 'pick up' python for almost a year now. you wanna know whose fault it is your hands are painfully numb?
yeah.

afaik, nobody is doing anything for numa. i offered to help out a while ago back when i wasn't slacking on projects i've said i'd work on due to irl job. i've pretty much retracted my previous offer of assistance to numa except for a few minor critical fixes (hint: .5 ratings aren't it, let it drop people. let it drop).

to people who seriously want to help instead of whining about it:
  • you have the resources (time and motivation)
  • you have incredible documentation on the GAE and its peripheral concepts
  • you have arachnid's email address
  • you have the community to back your contributions
go get 'em, tiger
spoiler

辻菜摘が好きじゃー ヽ(´ー`)ノ sig by peking duck


dreams slip through our fingers like hott slut sexxx
Posts: 3896
Joined: 2009.01.14 (15:41)
NUMA Profile: http://nmaps.net/user/Tunco123
MBTI Type: INTJ
Location: Istanbul

Postby Tunco » 2010.05.19 (19:12)

I remember you posting this a few months ago. You deleted the post a minute later, what happened, Arachnid?
spoiler

Image


User avatar
Mr. Glass
Posts: 2019
Joined: 2008.09.27 (20:22)
NUMA Profile: http://nmaps.net/user/astheoceansblue
MBTI Type: ENTP
Location: up down left right start A start

Postby a happy song » 2010.05.19 (20:33)

Tunco wrote:I remember you posting this a few months ago. You deleted the post a minute later, what happened, Arachnid?
Check the date of the post, it's the same one. It was moved to another forum, then returned upon Arachnid's request. No conspiracy.

And mailing Arachnid is proving to be fruitless. He told me he'd get on a few fixes and that I should remind him via his email, but I've sent a few mails now and not received any notification.

I'll keep at it.
click sig :::
spoiler


n
::: astheoceansblue
::: My eight episode map pack: SUNSHINEscience
::: Map Theory: The Importance of Function & Form

-
M U S I C
::: The forest and the fire: myspace
::: EP available for FREE download, here.

-
A R T
::: Sig & Avatar Artwork by me - see here!

-
G A M I N G
::: Steam ID: 0:1:20950734
::: Steam Username: brighter


http://www.starstore.com/acatalog/Planet_Apes-lawgiver-statue.jpg
Posts: 514
Joined: 2008.09.26 (19:13)
MBTI Type: ISTJ

Postby Pikman » 2010.05.21 (00:12)

*sigh*
/me continues whining.

The last email I have for Arachnid is [email protected]. (The forum can't handle the awesomeness of mailto: addresses?)
makinera wrote:afaik, nobody is doing anything for numa.
That's what I wanted to hear, but apparently I don't qualify to do anything about it.


Who is online

Users browsing this forum: No registered users and 7 guests