Metanet Forums

An Interactive Graphical History of Highscoring

I have made a couple interactive graphs that many of you might be interested in. They show the history of both 0th and Top-20 highscorers. I suppose this builds on Seifer's Highscore Rankings Through History.

I have hosted them on my website:

http://www.jamesinman.ca/n/

The graphs should scale to the width of your browser window. Further, they are interactive in two ways: First, you can click certain people on and off in the legend on the right, to just display certain scorers. The graph should rescale itself to the scale of the chosen scorers. Second, by clicking and dragging you can zoom to certain time periods and/or score ranges to get a closer look. Use the reset zoom button to get back out to full size.

Further steps

These were just the result of one evening's work. I plan to continue improving them in several way:

More people! | I'm obviously missing lots of important people, on both graphs. I didn't want to make them too crazy at first. Still, you can see how busy the top-20 graph is with just 14 scorers. Imagine 60.

Higher sampling rate | As the graphs are right now, each player's score total is sampled just once every 6 months. While these graphs give a decent overview of trends, they miss lots of details. I doubt you would you want all the hours and effort you put into scoring for 6 months reduced to a single number. Furthermore, highscoring battles, like those between xaelar/vankusss, johnny/trib4l, and Lim/xaelar would be much more interesting if they could be graphed week by week (or even day by day).

More graphs | How about one for Total Level/Episode Score? Top-10 or Top-5 Highscores? What about a graph for an individual level, showing how the top-20 leaderboard for just that level over time. I think that wold be really cool...

Anyone want to help?

Messing around with NHigh files, combing through ancient Highscore Ranking Threads and making sure the Hackers are properly ignored (and deciding whether certain players should be ignored) is a lot of work. If anyone else sees value (or is just interested) in having these stats collected, I would welcome a helping hand. We can organize ourselves in IRC if so.

Wow!!! This is an amazing interactive chart! I would recommend a quick "solo" button (that saves you having to manually uncheck each player.) I also notice the big bump on my top-20 tally between Late-2011 and early 2012. I was like, damn, I didn't realise I did so much highscoring then. Then I was even more shocked when I toggled JF's chart and saw something similar, and I'm like, "damn, I didn't know Johnny made a comeback". And then, soon after, the penny dropped, and I recalled that that would have been the time when Mare cleaned up the boards, which would have given everyone a significant score boost. Keep it up... I think total game score would be of particular interest... (Although it's accuracy will be dubious, since people like me have had their sol reset numerous times, and online submissions may be incomplete for players who don't have close to 600 scores.)

I'm loving this Hendor, how informative it is and how clean everything is presented. Thanks a lot for putting the effort into creating this project. I'm willing to help in my free time, preferably with some kind of programming to extract info from .hs files, which is kinda fun, but if that doesn't turn out well I can just add info manually too. I'll see how it goes.

Adding more rankings would rock indeed, like top10s, top5s, total/avg points, total lvl/ep score, etc. Biggest 0th-1st/19th differences over time would rock too, to see how they got reduced as runs got refined. Cleanest episode too, as they have tended to get cleaner with time, obviously. Rankings regarding averages would be quite interesting. And hell, community total level/ep score evolution too! And I agree 0th evolution for each level would kick ass. Maybe a graph showing the max/min 0th/top20/top10/top5 count over time... NHigh can get all this info for us (for each date that is), except for total game score.

Regarding this last question, and answering ska, I'd use the old forums total level/game score thread to gather the info. The problem is, then there is a huge gap until TRO/myself started posting this rankings too, because eru hadnt developed that function yet and there was no thread created, something that really dissapoints me. That nobody created that thread is still a mystery to me, considering how interesting and fun it is to compete for total game score. I've thought on creating it several times to encourage competition there, but always thought it was already too late since there's barely any competition. Oh, this gives me an idea, what about community total level/ep 19th score, which would be the technical minimum to get all top20 scores.

Finally, some ideas/fantasies for the user interface include the "solo" button that ska suggests, or even selecting a range of players to get displayed (from player A to player B), and maybe an option to invert the range afterwards (that is, display all players except the ones between A and B). Anyways, as a stat lover myself my head is just getting overloaded with options and ideas xD I'll just try to give a hand automatizing this info extraction process first or everything will just stay as fantasies forever :P

EDIT: I found something weird, trib4l had 267 0ths in his prime, while on the graph it is shown as 177. I'm guessing this is just a time precision thing right? In that case, its interesting how quick he lost most of his 0ths.

Really cool.

…And now I want to code more, more, moooore! \o/ Hopefully I will find time to do this soon.
What do you think about adding a backend that loads info from a database, with automatic extraction and everything? If we do it in python, we can have some of this already written in NHigh.

EddyMataGallos wrote:EDIT: I found something weird, trib4l had 267 0ths in his prime, while on the graph it is shown as 177. I'm guessing this is just a time precision thing right? In that case, its interesting how quick he lost most of his 0ths.

Same for my top 20 scores, the fact that I got as high as 570 for several months doesn't appear, and it seems I barely overtook JF for a short time :/

Weee long post. Thanks for all the feedback people.

ska wrote:The bump

Yeah... the bump. It was pretty evident as I put the numbers together. It's also pretty annoying, as it distracts from the actual message of the graph. Plus, it happened right during the period when you were most active ska, I believe. A good chunk of your increase at the time was due to your playing, not the hacker deletion. I had thought about two ways to eliminate this bump: Using NHigh files, note when a hacker knocks someone off a board and give them credit for the level anyways. That would be a lot of work just to fix a minor inconvenience on just the top-20 board, but might be something you could program Eddy/zapkt...? The second way would just be to mathematically model the effect of the hackers, in order to apply a scaling factor to everyone's top-20 total pre 2012, perhaps using something like Total Level Score, Total Points Ranking, or Score Distribution as a metric for how likely someone's scores were to be knocked off. Again, a bit complicated for such a minor factor, but I think it would work well. Get rid of the bump at least.

I also agree that Total Level Score would be somewhat of a challenge to create a graph for, as Eddy said, NHigh saves don't get you the player's total score (unless the player has 590+ highscores and we could figure it out pretty easily to within a couple seconds error). At least we have Total Score/Episode threads that go back pretty far, again, as Eddy said, with a hiatus.

EddyMataGallos wrote:Adding more rankings would rock indeed, like top10s, top5s, total/avg points, total lvl/ep score, etc. Biggest 0th-1st/19th differences over time would rock too.

Good ideas. I think all of these would be interesting and quite feasible to produce.

EddyMataGallos wrote:Finally, some ideas/fantasies for the user interface include the "solo" button that ska suggests, or even selecting a range of players to get displayed (from player A to player B), and maybe an option to invert the range afterwards (that is, display all players except the ones between A and B)

zapkt wrote:What do you think about adding a backend that loads info from a database, with automatic extraction and everything? If we do it in python, we can have some of this already written in NHigh.

Yes, my initial thought was to have each player's top-20 and 0th data in a database and then have a new webpage before the graphs show that asks for input. You would go through a list and check off the players you want. and then the graph would just pull the info for those players. With the addition of this new page, you could do some other stuff too: You could check whether you want the graph to show points at each data point, or just a line like they are now.... Whether you want the graph to be splined as it is now or more straight and precise... You could specify colors and backgrounds and all kinds of neat visual stuff.

I don't know any Python :( My thought was to do it with php, though, hell, just some javascript would be enough.

One great thing I just found out is that the graph does not need player's data points aligned. That is, the graph can take irregular time intervals... If we want to show Eddy's increase from 18 to 400 0ths in 50 steps (data points)) as opposed to 3, we don't need to collect 50 steps for johnny_faneca or zapkt over the same time period. Same goes for all of us modern players during trib4l and johnny's heyday. We can produce one dataset for each player, centered around the time they were most active, and then have the graph put them together. Props to Highchartsfor that.

EddyMataGallos wrote:I found something weird, trib4l had 267 0ths in his prime, while on the graph it is shown as 177.

zapkt wrote:Same for my top 20 scores, the fact that I got as high as 570 for several months doesn't appear

Yes, as Eddy said, I think this is a time precision thing. Though I tried to get as close as I could to 6 month intervals, I sometimes had to compromise to 8/9 months when I couldn't find anything. I had noticed that your top-20s didn't go as high as they should zapkt, briefly tried to figure out why and then gave up. Figured you guys would be interested to see it as it was. I have corrected both your top-20 total zapkt and trib4l's 0ths.

Alright, so I made a big step towards automatizing the process. Created an NHigh function that reads all NHigh files in a folder, extracts the 0th count for a certain player (and the date) from each file, and adds everything to a list that is outputted as text is chronological order. This can then be easily ported into an Excel file, the first column being the dates and the second column being the 0th count. Each row being a file. This is my resulting file:
Is that format optimal for you to then port it to your interactive graph Hendor? If not, just tell me. Changing the text output style is straightforward now, and hence the resulting spreadsheet. The date format might vary depending on your system, so I'm not quite sure what you'll see, hopefully Excel is clever enough to automatically adapt that. I can even include hrs/mins/secs and so on if you wish.

As of right now I've only done it for 0ths, but that was the big step, adding any other information (top20's, top10's, top5's, total/avg points, etc; basically anything NHigh can provide) should just imply slight changes in the function, so expect them quite soon. I guess I could also do it for all players at once, but only a player already takes a decent amount of time (~40 secs in my PC) because it evidently has to load a ton of files (277 as of right now), and it can even lag the PC. Whatever suits you the most Hendor.

Everything is quite rough yet though, I've barely had time to polish this, but when I do I'll post the updated NHigh so you can use it to update your Graph. Just inform me with your preferred data output format. You'll have to do the text to spreadsheet conversion yourself, but because of the output format it is fairly simple.

EDIT: Superquick update for top20 evolution. Literally 1 character change required in the code!

Haha, wow Eddy, that is huge. I love you.

The exact format I will use is probably going to be this: I'm afraid I'm going to be out for the rest of the evening here (Seeing Interstellar!) but I will work on this a bunch when I get back. I'm tempted to just give you access to my site Eddy haha.

Edit -

My plan was just to create the full graph dataset as it appears above, store it and then call it when needed.

But now I see where you are going with this... I wasn't planning on making a database of scores and dates as I can't see a need for pulling out specific dates for my purpose of graphing. However, having such a database would add another whole scope to the project... Essentially it could replace NHigh, with a full history of scores, daily NHigh updates and live automatic updating to the graphs. Cooool!!! But a lot of work, don't know if my programming skills up to it.

Wow great job guys.

About the format, what about the simple, open and easy to use CSV format?

EddyMataGallos wrote:Alright, so I made a big step towards automatizing the process. Created an NHigh function that reads all NHigh files in a folder, extracts the 0th count for a certain player (and the date) from each file, and adds everything to a list that is outputted as text is chronological order. This can then be easily ported into an Excel file, the first column being the dates and the second column being the 0th count. Each row being a file. This is my resulting file:

Can you share the code somewhere?

I don't know whether I will find time to work on this before the week-end. Don't finish everything in 2 days!

This is awesome! Thank you so much Hendor and Eddy for your work on this. So very cool to look through.

zapkt wrote:About the format, what about the simple, open and easy to use CSV format?

I did generate the output in CSV with python (well actually just plain text with separators, like CSV), but I read somewhere that Hendor was talking about Excel files so thats why I did the conversion, thinking that was the way he implemented the data. Nonetheless, now that I have the exact output he needs, the easiest thing to do it just to make python output that exact thing, of course.

zapkt wrote:Can you share the code somewhere?

Sure man, here you go:

http://pastebin.com/nPsZaDgC

I only coded the "skeleton" of the method in the little time I had. I'm not even popping up a player selection dialog, nor folder selection dialog, nor checking which files/players are valid. Those things are just superficial though, I'll do them when I'm done with the core. After the polishment all we need to do is just generalize the function to look up all players and elaborate the whole rankings, which again is simple enough.

I need to polish it a lot before I post the whole program though, implement all functions and so on, and its getting late already, so I'll get things finished in the following days. I also made several modifications a while ago so I'll probably publish the whole program in eru's thread or something in the future.

Also my python coding is probably very noobish, in reality I've barely worked with python, so I had to spend a good deal of time seeing how things work here first. So maybe you can optimize that method? If you start working on anything outside that method, remember we should get coordinated, so we don't overwrite each others edits haha.

Hendor wrote:Haha, wow Eddy, that is huge. I love you.

The exact format I will use is probably going to be this:

Thanks man, now that I know which output you want, I'll implement it to get it properly from python tomorrow. If its ready enough I'll post a quick NHigh update so you can download it and use it, at least for the basic functions at first.

This is pretty amazing, especially all the work everybody else has put out on this already o.O For a couple reasons, I won't be able to help out with this more than what is already being done. Firstly, I don't have the programming knowledge needed, based on what I've read. Second, I'd like to refer to my edit to the #n-highscores topic from tonight:

golfkid will be available again sometime after December 25

I've already put in almost 40 hours this week, and haven't been able to see the sun at all :( If anything, it will just get worse as we get closer to Christmas... Boys and girls, this is why you should stay in school, so you don't have to work a physical labor job (or 2) that gets crazy busy during any particular time of the year.

I'm amazed at how this thread has blown up. It's good to see there are still a bunch of people who care about this sub-community.

Hendor wrote:...Score Distribution as a metric for how likely someone's scores were to be knocked off. Again, a bit complicated for such a minor factor, but I think it would work well. Get rid of the bump at least.

Yeah, this was one of my ideas, too. I would suggest that you give an estimated progression of hackers over time that can be factored into the timeline. (That way you don't need to directly calculate the 'impact' that hackers have had on an individual basis.) I also recall that Metanet kept the top 40 scores in their database. Is there some way to historically extract that information, sans the hackers' fake scores?

Also, I wish I could help, but my aptitude for programming leaves much to be desired. (I'm more or a design person.)

P.S. My novel, Counting on Silence, is almost complete... Just a few more chapters to edit and an epilogue to write. I'll happily PM anyone who wants to check out the first few chapters in any electronic format they desire.

A minor update:

I've updated the graphs to be able to take irregular time intervals, so each player will just show on the graph for the time period they have scores. Here is a sample with the data Eddy posted for his own scores, which is pretty interesting in itself:

0th
Top-20

This is really interesting, I had no idea such detailed statistics were kept by the community. Since I didn't join the competitive community until 2010, it's cool to see some earlier history.

I'm guessing not, but as a speedrunner I have to ask, are there any similar statistics on speedrun mode?

ska wrote:I'm amazed at how this thread has blown up. It's good to see there are still a bunch of people who care about this sub-community.

Metanet, the place were 10 posts means blown up.

ska wrote:I also recall that Metanet kept the top 40 scores in their database. Is there some way to historically extract that information, sans the hackers' fake scores?

I really, really don't think so. We haven't downloaded such scores and so they are gone (even assuming the hypothesis that we could access M&R database and download all 40 scores, or as many as there are). That could only be done if they kept all submitted runs from each player and each level, without overwritting the previous ones. We could then rebuild the boards for a specific date based on the run dates. But I highly doubt this, they only save the personal best.

I'm talking about the database entries that is. We can check HarveyCartel's website with the Wayback Machine and there are some cool things to see, yet incomplete, as levels were crawled in differentes dates and so on xD

EddyMataGallos wrote:

ska wrote:I'm amazed at how this thread has blown up. It's good to see there are still a bunch of people who care about this sub-community.

Metanet, the place were 10 posts means blown up.

ska wrote:I also recall that Metanet kept the top 40 scores in their database. Is there some way to historically extract that information, sans the hackers' fake scores?

I really, really don't think so. We haven't downloaded such scores and so they are gone (even assuming the hypothesis that we could access M&R database and download all 40 scores, or as many as there are). That could only be done if they kept all submitted runs from each player and each level, without overwritting the previous ones. We could then rebuild the boards for a specific date based on the run dates. But I highly doubt this, they only save the personal best.

I'm talking about the database entries that is. We can check HarveyCartel's website with the Wayback Machine and there are some cool things to see, yet incomplete, as levels were crawled in differentes dates and so on xD

Yeah, I figured that... I still to this day don't know what my first top-20 score was...

This just awesome! Thanks for creating this. Nothing else to say here from my part. ;)

Alright Hendor, next update here. I've polished the functions for both 0th and Top20 score evolution over time, and there's 2 kinds of them now. Current state:

The middle 3 functions output a regular list following the same style as the rest of NHigh functions.
The final 3 functions are the very same ones, but they output them in the exact format you wanted.
I'll work on the rest of the functions and the polishment when I come back ;)
There's a blank line that I'm not outputting, the rest is identical to the input you require. If you do require that blank line (the one after all dates, but before closing the braces), just tell me and I'll add it, its just adding 2 characters ;)

I'll work on generalizing the function too. I can't post the code yet because at the moment I have things "on construction" and it doesn't run :P

Holy macaroni... how long did you spend putting all this together?

ska wrote:Holy macaroni... how long did you spend putting all this together?

Not so much really :P all 6 use the same function, but depending on the parameters I send it does one thing or the other. I'm really liking the final result though, coding is fun ^^

I rushed the previous post too much, didn't even post an output comparison. Here you go:

Thats a comparison of the function "Both reports" in normal mode and graph mode, with 5 test files and my name. Is that output format optimal now Hendor? Ideally, it should be ready to copy-paste to your code, to avoid manual work as much as possible.

I have so many ideas... I started creating a database, this week-end I will start the interactive web application!

Awesome news zapkt :D Really looking forward to seeing it. On the meantime, I finished my thing, so now Hendor you can easily implement the info to your graph. It only has the basic functions as of right now, but I'll keep implementing the others. I'll copy the post I had prepared for this update:

I just finished doing some polishing and basic function implementation. I guess I'll post the resulting version now (only the updated files, replace your old ones), so that Hendor can use it for his graph already, and maybe for the people's curiosity. All you have to do is extract the rar and substitute 2 of the files in NHigh folder with the ones inside the rar. Obviously, I'd suggest doubling your NHigh folder first and doing this on the second one, or at the very least, backuping your old files.
For anybody interested: I've flagged all my edits so you can see the changes. The main addition is lines 1610-1744, which contains the new functions. I also flagged my edits on the library file.

This is basically what's added:
- Player's 0th evolution
- Player's Top-20 scores evolution
- Player's X-Y scores evolution
- Each of those 3 reports but with the graph code output Hendor requires.
- Each of those previous 6 (3*2) reports, but for a multiple amount of people n, which are the n top players in the top-20 rankings. Just a way to speed up things. The final output should be proper in all cases.

That makes it a total of 12 functions. You DON'T need to have highscores loaded to do the individual player's reports, because the program automatically scans the folder for files. On the other hand, you DO need it for the multiple (global) player selection since it needs the top-20 rankings to extract the players from it and do the report for each.

Player's X-Y scores is basically the one already implemented in NHigh, but only retrieving the total number evolution. So you can get Top-10 scores by choosing rank1=0 and rank2=9, or 2nd scores by choosing both ranks equal to 2, or 3rd-5th scores, etc. That should be enough for implementing the 0th, Top-20, Top-10 and Top-5 graphs we were thinking on. Output example: http://pastebin.com/jUcH736F

I've done some corrections too. There's a player selection dialog now, and it will ignore non-.hs files in the folder (it used to crash before xD), so there can be other type of files on the folder now. I haven't added a folder dialog, unfortunately, since there wasn't a folder selection dialog already designed and I'm way too lazy to design the widget now. I tried the TkInter standard one, but it sucks ass.

So, how you select the folder then? Well sorry, you can't. You have to create a folder named "scores" (no quotation marks of course), right inside your NHigh folder (where all the code files are, too), and put the files inside there. If somebody could design this thing to match the rest of the selection dialogs, that would be great haha :P On the meantime though, I'll dedicate to implementing more functions, which I think is more useful. Just remember to create the folder and put all files there (there are over 800 now). In case you haven't the folder ready, I just uploaded it to MEGA, it contains all 807 files (zapkt, TRO, ska, and myself files), with duplicates deleted and already inside the folder "scores", so all you'd require to do it to extract it on NHigh folder. I compressed it with 7z so its only 6.3MB, but be aware its 220MB uncompressed.

https://mega.co.nz/#!1UIkBCDQ!TuLZMqRdz ... lnpFKzrFK0

Finally, be aware, scanning all 800 files might take a long time just for a single player (a couple minutes maybe? depends on the computer, mine's quite old already). So of course, selecting muliple players will take multiple times that amount of time.

I'm looking forward to seeing all these new rankings implemented in your graph Hendor, and with more players now!

This is great Eddy, nice work. Your NHigh is working perfectly. Scores are going pretty quickly.

One thing to change though is that Javascript uses 0 for January and 1 for February, and etc, which means that all dates are off by exactly one month (which made troubleshooting the below problem confusing). I looked into the Python file and found the relevant if(graph) and datetime bits, but don't know enough (any) Python to change it. Could you please just add a -1 in there at some point for the Graph Lists? Thanks!

One more issue is that some of the NHigh saves are messed up and I can't figure out why or which they are. They are all named differently, and many of the file names don't agree with the Date File Modified. I think the bad files must all come from the same person/source as they are often kinda clumped up. See this screenshot

EDIT - I figured out some of them. I think. No. No I didn't. Wait, yes I did. They're your files Eddy. Check out the ones in your folder named "10-05-31" through "10-06-20". There are several others at later dates as well. They've got all the cheaters still in them. I'm gonna try and exterminate the bad files.

EDIT2 - K, more or less got rid of them.