If you lookup statistics on the armory under BC raids turns out the armory has no data on the Hyjal Summit raid to defeat Archimonde. Wierd huh?

• Finished downloading new armory data for this week
• Integrated google charts into the member boss kill data for all WOLK bosses

Some of the things I am still woring on ...

• Realm stats like population count, guild count etc
• More guild stats and more ways to view the data. Guild population, class percentages, member experience rankings
• Start working on a character page with ability to see talent spec, gear changes over time etc. Ever wish you could go back in time and review a talent build what gear you had? You will be able to look it up!
• Guild AND Character AND Realm progression stats! Ever wonder what realms or guilds have the fastest progression velocities? I do!
• Implement character and guild searches into the search box (currently it only searches blog posts)
• Guild leader/officer tools for tracking and managing guild activities and members
• Character gear score system
• Frankly too many to list here. All in due time ....

If you have any suggestions or comments please feel free to use the contact page and submit your ideas so I can add them to the list of things to do !

• My code had a bug in the way it was presenting class, faction, gender, race id's for characters. The armory uses specific numbers to represent these id's unfortunatley I neglecte to properly match those id's with my own. This has now been fixed.
• I've updated and added some optimizations on how the data is saved, to include versioning and to reduce the storage requirements.

Added more ways to view the member list for a guild (only for icc at the moment). One problem I've run into in presenting the data in this manner is that I had to use abriviated boss names for the headers. If I didnt the width of the columns would be too big and spread out to fit on the page properly.

I'm going to have to start adding some kind of graphs to present the data in a more easy to see format. I'm looking into using google charts, silverlight or maybe rolling my own solution, so I will have to wait to see how that turns out.

• Uploaded this weeks armory data
• Fixed an issue with the Date selector not working properly ( but dates are not carried across page views still working on fixing that )
• Added a members link on the guild stats page to view the list of guild members

I have a desktop app I made to crawl the armory from a master list of toon names I have been building. Last night I crawled a few thousand characters and everything went fine but about 5% came back as 503 Server errors. IE: The server denied my request. This is normal and expected based on passed experience. So this morning I used that same program to try and re-download all the toon names that came back a 503 errors. But this morning only a dozen or so characters were crawled before I started to get 503 errors for every single request after that.

I cant even use firefox to look up the characters anymore, and was locked out of the armory fer a few hours.

So obviously blizz has flow control built into the armory to help control the volume of requests that come in. Each character requires about 10-12 requests 1 character page, 9 or so statistic pages etc, multiply that by thousands of character names and a 50ms delay between each request and it takes a very very long time.

This is not to say that there are not ways around this and I am working on a system to distribute the crawling of the armory across any number of machines but that is still a long way off.

I just find it utterly confrusing to be able to make thousands of requests over night but then come morning, or what I assume are peak armory hours, I can hardly make a few dozen requests before being 503 denied.

Calculating and presenting character/guild/realm statistics is easy mode trying to get the character data from the armory is a real pain in the ass. :(

After some searching I came across this article about how the armory throttles requests. Gonna have to do more looking into how to over come that because it takes me 20+ Hrs already and it will take me 3 days if I delay each request by 1.5 secs. I've been tracking what characters return 503 errors but if I'm not carfull the armory will block me for a long time for making too many requests. I have implemented a auto throtlling system in the crawler app now so hopefully it will automatically throttle the number of requests per second that the app is making.

After having downloaded last weeks character data from the armory I almost had a close call thinking the data files were corrupted. When I read them back I would get end of file errors. Thought I had fixed it but turns out there was nothing wrong with the way the files were saved my reader code was missing a line thus the next pieces of data that got read were corrupted. Phewf ! For about thirty minutes there I was concerned that all those 13k+ files I had were corrupted. But they are not so I'm safe :P

I also had to turn of comments so people can't add make comment posts. I was getting too much spam from those pesky comment spammers. I'm a gonna have to integrate a captcha system to stop that from happening.

Also there is some preliminary statistics now availible for browsing. Just click the Realm List Link on the navigation bar.

Almost finished crawling the armory again. After having reworked my entire back end storage strategy. Was originally storing data in the database but quickly realized that the DB size was gonna blow up after a few weeks and in order to store the data in a DB I'd have to write an entire automated db creation and management system, where each weeks data crawl from the armory would get stored in it's own db. Then the problem arises how to organize the data and write the code that will properly pull the data back out of the proper db's no less. *Sigh*

So I went with a far simpler strategy and simply store the data in my own custom file format, giving me flexibility and more control, as well as making it way easier to manage and backup. So I just need to make a few more minor adjustments to my armory crawling code and I can get back to the easy and fun stuff which is taking that data and presenting interesting and informational statistics!

Keep meaning to get around to finish work on the sites look and feel but meh it is low on the priority list right now. There is still some white text over a white background here and there >.<