In the last episode of Wikicounting I investigated how Wikipeia treated history, by calculating the size (in characters) of the Wikipedia entries for each individual year. The results were very interesting. In the comments of that post PEB mentioned that it would be fun to see how the size of the Wikipedia article for a specific year is correlated with the number of links that Google turns up.
Here is what you find if you plot the number of links turned up by Google against the number of characters on the wikipedia page:
I used google search queries of the form "year XXXX" (with the quotes), which should ensure that most of the results we return are relevant to the given year. There is a really good correlation (the odd points at the lower end of each spectrum are mainly years around the 1500s).
It's interesting to note that there are only two points with over ten million google links, they are 2000 and 2005 (the graph stops at 2005). In the year 2000 every single blog, news site and poorly designed geocities webpage was ranting about the y2k bug and so that point got boosted massively above the others around it. Just look at the numbers
(year, number of google links)
1998 1,520,000
1999 1,550,000
2000 15,600,000 <-- !!!!
2001 2,760,000
2002 4,480,000
2003 5,890,000
2004 8,100,000
2005 15,400,000
2006 37,500,000
2007 23,200,000 (and we're still in January!)
It looks like post-2000 number of links is almost doubling every single year, some may say that Google is growing at a geometric rate!
next time: We learn which deity is preferred by the people of the internet!
Also give me more ideas please
Here is what you find if you plot the number of links turned up by Google against the number of characters on the wikipedia page:
I used google search queries of the form "year XXXX" (with the quotes), which should ensure that most of the results we return are relevant to the given year. There is a really good correlation (the odd points at the lower end of each spectrum are mainly years around the 1500s).
It's interesting to note that there are only two points with over ten million google links, they are 2000 and 2005 (the graph stops at 2005). In the year 2000 every single blog, news site and poorly designed geocities webpage was ranting about the y2k bug and so that point got boosted massively above the others around it. Just look at the numbers
(year, number of google links)
1998 1,520,000
1999 1,550,000
2000 15,600,000 <-- !!!!
2001 2,760,000
2002 4,480,000
2003 5,890,000
2004 8,100,000
2005 15,400,000
2006 37,500,000
2007 23,200,000 (and we're still in January!)
It looks like post-2000 number of links is almost doubling every single year, some may say that Google is growing at a geometric rate!
next time: We learn which deity is preferred by the people of the internet!
Also give me more ideas please
Labels: google, wikicounting, wikipedia
After introducing my wiki wordcount script in this post I solicited for ideas of things to count. One great suggestion from Abbie was to do 'years'.
Here it is.
Here we have plotted year against 'number of characters in the wikipedia entry'. There are a couple of lines marked on the graph (I gave them fancy names because that's just the way I roll).
The dark, horizontal line is the 'Base Historical Interest' (BHI). Beyond a certain point (around 1850) 'normal people' stop being interested in what happened on a month by month basis in a certain year and so the articles are written by and for a vast minority (historians? those with family connections to a certain event?). The BHI level represents the amount of work that Wikipedians put into an average, uninteresting year. Most of the letters are actually in the form of the sidebar and headers, which my script does not yet automatically remove.
The light, diagonal line is the 'Linear Interest Growth' (LIG) trend. As we get closer and closer to the present day people become more involved with the subject and there is a corresponding increase in the amount of words that get written.
I find it really interesting that when something historically significant happens, the trend lifts itself from the LIG and we see an increase in wikipedia interest. I have labelled a few LIG-excess events here:
The mystery peak of 1903 is due to somebody having messed up the formatting on the 1903 wikipedia page so all the tags get counted in the wordcounter. Naturally after wikipedia was introduced (2001) the number of entries on the year page exploded as people began to add things in real-time.
late edit: I just ran the script all the way back to 1500. The results are here. With this extra data I'm beginning to think that an exponential may make for a better fit.
The point is moot really, as just like trying to observe an uncertain-electron, the act of using wikipedia as a measure of history has been affected by the existence of wikipedia itself, and any and all trends will have been smashed by 2001
late edit 2: Give me more ideas for things to count
Here it is.
Here we have plotted year against 'number of characters in the wikipedia entry'. There are a couple of lines marked on the graph (I gave them fancy names because that's just the way I roll).
The dark, horizontal line is the 'Base Historical Interest' (BHI). Beyond a certain point (around 1850) 'normal people' stop being interested in what happened on a month by month basis in a certain year and so the articles are written by and for a vast minority (historians? those with family connections to a certain event?). The BHI level represents the amount of work that Wikipedians put into an average, uninteresting year. Most of the letters are actually in the form of the sidebar and headers, which my script does not yet automatically remove.
The light, diagonal line is the 'Linear Interest Growth' (LIG) trend. As we get closer and closer to the present day people become more involved with the subject and there is a corresponding increase in the amount of words that get written.
I find it really interesting that when something historically significant happens, the trend lifts itself from the LIG and we see an increase in wikipedia interest. I have labelled a few LIG-excess events here:
The mystery peak of 1903 is due to somebody having messed up the formatting on the 1903 wikipedia page so all the tags get counted in the wordcounter. Naturally after wikipedia was introduced (2001) the number of entries on the year page exploded as people began to add things in real-time.
late edit: I just ran the script all the way back to 1500. The results are here. With this extra data I'm beginning to think that an exponential may make for a better fit.
The point is moot really, as just like trying to observe an uncertain-electron, the act of using wikipedia as a measure of history has been affected by the existence of wikipedia itself, and any and all trends will have been smashed by 2001
late edit 2: Give me more ideas for things to count
Labels: history, wikicounting, wikipedia
It is a sad reality of life that my usual routine of 'doing shit on the internet' has been disrupted by 'doing work' and as such the update rate on the blog has dropped.
I thought that today I would share something I have been doing in my spare time for the past couple of days: Wordcounting Wikipedia. I put together a little script that takes a list of words, finds the corresponding wikipedia page for each word and counts the size of it. Here is an example (click for full size), it contains the size of the wikipedia entry for every country with a name starting in the range A-K (clicky for massive version).
Cool, Eh? I have had far too much fun just seeing who the big winners and losers are in the world of internet. I'll finish off the graph soon, but I'm feeling lazy right now. Here are a few facts I've learnt about the world (A through K inclusive)
What about digging deeper into the data, using new and powerful statistics to let us view the world in a whole new way. The statistic I'm going to introduce now is probably as powerful as the legendary MEPP. It is called the WDQ or Wiki Drama Quotient. Every page on wikipedia has an associated 'Talk' page, where beardy internet people can argue about what gets put in the article itself. I would contend that the ratio of the length of a discussion page to its article gives a measure of how much drama a particular term carries with it. For example lets take a look at the age old battle between Tea and Coffee:
Coffee, 29027
Talk:Coffee, 64620
Tea, 43991
Talk:Tea, 35143
Although Tea has a longer article, its WDQ is much lower than that of coffee (0.79 vs. 2.23) suggesting that in the internet world tea is more popular than coffee, but it has a calmer history and there is not much in the way of disagreement about it (although on the talk pages there are a few sharp exchanges about the merits of different shapes of teabag).
What about figuring out what the best colour is?
Blue. Obviously. Anybody that disagrees is fooling themselves.
The main point of this blog post is that I need ideas for things to investigate: places? football teams? branches of the physical sciences? Using this stunning technique we can, once and for all, settle a lot of arguments
p.s. also could somebody write a guest post for me, I'll buy you a beer. You can even be anonymous if you wish.
I thought that today I would share something I have been doing in my spare time for the past couple of days: Wordcounting Wikipedia. I put together a little script that takes a list of words, finds the corresponding wikipedia page for each word and counts the size of it. Here is an example (click for full size), it contains the size of the wikipedia entry for every country with a name starting in the range A-K (clicky for massive version).
Cool, Eh? I have had far too much fun just seeing who the big winners and losers are in the world of internet. I'll finish off the graph soon, but I'm feeling lazy right now. Here are a few facts I've learnt about the world (A through K inclusive)
- Argentina has a freakishly long wiki entry, I have no idea why
- It took me three attempts to spell Kyrgyzstan correctly
- There are no countries that start with the letter X, an only one that starts with Q,O and Y
- The peoples of Brunei do not enjoy writing on wikipedia
What about digging deeper into the data, using new and powerful statistics to let us view the world in a whole new way. The statistic I'm going to introduce now is probably as powerful as the legendary MEPP. It is called the WDQ or Wiki Drama Quotient. Every page on wikipedia has an associated 'Talk' page, where beardy internet people can argue about what gets put in the article itself. I would contend that the ratio of the length of a discussion page to its article gives a measure of how much drama a particular term carries with it. For example lets take a look at the age old battle between Tea and Coffee:
Coffee, 29027
Talk:Coffee, 64620
Tea, 43991
Talk:Tea, 35143
Although Tea has a longer article, its WDQ is much lower than that of coffee (0.79 vs. 2.23) suggesting that in the internet world tea is more popular than coffee, but it has a calmer history and there is not much in the way of disagreement about it (although on the talk pages there are a few sharp exchanges about the merits of different shapes of teabag).
What about figuring out what the best colour is?
Blue. Obviously. Anybody that disagrees is fooling themselves.
The main point of this blog post is that I need ideas for things to investigate: places? football teams? branches of the physical sciences? Using this stunning technique we can, once and for all, settle a lot of arguments
p.s. also could somebody write a guest post for me, I'll buy you a beer. You can even be anonymous if you wish.
Labels: statistics, wikicounting, wikipedia
