How do different traffic measurement tools work?

A good way to illustrate how different measurement methodologies produce different internet traffic figures it to look at the question 'which is the largest job board in the UK'? Before going on it's worth pointing out that this isn't necessarily the most useful question to ask when evaluating the merits of different online recruitment sites as surely what really matters is the number of applications sites can generate and the type of candidates they can offer you rather then the total size of the site? However in order to illustrate the differences between the methodologies described below we'll stick with that question for now and look at three commonly quoted sources of internet traffic figures.

Nielsen Net Ratings (NNR)

NNR shows that Monster is the UK's largest job board with a market share of 26% of all traffic going to online recruitment sites (Source: NNR figures in Monster Matters email newsletter). NNR is a panel based measurement tool that works in the following way. Firstly NNR establish a group, or panel, of internet users. Members of the panel then install a piece of software on their computers that records all of their online activity. Based on these records NNR extrapolate the activity of members of their panel to represent the activities of all internet users in the UK.

In theory this methodology should give an accurate picture of internet activity in the UK, however there are a couple of problems regarding the methodology in practice. Firstly the effective extrapolation of the activities of a small group to represent the activities of a larger group relies on the first group accurately representing the second group. Although it's easy to establish a panel of home based internet users it is much more difficult to establish a panel of work based internet users as employers are very reluctant to allow a third party to install software that monitors online activity on their employee's PCs. Because of this panel based research techniques can under count work based internet use and therefore produce inaccurate results. Panels can also be unrepresentative because of the methods used to recruit panel members.

In addition to these issues panel based measurement techniques can also be affected by the size of the panel used as small panels can effectively exclude smaller sites from the measurements and this can lead to the generation of an incomplete picture of the overall market.

Hitwise (HW)

HW is a competitive ranking tool that shows that Job Centre Plus is the UK's largest recruitment site (Source: HW week ending 5th June). Although not a panel based measurement tool HW uses similar principles to NNR to show the relative size of internet sites. HW use the records of internet service providers (ISPs - examples of ISPs include Wanadoo and Tiscali) to discover the different internet sites visited by subscribers to these ISPs.

Again this technique is fine in theory, however the validity of the rankings produced by HW depends on the activity of the users of HW's ISPs accurately representing the activity of all internet users (unfortunately HW is unable to make public which ISPs it partners with and so its difficult to evaluate whether or not this is the case). In addition to this issue ISP records sometimes fail to identify visits to certain sites because of the URL structure of that site and this again can lead to an incomplete picture of the market place.

ABC ELECTRONIC (ABCE)

ABCE figures show that Fish4Jobs is the biggest UK recruitment site with 1,217,481 Unique Users (Source: ABCE audit). ABCE is different from NNR and HW in that it is an industry owned body (the electronic arm of the Audit Bureau of Circulation that has been auditing newspaper circulations for over 70 years) that is responsible for auditing internet traffic figures to a set of pre agreed industry standards rather than a measurement tool.

Sites that want to be audited by ABCE are responsible for producing their own traffic figures. They then submit these figures to ABCE who independently audit them, making sure that the software used to produce the figures has been configured correctly and that all robotic traffic (such as automated visits from search engines) has been excluded from the figures. Once they are satisfied that the figures are genuine ABCE issue the site with an audited traffic certificate.

ABCE audit Page Views, Visits and Unique Users. One advantage of the ABCE data is that Unique User numbers are easily compared with viewers, readers and listeners thus enabling cross media comparisons. It's also possible to view site's individual audit certificates on the ABCE website and see which elements of the site have contributed towards the final audited figure (increasingly important as the number of partnerships and content deals increases) making it a more transparent tool than some other measurement systems.

Another advantage that ABCE figures have over NNR and HW data is that rather than being based on extrapolations from samples they are based on the counting and measurement of actual activities on audited web sites. This counting technique can have its own issues, for example the undercounting of pages by log file based software due to 'caching'. Additionally not all sites are willing or able to have their traffic figures scrutinised by a third party and again this can lead to an incomplete picture of the market.

So why do these different methodologies produce different figures and which should be regarded as the most reliable?

In short the different systems produce different figures because they work them out in different ways. There are advantages and disadvantages to all three and for different reasons no one system offers a 100% complete picture of the market. In light of this we'd recommend using the system that produces the most credible and comparable data and when it comes to credible data then independently audited figures that are based on counting have to be regarded as more useful than extrapolated numbers. There is a place for extrapolation, for example when it is not possible to measure activity or conduct interviews with an entire universe, but when it is possible to measure activity then this has to be the preferred option. Because of this we'd recommend using ABCE audited figures whenever possible.

For more information about some of the technical terms or acronyms used in this newsletter please visit the IAB's internet jargon buster or Whatis.com.