Television Audience Measurement Issues
TAM. No, this is not a Scottish headwear. This is the acronym for Television Audience Measurement, a rather obscure business sector that nevertheless has a tremendous impact on advertising expenditure. In most countries in the Americas, the bulk of advertisements is invested in television (with possibly the sole exception of the Dominican Republic, where radio is king). Within each market, advertisers have many television choices and the allocation of their budgets is largely determined by television audience estimates provided by ratings companies. In most markets, there is only one ratings company because nobody likes to have duplicated efforts that yield divergent ratings. Therefore, the integrity of the work by the ratings companies is extremely important.
Periodically, an outsider is brought in to evaluate the work of the incumbent ratings company. The work of a ratings company is specialized and complicated, quite often beyond the comprehension of inexperienced outsiders. The experts are those people who have worked for ratings companies. Unfortunately, there are not too many of those around to serve as independent auditors.
The top local newspaper El Tiempo in Bogota (Colombia) carried an article about a recent evaluation of the local ratings company, IBOPE Colombia.
"Las cifras de audiencia no son confiables y por tanto no son válidas para hacer transacciones...". "...La probabilidad de estimaciones acertadas y por ende la confiabilidad del rating producido por Ibope está muy por debajo de lo que se considera aceptable estadísticamente...". [translation: "The audience figures are not reliable and therefore not valid to conduct transactions ... "The likelihood of accurate estimates and therefore the reliability of the ratings produced by IBOPE are much lower than that which is considered statistically acceptable ...]
Estas son apenas dos conclusiones de la auditoría que durante ocho meses la Universidad Nacional le realizó al sistema de medición de audiencia efectuado por Ibope y que, en tres tomos, fue presentada al Comité Intergremial Colombiano de Medición de Audiencia (Cicma), compuesto por la Asociación Nacional de Anunciantes (Anda), la Asociación Nacional de Medios Audiovisuales (Asomedios) y la Unión Colombiana de Empresas Publicitarias (Ucep). [translation: These are some of the conclusions of the audit report that the National University has done over an eight month period of the audience measurement system operated by IBOPE and which have been presented in three volumes to the Joint Committee of Audience Measurement composed of the National Association of Advertisers, the National Association of Audio-Visual Media and the Colombian Group of Advertising Companies.]
... El informe final también concluye que los datos suministrados por la firma medidora durante los ocho meses que duró la auditoría no son confiables para que los anunciantes pauten en los programas de televisión, ni para que los canales y concesionarios de espacios armen su programación. [translation: The final information also concludes that the data offered by the company during the eight months of the audit period were not reliable for the advertisers to select television programs, neither for the television channels nor for the television programmers who sell their programs.]
Según expertos, la confiabilidad estadística de cualquier proceso no depende de la tecnología sino de la calidad de la muestra maestra (un gran paquete de hogares representativos de estratos, sexos, edades y otras variables) y del panel (los 900 hogares que reportan diariamente su sintonía). El informe de la auditoría revela que esa muestra no está bien tomada y que no se explica cómo y por qué se escogió el panel. [translation: According to experts, the statistical reliability of this process does not depend on the technology but on the quality of the master sample (a large number of households which represent the social classes, gender, age and other variables) and the panel (the 900 households which report their television behavior every day). The audit information revealed that this sample was not well executed and that does not account for how and why the panel was chosen.]
Dentro del proceso metodológico también se observó una concentración de dicho panel en estrato pobre (el 42 por ciento), lo que a juicio de algunos consultados no concuerda con la situación demográfica del país. Además de esto se presentan situaciones en las que el estrato 6 está representado por alguien del estrato 1. Por ejemplo, se encontró que en una manzana de estrato 6, un people meter (aparato que se instala en el televisor para medir la audiencia) está en la casa de un vigilante de estrato uno. [translation: On the methodology of the procedure, it was also observed that the said panel was heavily concentrated in the poor stratum (42%), which does not correspond to the demographic situation in the country according to certain consultants. Moreover, there were some situations in which someone in stratum 6 was represented by someone in stratum 1. For example, there was a stratum 6 neighborhood in which a people meter (the instrument that is installed on the television in order to measure tuning/viewing) was in the home of a person in stratum 1.]
Otro punto que alarmó tiene que ver con el denominado factor de expansión, que consiste en la representación que una persona de ese panel tiene del resto de la población. En el mundo, este factor, según las estadísticas, debe estar entre uno y máximo 4 mil y en esta auditoría se encontró que varía hasta 45 mil. Lo que significa que la representatividad del panel no está bien ajustada. [translation: Another alarming point was about the projection factor, which is the number of persons that a person in this panel represents in the population. In this world, according to statisticians, this projection factor should be between 1 and a maximum of 4,000, but there were values as large as 45,000 encountered during the audit. This means that the panel was not quite representative.]
La llamada tasa de respuesta es otro aspecto que llama la atención. Mediciones, entre el 10 y el 25 de febrero pasados, mostraron que, durante estos días, de los 900 hogares solo 720 reportaron audiencia diaria en promedio, que 45 no informaron y que 77 solo lo hicieron durante unos pocos días. "La región más afectada por esta ausencia es la Caribe, donde el cubrimiento medio no llega al 75 por ciento", dice la auditoría. Incluso, una persona que tuvo acceso a la auditoría, comentó que se encontró que un solo día la medición del rating se hizo con solo ocho hogares. [translation: The so-called response rate is another aspect that called for attention. During the measurement period between the 10th and 25th of February, it was reported that only 720 of the 900 households reported on the average, with 45 not reporting at all and 77 reporting for only a few days. "The region most affected by the missing homes was the Caribe, where the average coverage was no better than 75%," said the audit report. According to someone who has access to the audit report, there was one measurement day in which the rating was based only upon 8 households.]
Por este tipo de hallazgos se concluye que los márgenes de error son tan altos que están muy por encima de lo que estadísticamente se podría o debería aceptar. Hasta por encima del 50 por ciento, en algunos casos. [translation: Because of this type of shortcoming, it was concluded that the margins of error are so high that they exceed the threshold of acceptabilty. In some cases, this was more than 50% higher than the threshold.]
Now I don't have access to this audit report. However, I have worked on various television audience measurement systems (including the IBOPE Colombia system) that I can make some comments on the facts as given in this newspaper report. I am always motivated to do so when I see TAMs being abused anywhere in the world.
I am going to skip over the first four paragraphs because they don't contain any meaningful technical information.
In paragraph five, the subject was the socio-economic characteristics of this panel. Since this panel is projected daily on socio-economic level to universe estimates that are published by the national census department, any disproportionality is immaterial. Repeat, there cannot be any effect!
As for the example of crossing of the stratum 1 and stratum 6 cases, I suspect that this has plenty to do with the particular definition of socio-economic level that is used in Colombia. In all other countries in the Americas, socio-economic level is defined at the individual household level. In USA and Canada, it is based upon the household income. In the rest of Latin America, it is based upon a combination of household goods and services together with the education level and occupation of the head of household. Colombia is the only country in which socio-economic level is defined at the local neighborhood level. Each local neighborhood is assigned to a stratum (1, 2, 3, 4, 5, 6). If neighborhoods are highly segregated and homogeneous, then this might make sense. There are mixed neighborhoods in Colombia where wooden lean-to shacks stand next to palatial mansions but this socio-economic scheme ends up assigning the same socio-economic level to all households in the same neighborhood. That is why the Colombian socio-economic level is relatively undiscriminating and ineffective for targeting advertisements.
In paragraph six, the subject was the projection factor (known elsewhere as the case weight). This sample of 900 households is used to represent several million households in the country. The only way that a projection factor of 1 can occur is that the sample represents 100% of the universe (ergo, each household represents one household, namely itself), and this is definitely not the case. The whole point about sampling is that it is neither necessary nor feasible to measure the entire universe. As for the large projection factor that is reported, it is not clear whether this had occurred by design. The Colombian sample is designed to produce national television audience estimates for the national networks (such as RCN and Caracol) as well as regional channels (such as City TV in Bogota). It may be necessary to allocate the sample disproportionately across the geographical regions in order to report these disparate estimates, as is done in any number of other countries (e.g. Argentina, Brazil, Canada, Chile, Ecuador, Mexico and Venezuela), which means that the maximum value would be different from an average value calculation by purposive design.
In paragraph seven, the report is not referring to a true response rate. Instead, this is just narrowly known as the intab rate, which is just one component of the overall response rate. The Colombian sample is a geographically dispersed sample that covers multiple regions of the country, some of which are not easily accessible. On any given day, any such system will encounter problems with individual households. For example, it may be observed that a household is not returning viewing data for a sustained period of time and different from its usual norm. You can choose to continue to keep the household in the report sample with zero viewing. But the more conservative action is to hold the household out of the report sample, set up an appointment to visit the household and then send a technician over. You may find that the household has unplugged the people meter because they bought a new television set and dumped the old one. It would be a grave error to have assigned zero viewing to this household.
As for that one day in which there were only 8 intab households in the Caribe region, this is a self-similar phenomenon that occurs everywhere. In the USA, the national television audience measurement is operated by the Nielsen Media Research with an average of over 5,000 intab households per day to represent the 110 million households in the country. The purpose is to report on national television ratings. If you ask for the number of cases in the local New York City market, it will be about 350 households which is marginally adequate. If you ask for the number of cases in the local market of Alpena (MI), there may only be 5 households which is inadequate. If you ask for the number of cases in Love (TX) county (population: 100 households in total), you may find zero representation. If you ask for the number of cases on your local street block, you will most probably not find anyone. But the point about the Nielsen sample is that it is NOT intended to measure either New York City or Alpena (MI) or Love (TX) or your local street block. Above all, it is a national sample that is composed of many local markets. When you see 8 households as the base for a rating, you are not supposed to be looking at that level of detail. If your business needs require that you examine at that level, you should be paying for a larger sample.
The last quoted paragraph refers to a mythical threshold of reliability being exceeded. The television audience measurement system does not produce one and only one television rating that can be benchmarked. Rather, each report contains millions and millions of numbers (organized by demographic groups, television channels, television programs and time period). Each of these ratings has its own margin of error because of its unique characteristics (e.g. a smaller demographic group has a higher margin of error, a longer time period has a lower margin of error, etc). A number such as the rating among boys 12 to 17 years old watching Telecaribe at between 6am-615am on the first Monday of February will have a very large margin of error; for example, there may be only one viewer, and the rating would be doubled if only one other person tuned in. But a number such as the rating for adult women to the weekday telenovela between 8pm-9pm on Caracol over a 3 month period would be very robust. So I have no idea how the threshold is operationalized with respect to these finer points.
This has been a fairly technical discussion, which should interest the known number of TAM industry people who visit this page. For everybody, though, the point is that TAM is a highly technical subject which involves issues that are related to industrial practice and not obvious to outsiders. Quite often, academic researchers and reporters rush into this subject and miss the mark on some key issues.
(posted by Roland Soong, 8/12/2003)
(Return to Zona Latina's Home Page)