Ultimate Guide
Home | Articles | Headlines | Searches | Tools | About | Suggest

Get the figures right and the facts will look after themselves

By Michael Lund

1 December 2007

It's a year now since the ABC was told that there was evidence of a cancer cluster at the Toowong studios in Brisbane, Queensland.

It was a day of mixed emotions for those who'd fought long and hard to prove that something was wrong in their workplace. It brought a sense of relief for some, and a sense of sadness at the same time.

I can't imagine what that must be like, to be told that your workplace has damaged your health, potentially threatened your life. I can only offer my support and sympathies to those involved.

But I also remember the day with some frustration too, because it was day some inaccurate reporting on the scale of the cancer cluster began.

Mid-morning on Thursday December 21, 2006, about a hundred-plus ABC staff gathered in the Riverside canteen to hear the latest findings from cancer expert Prof Bruce Armstrong.

The ABC's managing director Mark Scott had asked the professor to investigate the staff concerns and now he revealed the findings of his research team, which included the ABC's own health expert, Dr Norman Swan.

After some considerable work checking and double checking the medical and personal histories of the effected staff, and efforts to establish the impact of breast cancer on a comparable population, Prof Armstrong said the research team had finally found something that previous investigations hadn't.

Of 550 women identified as working at the site between January 1994 and June 2006, 13 had been diagnosed with some form of breast disease. Three were considered to have conditions outside the scope of the inquiry, leaving 10 with breast cancer for further investigation.

The research team had then calculated the expected incidence of breast cancer among a similar average population and found it to be 1.6, so roughly one or two cases. Not 10.

A simple piece of arithmetic by the research team (10÷1.6 = 6.25) established that the incidence of breast cancer at the ABC studios in Brisbane was six times the norm. That's a six fold increase.

Prof Armstrong went on to say the team had tested the validity of that figure to the best of their statistical ability and he was confident there was a one in a million chance that such a cluster could have occurred by chance.

So how accurate was this figure? Statisticians like to work on a 95 per cent confidence interval, by that they mean than if the study was repeated 20 times then on 19 occasions they would probably get the same result within a certain range. That range is often referred to as the error margin.

In the ABC breast cancer study the confidence interval was between three and 11.5. That means the number of breast cancer cases found at the ABC was between three times and 11 times higher than expected.

But the research team had calculated the figure at 6.25.

Some journalists took this to mean that the incidence of breast cancer among women was "up 11 times" higher than average. In once sense they are correct, but only in reporting at the extreme.

The first report to make this claim was the ABC's own lunchtime radio news and current affairs program, The World Today:

"The report showed women who work there have reported breast cancer at a rate 11 times higher than the general working community."
The World Today, Thursday 21 December 2006

Other news organisations (not all) followed.

No journalist has ever said (as far as I'm aware) that the incidence of breast cancer at the ABC studios was "three times" higher than average.

That would be a significant story in itself, as what Prof Armstrong and his team had succeeded in doing was proving that the incidence of breast cancer on the site was higher than average, and that's enough to have it termed a cancer cluster.

Prof Armstrong's team had finally confirmed what many staff had suspected for some time, that there was a cancer cluster at the ABC studios, and to the best of their ability they discovered the risk of breast cancer among women on the site was six times higher than average. Six times!

The team had not identified any cause but this figure was enough for Mark Scott to announce the ABC would abandon the Toowong site as soon as was possible. News broadcasts in Brisbane were being cancelled with bulletins would come from Sydney.

By the PM program the ABC was correcting itself:

"Ten breast cancer cases have surfaced at the Toowong studios in a decade and the experts say that means women working there are six times more likely to develop breast cancer than women in the general community."
PM, Thursday 21 December 2006

Other news organisations (not all) were still reporting the "11 times" claim.

The story quickly travelled around the globe as evidence of a cancer cluster was hot news. The "11 times" figure popped up in many overseas news reports.

By the New Year the ABC was again quoting the "11 times" figure, this time on ABC Radio National's The Media Report.

"The reason for the sudden evacuation was a finding that breast cancer levels among staff at the site, were up to 11 times the national average, and that something on the site was the cause of the ongoing cancer problem."
ABC Radio National, Media Report, Thursday 1 February 2007

In March the ABC was preparing to broadcast a two-part Australian Story "Million to One" and the public broadcaster's reporting was more accurate on Prof Armstrong's report.

"The report found that the incidence of breast cancer among women working at the ABC in Brisbane was around six times higher than expected. The chances of this being a statistical fluke were put at 'one in a million'."
Australian Story, "Million to One", Monday 12 March 2007

However, the documentary prompted a rush of other news stories on the ABC's cancer cluster and the "11 times" claim returned in many (not all) news reports.

Again, in October this year it was announced a new study into the ABC cancer cluster funded by the National Breast Cancer Foundation, and carried out by the Queensland University of Technology, Queensland Institute of Medical Research and the University of Queensland.

This time the ABC's online news reported:

"The Toowong site was shut down late last year after a report confirmed breast cancer rates were up to 11 times higher than normal."
ABC News Online, Thursday 25 October 2007

Other news organisations (not all) repeated the "11 times" claim.

I'm not picking on the ABC. I use its reporting to show how even when a story is in your own backyard, journalists can get it wrong.

Why do I think they're wrong? What's wrong with reporting the "11 times" claim, especially when pre-fixed by "up to"?

I'll let Prof Armstrong explain.

"It could be considered to be the highest plausible estimate of the actual increase in risk, while the best estimate was 6.25," he told me.

"It would be quite misleading about the extent of the risk to quote a figure of 11.5 (or 11) without explaining exactly what it was and that the best estimate was 6.25."

Reporting figures can be a nightmare for journalists if they do not understand the background to those figures, where they come from and what they mean.

Experts are not always skilled in explaining their research in clear and simple terms and their reports can be even more confusing. Trying to read such reports when deadlines are pressing can be difficult for journalists.

The Breast Cancer at the ABC Toowong Queensland - Third Progress Report from the Independent Review and Scientific Investigation Panel - 21st December 2006 was 62 pages long and was written more as a scientific journal that a document for journalists.

Maybe there is a message here for those who hope for fair and accurate reporting of any complex reports they release to journalists. If they are to make claims, then make then clear and understandable to an average audience.

Reading through Prof Armstrong's report does eventually provide the answers at page 48, and even this report errs on the lower side of caution.

"These results suggest that the incidence of breast cancer among female staff working at the ABC Toowong Studios may be about 6 times higher than what could be expected based on the age-specific breast cancer incidence rates of the Queensland female population. Taking the statistical uncertainty into account, we have high confidence that the breast cancer rates among female staff at ABC Toowong studios are at least 2.9 times higher than expected."

To fully understand my concerns at reporting at extremes of error margins take the reporting of opinion polls for the recent federal election. Would it have been acceptable for a journalist to report any poll figures at the extreme of error margins, usually +/-3 per cent the actual result?

If an opinion poll says Kevin Rudd is at 54 per cent and John Howard at 46 per cent you would not expect to read a report saying support for Kevin Rudd was "up to 57 per cent" while support for John Howard was "as low as 43 per cent", effectively giving Labor a 14 per cent lead over the Coalition.

Perhaps more worrying though is that reporting at the extremes of error margins makes it difficult for others in any future investigations.

What if a cancer cluster study on another case finds a best estimate of an increased risk at seven times the average, but has a narrower margin of error of between five and nine.

Any reports then of a risk "up to nine times" higher would make it sound not as serious as the ABC case, and yet the best estimate is worse.

As journalists we have a responsibility to get it right. We should stick to the facts when reporting and let those facts speak for themselves.

This brings me to the final report on the ABC's breast cancer study, released this June. The research team considers if anything has been gained by the study given that no cause has been found.

The team found evidence that the breast cancer cases were related to the length of employment of the women, and it says removing staff from the Toowong site has addressed staff concerns as well as taking pressure of management for any further action on the cancer cluster.

So does that mean there will be no new breast cancer cases among ABC staff?

"Time will tell if it has also brought an end to clustering of breast cancer in the ABC female workforce formerly at Toowong," says the report, on page 39.

Useful Links

ABC - General Information - Incidence of Breast Cancer, ABC Toowong

Medical Journal of Australia - "There will be no more!": the legacy of the Toowong breast cancer cluster