The phrase “Lies, damned lies, and statistics” has been well known and widely used for over 100 years.
The debate about who first coined the phrase is unresolved but the leading candidate seems to be the British Prime Minister Benjamin Disraeli – although he may simply have been the first prominent politician to use it.
It is usually rolled out in the heat of public, political or academic debate to highlight an opponent’s attempts to bolster an often dubious argument via the use of selective data – which brings me to my theme for this newsletter.
I was forced to turn that phrase over in my mind as I sifted through the many documents produced by government regulators in the last month or so. Not to put too fine a point on it, some of the data advanced in support of certain areas of investigation seem – well…selective?
At AADA we are doing a lot of work in our formal submissions to provide ACCC and ASIC with accurate, statistically significant information to help guide sensible outcomes when regulations are made. That said, often we see a surprising reliance on small data sets that drive big conclusions – and that’s worrying.
Take, for example, matters dealt with on page 10 of the ACCC’s Issues Paper (published on October 17 in support of the investigation into new car retailing). This section of the paper highlights concerns about persistent high levels of complaints received by the ACCC and ACL from new car buyers.
It includes the statement that ‘two thirds’ of new car buyers experienced problems ‘within five years’ of purchasing their new car. When you actually dig into the basis for this assertion you find that the ACCC is relying on research from …guess who…?
In a study earlier this year entitled “Lemons On Wheels” Choice surveyed 1505 consumers who said they purchased a new car between January 2011 and January 2016 – a period during which we sold about 5 million new cars. This is a sample rate of 0.03% of all new car sales – but let’s be generous and call it 0.06% by excluding fleet buyers.
That is a very low sample size.
Choice found that 66% of these 1505 consumers (about 990 owners) had experienced a problem with their cars in the last five years. These owners provide the statistical basis for the ‘two thirds’ of new car buyers that the ACCC refers to in the paper. Choice also found (not mentioned in the ACCC paper) that only 9% of those complaints could be classified as ‘major’ (with ‘major’ not defined).
In other words, the ACCC assertion that two thirds of new car buyers experience problems in their first five years of ownership is based on unverified information from about 990 new car buyers, 90 of whom might have had something ‘major’ go wrong (although we can’t actually know what ‘major’ is).
That’s 0.0036% of all retail customers over the period who might have had a ‘major’ issue.
That is a very low percentage.
Turning to warranties, the ACCC makes the statement that in 2014-15 regulators received about 1800 consumer complaints about warranty and guarantee issues on cars. There is no analysis provided as to the nature of these complaints and whether or not they were ultimately justified. If all these complaints related to cars sold during that period, then the complaint rate is about 0.16% of the number of new cars sold. If the complaints relate to a longer period, the number is even lower.
Either way, these are also very low percentages.
As regards manufacturer recalls, the ACCC points out that cars were the ‘most recalled’ products in the 2015-16 financial year – as if this is automatically and necessarily a negative outcome for customers.
Of course, many recalls are conducted by manufacturers as precautionary measures, to make adjustments to software and hardware, to improve the customer experience or to upgrade specifications at low or no cost to owners. Safety recalls are subject to a rigorous process and aren’t under review.
Again we are drawn to the question of how data is assembled and used in this debate and that in turn leads us back to the beginning of this newsletter – you can do anything with statistics if you have a mind to.
It is surely reasonable to expect a powerful government regulator to do more than rely on data samples of this size. We know from independently managed new car customer surveys that customer satisfaction scores across the industry have been on a continuous upwards trajectory for the last ten years. We also know from independent quality surveys conducted by well-respected firms like JD Power that new cars have never achieved higher quality levels or lower defect rates than they do today.
It is perplexing therefore that our industry’s relationship with our customers continues to be characterised negatively by the popular media, by consumer organisations like Choice and by regulators. In a world dominated by the 24/7 news cycle far too much weight is given to the spectacular outliers – the systemic oversight lapses that occasionally grab global headlines – and too little to the mass of data that support the industry’s success record with our consumers.
And while it’s easy to start the blame game, it’s far more productive if we also examine our own culpability in failing to set the record straight over many years.
So I ask us all to consider this question:
“What do we need to do to make sure that legislators, regulators and car buying consumers get the full facts about how efficient, responsible and consumer focused the modern retail car industry really is?”
In my view, the answer lies in consistent advocacy around the true facts by sharing accurate data with all stakeholders, including state and federal legislators and regulatory bodies. Alongside this we must also conduct a professionally managed communications strategy using traditional media and embracing all available forms of on-line social media.
It’s a work in progress that must be sustained and persistent, but it is the only way we will avoid continuing to be victims of…
‘lies, damned lies and statistics…’
As ever, I wish you…
Good Luck and Good Selling.
Chief Executive Officer
Australian Automotive Dealer Association Ltd