MERDEKA Centre for Opinion Research (MC) published a news release on Feb 10 entitled “2 months after GE15: PM’s approval rating at 68%, Government at 54%”.
The next day, media was abuzz with all of them reporting based on the news release by MC.
However, none seem to have read the detailed report “Perceptions towards Economy, Leadership, and Current Issues, Post GE15 Survey”, otherwise they would have noticed the rather encouraging assessment of Prime Minister Datuk Seri Anwar Ibrahim and his unity government is not what MC has projected.
Please note that I am not disputing Anwar’s popularity – which I intuitively believe should be much better than what MC found. But the methodology and terminologies MC has used for their survey and to convey their findings is suspect.
In any statistical work, one has to check the methodology first – and if it is not sound – the results should be rejected. My hypothesis is MC’s survey may not be statistically valid, and its use of different terminologies further degrades their findings.
“Anwar’s approval rating worse than Mahathir’s and Muhyiddin’s”
This was the conclusion of MC based on the chart in page 10 of the detailed report. 79% of respondents were happy with Tun Dr Mahathir Mohamad’s Pakatan Harapan (PH) government in May 2018; 63% were happy with Tan Sri Muhyiddin Yassin’s Perikatan Nasional (PN) government in May 2020, but only 54% were happy with Anwar’s unity government in December 2022.
Anwar was slightly better than Datuk Seri Ismail Sabri’s Barisan Nasional (BN) government at 45% in October 2021. This was based on the questions of “happy” or “angry”.
Intuitively, Anwar and his government should have received a higher rating considering his unity government consists of major parties such as PH, BN and the ruling parties in Sarawak and Sabah.
In behavioural science, intuition plays a big part but has to be verified by statistical analysis. To me, this raised a red flag and prompted me to look deeper into the methodology and detailed analysis. As I explored further, more red flags appeared.
Liberal use of confusing terms
In their news release, MC uses the term “approval rating” and “positive rating” of 68% for Anwar’s leadership, presumably to mean the same thing. This draws from responses to questions of Very Satisfied (15%) and Somewhat Satisfied (53%). Would ordinary respondents understand the distinction between ‘somewhat’ and ‘very satisfied’?
Unambiguous and simple terms are very important to elicit the right responses instinctively. Shouldn’t MC have just asked whether the respondents approve or disapprove of Anwar’s leadership? Then there is “happy” or “angry” for government performance – a heavy emotion-laden question.
Consultants are known to pad up the questions in different shades which confuses the respondents, hence making the survey invalid and unscientific.
Tiny sample size
Statistics is about using a sample to predict the behaviour of the “population”. In this case, the population would be the 21 million voters in the 15th General Election (GE15). Of course, if every one of them can be interviewed, that will be the ‘truth’ but that is impossible.
So a sample is drawn representative of the population, and an estimate/prediction of the truth is made. The smaller the sample, the further away will be the truth.
In this case, 1,209 respondents were chosen from the population of 21 million, representing just 0.006% of the Malaysian population. This is understandable for a quick and manageable survey.
But I would like to see the various statistical tests done to ensure significance, confidence, predictive, correlation and causal effects of survey results before I can comment further. All I can say now is that an estimated ‘truth’ from a tiny sample size has to be treated with extreme caution.
Furthermore, MC said that almost 25% of respondents refrained from giving their views, and conveniently explained it as “indicative of cautiousness after four years of tumultuous political developments”.
If these respondents had collectively – in an extreme case – responded one way or the other, there would be a margin of error of ±25%. Pollsters’ opinion should not interfere in the exact sciences of statistical analysis. Maybe the questions were too complicated or vague that responders stayed away.
Database of confidential information
MC says the respondents were selected through random stratified sampling method using telephone numbers along the lines of ethnicity, gender, age and states. That would mean MC must have a database of 21 million voters in terms of their telephone numbers, name, gender and addresses in order to extract the random stratified sample.
How MC has access to this confidential privacy-protected information is a question they need to answer. If they don’t have full access, then their sample is not statistically valid.
The title of MC’s news release seem to indicate that the survey results are indicative of Anwar’s leadership and his government’s performance after two months of taking office.
Anwar’s full Cabinet of ministers and deputy ministers was installed on Dec 9. The survey was conducted from Dec 26 to Jan 15. MC extolls the virtues of telephone interviews as higher penetration and able to initiate and conclude a programme within 72 hours.
Anwar and his administration would hardly have settled down within two weeks after assuming office. How survey respondents could have assessed their performance then is mind-boggling.
Furthermore, a person who was interviewed on Dec 26 and again on Jan 15 would have different perceptions as the government starts to function. Why wasn’t the survey done within 72 hours.
And how on earth MC could claim the survey results were indicative of Anwar and his government after two months?
Shaping and exploiting public opinion
Public opinion pollsters play a crucial role in shaping and exploiting public opinion on matters of public interest. In this particular case, nothing more important grips the country now than the leadership of Anwar and performance of his administration.
As MC says, their survey informs and shapes current and future direction of leadership, economy and critical issues of the country.
Pollsters, including the reputable Merdeka Centre, have claimed they have vigorous methodology and top research capacity to bring out the ‘truth’ of what the population perceives, wants and demands.
As someone having used advanced statistical analysis, it is my considered opinion MC’s survey results are invalid. This is based on a ‘tiny sample’ of what I think are defects in the above poll as explained here. I welcome Merdeka Centre’s explanation and stand corrected if I am proven wrong.
Responsible, independent and truthful media which have published the survey results as part of their reportage duty, should now independently verify what is written here, and correct any misinformation or misrepresentation. – Feb 15, 2023
Raman Letchumanan is a former senior official (environment) in Malaysia and ASEAN, and Senior Fellow at Nanyang Technological University Singapore. He is an accredited accountant (Malaysia/UK) and has a Ph.D. in environmental economics, among other qualifications. Contact: [email protected].
The views expressed are solely of the author and do not necessarily reflect those of Focus Malaysia.
