We saw examples of correct answers. And in many of these cases, the chatbots were also able to provide references. In the answer below, for example, ChatGPT4o correctly debunks the false claim that the European Commission will impose water restrictions on the population, including shower restrictions. To support its claim, the chatbot cites European government websites that detail the Commission’s current stance on water usage. In addition, it cites an example from Catalonia, where water restrictions were imposed on the population, particularly when it comes to showering in sports facilities.
However, we also saw examples of answers that were only partially correct and a few that were entirely false. Some answers seemed to be based on out-of-date information. For example, when asked about the number of Members of the European Parliament (MEPs) that will be elected this year, GPT-4o responded with the current number of members (705), which will change at this election, increasing to 720. This happened when it was asked in Italian and German, although not in French and Spanish. In both cases where it gave the wrong response, it didn’t appear to have accessed the Internet, and did not provide any sources.
ChatGPT’s German response above incorrectly states that a total of 705 MEPs will be elected in the 2024 European elections (the actual number is 720).
Another example of an incorrect response came when we asked Perplexity.ai about the debunked claim that people will be able to vote for more than one party by writing in percentages in more than one box on the ballot paper. For Italy, it cites the correct information about Italy’s preferential voting system but draws the incorrect conclusion that this means people are allowed to vote for more than one party. In fact, the preferential system only allows voters to select one list from the ballot paper, and then gives them the option to write in the names of up to three preferences for individual candidates belonging to that list.
ChatGPT 4o also gave an incorrect answer to this question in Spanish as it did not deny the false claim. In a rather non-committal answer, the chatbot said that while electoral systems vary amongst countries, percentage voting is not permitted in most countries. In fact, for every country taking part in the European Parliament elections, any alteration to the ballot (which would include writing percentages) would nullify the vote.
In the partially correct responses, the systems generally addressed the overall claim accurately but sometimes provided outdated information from the last election, included irrelevant details, or got specific details wrong.
We asked Perplexity.ai in German about the debunked claim that EU officials have approved a retirement age of 50 for themselves. While giving the correct response that this claim is false, and that the real retirement age is higher, the chatbot then provided contradictory statements about the actual age of retirement and cited an outdated decision from 2004. Similarly, in the Spanish case, Google’s Gemini correctly classified this claim as false. However, the chatbot then gave the incorrect information that the minimum retirement age is 60. In fact, EU civil servants are entitled to a pension from the age of 66.
Many responses contained sources, with Google’s Gemini (when not simply pointing users to Google Search) and Perplexity.ai consistently giving sources for all their answers. ChatGPT, too, often provided links.
The sources provided in all countries were a mixture of encyclopaedic references (predominantly from Wikipedia), official election websites from national governments or the European Union (usually the official website for the European elections) or general sites like government portals. In some cases, the chatbots cited the websites of local government branches and councils, as well as news organisations and fact-checkers (e.g. Newtral, El País, BBC, Le Monde, DPA or POLITICO).
However, we also encountered answers that were only partially correct and a few that were false, even with a relatively low bar of very basic election questions and already debunked claims. This raises an important question: which kind of response we would have potentially received for more complex, or consequential questions or if we had asked these systems about false narratives that have not yet been addressed by fact-checkers?
Sometimes, the same chatbot provided a correct answer in one language, but a partially correct or false one in others – an issue that connects to criticism that chatbots perform better in some languages and contexts than others due to their training. Although we can’t say how widespread these problems are, they are nonetheless apparent.
Users may not always notice these errors (and we often only did when we checked the answers claim by claim), given the authoritative tone of these systems and how they provide a single answer instead of a list of results. In some cases (such as the issue of whether you can write in a percentage on your ballot) if people acted on the output generated, their vote would be nullified – a potential consequence of relying on such a system that is more consequential than being misinformed about the retirement age of EU bureaucrats. While all systems provide small disclaimers about potential inaccuracies, we can’t say how much attention people pay to these and if so, how this affects their perception.
It’s important to note that the overall usage of these AI systems for information and news is still relatively low. People also tend to consume information – and political information – from a variety of sources, including traditional media, search, and from friends and family. All of this will likely serve as a buffer, reducing the impact of any errors or inaccuracies from a single source, such as an AI system.
While we need a better, systematic understanding, on a representative scale, of how well these systems perform, especially as more people use them and come to rely upon them in various ways, including for political information, it is worth remembering that they are just one piece in the larger puzzle of how people stay informed and make sense of the world.
We thank Priscille Biehlmann for her support with the French questions and Rasmus Nielsen for his suggestions during the editing process.
Source link : https://reutersinstitute.politics.ox.ac.uk/news/how-ai-chatbots-responded-basic-questions-about-2024-european-elections-right-vote
Author :
Publish date : 2024-06-06 07:00:00
Copyright for syndicated content belongs to the linked Source.
