Battle of AI bots: ChatGPT, Ernie Bot throw different punches

While Ernie Bot, the chatbot launched by Chinese tech giant Baidu, has a tendency to avoid certain political questions, it fares better than its OpenAI rival ChatGPT when it comes to providing up-to-date information, according to a random test conducted by the Post.

Conversations were conducted in English with ChatGPT and in Chinese with Ernie Bot to gauge the capabilities of the artificial intelligence-powered bots, and they scored differently across a range of topics including politics, economics and celebrities.

Politics proved a tricky topic for Ernie Bot, given that China is subject to strict censorship. When asked if China is a democratic nation, Baidu’s bot ducked the question by saying it “hasn’t learned how to answer this question yet”.

In contrast, ChatGPT stated that the country “does hold periodic elections at the local level” while clarifying that “ultimate decision-making authority is held by the Communist Party and its leaders”. It added that China has made “significant strides in economic development and poverty reduction” but “its political system is not considered to be democratic by international standards”.

Ernie Bot was able to provide a brief introduction when asked about Chinese political figures. However, the chatbot, which usually allows follow-up questions, asked users to “start over with a different topic in a new conversation” when the prompt included names like Chinese President Xi Jinping, newly-elected premier Li Qiang, Li’s predecessor Li Keqiang and Mao Zedong, who led the Communist Party to found the People’s Republic of China.

ChatGPT took this question on the chin, introduced these politicians and allowed longer follow-up questions.

Ernie Bot also went in the wrong direction on some questions. When asked how it would solve China’s problem of population decline, one of its suggestions was to “implement family planning and encourage couples to have only one child”. On the same question, ChatGPT suggested that the government “relax the one-child policy”.

However, ChatGPT has also made some missteps, apparently due to lack of current information as its data is currently limited to events up to 2021. It was not able to name any of this year’s Oscar winners or answer when Russia invaded Ukraine. When asked to rate Ernie Bot, it mistook it for a product from Alibaba Group Holding, the e-commerce giant that also owns the South China Morning Post.

Ernie Bot, developed by the company behind China’s largest search engine, was able to offer real-time information on events such as Russia’s invasion of Ukraine, and stated correctly that Michelle Yeoh became the first Asian woman to win an Oscar for best actress this year.

However, there were factual errors within Ernie Bot’s longer answers. For example, it stated that Russia has already taken over Kyiv, Ukraine’s capital, which is not true, and it said that Yeoh won the award for her performance in Crazy Rich Asians, instead of Everything Everywhere All at Once.

Baidu did not immediately respond to a request for comment on this issue.

At this month’s launch event for Ernie Bot, Baidu chief executive Robin Li Yanhong said that the product is not perfect but that demand for such a service is strong. US rival ChatGPT is not officially available in China or Hong Kong, for example.

Li said that Baidu’s product can “evolve faster with human feedback”. “We want [Ernie Bot] to improve faster, so it has to be launched extremely early to accelerate its ability to improve,” he said.

Ernie Bot is also able to generate images from text prompts, a feature that ChatGPT does not have. But Baidu has had to face criticism in the past week from some users who have expressed concerns that the bot may simply be taking text and putting it into foreign image generators.

For example, some netizens said that Ernie Bot drew a turkey, the bird, when asked for the country Turkey, and a crane – the bird – instead of the machine used for lifting heavy objects. These images were generated in spite of the Chinese-language prompts and use of completely different characters for each word.

Baidu asserted that its chat bot has been “totally self-developed” and that the text-to-image capabilities were trained by its Ernie-ViLG model, according to a post on its official Weibo account last Thursday.

By Thursday afternoon, Ernie Bot was able to differentiate between a crane that is a bird and the machine, and did not mix up a turkey bird with the country.

Baidu’s official product launch was also slightly overshadowed by OpenAI’s upgraded GPT-4 model, unveiled just two days earlier. Compared with GPT-3.5 that powers ChatGPT, GPT-4 is able to process images, although the feature is still being worked on by OpenAI and its partner Be My Eyes. South China Morning Post