Chatbot and
OCR technology

Learning the basics: how machines are able to converse with, read and even sense a customer's emotions

It does not take long for the ears of a non-Cantonese speaker in Hong Kong to prick up when overhearing two locals speak, struck by how many recognisable words in English will pepper the average conversation in our city.

Story #1

This is Kongish, the inevitable melding of the city’s official languages (and not to be confused with Konglish, which is where Korean and English meet). While the bilingual population of the city will have no problem sorting out this organically developing hybrid language, in terms of automation it proves more of a challenge.

ASTRI has purpose-built and successfully commercialised an AI chatbot with speech-to-text solutions that understands locally spoken Cantonese mixed with English – including jargon and slang – and Mandarin to English. It has been deployed in branches of the Industrial and Commercial Bank of China (ICBC), ranked the largest bank in the world in terms of assets by a 2019 S&P Global Market Intelligence report.

“ICBC were getting thousands of enquiries a day and they needed a faster way of dealing with them,” explains Co Fung, an ASTRI engineer specialising in linguistics, artificial intelligence and big data analysis. “We prepared a chatbot that can handle general banking questions, such as how to apply for a credit card or open an account. It freed up ICBC’s employees to attend to customers with complaints or more complex needs.”

The first stage of the project’s development was to collate all the questions from ICBC’s training manual normally given to new joiners and other internal documents. The ASTRI team began creating a tree spreadsheet with each question forming the initial trunk then branching off depending on the possible answers. Doing this from scratch took 18 months, Co explains, but now that the team has the structure and experience, making a new model would only take a third as long. This could be useful as other banks have also explored the technology’s potential.

The source document can also be updated quickly to meet the rapidly evolving terminology in both the banking world and the hybrid language, though that is where the project really got interesting.

“Teaching the chatbot Kongish was the most difficult part,” Co says. “We had very little data so our team had to record ourselves role-playing banking scenarios. Then we would transcribe and look for patterns.”

Now that the technology has grasped Kongish, the next step is to develop text to speech so the system can detect a natural way to answer the user and to teach it emotions.

To take the level of customer service further, the ASTRI team has developed a sentiment analyser that will be able to detect whether the customer is content or starting to lose their patience. If the chatbot senses rising irritation in the customer, it will be trained to reply in a slower and calmer tone. This will be assessed through both through what the customer says as well as how they say it.

Does this include profanity? “That is also under development,” Co replies.

The chatbot is not the only ASTRI technology capable of making sense quickly from the
complexities of Hong Kong’s unique linguistical circumstances.

ASTRI’s handwritten Chinese Optical Character Recognition (OCR) technology is capable of
recognising more than 8,000 traditional and simplified handwritten, English typed letters and numbers, reading a page in three seconds on average with an accuracy reaching 97.12 per cent, which outscores humans by a percentage point.

The OCR technology is also equipped with contextual intelligence capable of auto-correcting local addresses.  Previously deployed to financial service providers to minimise operating costs involving manual data entries, the technology is now being customised for the ITC to handle applications for Technology Voucher Programme for small and medium-sized enterprises (SMEs), as well as the disciplined services such as fire and police, and that form that all adult Hongkongers will be
familiar with, the recognizable green tax return.

The pain point for Hong Kong here becomes apparent fast. When ASTRI Deputy Director, Multimedia Systems & Analytics Dr. Arvin Tang searched on jobs website on one given day, he found more than 2,000 jobs for data entry personnel. Further calculations using expected salary and adjusted for turnover, arrived at the conclusion that more than HK$4 billion a year was being spent in Hong Kong on data entry.

“Our technology can help Hong Kong become more efficient and save money,” Arvin says.

The high accuracy rating in Chinese characters is the key selling point. It can read 4,000 simplified characters with 97 per cent accuracy, and 6,000 traditional characters with 98 per cent accuracy.


“The extra strokes in traditional Chinese character add more complexity and in this, actually make it easier for the machine to distinguish words and score higher,” Arvin explains.

With so many Chinese characters to learn and many similar in shape, it was a time-consuming
process to teach the machine. There are many old Chinese characters that remain in use in Hong Kong but have become obsolete or falling out of popular usage in the rest of the Chinese-speaking world. Then just as Co and the chatbot team needed to work on huge variations in language, Arvin’s team had a similar challenge in handwriting.

“We collected many different handwriting styles for every single character and trained the machine to adopt large variations between them,” Arvin says.

As for the next chapter in the OCR’s story, while the accuracy is high, the next development is robustness since the accuracy can drop if the quality of a document deteriorates, while the
ultimate end goal remains to

free all Hongkongers from the arduous and tedious task of data entry in the future.