Technology with Shaw Pence

This is one of my course reports at first sumbitted in April, 2023 and published here in Dec, 2023.

History

When ChatGPT has been introduced months ago, the current dynamics of chatbots advances Alan Turing (1950)’s ideal, where he proposed the question “Can machines think”. This retrospective will look back at the history of chatbots, from the early days of chatbots to the present day, highlighting the past milestones that have shaped today’s more powerful chatbots.

The first chatbot is called ELIZA (Weizenbaum, 1966) as a response to the Turing test (Shum et al., 2018). ELIZA was invented by Weizenbaum in the Artificial Intelligence Laboratory of MIT in 1964 (Skrebeca et al., 2021). This machine, named DOCTOR at that time, worked as a Rogerian psychotherapist to answer emotions-related questions. Because Weizenbaum only used a rule-based approach and limited patterns to generate answers in ELIZA, it is evident that these utterances deviate from humans’ words, especially from professional therapists.

PARRY is another robot in the early years created by Kenneth Colby, a Stanford psychiatrist as well as computer scientist, in 1972 (Colby, 1976). As a part of psychology research, this work assumes the role of a simulated paranoid schizophrenic patient rather than a doctor as ELIZA, equipping with effective emotion variables like anger, fear, and mistrust. Compared with ELIZA, beyond the similar pattern and rule-based approach shared by both of them, PARRY is more well-engineered with a more advanced structure (Thorat & Jadhav, 2020).

Ractor is another notable chatbot (Chamberlain, 1984). It was developed by William Chamberlain and Thomas Etter in 1983 for Amiga, Apple II, Macintosh platforms (Zemčík, 2019). The main function of Ractor is to create proses like the following one:

“A crow is a bird, an eagle is a bird, a dove is a bird. They all fly in the night and in the day. They fly when the sky is red and when the heaven is blue. They fly through the atmosphere. We cannot fly. We are not like a crow or an eagle or a dove. We are not birds. But we can dream about them. You can.”

(Chamberlain, 1984)

Some literature see it as a chatbots (Skrebeca et al., 2021; Zemčík, 2019), it mainly works, however, in generating texts, rather engaging in a conversation. Given that text generating is a crucial part of todays’ chatbots, Ractor is considered as a pioneer in this retrospective.

A.L.I.C.E (Artificial Linguistic Internet Computer Entity), created in 1995, is another influential chatbots in the whole history of the development of chatbots because of numerous innovations in this work (Adamopoulou & Moussiades, 2020). It is a practice adopted AIML (Artificial Intelligence Markup Language, Figure 1), an XML-based markup language (Thorat & Jadhav, 2020) facilitating pattern recognition in the development of AI agents (Bruno Marietto et al., 2013). Inspired by ELIZA, ALICE extends the development of pattern machine approch and the knowledge to 40, 000 categories (only about 200 for ELIZA) (Wallace, 2009). However, ALICE did not have enough intelligent to generate human-like answers with emotions (Adamopoulou & Moussiades, 2020).

Figure 1 An example of AIML

Chatbots discussed above are technological explorations and pioneers, where most are only laboratory products. When it comes to 2000s later, commercially used or mass-oriented chatbots started to sprout with the help of the Internet and big companies’ involvement. SmarterChild, developed in 2001, was available on messaging apps like AOL and MSN (Molnar & Szuts, 2018). As a commercial product, you could even some advertisements and sponsors’ information within the chats. Microsoft also attempts to launch a Chatbot, and then named as XiaoIce in 2014 (Shum et al., 2018). This is only the first attempt for this big company. In recent years, almost all tech giants, like Apple Siri, IBM Watson, Google Now, Microsoft Cortana, tried to build their chatbot products. Although chatbots nowadays seems so profitable, customers are reluctant to engage with them (Van Pinxteren et al., 2020), until the ChatGPT came into being in 2022 with peoples’ astonishing and welcoming.

Figure 2 SmartChild user interface

Since the first electronic computer was introduced, it only took 19 years to bring the first chatbot ELIZA to the reality. Computer scientists’ determined endeavor and enthusiasm to engineer a more powerful chatbot from that time onwards finally becomes more realistic than ever when the ChatGPT was launched by OpenAI and keeping evolved.

Technology

Pattern matching and rule-based approach

Pattern matching is a work to recognize whether a string is similar to a pattern in which a text sequence defines a string set by given fixed and free text positions.This technology is widely used in Natural Language Processing especially at early ages before Chomsky’s linguistic knowledge impacted computing linguistics and mathematical grammar models (e.g., CNF, CKY parsing, POS) were proposed. Regular Expression is the most popular and simple pattern matching method in nowadays. Both ELIZA and ALICE employed pattern matching in their system.

ELIZA is based on 200 patterns and rules. The rules are comprises decomposition template (Weizenbaum, 1966), the pattern for parsing users’ input, and reassembly rule, the pattern to generate output sentence. Both of these two patterns use numbers to mark the functions for specific words. The decomposition template is like “0 YOU 0 ME” where the digit 0 means a place for indefinite words. The reassembly rule is defined as “WHAT MAKES YOU THINK I 3 YOU” where the digit 3 represents the third word in the input sentence. By embedding the third word in the reassembly rule, a response is generated. When the sentence is long and complicated, it might fires multiple rules. When there is no pattern matched, ELIZA will respond a non-commital response like “I SEE”, “PLEASE GO ON”, “THAT’S VERY INTERESTING.”

While ELIZA’s architecture is easy and looks like it executes copy and paste only, as a chatbot at the early age, it even did not adopt the word “pattern”, it is an impactful work for the whole chatbot development. After about three decades, A.L.I.C.E. adopted a more complicated pattern matching.

A.L.I.C.E. is an implementation of Artificial Intelligence Markup Language, which is a specified format of XML file to define patterns. Enlightened by ELIZA, AIML shares some principles in defining patterns, such as digits marked blank position for tentative sentence parts. In AIML, basically, all the patterns are stored in a tree structure called Graphmaster (Wallace, 2009) for pattern matching. The structure of Graphmaster is shown as Figure 3.

Figure 3 Graphmaster (Wallace, 2014)

Retrieval-based approach

The retrieval-based approach regards users’ input as query and find the most similar response in a corpus. A tradition strategy is to use cosine similarity to calculate as the equation below.

\(\) \begin{aligned} response(q, C)=argmax \frac{q \cdot r}{|q||r|} \quad r \in C \end{aligned}

Where q and r represent tf-idf for query and candidate response respectively. Using encoder, like BERT (Devlin et al., 2019), to calculate the similarity is a more advanced approach. This method requires transform users’ input and candidate response to embeddings. Via input these embeddings into the BERT, we will get two representations. Furtherly, we can calculate the similarity by dot product on these two representations (Jurafsky & Martin, 2023).

Generation-based approach

Sequence to sequence model is a commonly used text generation model. This kind of models regards text as a sequential data, and then input these data into a sequential data-oriented neural network (RNN, GRU, LSTM). Generation model always trained with paired database to simulate human conversations. Google harnesses the seq2seq model in their translation product, which significantly improved the performance compared with all the other published methods (Sutskever et al., 2014). Microsoft Xiaoice is a foremost user of seq2seq model in chatbot system. The Neural Response Generator of Xiaoice is based on a GRU-RNN model where the model can linearly combine the users’ input query, the features of chatbot itself (Xiaoice) and users’ features into a vector representation (Zhou et al., 2020, figure 4) to generate text in the decoder.

Figure 4 Neural Response Generator of Xiaoice

Since Transformer (Vaswani et al., 2017) proposed and became evident, it is also utilized in chatbot. ChatGPT harness the transformer in its decoder to predict the next token of the output texts (OpenAI, 2023) and make great impact in recent months.

Slot filling approach for intent-oriented chatbots

Slot filling is a simple method for intent-oriented chatbots. This is a classification task on word embeddings targeting to identity all words in the user input whether it belongs to a specific slot where it is called BIO labels. BERT is the popular model in nowadays to address this classification task with a better performance than other encoders like RNN, GRU, or LSTM (Chen et al., 2019). The architecture is shown in figure 5 (Jurafsky & Martin, 2023).

Figure 5 The architecture of slot filling task

We revisited four types of Chatbot technology in this section: rule-based, retrieval-based, generation approach and slot filling. In practice, these four technologies are not used discretely. In a large, business chatbot, two or more strategies are always required to adopt all together. In Xiaoice’s research, a hybrid model is stronger than one way approach (Zhou et al., 2020). And intent-oriented chatbots are always combine the slot filling and rule-based approach.

Applications

Given that chatbot applications have varied so transiently since the dynamic after Transformers are introduced, especially ChatGPT triumphs in the market, people, not only those in academic or computer science industry professionals but also all kinds of individuals regardless of disciplines, jobs, or purposes, are so touched that they are now all engaging in exploring the new applications of chatbot and publishing their discoveries on social media and open source software platform which pushes the past research on the taxonomy of chatbot application obsolete so fast. We already knew that some promising applications would emerge soon, maybe tomorrow, which may challenge our discussions here. Therefore, it will be a better time to recall chatbot applications in a sorted way in the future. In the last of this passage, some chatbot applications will be presented individually in a tentative sort as a retrospective. Furthermore, we can also discuss the future of applications of chatbots.

FAQ chatbots and knowledge retrieval chatbots

Institutions constantly encounter intensively consultations, so they collect frequently asked questions from the history records and list them in a web page. This is a conventional approach to address this problem. Chatbot can also retrieve FAQs as a response to users’ queries (Ranoliya et al., 2017) to alleviate the workload of administrative staff (Lee et al., 2019). In this kind of application, chatbot assumes a role of the interface between users and database. By replacing clicks on windows and webpages, user can use their own words to fulfill their wishes to knowledge.

Tasks-oriented or intent-based chatbots

Businesses deploy tasks-oriented chatbots to receive and execute users’ instructions, like Siri and Google Now when they are asked to check the weather, find a path or a location and control smart appliances. Although it is believed that they are enabled to complete multiple tasks with the help of both sequential models and slot filling approaches, they are regarded as not intelligent enough due to functional defects like poor voice recognition, unnatural dialogue responses, and an inability to support mixed-language speech recognition (Bogers et al., 2019).

Free chat chatbots

When it is proposed, the first cohort of users surging into Xiaoice tried to test the chatbot with free chat questions. People crave for chatbots to be more humanoid with understanding human’s emotions and responding correctly. When it comes to this objective, it is quite a magic.

Future: What is possible and what is impossible?

All in all, chatbot is a human-computer interface which is capable, theoretically, replacing any other user interface could do and do more. Imagine that I am typing in a document, and you are designing a slide for tomorrow’s presentation, what about, leaving our mice and keyboards, just giving instructions by our voice. It is now being more realistic after Microsoft launched its Microsoft 365 copilot. And technically it is also possible by training a GPT model on a pairwise training dataset combining users’ linguistic inputs and a series instruction in the software.

However, there is something it couldn’t be practical in a foreseeable future. One instance is chatbots’ attempts to play a role as psychological therapist. We have failed since engineers’ first attempt bringing ELIZA to the world. It did not replace any therapist, but only resulting people’s surprises and motivations to develop more powerful chatbots. Emotional intelligent is too complicated to be complete by a robot because people can easily manipulate their input no matter facial expression or utterance.

Reference

Adamopoulou, E., & Moussiades, L. (2020). Chatbots: History, technology, and applications. Machine Learning with Applications, 2, 100006. https://doi.org/10.1016/j.mlwa.2020.100006

Bogers, T., Al-Basri, A. A. A., Ostermann Rytlig, C., Bak Møller, M. E., Juhl Rasmussen, M., Bates Michelsen, N. K., & Gerling Jørgensen, S. (2019). A Study of Usage and Usability of Intelligent Personal Assistants in Denmark (N. G. Taylor, C. Christian-Lamb, M. H. Martin, & B. Nardi, Trans.). 79–90.

Bruno Marietto, M. das G., Aguiar, R. V., Barbosa, G. de O., Botelho, W. T., Pimentel, E., Franca, R. dos S., & da Silva, V. L. (2013). Artificial Intelligence Markup Language: A Brief Tutorial. International Journal of Computer Science & Engineering Survey, 4(3), 1–20. https://doi.org/10.5121/ijcses.2013.4301

Chamberlain, W. (1984). The Policeman’s Beard is Half Constructed: Computer Prose and Poetry. Warner Books.

Chen, Q., Zhuo, Z., & Wang, W. (2019). BERT for Joint Intent Classification and Slot Filling (arXiv:1902.10909). arXiv. http://arxiv.org/abs/1902.10909

Colby, K. (1976). Artificial paranoia: A computer simulation of paranoid processes. Behavior Therapy, 7(1), 146. https://doi.org/10.1016/S0005-7894(76)80257-2

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805). arXiv. http://arxiv.org/abs/1810.04805

Jurafsky, D., & Martin, J. H. (2023). Speech and Language Processing. Stanford.

Lee, K., Jo, J., Kim, J., & Kang, Y. (2019). Can Chatbots Help Reduce the Workload of Administrative Officers? – Implementing and Deploying FAQ Chatbot Service in a University (C. Stephanidis, Trans.). 348–354.

Molnar, G., & Szuts, Z. (2018). The Role of Chatbots in Formal Education. 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics (SISY), 000197–000202. https://doi.org/10.1109/SISY.2018.8524609

OpenAI. (2023). GPT-4 Technical Report (arXiv:2303.08774). arXiv. http://arxiv.org/abs/2303.08774

Ranoliya, B. R., Raghuwanshi, N., & Singh, S. (2017). Chatbot for university related FAQs. 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 1525–1530. https://doi.org/10.1109/ICACCI.2017.8126057

Shum, H., He, X., & Li, D. (2018). From Eliza to XiaoIce: Challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19(1), 10–26. https://doi.org/10.1631/FITEE.1700826

Skrebeca, J., Kalniete, P., Goldbergs, J., Pitkevica, L., Tihomirova, D., & Romanovs, A. (2021). Modern Development Trends of Chatbots Using Artificial Intelligence (AI). 2021 62nd International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS), 1–6. https://doi.org/10.1109/ITMS52826.2021.9615258

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks.

Thorat, S. A., & Jadhav, V. (2020). A Review on Implementation Issues of Rule-based Chatbot Systems. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3567047

Van Pinxteren, M. M. E., Pluymaekers, M., & Lemmink, J. G. A. M. (2020). Human-like communication in conversational agents: A literature review and research agenda. Journal of Service Management, 31(2), 203–225. https://doi.org/10.1108/JOSM-06-2019-0175

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.

Wallace, R. S. (2009). The anatomy of ALICE. Springer.

Wallace, R. S. (2014, March 9). AIML 2.0 Working Draft. Gist. https://gist.github.com/onlurking/f6431e672cfa202c09a7c7cf92ac8a8b

Weizenbaum, J. (1966). ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45.

Zemčík, Mgr. T. (2019). A Brief History of Chatbots. DEStech Transactions on Computer Science and Engineering, aicae. https://doi.org/10.12783/dtcse/aicae2019/31439

Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020). The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. Computational Linguistics, 46(1), 53–93. https://doi.org/10.1162/coli_a_00368