Liu L. Sijia

2023

1. AAAI

   Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup

   Sijia Liu, Patrick Lange, Behnam Hedayatnia, and 5 more authors

   Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2023

   Abs Bib PDF Poster

   Evaluating open-domain conversation models has been an open challenge due to the open-ended nature of conversations. In addition to static evaluations, recent work has started to explore a variety of per-turn and per-dialog interactive evaluation mechanisms and provide advice on the best setup. In this work, we adopt the interactive evaluation framework and further apply to multiple models with a focus on per-turn evaluation techniques. Apart from the widely used setting where participants select the best response among different candidates at each turn, one more novel per-turn evaluation setting is adopted, where participants can select all appropriate responses with different fallback strategies to continue the conversation when no response is selected. We evaluate these settings based on sensitivity and consistency using four GPT2-based models that differ in model sizes or fine-tuning data. To better generalize to any model groups with no prior assumptions on their rankings and control evaluation costs for all setups, we also propose a methodology to estimate the required sample size given a minimum performance gap of interest before running most experiments. Our comprehensive human evaluation results shed light on how to conduct credible human evaluations of open domain dialog systems using the interactive setup, and suggest additional future directions.

   @article{Liu_Lange_Hedayatnia_Papangelis_Jin_Wirth_Liu_Hakkani-Tur_2023,
     title = {Towards Credible Human Evaluation of Open-Domain Dialog Systems Using Interactive Setup},
     number = {11},
     journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
     volume = {37},
     url = {https://ojs.aaai.org/index.php/AAAI/article/view/26557},
     doi = {10.1609/aaai.v37i11.26557},
     author = {Liu, Sijia and Lange, Patrick and Hedayatnia, Behnam and Papangelis, Alexandros and Jin, Di and Wirth, Andrew and Liu, Yang and Hakkani-Tur, Dilek},
     year = {2023},
     month = jun,
     pages = {13264-13272},
   }

2. Proceedings

   Advancing open domain dialog: The Fifth Alexa Prize SocialBot Grand Challenge

   Michael Johnston, Cris Flagg, Anna Gottardi, and 28 more authors

   In Alexa Prize SocialBot Grand Challenge 5 Proceedings, Sep 2023

   Abs Bib PDF

   Creating conversational dialog systems that are able to converse naturally and engagingly with humans on any topic remains one of the fundamental challenges of artificial intelligence. The Alexa Prize SocialBot Grand Challenge was launched in 2016 to take on the problem of enabling conversational systems to support natural, sustained, coherent, and compelling open-domain dialog. The competition enables university teams from around the world to test their innovations at scale with Alexa customers. The 5th SocialBot Grand Challenge (SGC5) expanded the competition to include both a live judged competition on system performance and a Science and Innovation prize to acknowledge the underlying scientific achievements. SGC5 also added multimodality to the challenge and encouraged teams to augment their open-domain conversations with multimedia content and multimodal interaction. The challenge included an extensively updated version of the CoBot (Conversational Bot) Toolkit, along with numerous models and APIs, including topic and intent classifiers, offensive content classifiers, pre-trained neural response generators and rankers, and multimodal support so that teams could land running and focus on building compelling multimodal conversational experiences. Use of large language models (LLMs) was a key theme in the fifth iteration of the competition and, in addition to neural response generators fine-tuned on previous Alexa Prize conversations, we provided APIs and fine-tuning capabilities enabling teams to make use of the 20 billion parameter Alexa Teacher Model LLM. The paper describes the operation of the competition and capabilities provided to teams. We outline and summarize the advances developed both by university teams and the Alexa Prize team in pursuit of the Grand Challenge objective, including use of LLMs and instruction prompting for dialog control, synthetic data and knowledge generation, multimedia response generation, and dialog evaluation.

   @inproceedings{Johnston2023,
     author = {Johnston, Michael and Flagg, Cris and Gottardi, Anna and Sahai, Sattvik and Lu, Yao and Sagi, Samyuth and Dai, Luke and Goyal, Prasoon and Hedayatnia, Behnam and Hu, Lucy and Jin, Di and Lange, Patrick and Liu, Shaohua and Liu, Sijia and Pressel, Daniel and Shi, Hangjie and Yang, Zhejia and Zhang, Chao and Zhang, Desheng and Ball, Leslie and Bland, Kate and Hu, Shui and Ipek, Osman and Jeun, James and Rocker, Heather and Vaz, Lavina and Iyengar, Akshaya and Liu, Yang and Mandal, Arindam and Hakkani-Tür, Dilek and Ghanadan, Reza},
     title = {Advancing open domain dialog: The Fifth Alexa Prize SocialBot Grand Challenge},
     year = {2023},
     month = sep,
     url = {https://www.amazon.science/alexa-prize/proceedings/advancing-open-domain-dialog-the-fifth-alexa-prize-socialbot-grand-challenge},
     booktitle = {Alexa Prize SocialBot Grand Challenge 5 Proceedings}
   }

2022

1. SigDial

   Improving Bot Response Contradiction Detection via Utterance Rewriting

   Di Jin, Sijia Liu, Yang Liu, and 1 more author

   In SIGDIAL 2022, Jul 2022

   Abs Bib PDF

   Though chatbots based on large neural models can often produce fluent responses in open domain conversations, one salient error type is contradiction or inconsistency with the preceding conversation turns. Previous work has treated contradiction detection in bot responses as a task similar to natural language inference, e.g., detect the contradiction between a pair of bot utterances. However, utterances in conversations may contain co-references or ellipsis, and using these utterances as is may not always be sufficient for identifying contradictions. This work aims to improve the contradiction detection via rewriting all bot utterances to restore antecedents and ellipsis. We curated a new dataset for utterance rewriting and built a rewriting model on it. We empirically demonstrate that this model can produce satisfactory rewrites to make bot utterances more complete. Furthermore, using rewritten utterances improves contradiction detection performance significantly, e.g., the AUPR and joint accuracy scores (detecting contradiction along with evidence) increase by 6.5% and 4.5% (absolute increase), respectively.

   @inproceedings{jin2022improving,
     title = {Improving Bot Response Contradiction Detection via Utterance Rewriting},
     author = {Jin, Di and Liu, Sijia and Liu, Yang and Hakkani-Tur, Dilek},
     year = {2022},
     month = jul,
     booktitle = {SIGDIAL 2022}
   }