805 Columbus Avenue
524 Interdisciplinary Science and Engineering Complex (ISEC)
Boston, MA 02115
ATTN: Timothy Bickmore, 524 ISEC
360 Huntington Avenue
Boston, MA 02115
- Human-computer interaction
- Dialogue systems
- Intelligent virtual agents
- Personal health informatics
- Human-robot interaction
- PhD in Media Arts & Sciences, Massachusetts Institute of Technology
- MS in Computer Science, Arizona State University
- BSE in Computer Systems Engineering, Arizona State University
Timothy W. Bickmore is a Professor in the Khoury College of Computer Sciences at Northeastern University. Prior to joining Northeastern in 2005, he was an assistant professor of medicine at the Boston University School of Medicine. He completed his PhD at the Massachusetts Institute of Technology Media Laboratory.
Professor Bickmore’s interdisciplinary research is at the intersection of human-computer interaction, natural language processing (dialogue systems), animation, and health/medical/behavioral informatics. The focus of his research is on the development and evaluation of computer agents that emulate face-to-face interactions between health providers and patients for use in health education and long-term health behavior change interventions, with a particular focus on the emotional and relational aspects of these interactions. His research has received funding from NSF, NIH, AHRQ, HRSA, and PCORI.
Professor Bickmore has chaired or co-chaired several meetings, including Intelligent Virtual Agents, and AAAI symposia and CHI workshops on health informatics and virtual agents. He is an associate editor of the Interacting with Computers journal.
Virtual Coach for Atrial Fibrillation Support
Virtual Coach for Atrial Fibrillation Support
When deployed on smartphones, virtual agents have the potential to deliver life-saving advice regarding emergency medical conditions, as well as provide a convenient channel for health education to improve the safety and efficacy of pharmacotherapy.
We are developing a smartphone-based virtual agent that provides counseling to patients with Atrial Fibrillation. Atrial Fibrillation is a highly prevalent heart rhythm disorder and is known to significantly increase the risk of stroke, heart failure and death. In this project, a virtual agent is deployed in conjunction with a smartphone-based heart rhythm monitor that lets patients obtain real-time diagnostic information on the status of their atrial fibrillation and determine whether immediate action may be needed.
This project is a collaboration with University of Pittsburgh Medical Center.
Relational Agent for Palliative Care
Relational Agent for Palliative Care
The purpose of this project is to develop a conversational agent system that counsels terminally ill patients in order to alleviate their suffering and improve their quality of life.
Although many interventions have now been developed to address palliative care for specific chronic diseases, little has been done to address the overall quality of life for older adults with serious illness, spanning not only the functional aspects of symptom and medication management, but the affective aspects of suffering. In this project, we are developing a relational agent to counsel patients at home about medication adherence, stress management, advanced care planning, and spiritual support, and to provide referrals to palliative care services when needed.
Transforming Scientific Presentations with Co-Presenter Agents
Transforming Scientific Presentations with Co-Presenter Agents
The PI’s goal in this project is to revolutionize media-assisted oral presentations in general, and STEM presentations in particular, through the use of an intelligent, autonomous, life-sized, animated co-presenter agent that collaborates with a human presenter in preparing and delivering his or her talk in front of a live audience.
Although journal and conference articles are recognized as the most formal and enduring forms of scientific communication, oral presentations are central to science because they are the means by which researchers, practitioners, the media, and the public hear about the latest findings thereby becoming engaged and inspired, and where scientific reputations are made. Yet despite decades of technological advances in computing and communication media, the fundamentals of oral scientific presentations have not advanced since software such as Microsoft’s PowerPoint was introduced in the 1980’s. The PI’s goal in this project is to revolutionize media-assisted oral presentations in general, and STEM presentations in particular, through the use of an intelligent, autonomous, life-sized, animated co-presenter agent that collaborates with a human presenter in preparing and delivering his or her talk in front of a live audience. The PI’s pilot studies have demonstrated that audiences are receptive to this concept, and that the technology is especially effective for individuals who are non-native speakers of English (which may be up to 21% of the population of the United States). Project outcomes will be initially deployed and evaluated in higher education, both as a teaching tool for delivering STEM lectures and as a training tool for students in the sciences to learn how to give more effective oral presentations (which may inspire future generations to engage in careers in the sciences).
This research will be based on a theory of human-agent collaboration, in which the human presenter is monitored using real-time speech and gesture recognition, audience feedback is also monitored, and the agent, presentation media, and human presenter (cued via an intelligent wearable teleprompter) are all dynamically choreographed to maximize audience engagement, communication, and persuasion. The project will make fundamental, theoretical contributions to models of real-time human-agent collaboration and communication. It will explore how humans and agents can work together to communicate effectively with a heterogeneous audience using speech, gesture, and a variety of presentation media, amplifying the abilities of scientist-orators who would otherwise be “flying solo.” The work will advance both artificial intelligence and computational linguistics, by extending dialogue systems to encompass mixed-initiative, multi-party conversations among co-presenters and their audience. It will impact the state of the art in virtual agents, by advancing the dynamic generation of hand gestures, prosody, and proxemics for effective public speaking and turn-taking. And it will also contribute to the field of human-computer interaction, by developing new methods for human presenters to interact with autonomous co-presenter agents and their presentation media, including approaches to cueing human presenters effectively using wearable user interfaces.
Utami, Dina and Timothy W. Bickmore. “Collaborative User Responses in Multiparty Interaction with a Couples Counselor Robot.” 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (2019): 294-303.
Intimate relationships are integral parts of human societies, yet many relationships are in distress. Couples counseling has been shown to be effective in preventing and alleviating relationship distress, yet many couples do not seek professional help, due to cost, logistic, and discomfort in disclosing private problems. In this paper, we describe our efforts towards the development of a fully automated couples counselor robot, and focus specifically on the problem of identifying and processing “collaborative responses”, in which a human couple co-construct a response to a query from the robot. We present an analysis of collaborative responses obtained from a pilot study, then develop a data-driven model to detect end of collaborative responses for regulating turn taking during a counseling session. Our model uses a combination of multimodal features, and achieves an offline weighted F-score of 0.81. Finally, we present findings from a quasi-experimental study with a robot facilitating a counseling session to promote intimacy with romantic couples. Our findings suggest that the session improves couples intimacy and positive affect. An online evaluation of the end-of-collaborative-response model demonstrates an F-score of 0.72.
Wang, C, Bickmore, T, Bowen, D, Norkunas, T, Campion, M, Cabral, H, Winter, M, Paasche-Orlow, M, "Acceptability and feasibility of a virtual counselor (VICKY) to collect family health histories", Genetics in Medicine, 17:822-830 (2015)
To overcome literacy-related barriers in the collection of electronic family health histories, we developed an animated Virtual Counselor for Knowing your Family History, or VICKY. This study examined the acceptability and accuracy of using VICKY to collect family histories from underserved patients as compared with My Family Health Portrait (MFHP).
Participants were recruited from a patient registry at a safety net hospital and randomized to use either VICKY or MFHP. Accuracy was determined by comparing tool-collected histories with those obtained by a genetic counselor.
A total of 70 participants completed this study. Participants rated VICKY as easy to use (91%) and easy to follow (92%), would recommend VICKY to others (83%), and were highly satisfied (77%). VICKY identified 86% of first-degree relatives and 42% of second-degree relatives; combined accuracy was 55%. As compared with MFHP, VICKY identified a greater number of health conditions overall (49% with VICKY vs. 31% with MFHP; incidence rate ratio (IRR): 1.59; 95% confidence interval (95% CI): 1.13-2.25; P = 0.008), in particular, hypertension (47 vs. 15%; IRR: 3.18; 95% CI: 1.66-6.10; P = 0.001) and type 2 diabetes (54 vs. 22%; IRR: 2.47; 95% CI: 1.33-4.60; P = 0.004).
These results demonstrate that technological support for documenting family history risks can be highly accepted, feasible, and effective.
ACM International Joint Conference on Pervasive and Ubiquitous Computing (Ubicomp) Trinh, H, Asadi, R, Edge, D, Bickmore, T (2017)
Rehearsing in front of a live audience is invaluable when preparing for important presentations. However, not all presenters take the opportunity to engage in such rehearsal, due to time constraints, availability of listeners who can provide constructive feedback, or public speaking anxiety. We present RoboCOP, an automated anthropomorphic robot head that acts as a coach to provide spoken feedback during presentation rehearsals at both the individual slide and overall presentation level. The robot offers conversational coaching on three key aspects of presentations: speech quality, content coverage, and audience orientation. The design of the feedback strategies was informed by findings from an exploratory study with academic professionals who were experienced in mentoring students on their presentations. In a within-subjects study comparing RoboCOP to visual feedback and spoken feedback without a robot, the robotic coach was shown to lead to significant improvement in the overall experience of presenters. Results of a second within-subjects evaluation study comparing RoboCOP with existing rehearsal practices show that our system creates a natural, interactive, and motivating rehearsal environment that leads to improved presentation quality.
Utami, D., Bickmore, T., Nikolopoulou, A., and Paasche-Orlow, M. (2017) "Talk About Death: End of Life Planning with a Virtual Agent" International Conference on Intelligent Virtual Agents (IVA)
For those nearing the end of life, “wellness” must encompass reduction in suffering as well as the promotion of behaviors that mitigate stress and help people prepare for death. We discuss the design of a virtual conversational palliative care coach that works with individuals during their last year of life to help them manage symptoms, reduce stress, identify and address unmet spiritual needs, and support advance care planning. We present the results of an experiment that features the reactions of older adults in discussing these topics with a virtual agent, and note the importance of discussing spiritual needs in the context of end-of-life conversations. We find that all participants are comfortable discussing these topics with an agent, and that their discussion leads to reductions in state and death anxiety, as well as significant increase in intent to create a last will and testament.
Perceived Organizational Affiliation and Its Effects on Patient Trust: Role Modeling with Embodied Conversational Agents
Zhang, Z., Bickmore, T, and Paasche-Orlow, M (2017) "Perceived Organizational Affiliation and Its Effects on Patient Trust: Role Modeling with Embodied Conversational Agents" Patient Education and Counseling (journal)
Utami, D, Bickmore, T, Kruger, L (2017) "A Robotic Couples Counselor for Promoting Positive Communication" IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN
Intimate relationships are crucially important in all human societies, yet many relationships are in some degree of distress. Couple psychotherapy has been demonstrated to be effective at reducing relationship distress, yet most couples do not seek help from professionals. Automated couples counselors could provide help to many couples who avoid professional help due to cost, logistics, or discomfort disclosing personal problems. We explore reactions to and acceptance of a humanoid robot that takes the role of a couples counselor in promoting positive communication skills among asymptomatic intimate couples. Couples were comfortable with the robot in this role; displaying intimate behavior during the counseling session. They followed the directions of the robot in practicing interpersonal communication skills, were largely satisfied with the experience, and described several advantages to working with a robot compared to human counselors or self-help materials
Improving Access to Online Health Information With Conversational Agents: A Randomized Controlled Experiment
Bickmore, T, Utami, D, Matsuyama, R, and Paasche-Orlow, M, "Improving Access to Online Health Information with Conversational Agents: A Randomized Controlled Experiment", Journal of Medical Internet Research (2016)
Conventional Web-based search engines may be unusable by individuals with low health literacy for finding health-related information, thus precluding their use by this population.
We describe a conversational search engine interface designed to allow individuals with low health and computer literacy identify and learn about clinical trials on the Internet.
A randomized trial involving 89 participants compared the conversational search engine interface (n=43) to the existing conventional keyword- and facet-based search engine interface (n=46) for the National Cancer Institute Clinical Trials database. Each participant performed 2 tasks: finding a clinical trial for themselves and finding a trial that met prespecified criteria.
Results indicated that all participants were more satisfied with the conversational interface based on 7-point self-reported satisfaction ratings (task 1: mean 4.9, SD 1.8 vs mean 3.2, SD 1.8, P<.001; task 2: mean 4.8, SD 1.9 vs mean 3.2, SD 1.7, P<.001) compared to the conventional Web form-based interface. All participants also rated the trials they found as better meeting their search criteria, based on 7-point self-reported scales (task 1: mean 3.7, SD 1.6 vs mean 2.7, SD 1.8, P=.01; task 2: mean 4.8, SD 1.7 vs mean 3.4, SD 1.9, P<.01). Participants with low health literacy failed to find any trials that satisfied the prespecified criteria for task 2 using the conventional search engine interface, whereas 36% (5/14) were successful at this task using the conversational interface (P=.05).
Conversational agents can be used to improve accessibility to Web-based searches in general and clinical trials in particular, and can help decrease recruitment bias against disadvantaged populations.
5. Kimani, E., Bickmore, T., Trinh, H., Ring, L., Paasche-Orlow, M. K., and Magnani, J. W. 2016. A smartphone-based virtual agent for atrial fibrillation education and counseling. 16th International Conference on Intelligent Virtual Agents, 120-127. Springer
When deployed on smartphones, virtual agents have the potential to deliver life-saving advice regarding emergency medical conditions, as well as provide a convenient channel for health education to help improve the safety and efficacy of pharmacotherapy. This paper describes the use of a smartphone-based virtual agent that provides counseling to patients with Atrial Fibrillation, along with the results from a pilot acceptance study among patients with the condition. Atrial Fibrillation is a highly prevalent heart rhythm disorder and is known to significantly increase the risk of stroke, heart failure and death. In this study, a virtual agent is deployed in conjunction with a smartphone-based heart rhythm monitor that lets patients obtain real-time diagnostic information on the status of their atrial fibrillation and determine whether immediate action may be needed. The results of the study indicate that participants are satisfied with receiving information about Atrial Fibrillation via the virtual agent.
Trinh, H., Ring, L. and Bickmore, T., 2015. DynamicDuo: Co-presenting with Virtual Agents. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), (pp. 1739-1748). ACM.
The quality of most professional oral presentations is often poor, owing to a number of factors, including public speaking anxiety. We present DynamicDuo, a system that uses an automated, life-sized, animated agent to help inexperienced speakers deliver their presentations in front of an audience. The design of the system was informed by an analysis of TED talks given by two human presenters to identify the most common dual-presentation formats and transition behaviors used. In a within-subjects study (N=12) comparing co-presenting with DynamicDuo against solo-presenting with conventional presentation software, we demonstrated that our system led to significant improvements in public speaking anxiety and speaking confidence for non-native English speakers. Judges who viewed videotapes of these presentations rated those with DynamicDuo significantly higher on speech quality and overall presentation quality for all presenters.
Ring, L., Shi, L., Totzke, K., Bickmore, T. (2014) "Social Support Agents for Older Adults: Longitudinal Affective Computing in the Home", Journal on Multimodal User Interfaces
Loneliness and social isolation are significant problems in older adult populations. We describe the design of a multimodal conversational agent-based system designed to provide longitudinal social support to isolated older adults. Results from a requirements analysis study and a remote “Wizard-of-Oz” study are presented that inform the design of the autonomous social support agent. An exploratory pilot study was conducted in which the agent was placed in the homes of 14 older adults for a week. Results indicate high levels of acceptance and satisfaction of the system. Results also indicate that when the agent proactively draws elders into interactions, triggered by a motion sensor, it is more effective at addressing loneliness than when the agent passively relies upon elders to initiate interactions. We discuss future research opportunities for affective computing to address this important societal problem.