Nanjie (Jimmy) Rao

Research · Embodied

Embodied and creative learning

Hand enactment, puppets, avatars and AR glasses in storytelling and everyday learning, and the systems I built or helped build along the way.

Hands as Narrators (ICIDS 2026)

A full paper on how writers use hand enactment in story writing. I built a remote web platform that logged every video scrub and keystroke alongside a WebGazer.js gaze stream, for a two-part study.

participants
24
of writing time went to returning to recorded hand enactments
about ¼
strategies the paper describes
4
behaviorally inferred reasons
5

The paper ends with design implications for authoring tools.

Puppets, avatars and emotion in storytelling

AR smart glasses for everyday learning (IEEE VR 2020)

A Latin-square, within-subjects study with 25 participants compared two ways for the glasses to show information. See the AR smart-glasses build.

Display shown on… Produced
spoken request (speech-triggered) better learning and task scores
object detection (object-anchored) higher immersion and usability

A humanoid telepresence robot

I was the on-site leader of Team Limitless, whose humanoid telepresence robot reached the semifinals of the ANA Avatar XPRIZE in 2021. See the robot build.

Publications

  1. Rao, N., Lanzino, A., Chenore, R., Adhikari, I., Brown, S. A., Quek, F., & Chu, S. L. (2026). Hands as Narrators: Characterizing Idea Reuptake from Hand Enactment in Story Writing. ICIDS 2026, Springer. Full paper, accepted.
  2. Rao, N., Chu, S. L., & Chenore, R. (2021). Flexible Low-Cost Digital Puppet System. HCI International (HCII 2021), Springer LNCS.
  3. Rao, N., Chu, S. L., Faris, R. W., & Ospina, D. (2019). The Effects of Interactive Emotional Priming on Storytelling: An Exploratory Study. ICIDS 2019, Springer, pp. 395–404.
  4. Zarei, N., Chu, S. L., Quek, F., Rao, N. J., & Brown, S. A. (2020). Investigating the Effects of Self-Avatars and Story-Relevant Avatars on Children’s Creative Storytelling. ACM CHI 2020.
  5. Rao, N., Zhang, L., Chu, S. L., Jurczyk, K., Candelora, C., Su, S., & Kozlin, C. (2020). Investigating the Necessity of Meaningful Context Anchoring in Augmented Reality Smart Glasses Interaction for Everyday Learning. IEEE VR 2020. Workshop paper and poster.

Builds