Unity puppet storytelling system

2018–21

3D-printed puppets animated in Unity in real time, built for the NSF project To Enact, To Tell, To Write.

Left: two hands hold 3D-printed puppets, a yellow block and a green figure, in front of a monitor showing a cartoon beach. Right: the same beach scene in Unity, titled “Act out your scene: beach”, with a wooden chest and a small cartoon character on the sand.

Datasheet

Puppets: 3D-printed puppets animated in Unity in real time
Position: YOLO detector trained on 148 labeled images
Rotation: BBC micro:bit and Aimxy IMUs over Bluetooth
Bridge: Python into Unity over UDP and shared memory
Evaluation: Perception study with 23 MTurk workers
Deployment: Supported on site with elementary-school teachers

Video

Play video: Unity puppet storytelling system (YouTube)

Research: Embodied and creative learning