Teaching Mandarin with a Robot That Thinks

When I joined the Instructional Design and Technology program at SUNY Polytechnic Institute, I was drawn to the edges of the field, the places where learning theory bumps into engineering. Enrolling in the AIXLab, and securing a New York State grant to fund the work, gave me the runway to pursue something genuinely novel: an embodied AI system for second-language instruction.

The project centers on a deceptively simple question: can a physical robot tutor Mandarin Chinese more effectively than a screen-based agent? The hypothesis draws on Human-Robot Interaction research, which consistently shows that embodiment, a body that gestures, turns, and occupies shared space, activates different attentional and social mechanisms in learners than text or voice alone. For tonal language acquisition like Mandarin, where prosody and mouth shape carry meaning, that physical presence may matter more than for other languages.

"Embodiment activates social mechanisms in learners that a screen simply cannot replicate, and for tonal languages, that difference may be decisive."

The platform I chose is the Reachy Mini, a compact desktop humanoid from Pollen Robotics with a four-microphone array, a 5W speaker, and a full Python SDK. It is small enough to sit on a student's desk, expressive enough to carry a conversational lesson, and open enough to integrate a full AI stack. Within it runs an agentic workflow: a speech recognition module pipes audio to a large language model that dynamically generates pedagogically sequenced Mandarin prompts, evaluates pronunciation attempts, adjusts difficulty in real time, and drives the robot's gestures and gaze without a rigid script. The loop closes on every utterance.

The technical constraints shaped every design decision. Onboard compute is limited, so latency management between the LLM API calls, audio pipeline, and motor commands required careful orchestration. Tone recognition for Mandarin, four lexical tones plus a neutral, demands audio preprocessing that consumer microphones handle poorly. The four-mic array helps, but the signal chain still required custom filtering. Localising the robot's head gaze to a learner's face in real time added a computer-vision dependency that had to be balanced against the processing budget.

Because the project is framed as a Research through Design investigation, a methodology that treats the prototype itself as a knowledge claim, human subjects testing was deliberately excluded from scope. Instead, I designed four detailed learner personas covering a range of Mandarin proficiency levels and learning styles, then ran simulation-based evaluations in the Reachy Mini's MuJoCo environment to stress-test timing, dialogue branching, and error-recovery behavior. A Mandarin-speaking teacher consultant reviewed the instructional logic, vocabulary sequencing, and feedback quality, grounding the pedagogical claims without requiring IRB review.

What this capstone ultimately demonstrates is not just a working prototype, but a design argument: that agentic AI, thoughtfully orchestrated inside an embodied platform, can instantiate second-language acquisition principles such as comprehensible input, corrective feedback, and interactional scaffolding in a form factor that travels to the learner's desk. The next step, now clearly scoped, is a formal efficacy study. The machine is ready to teach.