Turning text prompts into 3D character motion

The project

Animating a game character takes a lot of manual work. Someone designs the character, rigs it, then animates each movement by hand. At AI Singapore I worked on an MVP for a gaming company that skips the last step: you type “a man walking” and get back a walking animation, joints and mesh, ready to import into Unity.

What I built

Our motion capture data came from several sources, and each one stored joint positions, kinematics and velocities its own way. I built a modular pipeline that converts all of it into one training format, then trained diffusion models on it in PyTorch.

Some clips had missing or thin text descriptions. We ran a multimodal video LLM over those clips to write new ones, and that improved our text-to-motion metrics by 20% over baseline.

I also optimised our GitLab CI/CD pipelines. Testing and deployment now take a fifth of the time they used to.

Looking back

The biggest jumps in quality came from fixing the data: one consistent format, and descriptions for the clips that had none. If I did it again, I’d start there before touching the model.

My team had a fresh graduate, an experienced engineer and two career switchers. Explaining Linux and Git to people from other fields took more time than I expected, and it showed me where my own understanding was thin. I wrote onboarding guides, built a small interactive repo for practising Linux commands and version control, and added checks to CI/CD so fewer mistakes reached review.

I also learned when to stop. Early on I spent time polishing parts of the MVP that didn’t need it yet. On a proof of concept with a deadline, that time belongs to the riskiest part.