Supervising the Master's Thesis "Smoothing techniques for 3D animated human pose estimation data"
During the spring of 2024, August Bergöö and Oliver Lundgren have been working on their Master's Thesis titled "Smoothing techniques for 3D animated human pose estimation data" at Lunds Tekniska Högskola. I have been one of their supervisors at Spiideo during this time. I want to take a moment and find out what their experience was like. I also want to know if there is anything I could do better next time.
It was a pleasure to supervise August and Oliver. They're two competent people, and they have complementary strengths which made them self going to a large extent.
August and Oliver had two additional supervisors, Håkan Ardö from Spiideo and Magnus Oskarsson from Lunds Tekniska Högskola. They did a great job supervising the thesis, and their contributions were undoubtedly much more important than mine.
Summary of the thesis and results
In their thesis, August and Oliver study smoothing techniques for 3D models that contain jitter and unnatural movements. They look at filters (B-splines, Savitzky-Golay, Double Exponential Smoothing), and machine learning techniques and evaluate those against each other using quantitative and qualitative analysis.
Human representations in 3D created by motion capture or AI/ML models can contain jitter and unnatural poses:
Often these estimations of human poses contain noise, either in the form of small inaccuracies that result in jittering character models, or errors in the estimation which lead to limbs of the player models being represented poorly. The latter can cause phenomena which make the experience feel unrealistic, such as feet sliding on the floor or arms clipping through the body in unnatural ways.
The focus of this work will be to use different techniques to mitigate the effects of this noise.
To evaluate the different techniques the BEDLAM data set (A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion) was used. Noise was applied to the data, and finally different smoothing techniques were used.
Jitterbugs
These ladies were made to contort unnaturally, and to jitter. Then different smoothing techniques were applied. Visualized in the thesis using Bevy.
To visualize the results, and to do various data manipulation tasks, the Bevy game engine was used. For the AI/ML tasks, PyTorch was used.
Two measurement methods, Mean Per Joint Position Error (MPJPE) [Wang et al., 2021] and Absolute Acceleration Error (AAE) [Shao et al., 2023] were selected for the quantitative study. For MPJPE B-Spline filtering worked best, and for AAE Savitzky-Golay filtering worked best.
The LSTM (ML) model wasn't without benefits though:
The LSTM predictions generally did a better job filtering out pose errors created by our noise model, but worse in filtering out the high frequency noise like jitter. The classic filters on the other hand, were much better at reducing these errors whilst ignoring the pose errors. It would be of interest to combine the methods used in the thesis, to see if there is a way to create a combined solution that the effective at solving both types of issues.
Three questions
I asked August and Oliver three questions: Insights you had, improvements you would do if you had to do it again, and finally I asked them to let me know what I could do better in the form of advice.
Oliver graciously answered these questions on behalf of both of them.
Insights
Other than getting the title of Master of Science, what are you taking with you that you learned while doing the thesis work?
This is the first time I have worked on a project of this size, so it was much more important to have a clear structure and plan, documenting what we are doing and why. This way, both partners had a clear idea of what needed to be done and what was in progress. Learning to use BEVY’s Entity Component System was a lot of fun. Most of my programming projects have been object-oriented, so the shift in how to think and build our program was incredibly interesting.
Improvements
If you could do it again, what would you do differently?
The first few weeks were the hardest. We felt like we didn’t have a clear direction of what we wanted to do and what the subject of our thesis was going to be. We thought we had a plan, but once we started, we soon realized that the initial subject wasn’t right for us. If I could do it again, I would want to be much more certain about our research focus to avoid having to switch directions after starting. Now, after writing the actual thesis, we have a lot more knowledge of the real issues in this field. If we were to redo the same thesis, our noise estimations would be more precise and accurate, resulting in a more interesting outcome.
Advice to the supervisor
If you could give any advice to me as your supervisor, what would it be?
I felt like we were very happy with both our supervisors. Especially after a couple of months when it felt like we got to know you better outside of just thesis meetings. At the start, at least for me, I was worried that I might not live up to your expectations. It was difficult to gauge your thoughts and opinions since our meetings were, as they should be, very serious and focused. But as you occasionally came around to our part of the office and talked about things other than the thesis, we started to get to know you better. This eventually led to me understanding that my worries were mostly unfounded, and I really started to enjoy this "relationship." So, if there was one thing you could have done differently, it would be to come by a bit more frequently. At least at the beginning, so we could have gotten to know you faster. I know all the thesis workers really enjoyed those moments.
Closing thoughts
Two years ago I supervised Gustav Sjölin
and Daniel Pendse during
Covid lockdown, which was a lot of fun for me because lockdown was kind of depressing.
Some day I might write a little about that experience, and ask them some questions
too.
I had a lot of fun supervising August and Oliver. It was quite different from the time I spent online with Gustav and Daniel, and maybe because of that I neglected my social responsibilities with August and Oliver. I appreciate the feedback, and I promise that I'll do better with the next batch of students.
I'm happy all of their anguish was put to an end when their thesis presentation was over and their thesis was approved, which looked like this:
Good luck in the future guys!