The characters are making horse's ears as they dance, based on a mishearing of the of the Swedish lyrics as saying "uma uma" = "horse horse" in Japanese. Wikipedia's explanation is as good as any.

You can find caramelldansen versions of most characters from recent popular anime (and quite a few video games), if you dare to look. (I don't myself, as my ears can't take it.)

---L.