The short answer is, nobody has a good explanation. I have also wondered about this a lot.

One factor must have to do with selecting a mate - or appreciating beauty in the opposite sex - as you said, symmetry and clear skin being indicative of health and reproductive fitness. The reason that it's present in children and across gender lines is that although it may have been selected largely because of its utility in mate selection, it was probably easier for evolution to develop a general appreciation for beauty in the species than to work out how to make it specifically for sexual selection.

Once a trait is present, evolution often finds other uses for it. It's much easier to modify existing characteristics than to wait for a mutation for develop the same thing from scratch. So once this appreciation for beauty in humans was established for mate-selection purposes, presumably further mutations altered it and selected those variations that were useful. The pleasure we experience in observing a beautiful woman who is unavailable to us feels fairly similar to the pleasure we feel in seeing a beautiful landscape or figuring out some part of Einstein's theory or watching a great movie. There's pleasure and artistic appreciation of beautiful design in many areas, and somehow that's probably useful in devising solutions to problems, inventing better ways of doing things, and so on. Those feelings of exhilaration in the face of beauty and the desire to experience beauty also give us the energy and initiative we wouldn't have if all we were concerned with was survival.

It's mind boggling to try and understand how genetics controls so many personality characteristics by wiring our brains just so. Frustrating, but the effort is fun.