From what I've learned in Cognitive Science, we're actually not entirely sure what causes the memory loss of earlier years. The memory system appears to be functional before, we think it might be due to reorganizing once children learn to use language to express themselves. Most people cannot remember much before they were around four years old I think.

Emotional events are generally remembered much better, so maybe it is also easier to remember events that are highly emotional, even if you were much younger at the time?