Comment 14 by VrijVlinder :

Can't say it affects me adversely besides not forgetting everything that happens to me or that someone says.

That's great news.

The down side is that those who are part of my memories do not remember their own.

Experience is not the same thing as memory. This means that these people are part of your collective experience if not your collective recall. Also we process (those of us doing so in a standard way anyway) much more than we realise.

You could do an experiment to test this. How about plotting a fun day full of learning and enjoyment with someone close? You could enjoy the day in all its glory and then do something similar (but not the same) a few months later. Does the person that you are with react to the stimuli given as completely novel, or instead has the experience that you shared coloured their reactions? I think you will find that it has, well, barring concussion or adverse events in the interim. We do not have to pay absolute direct attention to take things in.

You are definitely not alone in the experience, and I am hoping that you might find that you are less alone than it seems with the recall.