I understand the problem you're suffering from. I think to be an attentive listener is the best way to solve it. Most occasionally, you are distracted by something else. For example, when you come to something you don't understand, you may stop for half a second. In fact, by doing this, you lost the track of the whole passage or conversation. So, keep following the track even you don't understand some points. In a word, be attentive when listening!