It is a fact that holding another's hand is an act which effects the heart rate. That shows it distracts one more, so that his concentration is taken away and towards that person holding his hand.
It's also a fact that stroking your pet cat or dog affects the heart rate and lowers the blood pressure.
I have often prayed to God while petting my cat or dog. I didn't find it distracting at all.
I also don't find it distracting to hold my Christian sister or brother's hand while praying. If that makes you uncomfortable, then you shouldn't do it. But you should not judge others for it. That's when you step over the line from Christian liberty to judgmental Pharisee.