The temporoparietal junction (TPJ) is a key node in the brain's ventral attention network (VAN) that is involved in spatial awareness and detection of salient sensory stimuli, including pain. The anatomical basis of this network's right-lateralized organization is poorly understood. Here we used diffusion-weighted MRI and probabilistic tractography to compare the strength of white matter connections emanating from the right versus left TPJ to target regions in both hemispheres. Symmetry of structural connectivity was evaluated for connections between TPJ and target regions that are key cortical nodes in the right VAN (insula and inferior frontal gyrus) as well as target regions that are involved in salience and/or pain (putamen, cingulate cortex, thalamus). We found a rightward asymmetry in connectivity strength between the TPJ and insula in healthy human subjects who were scanned with two different sets of diffusion-weighted MRI acquisition parameters. This rightward asymmetry in TPJ-insula connectivity was stronger in females than in males. There was also a leftward asymmetry in connectivity strength between the TPJ and inferior frontal gyrus, consistent with previously described lateralization of language pathways. The rightward lateralization of the pathway between the TPJ and insula supports previous findings on the roles of these regions in stimulus-driven attention, sensory awareness, interoception and pain. The findings also have implications for our understanding of acute and chronic pains and stroke-induced spatial hemineglect.