Sorting on id, not on name
Just an observation, as I can imagine it is perhaps working as designed:
When I sort on a Contact's Sort Name, then sorting is okay when the field type is Raw. When the field type is Link to view contact, the sorting is done on the Contact ID Field and not on the Contact Name Field.