Towards CRM-20155 remove duplicate calls to dedupe functions.

This is a partial commit which covers the best tested section of these & establishes the new functions.

I didn't want to make the commit too daunting. The bin file probably has not been used in years
hence all the require_once calls :-)