Group Admin are allowed to remove themselves and others
I noticed that via manage groups (wp-admin/admin.php?page=CiviCRM&q=civicrm%2Fgroup&reset=1) group admins can remove themselves and others as well as add them. This shouldn't be possible but (I think) I can't also give less permissions.
It's only available if you click here on "manage groups":
Possible workaround:
Update: Only Hide "Manage Groups" does not work. But it is possible to change the permission to "edit groups".