Tory activists unwilling to work for the party