The front end Directory/Directory Widget pull users in based on the Department(s), School Level(s) and Grade Level(s) they are tied to. 

If a user is associated with Departments or Levels, there is be no way to prevent them from showing in the front end Directory.