If grandparents are pulling into a smart group list that is set up for parents, this is because it's a type of parent relationship and they have "List as Parent" checked off in the relationship.

To remove this setting:
  1. Navigate to Core > User/Access > Profile
  2. Search for the user name and select the appropriate user profile
  3. Select Relationship Settings
  4. Select Edit 
  5. Unmark the checkbox "List as Parent"
  6. Save
User-added image
This will remove the grandparents from the Smart Group list and only keep those who are checked as "List as Parent".