While there are many factors that can contribute to an email group not showing the numbers you would expect (based on pulling members from other groups with stated amounts of members), The key thing to remember is to make sure to insert the following clauses:
  1. Email Status: Good (Found under Email -> Email Status: Good)
  2. Accept Email: TRUE (Found under Email -> Accept Email: TRUE)
With both of these parameters included, this will restrict the groups to only contain active users with active/good email addresses.