When a Query Group is rebuilt, it will populate constituents that specifically meet the Query logic criteria. Constituents should never manually be added to a Query group manually or via import because they will be removed next time it is rebuilt. If you must use constituents from your own list, then they should be put in to a non-query group and both groups can be used in whatever it is you wish to accomplish, such as sending an email.