Yes, that will not affect anything, as groups are simply used to hold records, and do not actually affect the record if they are removed or deleted.