A Group Activity report can at least show changes made to a group on a day-by-day, week-by-week, or month-by-month basis.

1. Go to Data Management -> Reports -> Report Writer -> Create a New Report.
2. Choose List Activity from the drop-down.
3. Choose Group Activity by Time and click Next.
4. Note the columns that are already chosen and add Group, then click Next.
5. Go through the remaining steps and filter by the Group in step 5.

This will show how many were added to the group and removed from the group over time.  The Total Count column, however, will only reflect the current number of members in the group.