Only Activity Group Managers can manage Activity Rosters through the group itself.

Activity Leaders can add and drop members under Extracurricular > Activity Management > Requests & Schedules > Activity Members > Edit