Please try the following:

1. Remove items from the report one at a time and see if the refresh time gets quicker.
2. After finding the field or fields that cause the report to refresh slowly, move those fields to a subreport.