I would like certain group of users to be able to process refunds, but no other access.
This permission is controlled by the Transaction Management type. If you select to give the admin group permission to Change Transactions it will allow the user to refund, edit recurring gift, change donor, add soft credit, and fund soft credit. This will be accessible by using the Fundraising tab and selecting transactions. Once here you would need to search for the constituent by name, or confirmation code.