Currently, you cannot delete or deactivate a fund for a giving form if it has donations submitted for it.
When you edit a giving form, funds with donations placed against them appear as In Use on the giving form's Fund tab. You would need to delete every donation placed against the fund in order to see the "Remove" button for that fund.
Since that would delete all donation history for that fund, we instead recommend creating a new giving form and retiring the giving form the fund is currently on (found under Form Properties while editing the form). Then, you must replace the old form with the new form where it appears on your website.