If a form has been created but never embedded or made live then there will not be an option to deactivate. 
That form essentially isn't active until it is live on a web page, so you have two options:
1. Keep the form as is, but it will remain in the list of forms and not hidden with deactivated forms.
2. Embed the form/go live and then you will see an option to "Deactivate" next to the options to edit and delete the form that you can select.