While we do offer the "auto-mailing" option when creating documents, there are certain sizing limitations involved with using this feature. It was not intended to be used for larger mailing tasks. Factors that come in to play here include the total number of Auto-mailing recipients, length of the email, and the size of any attached PDF documents...
There are multiple ways to reduce the total size of the job (so as to make it successfully fall within the 500 MB size limitation)... You could reduce the length of the email template, you could reduce the length of the attached PDF document, or you could reduce the number of auto-mail recipients.

The most effective way to avoid encountering this error would be to divide up the query that you're using for this job into 2 separate queries: (1) including accounts that DO have emails; and (2) including accounts that do NOT have emails.

For more detailed steps on how to query for accounts with and without emails, please see the following: How do I query on all accounts without an email address? - (Note: The first set of steps walks you through how to query for accounts WITH emails... If you are using this query to break apart an existing query that you have set up, just be sure to select your existing desired recipient query as the Starting Query during step #6. Similarly, when you go to query for accounts WITHOUT emails (which is provided in the second set of steps within that same article), instead of selecting Base/All Constituents in the Query 1 position, select your existing desired recipient query during step #5 when designating the query to be placed in the Query 1 position of the compound query!)

If you do this, then you can run the query containing accounts with emails through the "Mass Email" steps (so as to avoid the Auto-mailing feature altogether) per the steps included in the following resource: How to send an Advanced Mass Email (includes video demo). - (Note: The same steps apply for sending Basic Mass Email).

Once you've emailed everyone in your list that have emails associated with their accounts, then you can run the other query through the "Create Documents" steps (and you won't need to use the auto-mailing feature this time)... For more on these steps, please see the following: How to generate letters from a Communications template