It is not possible to limit the number of registrations in a Thon event, team or otherwise.  The only way to limit registrations is through the Stand-alone Registration form.

In Sphere, you will need to run a report on all teams periodically.  Once the desired number of teams is reached you must disable team registrations through Website Features.
  1. Navigate to Communities > Special Events > Friends Asking Friends > Kintera Thon.
  2. Hover over your event and select Website Info.
  3. Click Website Features.
  4. Locate the Event Teams section of the page and set the radio selection to "Users cannot form or request a team on the front end website."
  5. Click Save.
  6. Navigate back to Website Info and click Publish This Event Now.

This will prevent users from creating new teams.

If you are worried about going over, you might set the selection to "Users request to form teams" once you are nearing the limit.