Mail merge will try to add all Contacts it is aware of to the list of recipients.
By default, if you're using our Contacts feature, it will select All Contacts; and if you're not using Contacts, it will select All Columns from the selected boxes.
To customize where it pulls recipients from:
Click the text after Gather recipient from:
Unselect/select the relevant options for your mail merge
The list of recipients will update in realtime in the list on the right of the screen. For finer grain control, too, you can delete specific recipients from the list by hovering over the recipient: