I only want to get notified when a friend invites me to an event.
You can go to Talk Calendar settings and enable "Friend's Invites", which blocks Talk Calendar notifications if you are invited to events by people who are not your friends.

- Talk Calendar Main > "Settings" menu on the upper right > Notifications > Enable "Friend's Invites"