I only want to get notified when a friend invites me to an event.
You can go to Talk Calendar settings and enable "Friend's Invites", which blocks Talk Calendar notifications if you are invited to events by people who are not your friends.

- Talk Calendar Main > Hamburger menu on the top right > "Settings" menu on the upper right > Enable "Friend's Invites"