After upgrading to macOS Monterey, you might see notifications like the above.

If so, the simple solution is to logout and then back in (or restart) or alternatively to restart the Notification Center via:

/usr/bin/killall NotificationCenter

This seems to correct things, without any deeper delving into the issue.

Many thanks for @nstrauss and @Tyler Sparr for mentioning this on the macadmins slack, and raising an issue for Notifier too.