Refers to something uniquely irritating to an individual, not everyone. Mismatched socks might be your pet hate, but others may not care.
Pro Tip 2/3
Not Literal Hate
Doesn't mean 'hate' in a strong sense, but rather an exaggeration for dislike. People saying 'literally' incorrectly is my pet hate, although it's just a small annoyance.
Pro Tip 3/3
Conversational Context
Used in informal chats to express personal annoyances or quirks. In a casual meeting, Sara mentioned that her pet hate is when meetings start late.