The phrase is casual and might be considered disrespectful if used in formal situations. At the lunch table, they whispered about the new guy being an odd fish.
Pro Tip 2/3
Not Negative
While it points out someone's unusual traits, it doesn't necessarily mean it's bad to be an odd fish. His unique way of dressing makes him an odd fish, but in a good way.
Pro Tip 3/3
Cultural Nuance
The phrase could be cultural as some societies celebrate odd fish more than others. In our town's artist community, almost everyone is a bit of an odd fish.