
The cards/odds are stacked against you
Metaphorical Use
Used metaphorically to describe an unfavorable situation, not only involving literal cards or games. 
Idiomatic Nuance
Understand it's an idiom implying unfairness or difficulty, not just numerical disadvantage. 
Non-Literal Odds
The 'odds' may involve non-quantifiable factors like bias, not just calculable probabilities. 