Leave somebody holding the baby
Definition 1 of 1
Pro Tip 1/3Unexpected Responsibility
The phrase often implies a sudden or unexpected transfer of responsibility. 
I was enjoying my vacation when I received an email that left me holding the baby for the project.
Pro Tip 2/3Negative Connotation
It generally has a negative connotation, suggesting abandonment or unfairness. 
The manager took another job and left me holding the baby with the angry clients.
Pro Tip 3/3Informal Usage
The phrase is informal and best suited for casual conversations, not formal writing. 
I can't believe you're going on leave now, you're going to leave me holding the baby!