The phrase is used when someone is upset or embarrassed and becomes very quiet and less communicative. He's been quiet all day; I think the criticism made him retreat into his shell.
Pro Tip 2/3
Temporary State
It implies a temporary state, suggesting the person may come out of their shell eventually. She's gone into her shell right now, but she'll be her usual self with some time.
Pro Tip 3/3
Not Literal
This phrase is figurative and does not refer to physically hiding within a shell like a turtle might do. Don't worry, he hasn't literally disappeared, he's just gone into his shell for a bit.