The phrase isn't about color or studying, but implies being lost in thoughts. During the meeting, I glanced over to see Jenna in a brown study.
Pro Tip 2/3
Archaic Usage
It's old-fashioned; use it to sound poetic or in historical contexts. The character from the Victorian novel remained in a brown study throughout the scene.
Pro Tip 3/3
Emotional Nuance
Often implies a hint of melancholy or preoccupation when deep in thought. After hearing the news, he was left in a brown study, contemplating what was to come.