Use 'on purpose' to stress that an action was intended, in contrast to being accidental or coincidental. I didn't trip you on purpose; it was an accident.
Pro Tip 2/3
Negative Connotation
'On purpose' may imply a negative intention, such as causing harm or inconvenience to someone. He left the door open on purpose to let the cold in.
Pro Tip 3/3
Emphasizing Intent
Use 'on purpose' for emphasis when you want to make it clear that an action was done with a specific intent. She chose those flowers on purpose for their scent.