Thank you for sharing your story and the challenges you're currently facing at work.
I understand that this particular instance was a frustrating experience for you as the person refused to respect your boundaries. This actually happens quite a lot at work and I have myself faced similar situation many times.
My only advice to you is to see if it's a one-off instance and take it as such. As long as your manager trusts and respects you and understands why you're making such decisions, it all works out well. Don't let one person's mindset get to you or affect you too much. Ignore them and move on.
If the person appears open to having a chat, discuss your challenges with them. Explain why certain timings don't work out for you and see if they understand.
Also, choose which battles you want to fight. Not all battles are worth it.