Rightly said Vincent. The issue was with my manager's reaction and not my saying no.

Learning to say no is so important. It's hard in the beginning, but it gets easier once you learn to say it.

It also helps us focus our energies in the right direction.