If you say yes, it can be interpreted as “yes I mind” or “yes, I give you permission”, if you say no, it can be interpreted as “no, I don’t give you permission” or “no, I dont mind”. You always end up having to clarify.