It is always correct to refuse food if you really do not want to eat it. Simply say, "No thank you" and that should suffice. You are not required to explain, and if you are asked for one...simply reply that you do not feel like having one right now.
That's because if you say you're on a diet or that you want to lose weight, you might have opened topics of unwanted conversation.