Is it morally wrong to say ‘no’ to the volunteers on the street?

I surely wouldn’t mind or be angry at all if I were in their position. But I always feel guilty every single time I pretend to play with my phone and ignore them.