Feeling sexually attracted to men of a certain race rather than to men of another race has nothing to do with racism I think.
But 'saying no' can be done in a blunt and indeed racist way instead of in a neutral, polite way. Some people are simply blunt and racist in the way they express themselves about a sexual preference which in itself is not racist.
Personnally I've always been more attracted to latin, arab, south-east asian and black men instead of white men like myself, but there have always been exceptions both ways and with time shifts in sexual taste may happen. Lately, it's more about sexual attitude and mindset for me and less about race.