I think people playing this game have mostly no opinion at all on this non-existent problem, and would not be quick to listen to a guy that has only one dollar in his pockets anyway.
Also, the guesser does not have a 100% chance to get the dollar 25% of the time, it depends the sample and the biases of the guesser.
What you don't seem to get is that you can tell me you have one dollar in your pocket BUT that is not a FACT. So that skews the percentage. You tell one thing and then another, so who tells me you will give me the dollar if I find it? You're just not somebody one can trust, and that's the crux of the matter.