You have to give at least the offered price.

Points in bartering (and attitude) affect the price the merchant quotes you but you can't offer less.

It confused me also.