Charles is a bit proudish. When someone doubts his word, it can sting on his pride, as if implying he doesn't inspire trust. I think he learned to understand it takes more than your word after being swindled a few too many times throughout the series, so he gets better n that regard.
Oddly, Mrs. Oleson looks older in this first episode than she will in the next seasons.