Breadboards can cause problems indeed with connections. In your case I'd suspect the pot itself, though.