I almost completely agree with you, this much talk also annoys me terribly. I think this situation applies to the majority of podcasts, but there are some legit ones.
Anyway, I mostly don't listen to podcasts, basically because I find really boring to concentrate on someone (or some people) talking while doing nothing, I'd rather read, reading is much more agreeing with me. I know, most people listen to podcasts while doing something else entirely, like repetitive chores, but then I don't get it as well, I don't need to cram information on my brain every waking hour of the day, and I much prefer listening to music if I want to listen to something while doing other stuff. By the way, music is the one thing that I listen endlessly without doing anything else at all, it is just sublime.
I'm lucky to live within WRTI's broadcast area, so I get free classical music from 6am to 6pm and free jazz from 6pm to 6am over the radio.