This is a very interesting topic to me. Something I've spent a lot of years thinking about, as well as researching. The "final word" I've seen is not intuitively obvious, but very compelling. If you'd like the details, you can look up the Dunning-Kruger Effect, but the elevator pitch goes like this: 1. people compute their assessment of "competence", both in themselves and in others, with the same part of the brain the computes the actual skill. Therefore: 2. incompetent people are completely incapable of understanding their own lack of competence. 3. Competent people tend to believe, through "false consensus", that everyone else is just as competent as they are. The old saying "the more I know, the more I know I don't know" is actually a truism, along with "the less someone knows, the less they know how little they know."
I don't think Dunning-Kruger is the final word on this, because while it explains the tendency for unknowledgeable people to be less likely to acquire more knowledge, it does not explain how people transition from unknowledgeable to knowledgeable in the first place.
When I learned, or rather taught myself to be a public speaker, one of the things I both learned and discovered is that rehearsal is not just about practice. Some people think that the benefit of rehearsal is identical to the benefit of practicing juggling. The more you do it, the better you get. That's true, but that's not even the most important part of rehearsing speeches. The real benefit comes from
listening to yourself talk. We use a completely different set of cognitive processes when we compose speech and when we deliver speech and when we listen to speech. When we're talking, we are focused on delivery. When we are listening, we are focused on comprehension. Stuff that sounds great in our heads when we are thinking it or saying it can sound horrible when we are listening to it because we are intrinsically more critical of the things we hear as opposed to the things we think.
When you rehearse a speech, you first are just trying to get the words out. Its often much more difficult than it seems, for the same reason we think we know the words to the song until we try to sing along with the radio. Thinking and speaking are different mental skills. But once you get comfortable delivering and can do it without it taking up 100% of your concentration, you can start to listen to yourself speak. And that's when the real benefit kicks in: you start to hear everything you are saying that is inarticulate, confusing, nonsensical, illogical, nonsequitor, or just plain goofy. You can start to criticize your own thoughts as they are spoken back to yourself. And that can make you a much better speaker. With enough practice, you can start to do this inside your own head, but no one starts off being able to do that, and no one does it as well as they criticize what they hear. There's a reason why I tend to read these words as I type them, and its the same reason. This sentence is being typed into this paragraph on my
third reading pass through this post.
I think cognition in general is like this. There's thinking, and then there's thinking about thinking. What's interesting about Dunning-Kruger is that we're far more vulnerable to its effects on what we know, but less vulnerable to its effects on we're told others know. If we don't know a lot about X, we can make mistakes about how far we can push our intuition about X. But even though we don't know a lot about X, we tend to be, on average, more skeptical about other people who make claims about X, even though we have no way to judge their competency any better than our own. And that's the secret to beating Dunning-Kruger in my opinion. A part of you has to be skeptical about you. You have to be able to take a step back, consciously or not, and evaluate your own thoughts as if a random stranger were trying to convince you of them. I think people we classify as "fast learners" tend to be this kind of person: someone that is able to switch back and forth between being confident enough in what they know to use it effectively without fear, and yet at other times being highly skeptical about what they know so that they can find the weaknesses in what they know and learn from those weaknesses. It is a skill I try to teach in my profession specifically.
I often tell people that what you are certain about is useful, what you know you don't know is useful, what you think you know but aren't sure about is often less than useless. Knowledge you can rely on is always useful. And definite uncertainty is something you can always work around. But the stuff you think you know but aren't sure about can be paralyzing. You know, so it is hard to conceive of anything else which blinds you to other possibilities. And yet you aren't sure, so you won't bet on it confidently either. "I know" tells you what to do. "I know I don't know" tells you what to learn. "I think" tells me to hit you until you change your mind. The world's not that simple of course, but as a general principle I think it works.