The definition
By definition, there are two types of uncertainty, namely
- Aleatoric Uncertainty (or Irreducible Uncertainty / Data Uncertainty): This type of uncertainty arises from the inherent randomness or variability in the data-generating process itself. It’s the “noise” that you can’t eliminate, even if you had a perfect model and infinite data. Think of it as the unpredictable factors that cause outcomes to vary, even under seemingly identical conditions.
Examples: The outcome of a fair dice roll, the inherent sensor noise in a measurement device, or the natural variability in weather patterns.
Characteristic: It cannot be reduced by collecting more data or improving the model.
- Epistemic Uncertainty (or Reducible Uncertainty / Model Uncertainty): This type of uncertainty arises from a lack of knowledge or information about the system, model, or parameters. It’s the uncertainty due to what we don’t know. This uncertainty can, in principle, be reduced by acquiring more data, improving the model, or gaining more knowledge about the underlying process.
Examples: Uncertainty in a model’s parameters because you only have a limited dataset to train it, uncertainty about which model structure is best to describe a phenomenon, or uncertainty due to unmeasured variables.
Characteristic: It can be reduced by collecting more data, refining the model, or improving our understanding.
Relation to medical imaging
As I mentioned in The challenge of medical image analysis, medical imaging has the problem of the “high dimensionality, low sample size” issue, we are certainly facing epistemic uncertainty. There’s always a craving for more data, better quality data, and samples with greater diversity and rare diseases. Because data is the most effective way to improve the model—more so than smart tricks (augmentation, loss functions, etc.), which certainly help but eventually yield diminishing or even negative returns as complexity stacks up. And model size and “architectural improvements” do not seem to scale well given the sample size; the added capacity seems to memorize (overfit) rather than generalize.
But I do believe that aleatoric uncertainty is creeping into the picture, more than we expect. Once the data covers a good amount of variation—to a point where a bad case actually looks like an annotation error—it can even start to confuse algorithm developers and entry-level annotators. I used to think this must be the result of an annotation quality issue (which is still reducible epistemic uncertainty), but the complex and obscure nature of medical images means there will always be irreducible variance.
This leads to a fascinating tension in how we should handle “context” when building models.
The hypothetical case
Considering this scenario: you are building a lung nodule detector, which is essentially searching for a white, isolated blob in a dark space (the lung) full of white branches (bronchi).
You want to limit the contexts
You are happy with the current version and deploy it to production. But suddenly, your PM gets pinged by unhappy doctors who complain that your system is making stupid false positives by reporting food residuals in patients’ stomachs, which are located not far below the lungs and also happen to be white blobs in a dark space. Adding a little data helps little in this case, because within the context of the detector’s bounding box, they are INDISTINGUISHABLE. Adding LOTS of samples may help, where you essentially force the model to learn to recognize the lung versus the stomach from a nodule’s point of view.
But on second thought, you are smart enough to solve the problem by masking out the region outside of the lungs and only training on the region inside. This is a classic example of reducing the problem space. We are simplifying the model’s task from “Find the lungs AND find nodules” to just “Find nodules in this specific area.” By removing the irrelevant context of the stomach, we are directly attacking the model’s ignorance of anatomy—its epistemic uncertainty. We are making the model smarter by giving it a critical piece of prior knowledge.
You want to increase the contexts
So, does this mean narrowing context is always the answer? Not at all. This is where we confront the other side of the coin: aleatoric uncertainty.
Imagine a different problem. A model is analyzing a lesion on a CT scan to determine if it’s malignant. The CT scan alone might be inherently ambiguous; a benign inflammation and a cancerous lesion could appear structurally identical. No matter how much CT data you collect or how perfect your model is, it can’t distinguish between them with certainty. This is true data ambiguity—aleatoric uncertainty.
To solve this problem, you can’t limit the context; you must increase it.
You have to look for clues outside the nodule region, and more likely, you have to assemble information from multiple sources . By adding a new modality, like a PET scan, we give the model new context—metabolic activity. The cancerous lesion will likely light up on the PET scan while the inflammation won’t. This new, information-rich context resolves the ambiguity that was inherent in the CT data alone, directly reducing the system’s aleatoric uncertainty and leading to a more robust prediction.
The Takeaway
The art of building robust models lies in this balance. We must be ruthless in excluding irrelevant context to simplify the model’s task (reducing epistemic uncertainty) while actively seeking inclusive, relevant context to clarify the data itself (reducing aleatoric uncertainty). Choosing the right context to model is, therefore, not just important—it is the central challenge in building effective AI systems. And how to choose the right context? We shall discuss this more in leveraging prior knowledge.