What is an information gain?
In decision forests, the difference between a node's entropy and the weighted (by number of examples) sum of the entropy of its children nodes.
information gain explained in plain English
In decision forests, the difference between a node's entropy and the weighted (by number of examples) sum of the entropy of its children nodes. A node's entropy is the entropy of the examples in that node. For example, consider the following entropy values: - entropy of parent node = 0.6 - entropy of one child node with 16 relevant examples = 0.2 - entropy of another child node with 24 relevant examples = 0.1 So 40% of the examples are in one child node and 60% are in the other child node. Therefore: - weighted entropy sum of child nodes = (0.4 * 0.2) + (0.6 * 0.1) = 0.14 So, the information gain is: - information gain = entropy of parent node - weighted entropy sum of child nodes - information gain = 0.6 - 0.14 = 0.46 Most splitters seek to create conditions that maximize information gain.
Example
Practitioners refer to information gain when building, training, or evaluating machine learning systems. It appears in research papers, product documentation, and technical discussions about AI capabilities and limitations.
People also read
- A/B testing
A statistical way of comparing two (or more) techniques—the A and the B.
- ablation
A technique for evaluating the importance of a feature or component by temporarily removing it from a model.
- accuracy
The number of correct classification predictions divided by the total number of predictions.
- activation function
A function that enables neural networks to learn nonlinear (complex) relationships between features and the label.
- active learning
A training approach in which the algorithm chooses some of the data it learns from.
- adaptation
Synonym for tuning or fine-tuning.
- agglomerative clustering
See hierarchical clustering.
- anomaly detection
The process of identifying outliers.
- area under the PR curve
See PR AUC (Area under the PR Curve).
- area under the ROC curve
See AUC (Area under the ROC curve).