What is a Low-Rank Adaptability?
A parameter-efficient technique for fine tuning that "freezes" the model's pre-trained weights (such that they can no longer be modified) and then inserts a small set of trainable weights into the model.
Low-Rank Adaptability explained in plain English
A parameter-efficient technique for fine tuning that "freezes" the model's pre-trained weights (such that they can no longer be modified) and then inserts a small set of trainable weights into the model. This set of trainable weights (also known as "update matrixes") is considerably smaller than the base model and is therefore much faster to train. LoRA provides the following benefits: - Improves the quality of a model's predictions for the domain where the fine tuning is applied. - Fine-tunes faster than techniques that require fine-tuning all of a model's parameters. - Reduces the computational cost of inference by enabling concurrent serving of multiple specialized models sharing the same base model.
The update matrixes used in LoRA consist of rank decomposition matrixes, which are derived from the base model to help filter out noise and focus training on the most important features of the model. ---
Example
Practitioners refer to low-rank adaptability when building, training, or evaluating machine learning systems. It appears in research papers, product documentation, and technical discussions about AI capabilities and limitations.
People also read
- pre-trained model
Although this term could refer to any trained model or trained embedding vector, pre-trained model now typically refers to a trained large language model or other form of trained generative AI model.
- AUC
A number between 0.
- auxiliary loss
A loss function—used in conjunction with a neural network model's main loss function—that helps accelerate training during the early iterations when weights are randomly initialized.
- Backpropagation
The process that tells a neural network which internal settings caused an error and how to adjust them, working backwards through layers.
- bag of words
A representation of the words in a phrase or passage, irrespective of order.
- base model
A pre-trained model that can serve as the starting point for fine-tuning to address specific tasks or applications.
- Bayesian neural network
A probabilistic neural network that accounts for uncertainty in weights and outputs.
- Bayesian optimization
A probabilistic regression model technique for optimizing computationally expensive objective functions by instead optimizing a surrogate that quantifies the uncertainty using a Bayesian learning technique.
- bidirectional language model
A language model that determines the probability that a given token is present at a given location in an excerpt of text based on the preceding and following text.
- broadcasting
Expanding the shape of an operand in a matrix math operation to dimensions compatible for that operation.