transfer learning

(Ruder) (ML Spec)

transfer learning uses the knowledge (artificial neural network model, data, etc) when solving one problem in one domain to apply to a different, but related problem in another domain .

Type (specifically in NLP)

Sequential transfer learning generally have the procedure: supervised pre-training (it might be done previously) then fine tuning (in later layers of artificial neural network and with small learning rate).

Example:

• GPT-3
• BERTs
• CNN using pre-trained parameters fixed or fine tuning with small learning rate, for computer vision tasks

Application

• Learning from simulations
• Adapt to new domains that require special knowledge
• Transfer knowledge across languages, modes, etc