
Transfer Learning
This page contains resources about Transfer Learning and Inductive Transfer.
Subfields and Concepts[edit]
- Direct Pretraining
- Learning to Learn / Meta Learning
- Multi-task Learning
- Collaborative Filtering
- Lifelong Learning / Continual Learning
- Variational Continual Learning (VCL)
- Reinforced Continual Learning (RCL)
- Dynamically Expandable Network (DEN)
- Progressive Neural Network (PGN)
- Automatic Machine Learning (AutoML)
- Few-shot Learning
- One-shot Learning
- Zero-shot Learning
- Catastrophic forgetting
- Neural Architecture Search (NAS) for ANNs
Online Courses[edit]
Video Lectures[edit]
Lecture Notes[edit]
Books and Book Chapters[edit]
- Chen, Z., & Liu, B. (2018). Lifelong machine learning. 2nd Ed. Morgan & Claypool.
- Sarkar, D., Bali, R., & Ghosh, T. (2018). Hands-On Transfer Learning with Python. Packt Publishing.
- Thrun, S., & Pratt, L. (Eds.). (1998). Learning to learn. Springer Science & Business Media.
Scholarly Articles[edit]
- Ravanelli, M., Zhong, J., Pascual, S., Swietojanski, P., Monteiro, J., Trmal, J., & Bengio, Y. (2020). Multi-Task Self-Supervised Learning for Robust Speech Recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6989-6993).
- T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, S. Levine, and C. Finn, (2019). Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. In 3rd Conference on Robot Learning (CoRL 2019).
- Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks.
- Wang, Y., & Yao, Q. (2019). Few-shot Learning: A Survey. arXiv preprint arXiv:1904.05046.
- Pham, H., Guan, M., Zoph, B., Le, Q., & Dean, J. (2018). Efficient Neural Architecture Search via Parameter Sharing. In Proceeding of the 35th International Conference on Machine Learning (pp. 4092-4101).
- Xu, J., & Zhu, Z. (2018). Reinforced continual learning. In Advances in Neural Information Processing Systems (pp. 899-908).
- Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (pp. 4077-4087).
- Munkhdalai, T., & Yu, H. (2017). Meta networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 2554-2563).
- Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1126-1135).
- Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098.
- Zoph, B., & Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.
- Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016). A Survey of Transfer Learning. Journal of Big data, 3(1), 9.
- Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016). Meta-learning with memory-augmented neural networks. In Proceedings of the 33rd International Conference on Machine Learning (pp. 1842-1850).
- Vinyals, O., Blundell, C., Lillicrap, T., & Wierstra, D. (2016). Matching networks for one shot learning. In Advances in Neural Information Processing Systems (pp. 3630-3638).
- Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How transferable are features in deep neural networks?. In Advances in Neural Information Processing Systems (pp. 3320-3328).
- van Rijn, J. N., Holmes, G., Pfahringer, B., & Vanschoren, J. (2014). Algorithm selection on data streams. In International Conference on Discovery Science (pp. 325-336). Springer.
- Vanschoren, J., Blockeel, H., Pfahringer, B., & Holmes, G. (2012). Experiment databases. Machine Learning, 87(2), 127-158.
- Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345-1359.
- Parameswaran, S., & Weinberger, K. Q. (2010). Large margin multi-task metric learning. In Advances in Neural Information Processing Systems (pp. 1867-1875).
- Torrey, L., & Shavlik, J. (2010). Transfer learning. In Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques (pp. 242-264). IGI Global.
- Taylor, M. E., & Stone, P. (2009). Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10, 1633-1685.
- Dai, W., Jin, O., Xue, G. R., Yang, Q., & Yu, Y. (2009). Eigentransfer: A unified framework for transfer learning. In Proceedings of the 26th International Conference on Machine Learning (pp. 193-200).
- Weinberger, K., Dasgupta, A., Langford, J., Smola, A., & Attenberg, J. (2009). Feature hashing for large scale multitask learning. In Proceedings of the 26th International Conference on Machine Learning (pp. 1113-1120).
- Caruana, R. (1997). Multitask learning. Machine Learning, 28(1), 41-75.
Software[edit]
- Multi-Task Learning via Structural Regularization (MALSAR) - MATLAB
- autokeras - Python
- auto-sklearn - Python
- nni - Python
- BayesianOptimization - Python
- hyperopt - Python
- ATM - Python
- Auto-PyTorch - Python
See also [edit]
Other Resources [edit]
- A Gentle Introduction to Transfer Learning for Deep Learning
- Hands-On Transfer Learning with Python (GitHub) - code of the book
- Continual AI - Tools and Tutorials
- Continual AI - blog
- Continual learning Workshop at NIPS 2018
- Workshop on Meta-Learning at NIPS 2018
- The 3rd AutoML Challenge: AutoML for Lifelong Machine Learning - NIPS 2018 Challenge
- Transfer Learning With Convolutional Neural Networks In Pytorch - blog post
- Multitask learning in TensorFlow with the Head API - blog post
- Transfer Learning (Part 1, Part 2) - blog post
- Transfer Learning - blog post
- variational-continual-learning (GitHub) - code
- metaworld (GitHub) - code
- RLBench (GitHub) - code
- neural-architecture-search (GitHub) - code
- bayesopt (GitHub) - code
- A Beginner's Guide to Automated Machine Learning & AI - blog post
- Awesome-NAS (GitHub) - blog post
- awesome-automl-papers (GitHub) - blog post
- awesome-architecture-search (GitHub) - blog post
- The State of Transfer Learning in NLP - blog post
- Why Continual Learning is the key towards Machine Intelligence - blog post
- An Overview of Multi-Task Learning in Speech Recognition