【英文】AI 全景报告 2020（178页）

英文研究报告 2020年10月21日 07:52 管理员

47% of these implementations are based on PyTorch vs. 18% for TensorFlow. PyTorch offers greater flflexibility and a dynamic computational graph that makes experimentation easier. JAX is a Google framework that is more math friendly and favored for work outside of convolutional models and transformers. Empirical scaling laws of neural language models show smooth power-law relationships, which means that as model performance increases, the model size and amount of computation has to increase more rapidly.

PolyAI, a London-based conversational AI company, open-sourced their ConveRT model (a pre-trained contextual re-ranker based on transformers). Their model outperforms Google’s BERT model in conversational applications, especially in low data regimes, suggesting BERT is far from a silver bullet for all NLP tasks.GPT-3, T5, BART are driving a drastic improvement in the performance of transformer models for text-to-text tasks like translation, summarization, text generation, text to code.