Advancing AGI: On the Generality, Generalizability, and Adaptability of AI

Publication

2023

Language Is Not All You Need: Aligning Perception with Language Models

A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot).
February 27, 2023
#AGI #MLLM Kosmos-1

2023

Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

We extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks.
March 7, 2023
#Speech VALL-E

2023

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

We train a neural codec language model (called VALL-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. VALL-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt.
January 6, 2023
#Speech VALL-E

2022

A Length-Extrapolatable Transformer

In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution.
December 20, 2022
#TorchScale Length-Extrapolatable Transformers

2022

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta Optimizers

This paper explains language models as meta optimizers and understands ICL as a kind of implicit finetuning. Theoretically, we figure out that the Transformer attention has a dual form of gradient descent based optimization.
December 20, 2022
#Fundamentals In-Context Learning

2022

Optimizing Prompts for Text-to-Image Generation

We propose prompt adaptation, a general framework based on reinforcement learning that automatically adapts original user input to model-preferred prompts.
December 19, 2022
#LMOps #Prompt_Intelligence Promptist

2022

Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples. In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples.
December 13, 2022
#LMOps #Prompt_Intelligence Structured Prompting

2022

Extensible Prompts for Language Models

We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible vocabulary of imaginary words that are introduced to help represent what NL words hardly describe, allowing a prompt to be more descriptive.
December 1, 2022
#LMOps #Prompt_Intelligence X-Prompt

2022

Text Embeddings by Weakly-Supervised Contrastive Pre-training

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs).
December 7, 2022
E5

2022

TorchScale: Transformers at Scale

We present TorchScale, an open-source toolkit that allows researchers and developers to scale up Transformers efficiently and effectively. TorchScale has the implementation of several modeling techniques, which can improve modeling generality and capability, as well as training stability and efficiency.
November 23, 2022
#TorchScale #Foundation_Transformers Magneto DeepNet X-MoE

2022

Foundation Transformers

We call for the development of Foundation Transformer for true general-purpose modeling, which serves as a go-to architecture for various tasks and modalities with guaranteed training stability. In this work, we introduce a Transformer variant, named Magneto, to fulfill the goal. Specifically, we propose Sub-LayerNorm for good expressivity, and the initialization strategy theoretically derived from DeepNet for stable scaling up. Extensive experiments demonstrate its superior performance and better stability than the de facto Transformer variants designed for various applications, including language modeling (i.e., BERT, and GPT), machine translation, vision pretraining (i.e., BEiT), speech recognition, and multimodal pretraining (i.e., BEiT-3).
October 12, 2022
#TorchScale #Foundation_Transformers Magneto DeepNet

2022

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

CVPR'23 We present BEiT-3, a general-purpose multimodal foundation model which achieves SOTA across 10+ major vision and vision-language benchmarks. BEiT-3 = Multiway Transformers + Masked Data Modeling + Scaling Up.
August 22, 2022
#multimodal #vision BEiT-3

Language Models are General-Purpose Interfaces

We propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer.
June 13, 2022
#multimodal #general-purpose-interface MetaLM

BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

Use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level.
August 12, 2022
#vision BEiT(-2)

A Unified View of Masked Image Modeling

We propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images.
October 19, 2022
#vision MIM

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

We propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Leveraging only 10K text sentences, our SpeechLM gets a 16\% relative WER reduction over the best base model performance (from 6.8 to 5.7) on the public LibriSpeech ASR benchmark.
September 30, 2022
#speech SpeechLM

SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

We propose a simple yet effective pre-training method for dense passage retrieval.
July 6, 2022
#language SimLM

Lossless Acceleration for Seq2seq Generation with Aggressive Decoding

We study lossless acceleration for seq2seq generation with a novel decoding algorithm -- Aggressive Decoding.
May 20, 2022
#language

On the Representation Collapse of Sparse Mixture of Experts

NeurIPS'22 Our method alleviates the representation collapse issue and achieves more consistent routing than the baseline mixture-of-experts methods.
April 20, 2022
#TorchScale #MoE X-MoE

StableMoE: Stable Routing Strategy for Mixture of Experts

ACL'22 We propose StableMoE with two training stages to address the routing fluctuation problem.
April 18, 2022
#language #MoE

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

ACM MM'22 Pre-train multimodal Transformers for Document AI with unified text and image masking.
April 18, 2022
#multimodal LayoutLM

DiT: Self-supervised Pre-training for Document Image Transformer

ACM MM'22 A self-supervised pre-trained DiT ransformer model using large-scale unlabeled text images for Document AI tasks.
March 4, 2022
#vision DiT BEiT

DeepNet: Scaling Transformers to 1,000 Layers

We introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization.
March 1, 2022
#TorchScale DeepNet

EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

EMNLP'22 We propose EdgeFormer -- a parameter-efficient Transformer of the encoder-decoder architecture for on-device seq2seq generation, which is customized under strict computation and memory constraints.
February 16, 2022
#language

2021

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

NeurIPS'22 We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network..
November 03, 2021
#multimodal VLMo Multiway Transformers

2021

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

JSTSP'22 Pre-training for full-stack speech processing tasks. WavLM jointly learns masked speech prediction and denoising in pre-training, and it achieves state-of-the-art performance on the SUPERB benchmark.
October 26, 2021
#speech WavLM

2021

MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding

ACL'22 We propose MarkupLM for document understanding tasks with markup languages as the backbone, such as HTML/XML-based documents, where text and markup information is jointly pre-trained.
October 16, 2021
#language MarkupLM

2021

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

ACL'22 We propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
October 14, 2021
#speech SpeechT5

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

EMNLP'21 We propose an algorithm VoCap to determine the desired vocabulary capacity of each language.
September 15, 2021
#multilingual XLM-E

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

ACL'22 ELECTRA-style tasks to cross-lingual language model pre-training.
June 30, 2021
#multilingual XLM-E

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

ACL'21 We present a general approach to developing small, fast and effective pre-trained models for specific domains.
June 25, 2021
#language AdaLM incr_bpe

2021

BEiT: BERT Pre-Training of Image Transformers

ICLR'22 (Oral) We propose Maskded Image Modeling to pretrain vision Transformers.
June 15, 2021
#vision BEiT

2021

Consistency Regularization for Cross-Lingual Fine-Tuning

ACL'21 (Oral) We propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation.
June 15, 2021
#multilingual xTune

LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding

ACL'22 We present a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding.
April 18, 2021
#multimodal #multilingual Layout(X)LM

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

EMNLP'21 Improve multilingual text-to-text transfer Transformer with translation pairs.
April 18, 2021
#multilingual DeltaLM

Knowledge Neurons in Pretrained Transformers

ACL'22 A preliminary study on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.
March 10, 2021
#language

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

ICML'21 we propose a unified pre-training approach to learning speech representations with both unlabeled and labeled data.
January 19, 2021
#speech

2020

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

ACL'21 Employ multi-head self-attention relations (scaled dot-product between the pairs of query, key, and value vectors within each self-attention module) to train the student model.
December 31, 2020
#language MiniLM

2020

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

ACL'21 We propose a new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework.
December 29, 2020
#multimodal LayoutLM

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

NAACL'21 An information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
July 15, 2020
#multilingual InfoXLM

2020

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

AAAI'21 (Best Paper Runner Up) A self-attention attribution method to interpret the information interactions inside Transformer..
April 23, 2020
#language

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

ICLM'20 Pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model.
February 28, 2020
#language UniLM

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

NeurIPS'20 A simple and effective approach to compress large Transformer based pre-trained models, termed as deep self-attention distillation.
February 25, 2020
#language MiniLM

2019

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

KDD'20 Jointly model interactions between text and layout information across scanned document images.
December 31, 2019
#multimodal LayoutLM

Unified Language Model Pre-training for Natural Language Understanding and Generation

NeurIPS'19 A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
May 8, 2019
#language UniLM