Advancing AI for humanity | Foundation of AI

Publication

2023

Kosmos-2.5: A Multimodal Literate Model

We present Kosmos-2.5, a multimodal literate model for machine reading of text-intensive images. Pre-trained on large-scale text-intensive images, Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1) generating spatially-aware text blocks, where each block of text is assigned its spatial coordinates within the image, and (2) producing structured text output that captures styles and structures into the markdown format. This unified multimodal literate capability is achieved through a shared Transformer architecture, task-specific prompts, and flexible text representations. We evaluate Kosmos-2.5 on end-to-end document-level text recognition and image-to-markdown text generation. Furthermore, the model can be readily adapted for any text-intensive image understanding task with different prompts through supervised fine-tuning, making it a general-purpose tool for real-world applications involving text-rich images.
Sep 20, 2023
#MLLM Kosmos-2.5

2023

Large Language Model for Science: A Study on P vs. NP

We propose Socratic reasoning, a general framework that promotes in-depth thinking with LLMs for complex problem-solving. Socratic reasoning encourages LLMs to recursively discover, solve, and integrate problems while facilitating self-evaluation and refinement. Our pilot study on the P vs. NP problem shows that GPT-4 successfully produces a proof schema and engages in rigorous reasoning throughout 97 dialogue turns, concluding "P ≠ NP". The investigation uncovers novel insights within the extensive solution space of LLMs, shedding light on LLM for Science.
Sep 13, 2023
LLM4S

2023

Retentive Network: A Successor to Transformer for Large Language Models

We propose retentive network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. The intriguing properties make RetNet a strong successor to Transformer for large language models.
July 18, 2023
RetNet

2023

LongNet: Scaling Transformers to 1,000,000,000 Tokens

We introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows.
July 6, 2023
LongNet

2023

PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training

We introduce Positional Skip-wisE (PoSE) training for efficient adaptation of large language models~(LLMs) to extremely long context windows. PoSE decouples train length from target context window size by simulating long inputs using a fixed context window with manipulated position indices during training. Concretely, we select several short chunks from a long input sequence, and introduce distinct skipping bias terms to modify the position indices of each chunk. These bias terms, along with the length of each chunk, are altered for each training example, allowing the model to adapt to all positions within the target context window without training on full length inputs. Experiments show that, compared with fine-tuning on the full length, PoSE greatly reduces memory and time overhead with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens.
September 19, 2023
#Long-context-modeling

2023

Adapting Large Language Models via Reading Comprehension

We explore how continued pre-training on domain-specific corpora influences large language models, revealing that training on the raw corpora endows the model with domain knowledge, but drastically hurts its prompting ability for question answering. Taken inspiration from human learning via reading comprehension--practice after reading improves the ability to answer questions based on the learned knowledge--we propose a simple method for transforming raw corpora into reading comprehension texts. Each raw text is enriched with a series of tasks related to its content. Our method, highly scalable and applicable to any pre-training corpora, consistently enhances performance across various tasks in three different domains: biomedicine, finance, and law. Notably, our 7B language model achieves competitive performance with domain-specific models of much larger scales, such as BloombergGPT-50B.
September 18, 2023

2023

WavMark: Watermarking for Audio Generation

This paper introduces an innovative audio watermarking framework that encodes up to 32 bits of watermark within a mere 1-second audio snippet. The watermark is imperceptible to human senses and exhibits strong resilience against various attacks. It can serve as an effective identifier for synthesized voices and holds potential for broader applications in audio copyright protection. Moreover, this framework boasts high flexibility, allowing for the combination of multiple watermark segments to achieve heightened robustness and expanded capacity.
August 24, 2023
#speech VALL-E

2023

Learning to Retrieve In-Context Examples for Large Language Models

We propose a novel framework to iteratively train dense retrievers that can identify high-quality in-context examples for LLMs. Our framework initially trains a reward model based on LLM feedback to evaluate the quality of candidate examples, followed by knowledge distillation to train a bi-encoder based dense retriever. Our experiments on a suite of 30 tasks demonstrate that our framework significantly enhances in-context learning performance. Furthermore, we show the generalization ability of our framework to unseen tasks during training.
July 14, 2023
#LLMOps

2023

In-context Autoencoder for Context Compression in a Large Language Model

We propose the In-context Autoencoder (ICAE) for context compression in a large language model (LLM). Our experimental results demonstrate that the ICAE learned with our proposed pretraining and fine-tuning paradigm can effectively produce memory slots with 4× context compression, which can be well conditioned on by the target LLM to respond to various prompts.
July 13, 2023
#LLMOps

2023

Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration

We propose Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas. A cognitive synergist refers to an intelligent agent that collaborates with multiple minds, combining their individual strengths and knowledge, to enhance problem-solving and overall performance in complex tasks. By dynamically identifying and simulating different personas based on task inputs, SPP unleashes the potential of cognitive synergy in LLMs.
July 11, 2023
Agent | Autonomous System

2023

Kosmos-2: Grounding Multimodal Large Language Models (MLLMs) to the World

We introduce KOSMOS-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. In addition to existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), KOSMOS-2 integrates the grounding capability into downstream applications.
June 26, 2023
#MLLM Kosmos-2

2023

Knowledge Distillation of Large Language Models

We propose MiniLLM that distills smaller language models from generative larger language models. We first replace the forward Kullback-Leibler divergence (KLD) objective in the standard KD approaches with reverse KLD, which is more suitable for KD on generative language models, to prevent the student model from overestimating the low-probability regions of the teacher distribution.
June 14, 2023
MiniLLM

2023

Augmenting Language Models with Long-Term Memory

NeurIPS'23 We propose a framework, Language Models Augmented with Long-Term Memory (LongMem), which enables LLMs to memorize long history. We design a novel decoupled network architecture with the original backbone LLM frozen as a memory encoder and an adaptive residual side-network as a memory retriever and reader. Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness.
June 12, 2023
#Long-term-Memory

2023

TextDiffuser: Diffusion Models as Text Painters

NeurIPS'23 Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds.
May 16, 2023
#AIGC TextDiffuser

2023

Pre-Training to Learn in Context

ACL'23 We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability by pre-training the model on a large collection of "intrinsic tasks" in the general plain-text corpus using the simple language modeling objective. PICL encourages the model to infer and perform tasks by conditioning on the contexts while maintaining task generalization of pre-trained models.
May 16, 2023

2023

Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting

We introduce a simple yet effective method, called cross-lingual-thought prompting (XLT), to systematically improve the multilingual capability of LLMs. Specifically, XLT is a generic template prompt that stimulates cross-lingual and logical reasoning skills to enhance task performance across languages.
May 11, 2023
#LLMOps #Cross-Lingual-Thought

2023

Inference with Reference: Lossless Acceleration of Large Language Models

We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios (e.g., retrieved documents). The improved computational parallelism allows LLMA to achieve over 2x speed-up for LLMs with identical generation results as greedy decoding in many practical generation scenarios.
April 10, 2023
#LLMOps LLM Accelerator

2023

On the Pareto Front of Multilingual Neural Machine Translation

NeurIPS'23 We study how the generalization performance of a given direction changes with its sampling ratio in Multilingual Neural Machine Translation (MNMT). By training over 200 multilingual models with various model sizes, directions, and total numbers of tasks, we find that scalarization leads to a multitask trade-off front that deviates from the traditional Pareto front when there exists data imbalance in the training corpus. That is, the performance of certain translation directions does not improve with the increase of its weight in the multi-task optimization objective, which poses a great challenge to improve the overall performance of all directions. Based on our observations, we propose the Double Power Law to predict the unique performance trade-off front in MNMT, which is robust across various languages, data adequacy, and the number of tasks.
April 6, 2023
#language #multilingual

2023

Language Is Not All You Need: Aligning Perception with Language Models

NeurIPS'23 A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot).
February 27, 2023
#MLLM Kosmos-1

2023

Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling

We extend VALL-E and train a multi-lingual conditional codec language model to predict the acoustic token sequences of the target language speech by using both the source language speech and the target language text as prompts. VALL-E X inherits strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translation tasks.
March 7, 2023
#speech VALL-E

2023

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

We train a neural codec language model (called VALL-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. VALL-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt.
January 6, 2023
#speech VALL-E

2022

A Length-Extrapolatable Transformer

ACL'23 In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution.
December 20, 2022
TorchScale XPos

2022

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta Optimizers

ACL'23 This paper explains language models as meta optimizers and understands ICL as a kind of implicit finetuning. Theoretically, we figure out that the Transformer attention has a dual form of gradient descent based optimization.
December 20, 2022
In-Context Learning

2022

Optimizing Prompts for Text-to-Image Generation

NeurIPS'23 We propose prompt adaptation, a general framework based on reinforcement learning that automatically adapts original user input to model-preferred prompts.
December 19, 2022
#LLMOps Promptist

2022

BEATs: Audio Pre-Training with Acoustic Tokenizers

ICML'23 We propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations.
December 18, 2022
#speech

2022

Structured Prompting: Scaling In-Context Learning to 1,000 Examples

Conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples. In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples.
December 13, 2022
#LLMOps Structured Prompting

2022

Extensible Prompts for Language Models

NeurIPS'23 We propose eXtensible Prompt (X-Prompt) for prompting a large language model (LLM) beyond natural language (NL). X-Prompt instructs an LLM with not only NL but also an extensible vocabulary of imaginary words that are introduced to help represent what NL words hardly describe, allowing a prompt to be more descriptive.
December 1, 2022
#LLMOps X-Prompt

2022

Text Embeddings by Weakly-Supervised Contrastive Pre-training

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs).
December 7, 2022
E5

2022

TorchScale: Transformers at Scale

We present TorchScale, an open-source toolkit that allows researchers and developers to scale up Transformers efficiently and effectively. TorchScale has the implementation of several modeling techniques, which can improve modeling generality and capability, as well as training stability and efficiency.
November 23, 2022
TorchScale Magneto DeepNet X-MoE

2022

Magneto: A Foundation Transformer

ICML'23 We call for the development of Foundation Transformer for true general-purpose modeling, which serves as a go-to architecture for various tasks and modalities with guaranteed training stability. In this work, we introduce a Transformer variant, named Magneto, to fulfill the goal. Specifically, we propose Sub-LayerNorm for good expressivity, and the initialization strategy theoretically derived from DeepNet for stable scaling up. Extensive experiments demonstrate its superior performance and better stability than the de facto Transformer variants designed for various applications, including language modeling (i.e., BERT, and GPT), machine translation, vision pretraining (i.e., BEiT), speech recognition, and multimodal pretraining (i.e., BEiT-3).
October 12, 2022
TorchScale Magneto DeepNet

2022

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

CVPR'23 We present BEiT-3, a general-purpose multimodal foundation model which achieves SOTA across 10+ major vision and vision-language benchmarks. BEiT-3 = Multiway Transformers + Masked Data Modeling + Scaling Up.
August 22, 2022
#multimodal #vision BEiT-3

Language Models are General-Purpose Interfaces

We propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer.
June 13, 2022
#multimodal MetaLM

BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

Use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level.
August 12, 2022
#vision BEiT(-2)

A Unified View of Masked Image Modeling

We propose a unified view of masked image modeling after revisiting existing methods. Under the unified view, we introduce a simple yet effective method, termed as MaskDistill, which reconstructs normalized semantic features from teacher models at the masked positions, conditioning on corrupted input images.
October 19, 2022
#vision MIM

SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data

We propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Leveraging only 10K text sentences, our SpeechLM gets a 16\% relative WER reduction over the best base model performance (from 6.8 to 5.7) on the public LibriSpeech ASR benchmark.
September 30, 2022
#speech SpeechLM

SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

ACL'23 We propose a simple yet effective pre-training method for dense passage retrieval.
July 6, 2022
#language SimLM

Lossless Acceleration for Seq2seq Generation with Aggressive Decoding

We study lossless acceleration for seq2seq generation with a novel decoding algorithm -- Aggressive Decoding.
May 20, 2022
#language

On the Representation Collapse of Sparse Mixture of Experts

NeurIPS'22 Our method alleviates the representation collapse issue and achieves more consistent routing than the baseline mixture-of-experts methods.
April 20, 2022
#TorchScale #MoE X-MoE

StableMoE: Stable Routing Strategy for Mixture of Experts

ACL'22 We propose StableMoE with two training stages to address the routing fluctuation problem.
April 18, 2022
#language #MoE

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

ACM MM'22 Pre-train multimodal Transformers for Document AI with unified text and image masking.
April 18, 2022
#multimodal LayoutLM

DiT: Self-supervised Pre-training for Document Image Transformer

ACM MM'22 A self-supervised pre-trained DiT ransformer model using large-scale unlabeled text images for Document AI tasks.
March 4, 2022
#vision DiT BEiT

DeepNet: Scaling Transformers to 1,000 Layers

We introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization.
March 1, 2022
DeepNet

EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

EMNLP'22 We propose EdgeFormer -- a parameter-efficient Transformer of the encoder-decoder architecture for on-device seq2seq generation, which is customized under strict computation and memory constraints.
February 16, 2022
#language

2021

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

NeurIPS'22 We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network..
November 03, 2021
#multimodal VLMo Multiway Transformers

2021

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

JSTSP'22 Pre-training for full-stack speech processing tasks. WavLM jointly learns masked speech prediction and denoising in pre-training, and it achieves state-of-the-art performance on the SUPERB benchmark.
October 26, 2021
#speech WavLM

2021

MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding

ACL'22 We propose MarkupLM for document understanding tasks with markup languages as the backbone, such as HTML/XML-based documents, where text and markup information is jointly pre-trained.
October 16, 2021
#language MarkupLM

2021

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

ACL'22 We propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
October 14, 2021
#speech SpeechT5

Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training

EMNLP'21 We propose an algorithm VoCap to determine the desired vocabulary capacity of each language.
September 15, 2021
#multilingual XLM-E

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

ACL'22 ELECTRA-style tasks to cross-lingual language model pre-training.
June 30, 2021
#multilingual XLM-E

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

ACL'21 We present a general approach to developing small, fast and effective pre-trained models for specific domains.
June 25, 2021
#language AdaLM incr_bpe

2021

BEiT: BERT Pre-Training of Image Transformers

ICLR'22 (Oral) We propose Maskded Image Modeling to pretrain vision Transformers.
June 15, 2021
#vision BEiT

2021

Consistency Regularization for Cross-Lingual Fine-Tuning

ACL'21 (Oral) We propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation.
June 15, 2021
#multilingual xTune

LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding

ACL'22 We present a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding.
April 18, 2021
#multimodal #multilingual Layout(X)LM

mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

EMNLP'21 Improve multilingual text-to-text transfer Transformer with translation pairs.
April 18, 2021
#multilingual DeltaLM

Knowledge Neurons in Pretrained Transformers

ACL'22 A preliminary study on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons.
March 10, 2021
#language

UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data

ICML'21 we propose a unified pre-training approach to learning speech representations with both unlabeled and labeled data.
January 19, 2021
#speech

2020

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

ACL'21 Employ multi-head self-attention relations (scaled dot-product between the pairs of query, key, and value vectors within each self-attention module) to train the student model.
December 31, 2020
#language MiniLM

2020

LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding

ACL'21 We propose a new pre-training tasks to model the interaction among text, layout, and image in a single multi-modal framework.
December 29, 2020
#multimodal LayoutLM

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

NAACL'21 An information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts.
July 15, 2020
#multilingual InfoXLM

2020

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

AAAI'21 (Best Paper Runner Up) A self-attention attribution method to interpret the information interactions inside Transformer..
April 23, 2020
#language

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

ICLM'20 Pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model.
February 28, 2020
#language UniLM

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

NeurIPS'20 A simple and effective approach to compress large Transformer based pre-trained models, termed as deep self-attention distillation.
February 25, 2020
#language MiniLM

2019

LayoutLM: Pre-training of Text and Layout for Document Image Understanding

KDD'20 Jointly model interactions between text and layout information across scanned document images.
December 31, 2019
#multimodal LayoutLM

Unified Language Model Pre-training for Natural Language Understanding and Generation

NeurIPS'19 A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks.
May 8, 2019
#language UniLM