Our long-term mission is to advance AI for humanity. We are dedicated to pioneering a new paradigm of AI, focusing on both technologies and models. Previously, we worked on paradigm-shifting technologies for new generations of foundation models. [Last Update: January 2025]

Building Frontier Efficiency Model [03/2025 - ]

• 04/25: BitNet b1.58 2B4T
• more coming

A New Paradigm of Generative AI [01/2025 - ]

• coming soon

==

The Next Recipe [06/2024 - ]

• R1.1 (Efficiency Revolution of AI): BitNet (b1.58 | a4.8) + MH-MoE + YOCO
• R1.2 (Transformer in the Post-Transformer Era: Generally better with less hallucination & better long-sequence reasoning): DIFF Transformer
• R2.1 (A New Paradigm of Multimodality): LatentLM
• R2.2: coming soon
• R3 (New Generation of (M)LLMs and Generative AI): R1.1 + R1.2 + R2.1 + R2.2

==

The Second Curve of Scaling Law

• Scaling Law: The Second Curve of Scaling Law
Model Architecture - (The Revolution of Model) Efficiency
•     The Era of 1-bit LLMs: (BitNet b1.58) All Large Language Models are in 1.58 Bits | Training Tips, Code and FAQ
•     Fully & Extremely Sparsely-Activated LLMs: (Block) Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
•     You Only Cache Once: You Only Cache Once: Decoder-Decoder Architectures for Language Models | Gated RetNet / RetNet-3
Model Architecture - Capability
•     DIFF: Differential Transformer
• The Learning Law: Towards Optimal Learning of Language Models

==

Foundation Models

• The Evolution of (M)LLMs:
  Kosmos-1/2/2.5/G, VALL-E


• #TheBigConvergence of foundation models and large-scale self-supervised pre-training across tasks, languages, and modalities.
• Language: UniLM(-2)
• Multilingual: InfoXLM, XLM-E
• Vision: BEiT(-2)
• Speech: WavLM, SpeechLM, VALL-E
• Multimodal: VLMo, BEiT-3
• Document (Multimodal): Layout(X)LM(-2/3)
• General-purpose: MetaLM

Foundation Architecture

• The Revolution of Model Architecture:
  RetNet, BitNet, LongNet
  TorchScale (Library)


• Fundamental research on modeling generality and capability, as well as training stability and efficiency.
• DeepNet: training stability
• Magneto: modeling generality
• Multiway Transformers: multimodal modeling
• X-MoE: efficiency
• LEX Transformer: better position embedding and length extrapolation

Science of Intelligence

• Understanding the principles and theoretical boundary of (artificial general) intelligence
• Why Can GPT Learn In-Context?

• The Learning Law
• Towards Optimal Learning of Language Models

LLMOps: Research and technology for building AI products w/ foundation models.

• General technology for enabling AI capabilities w/ (M)LLMs
• MiniLLM: LLM Distillation
• AdaLLM: LLM Adaptation
• LLM Accelerator
• Promptist, Extensible Prompts, Structured Prompting


• Effective and efficient approaches to deploying large AI models in practice
• MiniLM(-2), xTune, EdgeFormer, Aggressive Decoding

In addition to the research achievements, these models are significant parts of Microsoft's own family of large AI (foundation) models powering language and multimodal tasks and scenarios across products in Microsoft. Moreover, our research tops public benchmarks and leaderboards across language, vision, speech, and multimodal tasks, and hugely contributes to the open source community through GitHub and Hugging Face.

More information about our Research.

microsoft/unilm: Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

microsoft/BitNet: 1-bit AI Infra: Official inference framework for 1-bit LLMs

We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models and General AI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to fuwei@microsoft.com.