Our long-term mission is to advance AI for humanity. We are currently committed to innovating the next scaling recipe(s) w/ paradigm-shift technologies for new generations of foundation models.

The Next Recipe [08/2024 - ]

• All coming soon
• R1.1 (The Revolution of Model Efficiency): BitNet (b1.58 | a4.8) + MH-MoE + YOCO
• R1.2
• R2.1
• R2.2
• R3

The Second Curve of Scaling Law

• Scaling Law: The Second Curve of Scaling Law
Model Architecture - (The Revolution of Model) Efficiency
•     The Era of 1-bit LLMs: (BitNet b1.58) All Large Language Models are in 1.58 Bits | Training Tips, Code and FAQ
•     Fully & Extremely Sparsely-Activated LLMs: (Block) Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
•     You Only Cache Once: You Only Cache Once: Decoder-Decoder Architectures for Language Models | Gated RetNet / RetNet-3
Model Architecture - Capability
•     DIFF: Differential Transformer
• The Learning Law: Towards Optimal Learning of Language Models
• More (coming)

Foundation Models

• The Evolution of (M)LLMs:
  Kosmos-1/2/2.5/G, VALL-E


• #TheBigConvergence of foundation models and large-scale self-supervised pre-training across tasks, languages, and modalities.
• Language: UniLM(-2)
• Multilingual: InfoXLM, XLM-E
• Vision: BEiT(-2)
• Speech: WavLM, SpeechLM, VALL-E
• Multimodal: VLMo, BEiT-3
• Document (Multimodal): Layout(X)LM(-2/3)
• General-purpose: MetaLM

Foundation Architecture

• The Revolution of Model Architecture:
  RetNet, BitNet, LongNet
  TorchScale (Library)


• Fundamental research on modeling generality and capability, as well as training stability and efficiency.
• DeepNet: training stability
• Magneto: modeling generality
• Multiway Transformers: multimodal modeling
• X-MoE: efficiency
• LEX Transformer: better position embedding and length extrapolation

Science of Intelligence

• Understanding the principles and theoretical boundary of (artificial general) intelligence
• Why Can GPT Learn In-Context?

• The Learning Law
• Towards Optimal Learning of Language Models

LLMOps: Research and technology for building AI products w/ foundation models.

• General technology for enabling AI capabilities w/ (M)LLMs
• MiniLLM: LLM Distillation
• AdaLLM: LLM Adaptation
• LLM Accelerator
• Promptist, Extensible Prompts, Structured Prompting


• Effective and efficient approaches to deploying large AI models in practice
• MiniLM(-2), xTune, EdgeFormer, Aggressive Decoding

In addition to the research achievements, these models are significant parts of Microsoft's own family of large AI (foundation) models powering language and multimodal tasks and scenarios across products in Microsoft. Moreover, our research tops public benchmarks and leaderboards across language, vision, speech, and multimodal tasks, and hugely contributes to the open source community through GitHub and Hugging Face.

More information about our Research and Highlights.

microsoft/unilm: Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

microsoft/torchscale: Neural Architecture for General AI

microsoft/lmops: General technology for enabling AI capabilities w/ (M)LLMs

We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models and General AI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to fuwei@microsoft.com.