Our long-term mission is to advance AI for humanity. We focus on fundamental and disruptive research of the Foundation of AI. We are also committed to building the new foundation of AI, including the New Generation of Foundation Models.

Foundation Models: #TheBigConvergence of foundation models and large-scale self-supervised pre-training across tasks, languages, and modalities.

• Language: UniLM(-2)
• Multilingual: InfoXLM, XLM-E
• Vision: BEiT(-2)
• Speech: WavLM, SpeechLM, VALL-E
• Multimodal: VLMo, BEiT-3
• Document (Multimodal): Layout(X)LM(-2/3)
• General-purpose: MetaLM
• (M)LLMs: Kosmos-1/2/2.5, VALL-E

Foundation Architecture: Fundamental research to develop new architectures for foundation models and General AI, focusing on modeling generality and capability, as well as training stability and efficiency.

• DeepNet: training stability
• Magneto: modeling generality
• Multiway Transformers: multimodal modeling
• X-MoE: efficiency
• LEX Transformer: better position embedding and length extrapolation
• TorchScale (Library)
• Revolutionizing Transformers for (M)LLMs and AI: RetNet, LongNet

Science of Intelligence: Understanding the principles and theoretical boundary of (artificial general) intelligence.

• Why Can GPT Learn In-Context?

LLMOps: Research and technology for building AI products w/ foundation models.

• General technology for enabling AI capabilities w/ (M)LLMs: MiniLLM (LLM Distillation), AdaLLM (LLM Adaptation), LLM Accelerator, Structured Prompting, Extensible Prompts, and Promptist.
• Effective and efficient approaches to deploying large AI models in practice: MiniLM(-2), xTune, EdgeFormer, and Aggressive Decoding

In addition to the research achievements, these models are significant parts of Microsoft's own family of large AI (foundation) models powering language and multimodal tasks and scenarios across products in Microsoft. Moreover, our research tops public benchmarks and leaderboards across language (GLUE, XTREME), vision (ADE20k, COCO), speech (SUPERB), and multimodal (NLVR2, VQAv2) tasks, and hugely contributes to the open source community through GitHub and Hugging Face.

More information about our Research and Highlights.

microsoft/unilm: Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

microsoft/torchscale: Neural Architecture for General AI

microsoft/lmops: General technology for enabling AI capabilities w/ (M)LLMs

We are hiring at all levels (including FTE researchers and interns)! If you are interested in working with us on Foundation Models and General AI, NLP, MT, Speech, Document AI and Multimodal AI, please send your resume to fuwei@microsoft.com.