Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Large Language Models for Code

Table of contents

  1. Decoder-only Models
    1. GPT-1
    2. GPT-2
    3. GPT-3
    4. Codex
    5. GPT-NeoX
    6. GPT-Neo
    7. CodeGen
    8. InstructGPT
    9. CodeGeeX
    10. GPT-J
    11. LLaMA
    12. ChatGPT
    13. StableLM-Alpha
    14. InCoder
    15. GPT-4
    16. WizardCoder
    17. PanGu-Coder
    18. OPT
    19. StarCoder
    20. SantaCoder
    21. PaLM
    22. Vicuna
    23. Flan-UL2
    24. CPM-Bee
    25. MT-NLG
    26. GLM
    27. YaLM
    28. Alpaca
    29. RWKV-4
    30. Sparrow
    31. Falcon
    32. Code Llama
    33. RedPajama-INCITE
    34. DeciCoder-1B
    35. OpenLLaMA
    36. CodeGPT
  2. Encoder-only Models
    1. BERT
    2. ALBERT
    3. RoBERTa
    4. CodeBERT
    5. GraphCodeBERT
  3. Encoder-decoder Models
    1. AlphaCode
    2. T5
    3. CodeT5
    4. CodeT5+
    5. UnixCoder
    6. PLBART
    7. CodeReviewer

Language models have shown impressive capabilities for various natural language processing and software engineering tasks. This page catalogs key details and papers for influential LLMs that have been applied to software tasks.

Distribution of LLMs

Decoder-only Models

GPT-1

GPT-2

GPT-3

Codex

GPT-NeoX

GPT-Neo

  • Release Date: 2021-03
  • Source: Github

CodeGen

InstructGPT

CodeGeeX

  • Title: CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X
  • Year: 2023
  • Paper: Link

GPT-J

LLaMA

ChatGPT

StableLM-Alpha

InCoder

  • Paper: “InCoder: A Generative Model for Code Infilling and Synthesis”
  • Authors: Daniel Fried et al.
  • Release Date: 2023
  • Paper: Link

GPT-4

WizardCoder

PanGu-Coder

OPT

StarCoder

SantaCoder

PaLM

Vicuna

  • Release Date: 2023/03
  • Blog: Link

Flan-UL2

CPM-Bee

MT-NLG

GLM

YaLM

  • Release Date: 2022-06
  • Institute: Yandex
  • Blog: YaLM Blog

Alpaca

  • Release Date: 2023-03
  • Institute: Stanford University
  • Access: Alpaca GitHub

RWKV-4

  • Release Date: 2022-09
  • Institute: Independent (BlinkDL)
  • Access: RWKV-4 GitHub

Sparrow

Falcon

  • Release Date: 2023-05
  • Institute: Technology Innovation Institute (TII)
  • Access: Falcon Homepage

Code Llama

RedPajama-INCITE

DeciCoder-1B

OpenLLaMA

CodeGPT

Encoder-only Models

BERT

ALBERT

RoBERTa

CodeBERT

GraphCodeBERT

Encoder-decoder Models

AlphaCode

  • Release Date: 2022/02
  • Access: AlphaCode
  • Institute: DeepMind

T5

CodeT5

CodeT5+

UnixCoder

PLBART

CodeReviewer