Using GitHub Copilot's LLM in your VS Code extension

In the July 2024 release of Visual Studio Code, extension developers will be able to use GitHub Copilot’s Language Model API (LLM) to their advantage by using the Language Model API. This new Language Model API (LLM) allows you to retrieve the available LLMs in Visual Studio Code and make requests.

The nice thing about this new API is that you can use it in your extension any way you want. For instance, you can integrate your extension into the GitHub Copilot chat, perform requests from your extension commands, create your own code completion logic, etc.

Waldek Mastykarz gave me the idea to test this out in the Front Matter CMS extension. Front Matter already had an AI integration for title, description, and taxonomy suggestions, but it was only available to the project’s sponsors. The idea is to change this to allow users to use the GitHub Copilot LLM when it’s available in their editor.

Using the Language Model API

In the upcoming version of Front Matter CMS, the GitHub Copilot LLM will be used when creating a new article to suggest a title and allow the user to generate a description and taxonomy based on the title and contents of the article. Extension commands trigger these actions.

In the example, I will show you how to interact with the LLM from within your extension commands.

Retrieving the available LLMs

To retrieve the available LLMs in Visual Studio Code, use the vscode.lm.selectChatModels method. This method returns a promise that resolves to the available LLM’s.

1
// Retrieving all models
2
const models = await vscode.lm.selectChatModels();
3

4
// Retrieving a specific model
5
const [model] = await vscode.lm.selectChatModels({
6
  vendor: "copilot",
7
  family: "gpt-3.5-turbo"
8
});

Creating a prompt

Once you know the model is available, we can create a prompt to interact with the LLM. The prompt is created by one or multiple vscode.LanguageModelChatMessage strings.

1
const title = await vscode.window.showInputBox({
2
  placeHolder: "Enter the topic",
3
  prompt: "Enter the topic of the blog post",
4
});
5

6
if (!title) {
7
  return;
8
}
9

10
const messages = [
11
  vscode.LanguageModelChatMessage.User(`For an article created with Front Matter CMS, create me a SEO friendly description that is a maximum of 160 characters long.`),
12
  vscode.LanguageModelChatMessage.User(`Desired format: just a string, e.g., "My first blog post".`),
13
  vscode.LanguageModelChatMessage.User(`The article topic is """${title}"""`),
14
];

Sending the prompt

Once the prompt is created, you can send it to the LLM using the model.sendRequest method.

1
let chatResponse: vscode.LanguageModelChatResponse | undefined;
2

3
try {
4
  chatResponse = await model.sendRequest(
5
    messages,
6
    {},
7
    new vscode.CancellationTokenSource().token
8
  );
9
} catch (err) {
10
  if (err instanceof vscode.LanguageModelError) {
11
    console.log(err.message, err.code, err.cause);
12
  } else {
13
    throw err;
14
  }
15
  return;
16
}
17

18
let allFragments = [];
19
for await (const fragment of chatResponse.text) {
20
  allFragments.push(fragment);
21
}
22

23
vscode.window.showInformationMessage(allFragments.join(""));

The result of this code looks like this:

Conclusion

The Visual Studio Code team made it easy for extension developers to use the GitHub Copilot LLM in their extensions. This LLM API opens up many possibilities for extension developers to enhance their extensions with AI capabilities.