Create your own Visual Studio Code Chat Copilot Agent with Phi-3.5 by GitHub Models

Are you using Visual Studio Code Copilot? In particular, you can enhance the ability to create, author, and maintain projects in Visual Studio Code by using various agents in Chat. Visual Studio Code provides an API that allows companies and individuals to create various agents according to their business, expanding their capabilities in various proprietary areas. In this article, we will focus on: Build your own Visual Studio Code agent using the GitHub models Phi-3.5-mini-instruct (128k) and Phi-3.5-vision-instruct (128k).

Phi-3.5 information for GitHub models

We know that Phi-3/3.5-mini-instruct from the Phi-3/3.5 family has powerful code understanding and generation capabilities and has many advantages over Gemma-2-9b and Mistral-Nemo-12B-instruct-2407.

The latest GitHub models already provide access to the Phi-3.5-mini-instruct(128k) and Phi-3.5-vision-instruct(128k) models. Developers can access them via the OpenAI SDK, Azure AI Inference SDK, and REST API.

memo: We recommend using the Azure AI Inference SDK as it provides better transitions with Azure Model Catalog in production environments.

The results are as follows: Phi-3.5-mini-instruction (128k) and pie-3.5-vision-directive (128k) After docking with the GitHub model, we also prepare the following example in the code generation scenario.

Demo: Generate code from the prompt with GitHub Models Phi-3.5-mini-instruct (128k) (Click this link)

Demo: Generating code from the GitHub Models Phi-3.5-vision-instruct (128k) image (Click this link)

About GitHub Copilot Chat Agent

GitHub Copilot Chat Agent can complete a variety of tasks in different project scenarios based on your code. There are four agents in this system: workspace, github, terminal, and vscode.

Adding an agent name with @ will help you get the job done quickly. For businesses, adding your own business-related content, such as requirements, coding, test specs, and releases, will give you even more powerful enterprise-specific features based on GitHub Copilot.

Visual Studio Code Chat Agent has now officially released API, allowing enterprises or enterprise developers to develop agents based on various software business ecosystems. Based on the development method of Visual Studio Code Extension Development, you can easily access the interface of Visua Studio Code Chat Agent API. You can develop based on this process.

Development scenarios can support access to third-party model APIs (e.g. GitHub models, Azure model catalog, self-built services based on open source models), as well as the gpt-35-turbo, gpt-4, and gpt-4o models available from GitHub Copilot.

Added Phi-3.5 based agent @phicoding

We are trying to integrate the programming capabilities of Phi-3.5 to complete the code writing, image generation code, and other tasks. Complete the agent built on top of Phi-3.5 – @PHI, here are some features:

1. Write your resume using GPT-4o provided by GitHub Copilot. @phicoding /help command

2. Generates code for various programming languages. Phi-3.5-mini-instruction (128k) Through @phicoding /gen command

3. Generate base code pie-3.5-vision-directive (128k) And complete the image @phicoding /image command

1. Install Visual Studio Code extension development support using npm

npm install --global yo generator-code 

2. Create a Visual Studio Code extension plugin (using Typescript development mode, named phiext)

yo code 

3. Open the generated project and edit package.json. Here you will find the relevant instructions and configuration, as well as the GitHub Models configuration. You will need to add your GitHub Models token here.

{
  "name": "phiext",
  "displayName": "phiext",
  "description": "",
  "version": "0.0.1",
  "engines": {
    "vscode": "^1.93.0"
  },
  "categories": [
    "AI",
    "Chat"
  ],
  "activationEvents": [],
  "enabledApiProposals": [
      "chatVariableResolver"
  ],
  "main": "./dist/extension.js",
  "contributes": {
    "chatParticipants": [
        {
            "id": "chat.phicoding",
            "name": "phicoding",
            "description": "Hey! I am Microsoft Phi-3.5, She can help me with coding problems, such as generation code with your natural language, or even generation code about chart from images. Just ask me anything!",
            "isSticky": true,
            "commands": [
                {
                    "name": "help",
                    "description": "Introduce myself to you"
                },
                {
                    "name": "gen",
                    "description": "Generate code for you with Microsoft Phi-3.5-mini-instruct"
                },
                {
                    "name": "image",
                    "description": "Generate code for chart from image(png or jpg) with Microsoft Phi-3.5-vision-instruct, please add image url like this : https://ajaytech.co/wp-content/uploads/2019/09/index.png"
                }
            ]
        }
    ],
    "commands": [
        {
            "command": "phicoding.namesInEditor",
            "title": "Use Microsoft Phi 3.5 in Editor"
        }
    ],
    "configuration": {
      "type": "object",
      "title": "githubmodels",
      "properties": {
        "githubmodels.endpoint": {
          "type": "string",
          "default": "https://models.inference.ai.azure.com",
          "description": "Your GitHub Models Endpoint",
          "order": 0
        },
        "githubmodels.api_key": {
          "type": "string",
          "default": "Your GitHub Models Token",
          "description": "Your GitHub Models Token",
          "order": 1
        },
        "githubmodels.phi35instruct": {
          "type": "string",
          "default": "Phi-3.5-mini-instruct",
          "description": "Your Phi-35-Instruct Model",
          "order": 2
        },
        "githubmodels.phi35vision": {
          "type": "string",
          "default": "Phi-3.5-vision-instruct",
          "description": "Your Phi-35-Vision Model",
          "order": 3
        }
      }
    }
  },
  "scripts": {
    "vscode:prepublish": "npm run package",
    "compile": "webpack",
    "watch": "webpack --watch",
    "package": "webpack --mode production --devtool hidden-source-map",
    "compile-tests": "tsc -p . --outDir out",
    "watch-tests": "tsc -p . -w --outDir out",
    "pretest": "npm run compile-tests && npm run compile && npm run lint",
    "lint": "eslint src",
    "test": "vscode-test"
  },
  "devDependencies": {
    "@types/vscode": "^1.93.0",
    "@types/mocha": "^10.0.7",
    "@types/node": "20.x",
    "@typescript-eslint/eslint-plugin": "^8.3.0",
    "@typescript-eslint/parser": "^8.3.0",
    "eslint": "^9.9.1",
    "typescript": "^5.5.4",
    "ts-loader": "^9.5.1",
    "webpack": "^5.94.0",
    "webpack-cli": "^5.1.4",
    "@vscode/test-cli": "^0.0.10",
    "@vscode/test-electron": "^2.4.1"
  },
  "dependencies": {
    "@types/node-fetch": "^2.6.11",
    "node-fetch": "^3.3.2",
    "@azure-rest/ai-inference": "latest",
    "@azure/core-auth": "latest",
    "@azure/core-sse": "latest"
  }
}

4. Modify src/extension.ts

import * as vscode from 'vscode';
import ModelClient from "@azure-rest/ai-inference";
import { AzureKeyCredential } from "@azure/core-auth";


interface IPhiChatResult extends vscode.ChatResult {
    metadata: {
        command: string;
    };
}


const MODEL_SELECTOR: vscode.LanguageModelChatSelector = { vendor: 'copilot', family: 'gpt-4o' };

function isValidImageUrl(url: string): boolean {
    const regex = /^(https?:\/\/.*\.(?:png|jpg))$/i;
    return regex.test(url);
}
  



export function activate(context: vscode.ExtensionContext) {

    const codinghandler: vscode.ChatRequestHandler = async (request: vscode.ChatRequest, context: vscode.ChatContext, stream: vscode.ChatResponseStream, token: vscode.CancellationToken): Promise<IPhiChatResult> => {


        const config : any = vscode.workspace.getConfiguration('githubmodels');
        const endPoint: string = config.get('endpoint');
        const apiKey: string = config.get('api_key');
        const phi35instruct: string = config.get('phi35instruct');
        const phi35vision: string = config.get('phi35vision');
        
        if (request.command === 'help') {

            const content = "Welcome to Coding assistant with Microsoft Phi-3.5"; 
            stream.progress(content);


            try {
                const [model] = await vscode.lm.selectChatModels(MODEL_SELECTOR);
                if (model) {
                    const messages = [
                        vscode.LanguageModelChatMessage.User("Please help me express this content in a humorous way: I am a programming assistant who can help you convert natural language into code and generate code based on the charts in the images. output format like this : Hey I am Phi ......")
                    ];
                    const chatResponse = await model.sendRequest(messages, {}, token);
                    for await (const fragment of chatResponse.text) {
                        stream.markdown(fragment);
                    }
                }
            } catch(err) {
                console.log(err);
            }


            return { metadata: { command: 'help' } };

        }

        
        if (request.command === 'gen') {

            const content = "Welcome to use phi-3.5 to generate code";

            stream.progress(content);

            const client = new ModelClient(endPoint, new AzureKeyCredential(apiKey));

            const response = await client.path("/chat/completions").post({
              body: {
                messages: [
                  { role:"system", content: "You are a coding assistant.Help answer all code generation questions." },
                  { role:"user", content: request.prompt }
                ],
                model: phi35instruct,
                temperature: 0.4,
                max_tokens: 1000,
                top_p: 1.
              }
            });

            stream.markdown(response.body.choices[0].message.content);

            return { metadata: { command: 'gen' } };

        }



        
        if (request.command === 'image') {


            const content = "Welcome to use phi-3.5 to generate code from image(png or jpg),image url like this:https://ajaytech.co/wp-content/uploads/2019/09/index.png";

            stream.progress(content);

            if (!isValidImageUrl(request.prompt)) {
                stream.markdown('Please provide a valid image URL');
                return { metadata: { command: 'image' } };
            }
            else
            {

                const client = new ModelClient(endPoint, new AzureKeyCredential(apiKey));
    
                const response = await client.path("/chat/completions").post({
                    body: {
                      messages: [
                        { role: "system", content: "You are a helpful assistant that describes images in details." },
                        { role: "user", content: [
                            { type: "text", text: "Please generate code according to the chart in the picture according to the following requirements\n1. Keep all information in the chart, including data and text\n2. Do not generate additional information that is not included in the chart\n3. Please extract data from the picture, do not generate it from csv\n4. Please save the regenerated chart as a chart and save it to ./output/demo.png"},
                            { type: "image_url", image_url: {url: request.prompt}
                            }
                          ]
                        }
                      ],
                      model: phi35vision,
                      temperature: 0.4,
                      max_tokens: 2048,
                      top_p: 1.
                    }
                  });
    
                
                stream.markdown(response.body.choices[0].message.content);
    
                return { metadata: { command: 'image' } };
            }



        }


        return { metadata: { command: '' } };
    };


    const phi_ext = vscode.chat.createChatParticipant("chat.phicoding", codinghandler);

    phi_ext.iconPath = new vscode.ThemeIcon('sparkle');


    phi_ext.followupProvider = {
        provideFollowups(result: IPhiChatResult, context: vscode.ChatContext, token: vscode.CancellationToken) {
            return [{
                prompt: 'Let us coding with Phi-3.5 😋😋😋😋',
                label: vscode.l10n.t('Enjoy coding with Phi-3.5'),
                command: 'help'
            } satisfies vscode.ChatFollowup];
        }
    };

    context.subscriptions.push(phi_ext);
}


export function deactivate() {}

6. running

/help

@phicoding /help

@phicoding /gen

@phicoding /image

Get this demo: Click this link

resources

1. Join the GitHub model https://gh.io/models

2. Learn to develop extensions for Visual Studio Code https://code.visualstudio.com/api/get-started/your-first-extension

3. Learn about the Visual Studio Code Coilot Chat API https://code.visualstudio.com/api/extension-guides/chat

4. Learn more in the Phi-3 Cookbook https://aka.ms/phi-3cookbook