Using Phi3-vision and Status Graphs for Simple Linear Regression

Hi, my name is Jambo and I am a Microsoft Student Ambassador for the Department of Applied Mathematics. First of all, this article is not a tutorial on regression analysis. Instead, it aims to share some thoughts on applying Phi3-vision to regression analysis. We will cover some regression theory, but we will not focus on the theory itself. The main goal here is to share how to use Phi3-vision, so even if you do not fully understand the theory of regression analysis, this article is still accessible.

What is linear regression

Linear regression is a method used to analyze and predict data. Simply put, it attempts to describe the relationship between X and Y using a straight line.

https://en.wikipedia.org/wiki/Regression_analytic

In regression theory, the best fit line can be found when the data satisfies the following conditions:

– The residual (distance between Y and the line) should be: Normal distributionThat is, the Y points should be evenly distributed on both sides of the line, neither too far from the line nor too crowded on one side.
– The variance of the residuals should be constant, which indicates that the distribution of data points is consistent.

These conditions are ideal and difficult to fully meet in reality, so regression analysis often involves transforming the data to make these conditions as close as possible and finding latent variables.

yes

A simple example showing the distribution of data is as follows:

From the graph, we can easily draw two conclusions.

1. As X increases, Y tends to increase.
2. The points on the left are more concentrated than those on the right, which means that as X increases, the range of Y also increases.

Since this graph does not meet the conditions mentioned above, applying a regression line directly to this data may not produce ideal results.

The following image shows the results of calculating the regression line without any adjustments. Note: R-squared The closer the value is to 1, the better the model.

Which transformation method can we use to get a better model? There are many theories and tests in regression analysis to decide which transformation method to use, but practical applications still require subjective judgment and experience.

Why I think Phi3-vision is suitable for regression analysis

The theory of regression analysis is very rigorous, but real-world applications often require subjective judgment. Real-world data is not as perfect as the theory suggests and is full of unpredictable variables. Therefore, regression analysis is not about finding causal relationships between data, but rather providing a reliable way to “guess” data.

As the proverb says, “All models are wrong.“We are simply looking for a model that fits our data, and this requires a lot of subjective judgment and industry experience. Phi3-vision can quickly provide a “subjective” judgment based on the chart, which is very useful in real-world applications. Once the “subjective” judgment is made, some tests in regression theory can be used to check if this judgment is reasonable.

Regression Analysis Process

This is a simplified flow chart. It still looks complicated, but you don’t have to understand it completely. Just know that you can break down the process, break down a complex problem into several simple subproblems, and then follow the flow chart step by step.

Implementation of state diagram-based regression process using LangGraph + Phi3-vision

We can easily implement regression analysis using the above diagram. Langgraph. Below is an automatically generated state diagram based on the nodes added by LangGraph.

If you don’t have a local environment to run Phi3-vision, you can also conveniently call the model using LangChain’s Nvidia NIM integration. This allows you to quickly test if a particular model is suitable for your application. More information about Phi3-vision can be found here. NVIDIA NIMAfter you log in, you can find your API key at:

from langchain_nvidia_ai_endpoints import ChatNVIDIA
os.environ["NVIDIA_API_KEY"] = ""

llm = ChatNVIDIA(model="microsoft/phi-3-vision-128k-instruct")

For a branch path in a node, you can simply ask Phi3-vision the following simple question: Constant variance In the node, we ask “You are a data analysis expert. Does this dataset have constant variance? You can answer with just True or False.”

From NIM’s online test, we can see that the answer to the above example of Phi3-vision is False. We just need to decide the next step based on the True or False answer.

When verifying that the data meets the requirements Normal distribution Based on the criteria, I wrote a program to automatically generate a QQ plot (a type of chart used to determine whether data follows a normal distribution) from the data, and then asked Phi3-vision a question: “You are a data analysis expert. The attached figure is a QQ plot of this dataset. Does this dataset conform to the normal hypothesis? Please answer with True or False.”

We got True answer. According to the flow, we know that the next step is to perform a weighted regression algorithm. However, we can see from the graph that there are several weighting methods. We can list these methods and automatically generate the necessary charts with the program, and then let Phi3-vision give us the most likely option.

In this example, Phi3-vision considers the first weighting algorithm as the most suitable. Then it just automatically jumps to the corresponding algorithm function, and the rest of the work is done by the program to automatically compute the result.

A concrete code implementation can be found here: here. The following image shows the results of performing regression analysis on the initial example using LangGraph. R-squared The value increased from 0.408 to 0.521, and the entire process took less than 5 seconds. If the model is inferred in a local deployment environment, the time can be reduced even further.

Key points of the example

Decomposition problem

For complex projects like regression analysis, a single large model cannot provide a complete answer in one go. We train a large model that can complete the project in just a few steps (cot) is also very difficult, as it requires collecting a huge amount of complete analysis process data and may require explaining why a particular method was chosen over another.

Our ability to solve these problems quickly and automatically is largely due to the fact that we have broken the problem down into sufficiently detailed steps. Each node is a very simple question, and Phi3-vision only needs to answer True or False. This makes it easy to check whether Phi3-vision’s answer is reasonable.

Now that you’ve broken down the problem into sufficiently detailed steps, you can write prompt text or functions that target each question, which will better guide Phi3-vision’s answer.

Advantages of Phi3-vision

Since the problem is broken down into very simple steps, a small model that provides a fast response is more practical than a large model that excels at answering complex questions. In addition, Phi3-vision has a 128k context capacity, which allows it to provide a large number of reference examples when dealing with simple questions. Since the question is known, there is no need to hard-code examples and wait for the embedding model and vector database to match the appropriate examples.

Advantages of State Graphs

Regression analysis is complex because it requires not only finding correlations between various factors, but also continuously optimizing the model, transforming the data, and further optimizing based on the modified results. Many simple scenarios also rely heavily on loops, for example, continuously checking whether we have enough information to proceed, or continuously adjusting the parameters of the model. The looping feature is a disadvantage of pipeline tools, but state graphs can handle it well.

Most mainstream LLM-based visualization tools are pipeline-based. fast flow and Langflow. If looping is required, use: agents Or code implementation. This can somewhat limit our work and even complicate some problems. State graphs and pipelines are not mutually exclusive. State graphs can act as a higher-level abstraction of pipelines, allowing us to connect pipelines that handle different tasks. And a state graph without loops can be degraded to a pipeline, making it a more general tool.

Loops in the state graph can cause some problems, but agents can also face the problem of infinite loops. Compared to agents, the state graph can show the whole process more intuitively, making it easier to debug and verify.

Differences with agents

Agents allow the model to summarize current information and autonomously decide the next step, giving the model the ability to make decisions and actively gather information. In addition, Multi-agent While methods can solve complex problems, they are better suited to exploring solutions when there is no clear solution. Using agents in areas where there is an established methodology can be redundant and add to the complexity of the system.

Since each step is driven autonomously by the model, it must generate a lot of textual thinking processes to make the right decision, which leads to longer system runtimes, significant token consumption, and the risk of going off track. Rapid adjustments can influence the model’s decisions, but debugging prompts is more of a black box than training a model. It is difficult to know which parts of the model are affected by the prompts.

In the method discussed in this article, engineers design the framework process up front, and the model plays only a role in branch selection. Automatic verification ensures that the model’s selection is correct. The entire system is based on state diagrams, so not only the text output of the LLM, but also the entire process is structurally recorded, facilitating subsequent debugging and verification. The problems encountered by the model are controllable, and the output is simply a Boolean value or a branch selection, so the model’s selection can be adjusted using examples. Even if the model gives incorrect results, the process is within a predictable range, so the problem can be easily identified.

conclusion

Phi3-vision and LangGraph allow you to break down a complex regression process into several smaller problems and solve them step by step. Phi3-vision provides a “subjective” judgment at a particular node, and verifies it with a program that implements the theory. Regression is an iterative process, and by gradually solving the smaller problems, you can get a complete answer.

This approach not only allows you to stop at any time to review and manually intervene in the data, but also offers high scalability. Phi3-vision handles simple problems, so you can easily collect enough data for fine-tuning or RAG by taking advantage of multi-step analysis. Overall, Phi3-vision’s fast response and high context capacity make it suitable for complex but well-established methodologies.

Of course, this approach is not suitable for exploring solutions in unknown areas, since the entire process must be designed in advance. However, in areas where the problem has a known solution, using agents may be redundant, but using Phi3-vision and state graphs is more efficient. These two approaches are not mutually exclusive. You can choose the appropriate method depending on the specific situation.

This is just one of my ideas, and I hope it will inspire you. The lack of graph-based visualization tools today may be due to some considerations I haven’t thought of, or I may not know enough about existing tools. If you have any other ideas or suggestions, please leave a comment.