Ochirsaikhan's Research Notebook

[Ochirsaikhan]

This is the start of my research notebook for my senior comp.

[Ochirsaikhan]

9-18-2023

Completed

• Research about prompt engineering and products that use chatGpt as a VSCode extension: In my pursuit of enhancing the capabilities of ChatGPT, I have successfully conducted research into prompt engineering techniques. Additionally, I explored various products that leverage ChatGPT as a Visual Studio Code (VSCode) extension.

  • Prompt Engineering
  • ChatGPT Extension
  • Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language

• Utilize Customized Prompt Structure for Code Explanation: I used a customized prompt structure to conduct experiments with ChatGPT. Specifically, I aimed to evaluate ChatGPT's ability to explain code based on different prompt structures.

Prompt structure 1

You are an elite college professor who is teaching computer science to new students. You are trying to help students understand what coding is, its purpose, and how it can solve problems. As an elite computer science professor, explain the following code in the following way: explain the inputs and outputs of the code using relatable metaphors and analogies, tell how and why the following code might be useful in real-life, and use conversational and spartan tones.

ChatGPT Answer 1

Hey there, future coding wizards! Let's dive into this code and break it down into something you can easily relate to.

Code Purpose: This code is like a magic wand for sorting a jumbled-up list of numbers into a neat, ordered sequence. Think of it as cleaning up your messy room, but with numbers instead of toys.

Inputs and Outputs: Imagine you have a bag of different-sized marbles (the input list). You want to arrange them from the smallest to the largest. That's the job of this code. It takes your bag of marbles (the input list) and returns a new bag of marbles (the sorted list) in the right order.

How It Works: Let's say you have a big jar of marbles, and you want to sort them quickly. What you do is:

Pick one marble randomly (that's the pivot marble).
Create three empty jars: one for smaller marbles (low), one for marbles of the same size (same), and one for bigger marbles (high).
Now, you go through your jar of marbles, one by one.
If a marble is smaller than the pivot, you put it in the "low" jar. If it's the same size as the pivot, into the "same" jar it goes. If it's larger, it joins the "high" jar.
You do this until all the marbles are in their respective jars.
Then, you repeat the process for the "low" and "high" jars. You pick a pivot for each, create three new jars (for smaller, same, and larger), and sort the marbles again.
You keep doing this until all the jars have only one marble left (which means they are sorted).
Finally, you put all the marbles together in order, and you have a beautifully sorted jar of marbles (the sorted list).
Why It's Useful: Think of this like organizing your favorite books. It's much easier to find the book you want to read when they're neatly arranged from A to Z instead of being all over the place. In real-life coding, you often deal with tons of data, and sorting it helps you quickly find what you need. This code can be a lifesaver when you have a long list of numbers, names, or anything else that needs to be sorted.

So, remember, this code is like your sorting assistant, making your messy list of marbles (or data) neat and tidy. Happy coding! 🚀✨

Prompt structure 2

You are an elite college professor who is teaching computer science to new students. You are trying to help students understand what coding is, its purpose, and how it can solve problems. As an elite computer science professor, explain the following code in the following way: explain the inputs and outputs of the code, tell how and why the following code might be useful in real-life, and use conversational and spartan tones.

ChatGPT Answer 2

ure, let's break down this code step by step in a straightforward manner.

Code Purpose:
This code is a Python function called quick_sort that takes a list of integers called array as input and returns a new list of integers in sorted order. It uses the quick sort algorithm to achieve this.

Inputs:

array (List[int]): This is the input list of integers that you want to sort.
Outputs:

List[int]: The function returns a new list of integers, which is the sorted version of the input list.
How it Works:

First, the code checks if the length of the input list (array) is less than 2. If it is, it means the list is already sorted or empty, so it returns the input list as is.

If the length of the list is greater than or equal to 2, it proceeds with the sorting process.

It initializes three empty lists: low, same, and high. These lists will be used to partition the elements of the input list based on their relationship to a randomly chosen pivot element.

The code selects a pivot element from the input list array using randint(0, len(array) - 1). The choice of pivot element can affect the efficiency of the algorithm.

It then goes through each element in the input list array and categorizes them into three groups:

Elements smaller than the pivot are added to the low list.
Elements equal to the pivot are added to the same list.
Elements greater than the pivot are added to the high list.
After partitioning, it recursively applies the quick_sort function to both the low and high lists. This recursive sorting process continues until the base case (list length less than 2) is met.

Finally, it concatenates the sorted low, same, and high lists and returns the sorted list as the output.

Why it's Useful in Real-Life:
The quick sort algorithm is a widely used and efficient sorting algorithm. It's valuable in real-life situations where you need to sort a large amount of data quickly. For example:

Database systems often use sorting to retrieve data efficiently.
Search engines use sorting to rank search results by relevance.
It can be used in financial systems for sorting stock prices, transactions, and more.
Sorting is a fundamental operation in computer science, and quick sort is a versatile tool for solving sorting problems efficiently.

Prompt structure 3

You are an elite college professor who is teaching computer science to new students. You are trying to help students understand what coding is, its purpose, and how it can solve problems. As an elite computer science professor, explain the following code in the following way: explain the inputs and outputs of the code, tell how and why the following code might be useful in real-life, and use relatable metaphors and analogies that is understandable by a college student.

ChatGPT Answer 3

Alright, let's break down this code step by step in a way that's easy to understand, even for someone new to coding. Think of coding like a recipe – you give the computer a set of instructions to solve a problem.

1. Inputs and Outputs:

Input: The code takes in a list of integers (whole numbers) as its input. This list is called 'array,' and it can contain any number of integers.
Output: The code returns a new list of integers, which is a sorted version of the input list. The sorting is done using a method called the "quick sort algorithm."
2. Quick Sort Algorithm:

Imagine you have a bag of different-sized marbles (the input list). Quick sort is like a method for arranging those marbles in ascending order based on their size.
3. How the Code Works:

The code first checks if the input list 'array' has less than 2 marbles in it. If it does (meaning it's already sorted or empty), it simply returns the same list, as there's no need to sort it further.
If there are 2 or more marbles in the bag (the 'array'), it starts the sorting process.
It selects one marble from the bag (the 'pivot') randomly. The choice of this marble doesn't really matter; it's like picking any marble to compare others with.
Now, the code goes through each marble in the bag (the 'for item in array' loop):
If a marble is smaller than the pivot, it goes into a new bag called 'low' (smaller marbles).
If it's the same size as the pivot, it goes into a separate bag called 'same' (marbles of the same size).
If it's larger than the pivot, it goes into another new bag called 'high' (larger marbles).
So, after this process, you have three bags: 'low,' 'same,' and 'high,' with marbles sorted by size.
4. Recursion:

Now, here's where the magic happens. The code repeats this process for the 'low' and 'high' bags. It's like sorting the smaller and larger marbles again in the same way.
It does this sorting process over and over until all the bags have only 1 or 0 marbles left in them. At this point, all the marbles are sorted because each bag contains marbles of the same size.
5. Combining the Bags:

Finally, it combines the 'low,' 'same,' and 'high' bags in order: 'low,' 'same,' and then 'high.' This gives you a single bag (the sorted list) with all the marbles in ascending order.
6. Real-Life Use:

Think of this code like organizing a messy stack of papers. You randomly pick one paper (the pivot), and then you separate all the papers into three piles: smaller ones, the same size, and larger ones. You keep doing this for each pile until they're all sorted. Then you stack them up in order.
In real life, this sorting technique is used in various applications, such as organizing data in databases, sorting search results on websites, or even in your computer's file system to arrange files and folders.
So, this code is like your computer's way of quickly sorting things, just like you might sort marbles or papers to make finding what you need easier and faster. It's a powerful tool in computer science!

Each prompt generated completely different answers by ChatGPT, and I noticed that small tweak in the prompt structure would cause completely different answers.

To Do

• Conduct further research on whether natural language generative AI would be a feasible comp idea.

[Ochirsaikhan]

Week 4 & 5

During Week 4 and 5, I've met multiple times with my first comp reader, Prof. Kapfhammer to further discuss the feasibility of my previous senior thesis idea, polish the idea, and brainstorm new ideas. After meeting with Prof. Kapfhammer, I noticed that my previous idea was lacking clear direction. However, after reading papers suggested by Prof. Kapfhammer, I narrowed down my comp idea further: Now I'll focus on fine-tuning LLM models such as OpenAI's ChatGPT, Anthropic's Claude AI, and Google's Bard, etc and do experimental analysis on which customized/fine-tuned model produces the best result in explaining a piece of code to novice programmers. Before, I was just focusing on fine-tuning the model; now, I'm more interested in doing experiments with real students to find out if the fine-tuned LLM models can help them understand a piece of code better than a general LLM or just through Googling. I'm thinking that my experiment will involve students using three tools to compare their efficiencies: my fine-tuned model, general LLM model, and plain old Googling.

The research papers I read includes the following:

• https://www.oreilly.com/radar/real-real-world-programming-with-chatgpt/
• https://pg.ucsd.edu/publications/scientists-engineers-learning-programming-using-ai_CiSE-2023.pdf
• https://www.pbs.org/newshour/science/chatbots-can-make-things-up-can-we-fix-ais-hallucination-problem

After reading two papers written by Philip Guo, who is a leading researcher in areas spanning human-computer interaction, data science, programming tools, and online learning, I learned that ChatGPT is really good "doer" and not a good "explainer". Therefore, I decided to improve the explaining power of ChatGPT and other LLMs by fine-tuning/customizing it with additional training set to conduct research whether that model will be more effective in explaining code. Now, I'll focus more on reading papers and doing more research on fine-tuning a Large Language Model, which will involve reading OpenAI's documentation and finding out other scholarly articles about customizing a LLM.

Moreover, during Week 5, my mentor notified me that he will connect me with a leading researcher on LLM, and I'm hoping that our meeting will be fruitful in terms of discussing the feasibility and quality of my senior thesis idea.

[Ochirsaikhan]

Week 6 & 7

During Week 6 & 7, my progress was somewhat hindered due to my flu, but I still managed to make significant strides in my project. I continued my research on fine-tuning ChatGPT-3.5 Turbo, delved into OpenAI's Fine Tuning documentation, and started experimenting with the model using a $10 credit. Additionally, I successfully connected to the OpenAI API through my API key and demonstrated a working prototype in my project repository.

Research and Learning

1. Literature Review: Literature Review: I continued to deepen my understanding of fine-tuning large language models, with a particular focus on ChatGPT-3.5 Turbo. I delved into research papers, including those suggested by Prof. Kapfhammer, to gain insights into the nuances of fine-tuning and the use of AI in code explanation. Some of the key papers I explored include:

• https://towardsdatascience.com/your-own-personal-chatgpt-cb0512091e3f
• https://platform.openai.com/docs/guides/fine-tuning/preparing-your-dataset
• https://platform.openai.com/docs/guides/gpt-best-practices/strategy-write-clear-instructions

2. OpenAI Documentation: I thoroughly studied the OpenAI Fine Tuning documentation to gain insights into the technical aspects of fine-tuning ChatGPT-3.5 Turbo. This has provided me with the necessary foundation to carry out my project effectively.

3.** Practical Experimentation**: To facilitate practical experimentation with ChatGPT-3.5 Turbo, I purchased a $10 credit for access to the model. This will enable me to test and fine-tune the model for my specific project requirements.

4. API Integration: I successfully connected to the OpenAI API using my API key. This milestone allows me to seamlessly integrate ChatGPT-3.5 Turbo into my project repository and obtain responses from the model.

Next Steps

1. Data Acquisition: One of the key aspects of my project is obtaining high-quality examples of code explanations. I will be focusing on identifying and collecting suitable datasets to use for fine-tuning the model. This process is critical in ensuring that the model can effectively explain code to novice programmers.

2. Meeting with academic advisors: In Week 5, my mentor informed me of an upcoming meeting with a leading researcher in large language models. This meeting is anticipated to be a valuable opportunity to discuss the feasibility and the quality of my senior thesis idea. I am eager to gain insights and guidance from this expert.

With these recent developments and my continued commitment to research and experimentation, I am making steady progress towards my senior thesis goal of enhancing the explaining power of ChatGPT-3.5 Turbo and similar language models. I am excited about the potential impact of my project and look forward to the challenges and opportunities that lie ahead.

[Ochirsaikhan]

On November 6th, 2023, OpenAI announced the introduction of GPTs, which is more tailored version of ChatGPT. This solves the problem of sharing structured prompts. Now, with the help of GPTs, any GPT will work based on its internal prompt, meaning that we don't have to copy and paste long prompts any time we want ChatGPT to do specific tasks for us. Also, OpenAI announced the GPT Store where people can share their custom GPTs and potentially make money from it if a lot of people are using it. So, there is a chance for me to incorporate customized GPTs into my senior thesis project.

---

According to OpenAI, "GPTs will continue to get more useful and smarter, and you'll eventually be able to let them take on real tasks in the real world." Also, developers can connect GPTs to the real world through APIs, which I can explore to make my senior thesis project more comprehensive. Moreover, ChatGPT Plus users (GPT-4) now includes fresh information up to April, 2023 instead of January 2022 (GPT-3.5 Turbo).

Research Article

I wanted to get an expert's opinion on GPTs, so I read this article that was written by Ethan Mollick who is a professor at the Wharton School of the University of Pennsylvania, studying what our new "AI-haunted" era means for work and education.

• https://www.oneusefulthing.org/p/almost-an-agent-what-gpts-can-do

From the article, I understood that GPTs currently are a powerful tool for sharing a "GPT" that was built by someone with their own prompt. Essentially, it is more convenient way to share structured prompts, which are programs written in plain English that can get AI to do useful things.

Even though it can learn from a proprietary dataset, GPTs can still hallucinate and make information up. This is just the start of an era, and in the future more GPTs and agents will have access to more systems, meaning they can act more on their own. Since this is a new technology, I'm curious about how I can use this in my senior thesis project productively.

---

Custom GPTs vs Fine-Tuning. What's the difference?

According to a user named "Paul.lene" on OpenAI's community form:

• Fine-tuning is giving new knowledge to an AI by retraining it. You « feed it » new data that will now be part of the AI, therefore changing the core of that specific AI. Just like if you read book - it’s somewhere in your brain.
• Custom GPTS instead are all based on the same AI model that remains unaltered. You « just » give the GPT instructions and documents, they can be modified at anytime to update the GPT to your convenience. Like a human wearing a pair of glasses, it changes the way it sees the world, but not the person itself.

The one misleading difference between fine-tuning and GPTs or other agents/bots/AI (they can be called in a multitude of ways) is that in both case you give it new data. It is really in the way the data is used by the AI that makes the big difference. In the first case the AI is modified in its core, while in the second case it is really about providing instructions to guide the existing AI (without modifying the core).

Why do you want to use one or the other ?** Context & cost. Both methods aim to specialize an AI but the outcomes are different. Fine-tuning is more complex and expensive, you need new quality data that will lead to consolidation of knowledge for a tiny part the AI systems - you literally improve the system in a very incremental way (if done correctly). While the custom GPTs approach is much less expensive and more accessible (no-code/low-code - everybody can do it). You do not improve the system but you activate the proper part of the AI brain to get the best out of it.

[gkapfham]

Hello @Ochirsaikhan, your research notebook does not meet several of the baseline requirements for research notebooks in CMPSC 600. Please meet with me during the start of the Spring 2024 semester so that we can develop a plan for improving your work in this area. I encourage you to refer to the syllabus for more details about the requirements for a research notebook in a computer science senior thesis.

[Ochirsaikhan]

February 20, 2024

Last week, I met with Prof. Kapfhammer and my academic advisor, Prof. Luman, to discuss the shift in my senior thesis project. After much contemplation and consideration, I decided to pivot my senior thesis idea from Code Explanation with Large Language Models to Comprehensive Stock Trading System with Large Language Models. I am fully aware that switching ideas this late into the semester is risky and might complicate the timeline of my senior thesis project. However, I firmly believe the potential benefits outweigh the challenges based on the following reasons:

1. The subjective nature of code explanation was complicating the Methods and Experimental Results chapter of my comp. I was stuck on the problem of objectively defining a good code explanation for introductory students. I faced this problem early in the Fall 2023 semester, but I was hopeful of finding a way to get around it. However, for some reason, this problem made me procrastinate on my comp so much that I didn't make a lot of progress towards my computational artifact. So, transitioning to a new computational artifact won't be costly since my current artifact is not where I envisioned it to be.

2. Since I will be utilizing Large Language Models and Generative AI technologies such as GPT-4, all the research I did last semester would not go to waste. The fine-tuning and prompting techniques I learned last semester will be crucial in the development of this new project. Instead of utilizing LLMs towards code explanation, I will utilize them for building a comprehensive stock trading/signal system that can analyze a company based on company news (sentiment analysis), company fundamentals, market fundamentals (fundamental analysis), and stock chart patterns (technical analysis) to generate "buy,"  "hold,"  or "sell" signals with logical explanations to aid individual and beginner stock traders. Therefore, I just need to revise 50% of all the work I did during last semester instead of rewriting and redoing everything from scratch. This indicates that this idea shift is not a total overhaul but a necessary change to allow me to go forward with my senior comp.

3. My original Junior Seminar idea was focused on stock trading with AI. The final deliverable title of my junior seminar was "Predicting Stock Movement Using Artificial Intelligence and Machine Learning." Therefore, all the research work I did in CMPSC_580 will help me transition to my new idea pretty quickly. However, my new senior thesis idea is to build a comprehensive trading system with Generative AI technologies such as GPT-4 instead of training my own machine learning algorithm. That is the crucial difference between the ideas developed in CSMP_580 and CMPS_610. So, my new idea nicely combines the work I've done for CMPS_580 (stock market and AI research) and CMPS_600 (research done for LLMs and techniques such as zero-shot prompting and fine-tuning), meaning that I don't have to start completely from scratch.

After talking to my comp readers and my academic advisors about the aforementioned reasons, they approved of my new idea. Moreover, I discovered that my true passion lies in the stock market, programming, and AI rather than code explanation and pedagogy, because I noticed that I started to enjoy working on my senior comp. With the newfound passion, I hope to catch up with the current course schedule by March 11, 2024 (after Spring Break), and I'm planning to fully commit to catching up to my comp during Spring Break.

Completed

• Read academic papers on the new idea:
  • Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection
  • Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4 on mock CFA Exams
  • Large Language Models and Generative AI in Finance: An Analysis of ChatGPT, Bard, and Bing AI
• Get OpenAI API key from Prof. Luman
• Research frameworks and technologies to build software applications based on LLMs
  • Found the LangChain framework that gives developers tools to construct LLM-powered apps

TODO

• Initialize the computational artifact repository with the LangChain framework and set up the dev environment with the OpenAI API key
• Learn the LangChain framework
• Revise Chapter 1 & 2
  • Find more academic papers

New Goal 🚀: By March 11, revise Chapter 1 & 2, finish writing Chapter 3 & 4 and catch up with the course.

[Ochirsaikhan]

February 22, 2024

Meeting with JJ 🖥️

I had a productive meeting with my second comp reader today. I introduced my new focus on my comp idea, and JJ provided clear guidance on my next actionable steps so that I can catch up to the course by March 11. She suggested that I focus on building the artifact and directed me on how I should conduct my experiment and evaluation. Specifically, I would focus on building a comprehensive trading system, backtesting it on historical data to test the long-term trading prospects, and using it in real life to test the short-term prospects. JJ also provided me guidance on how I should test it specifically. Overall, the meeting was really productive, and I cleared up my confusion on a lot of things.

Completed ✅

• Read LangChain documents and tutorials
• Meet with my second reader
• Schedule a meeting with my first reader

TODO 🔨

• Re-initialize my artifact repo with LangChain
• Start regularly committing to my artifact and thesis chapters

[Ochirsaikhan]

February 27, 2024

Meeting with Prof. Kapfhammer 🖥️

I met with my first comp reader, Prof. Kapfhammer, today, and we talked about the implementation details of my senior thesis project. Specifically, we discussed how I should store all of my relevant data, such as company news, company fundamentals, and macroeconomic news, retrieve it, aggregate all the data, and give it to the LLM model. He recommended I use vector databases and embeddings in the last step to ensure the LLM model has the most relevant information. We also talked about the potential token limitation to give the LLM context, which is around 8,192 tokens in GPT-4. So, I have to limit and implement a logic that, with the user input and the context, doesn't overflow the token limit of the given LLM model. Lastly, we discussed what I can incorporate in my future works section in my comp and how I would experimentally evaluate my model through historical data and in real life.

Completed ✅

• Re-initialize my artifact repo with LangChain
• Set up my project repo with Poetry as the dependency manager
• Continue learning about LangChain by reading documents and watching tutorials
• Learn about vector databases and embeddings

TODO 🔨

• Build the pipeline of my system with few specific companies
  • Decide which database model (SQL, NoSQL, JSON, or Vector) I would use for individual company chain (e.g. company news, company fundamentals, and macroeconomics, etc.)
• Read more peer-reviewed papers related to the field

[Ochirsaikhan]

March 05, 2024

Progress Note 👨🏻‍💻

I've been working on my artifact and making progress.

Completed ✅

• Talked with the CEO of MarketsenseAI to talk about how they implemented their project.
• Fixed Python dependencies. Some packages were incompatible with Python 3.12, so I'm reverting back to Python 3.11
• Watched an expert video on LLM. Link: https://www.youtube.com/watch?v=zjkBMFhNj_g
  • Learned a lot about the current state of LLMs and the future of Generative AI, and security implications of LLMs.
• Continue learning about LangChain by reading documents and watching tutorials

TODO 🔨

• Make the MVP version of my artifact
• Do more research on GPT-4 128,000 token version.

[Ochirsaikhan]

March 06, 2024

Progress Note 👨🏻‍💻

Today, I focused on doing more research on the best Stock Market News & Fundamental Market news other than EODHD to get exposed to other API providers. I've found a few other compelling Stock API providers such as AlphaVantage and Polygon, which will be the data source for my system to make decisions, so the quality of the API is almost directly correlated with the quality of the system. Therefore, I need to be careful and thoughtful in choosing the best API provider. Moreover, I've dug deeper into the LangChain and OpenAI documentations. Specifically, I learned more about RAG (Retrieval Augmented Generation), and building complex Agents with Chains. In the upcoming few days, I need to focus more on building the pipeline for the system.

Completed ✅

• Cleaned up my machine Python environment and dependencies to resolve conflict in package compatibility.
• Did more research on GPT-4 128,000 token version.
• Learned more about the LangChain framework and OpenAI documentations.

TODO 🔨

• Choose the best Stock Market News & Fundamental News API provider.
• Build the "Chain" for each step of my system such as the company fundamentals analyzer and stock news summarizer, etc.
• Do more research on backtesting on historical data.

[Ochirsaikhan]

March 07, 2024

Progress Note 👨🏻‍💻

I have explored deeply what each Stock & Financial Market API provider offers. I now have API keys for Polygon, Alpha Vantage, Finnhub Stock API, and EODHD. I compared each provider's strengths and weaknesses and identified which area the specific provider is good at. For example, I've found that only EODHD provides the stock news article in its full length, which is crucial for the LLM to summarize on its own rather than using summaries that were already provided by the API provider. So, I've purchased EODHD's premium All-in-One Package to get access to their endpoints and make progress towards my artifact. Moreover, the quality of the API response is also different depending on what endpoint I'm using. For instance, I've found that when quoted with "MSFT" as a ticker, Alpha Vantage's news API responded with an [article](Alpha Vantage) that only mentioned Microsoft once (it only mentioned Microsoft Teams which is not even the company but its product), meaning that their API is not returning relevant and valid news for my system to use. That's why I've chosen EODHD for the news summarizer section of my system.

Completed ✅

• Obtained premium API key as the foundation of my comp
• Built a simple request module that writes a JSON file of the company news
• Read more finance API documentation

TODO 🔨

• Come up with a way to build a company profile. For example, when a user asks about the stock projection for Company A, how would my system build the profile for Company A? Is it going to utilize the LLM's own long-term memory to build the company profile, or should it programmatically gather Company A news from the API ranging 6 from months to 1 year (need to decide the range) and build the profile that way and make decisions based on that profile?
• Build the basic version of Gator News Summarizer & Gator Fundamentals Summarizer

[gkapfham]

Hello @Ochirsaikhan please note that the minimum number of research notebook entries for the Spring 2024 semester was 8 and you had 6 in total.