first draft of the walk-through

.gitignore

@@ -160,3 +160,9 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# Local Stuff
+secrets.env
+*.Rproj
+.Rproj.user
+.DS_store

README.qmd

@@ -0,0 +1,197 @@
+---
+title: "botex: a walk-through"
+format: gfm
+ipynb-shell-interactivity: all
+---
+
+### Idea
+
+Have you ever thought about how you can use LLMs like Chat GPT when designing and testing your (online) experiments? Are you interested in studying the ‘behavior’ of LLMs? Or do you want to assess how humans change their behavior when they interact with AI? 
+
+If your answer to any of these questions is yes (or if you are just curious): We, Victor Maas from the University of Amsterdam, and Fikir Worku Edossa as well as Joachim Gassen from Humboldt-Universität zu Berlin and the Open Science Data Center of TRR 266 Accounting for Transparency, have recently released [‘botex’](https://github.com/joachim-gassen/botex), a Python package that makes it straight-forward to use Chat GPT and even local LLM models as participants in oTree experiments. 
+
+This repository provides a short walk-through explaining how to use botex together with [oTree](https://www.otree.org) and the [OpenAI API](https://openai.com/api). 
+
+
+### Setup
+
+In order to use botex you first need to install it. We recommend to use a virtual environment to avoid conflicts with other packages.
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
+# this will change to pip install botex
+pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple botex==0.1.6
+```
+
+As we will be running a local oTree server in this walk-through, we install oTree as well. Also we will install some additional packages that are used in the example code. 
+
+```bash
+pip install otree
+pip install -r requirements.txt
+```
+
+BTW: At least for uns, installing otree causes a warning about a dependency conflict for the MarkupSafe package. We did not yet recognize any issues with this, but if you do, please let us know.
+
+
+### Configuration
+
+Copy the file `_secrets.env` to `secrets.env` and edit to your needs. Assuming that you will be using the OpenAI API, all you need to provide is your API key.
+
+
+### Starting the oTree server
+
+To start the oTree server, run the following command in the terminal:
+
+```bash
+cd otree
+otree devserver
+```
+
+Once the server has started, you can go to `http://localhost:8000` in your browser to see the oTree frontend. It should look like this:
+
+![](/media/otree_screenshot.jpg)
+
+
+### Running a human/bot experiment
+
+If you want, you can now try out the experiment (the split screen variant is nice for this) by clicking on the the experiment caption. After clicking on it, you will also see to single use links that you can use to participate in the experiment. The P1 link will be leading to the investor role, while P2 will be the role of the manager. 
+
+Time to face an experiment with an LLM participant. Pick the role that you want to play. Copy the other link into the code snippet below, which - like all other snippets - is also available in the `code` folder of this repository and run the code. Finally, click on your link to see the instructions of the experiment and to start your run.
+
+```python
+from os import environ
+import logging
+logging.basicConfig(level=logging.INFO)
+
+import botex 
+
+from dotenv import load_dotenv
+load_dotenv('secrets.env')
+
+# The missing config (botex_db, openai_api_key) will be read from the environment
+botex.run_single_bot(
+    url = "http://localhost:8000/InitializeParticipant/wojqsysx"
+)
+```
+
+While participating in the experiment, you should be able to observe the log messages of the bot in the terminal. If anything goes wrong, these log messages might help you to identify the issue.
+
+After completing the experiment, you could download the resulting data from oTree exactly like you would for a 'normal' experiment. But here, we are interested in the bot data. To get this data, you can run the following code snippet:
+
+```{python}
+#| output: asis
+import botex
+from tabulate import tabulate
+
+# Adjust this to where you stored the botex data 
+BOTEX_DB = 'data/external/botex_single_exp.sqlite3'
+part = botex.read_participants_from_botex_db(
+  botex_db = BOTEX_DB
+)
+disp_part = [
+  [r[v] for v in ('session_name', 'participant_id', 'time_in','time_out')] 
+  for r in part
+]
+print(tabulate(
+  disp_part, 
+  headers=["Session", "Participant ID", "Time in", "Time out"],
+  tablefmt="github"
+))
+```
+
+Well, this is interesting but frankly not that exciting. Let's take a look at the rationale data. For each answer that the bot provides, it also provides a rationale underlying its answer.
+
+```{python}
+#| output: asis
+# Reading response data from botex database
+responses = botex.read_responses_from_botex_db(
+  botex_db = BOTEX_DB
+)
+print(tabulate(responses, headers="keys" , tablefmt="github"))
+```
+
+Now, this is somewhat more interesting and clearly indicates that the bot got the gist of the experiment and the short post-experimental questionaire.
+
+
+### Running a bot only session
+
+If you want to run an oTree session using only LLM participants. You can do this using the `init_otree_session()` function and then use `run_bots_on_sesssion()` to run the bots. The code below runs a session on 5 bot investor/manager pairs.
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)
+
+import botex 
+
+from dotenv import load_dotenv
+load_dotenv('secrets.env')
+
+mftrust = botex.init_otree_session(config_name = "mftrust", npart = 10)
+
+botex.run_bots_on_session(
+  session_id = mftrust['session_id']
+)
+```
+
+
+### Take a look at the experimental results by using the botex data
+
+While in a real setting, you would most likely download the experiment's data from otree and work from there. However, you can also use the response data from the botex directly for some quick test. From our paper we know that Chat-GPT 4o bots are very likely to invest 50 in the first round of a standard framed trust game (See Figure 4 of the paper). How does providing the manager a communication option change the first round behavior of the investor?
+
+```{python}
+#| output: asis
+BOTEX_DB = 'data/external/botex_session_exp.sqlite3'
+responses = botex.read_responses_from_botex_db(
+  botex_db = BOTEX_DB
+)
+sent_amount_first_round = [
+    r['answer'] for r in responses 
+      if r['question_id'] == "id_sent_amount" and r['round'] == 1
+]
+print(tabulate(
+    {"Sent Amount": sent_amount_first_round}, 
+    headers = "keys", tablefmt="github"
+))
+
+from scipy import stats
+
+t_stat, p_value = stats.ttest_1samp(sent_amount_first_round, 50)
+
+print(f"t statistic: {t_stat:.2f} (p-value: {p_value:.3f})")
+```
+
+It certainly points into a direction. Not bad for a n=5 study, right?
+
+
+## Let me take a peak under the hood: the prompting level
+
+If you want to understand how the saussage is being made, it is ueful to analyze the prompt level. See below for the prompting sequence that generated the bot responses in the human bot setting.
+
+```{python}
+#| output: asis
+
+import json
+import re
+
+# Adjust this to where you stored the botex data 
+BOTEX_DB = 'data/external/botex_single_exp.sqlite3'
+
+# Reading response data from botex database
+conv = botex.read_conversations_from_botex_db(
+  botex_db = BOTEX_DB
+)
+ps = json.loads(conv[0]['conversation'])
+prompt_sequence = [
+  {"role": p["role"], "content": re.sub(r'[\s+]', ' ', p["content"])}for p in ps
+]
+print(tabulate(prompt_sequence, headers="keys", tablefmt="github"))
+```
+
+Quite a bit of tokens for running a three-round trust game, right? This is also why using Chat-GPT as an experimental participant on oTree sessions is not particularly cheap (but still much cheaper than using human participants). For example, the costs of running the one bot participant above where US-$ 0.11 and the cost for running the session with the five dyads (10 participants) were US-$ 1.30.
+
+
+## Get in touch!
+
+This concludes our little walk-through. If you are interested in this project or even have already tried it, we would love to hear from you. Simply shoot an email, comment on our linkedin post, or open an issue on GitHub, either here, of if your point is more related to the inner working of botex, directly in the botex repository.
+

_secrets.env

@@ -0,0 +1,30 @@
+# Configuration of passwords, key, etc.
+# Copy to secrets.env and edit
+# ___ DO NOT COMMIT 'secrets.env' ___
+
+OTREE_SERVER_URL="http://localhost:8000"
+
+# You can see the next ones to any value you like.
+# They are used to authenticate the bot with the oTree server.
+# When your oTree server is available in the network 
+# it is recommended to use 'real' passwords.
+OTREE_ADMIN_PASSWORD=****
+OTREE_REST_KEY=****
+
+# This one is required if you plan to use one of OpenAI's Chat-GPT models 
+OPENAI_API_KEY=******
+
+# Set this to where you want to store the data generated by the bots
+BOTEX_DB="botex.sqlite3"
+
+# --- Everything below is only needed if you use local LLMs ----
+
+PATH_TO_LLAMA_SERVER="llama.cpp/llama-server"
+LOCAL_LLM_PATH="models/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf"
+
+# The below should be 0 if you are GPU poor :(, 
+# otherwise, you can set it depending on your GPU memory size, 
+# you can also increase the number of slots for more parallelism if 
+# you have a plenty of GPU memory.
+NUMBER_OF_LAYERS_TO_OFFLOAD_TO_GPU=0
+NUM_SLOTS=1

code/display_botex_prompt_sequence.py

@@ -0,0 +1,15 @@
+import botex
+from tabulate import tabulate
+
+import json
+
+# Adjust this to where you stored the botex data 
+BOTEX_DB = 'data/external/botex_single_exp.sqlite3'
+
+# Reading response data from botex database
+conv = botex.read_conversations_from_botex_db(
+  botex_db = BOTEX_DB
+)
+
+prompt_sequence = json.loads(conv[0]['conversation'])
+print(tabulate(prompt_sequence, headers="keys"))

code/extract_botex_participants.py

@@ -0,0 +1,16 @@
+import botex
+from tabulate import tabulate
+
+# Adjust this to where you stored the botex data 
+BOTEX_DB = 'data/external/botex_single_exp.sqlite3'
+part = botex.read_participants_from_botex_db(
+  botex_db = BOTEX_DB
+)
+disp_part = [
+  [r[v] for v in ('session_name', 'participant_id', 'is_human', 'time_in','time_out')] 
+  for r in part
+]
+print(tabulate(
+  disp_part, 
+  headers=["Session", "Participant ID", "Is Human?", "Time in", "Time out"]
+))

code/extract_botex_rationales.py

@@ -0,0 +1,11 @@
+import botex
+from tabulate import tabulate
+
+# Adjust this to where you stored the botex data 
+BOTEX_DB = 'data/external/botex_single_exp.sqlite3'
+
+# Reading response data from botex database
+responses = botex.read_responses_from_botex_db(
+  botex_db = BOTEX_DB
+)
+print(tabulate(responses, headers="keys"))

code/parse_sent_amount_answers.py

@@ -0,0 +1,17 @@
+import botex
+from scipy import stats
+from tabulate import tabulate
+
+BOTEX_DB = 'data/external/botex_session_exp.sqlite3'
+responses = botex.read_responses_from_botex_db(
+  botex_db = BOTEX_DB
+)
+sent_amount_first_round = [
+    r['answer'] for r in responses 
+      if r['question_id'] == "id_sent_amount" and r['round'] == 1
+]
+print(tabulate({"Sent Amount": sent_amount_first_round}, headers = "keys"))
+t_stat, p_value = stats.ttest_1samp(sent_amount_first_round, 50)
+
+print("t statistic:", '{:.2f}'.format(t_stat))
+print("p-value:", '{:.3f}'.format(p_value))

code/run_botex_session.py

@@ -0,0 +1,13 @@
+import logging
+logging.basicConfig(level=logging.INFO)
+
+import botex 
+
+from dotenv import load_dotenv
+load_dotenv('secrets.env')
+
+mftrust = botex.init_otree_session(config_name = "mftrust", npart = 10)
+
+botex.run_bots_on_session(
+  session_id = mftrust['session_id']
+)

code/run_single_botex_bot.py

@@ -0,0 +1,12 @@
+import logging
+logging.basicConfig(level=logging.INFO)
+
+import botex 
+
+from dotenv import load_dotenv
+load_dotenv('secrets.env')
+
+# The missing config (botex_db, openai_api_key) will be read from the environment
+botex.run_single_bot(
+    url = "http://localhost:8000/InitializeParticipant/ca1cfrdk"
+)

otree/.gitignore

@@ -0,0 +1,14 @@
+venv
+staticfiles
+./db.sqlite3
+.idea
+*~
+*.sqlite3
+_static_root
+_bots*s
+__temp*
+__pycache__/
+*.py[cod]
+.DS_Store
+merge.ps1
+*.otreezip

otree/LICENSE

@@ -0,0 +1,26 @@
+Copyright (c) 2014 Daniel Li Chen, Martin Walter Schonger, Christopher Wickens.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+The Licensee undertakes to mention the name oTree, the names of the licensors
+(Daniel L. Chen, Martin Schonger and Christopher Wickens) and to cite the
+following article in all publications in which results of experiments conducted
+with the Software are  published: Chen, Daniel L., Martin Schonger, and Chris Wickens.
+2016. "oTree - An open-source platform for laboratory, online, and field experiments."
+Journal of Behavioral and Experimental Finance, vol 9: 88-97.

otree/Procfile

@@ -0,0 +1,2 @@
+web: otree prodserver1of2
+worker: otree prodserver2of2

otree/_templates/global/Page.html

@@ -0,0 +1,7 @@
+{{ extends "otree/Page.html" }}
+{{ load otree }}
+
+{{ block global_styles  }}
+{{ endblock }}
+{{ block global_scripts  }}
+{{ endblock }}