Merge remote-tracking branch 'origin/dev' into un-revert-backtesting-…

…optimizations

README.md

@@ -178,13 +178,18 @@ git commit -m "my changes"
 git push -u origin my-feature
 ```
 
-If work on main progressed while you were in another branch, this is how you rebase it into your branch:
+If work on main progressed while you were in another branch, this is how you rebase it into your branch. Note that
+since you've rebased your local branch, you'll need to force push your changes to update the remote branch. 
+The --force-with-lease option is a safer alternative to --force as it will abort the push if there are any new 
+commits on the remote that you haven't incorporated into your local branch
 ```shell
 git checkout dev
 git fetch origin
 git merge origin/dev
 git checkout my-feature
 git rebase dev
+git checkout my-feature
+git push --force-with-lease origin my-feature
 ```
 
 When ready to merge the branch into main, go into github, create a pull request, and await review. When your PR is approved it will automatically be merged into the dev branch remotely. Now, you can delete your local branch and the remote branch.

docs/_sources/brokers.interactive_brokers.rst.txt

@@ -1,46 +1,100 @@
 Interactive Brokers
-===================================
+===================
 
-To trade in your interactive brokers account, you must install Trader Workstation (or Gateway). Instructions for installation can be found here: https://ibkrcampus.com/campus/ibkr-api-page/twsapi-doc/
+Interactive Brokers is a brokerage firm that operates in most countries around the world. This makes it a great choice for investing and trading if you don't have access to our other supported platforms.
 
-Once installed, navigate in Trader Workstation to ``File -> Global Configuration -> API -> Settings`` The key settings required to trade using Lumibot are:
+**Note:** If you do have access to other supported brokers, compare their fee structures to see which one is the best deal for you.
 
-#. Enable ActiveX and Socket Clients
-#. Disable Read-Only API
-#. Socket port ``7496`` for live trading, ``7497`` for paper account trading.
-    * NOTE: It is highly recommended to thoroughly test your algorithm in paper trading mode before trading live.
-#. Master API Client ID: You can find in the Trader Workstation by going to ``File -> Global Configurations -> API -> Settings``, then looking for "Master API client ID". This can be any number you choose up to ``999``. You will use this in your configuration file to log in.
+Market Data Subscriptions
+-------------------------
+To access real-time market data, you need to subscribe to the appropriate market data packages. Interactive Brokers offers various market data subscriptions depending on the exchanges and types of data you need. Here are some common subscriptions:
 
-.. image:: interactive_brokers.png
+- US Securities Snapshot and Futures Value Bundle
+- US Equity and Options Add-On Streaming Bundle
+- NASDAQ (Network C/UTP) TotalView
+- NYSE (Network A/CTA) OpenBook Ultra
+- OPRA (US Options Exchanges)
 
-Set up your credentials as follows in ``credentials.py``:
+.. note::
 
-.. code-block:: python
+    Different strategies may require different market data subscriptions. For our top options-related strategies, the OPRA (US Options Exchanges) should suffice.
 
-    INTERACTIVE_BROKERS_CONFIG = {
-        "SOCKET_PORT": 7497,
-        "CLIENT_ID": "your Master API Client ID three digit number",
-        "IP": "127.0.0.1",
-    }
+**To subscribe to market data:**
 
-Set up your entry point file as above, except using Interactive Brokers. Here is an example of a completed file:
+1. Log in to the IBKR Client Portal.
+2. Navigate to the "Settings" menu.
+3. Under "Account Settings", find the "Market Data Subscriptions" section.
+4. Click "Configure" and select the desired market data packages.
+5. Follow the prompts to complete the subscription process.
 
-.. code-block:: python
-    :emphasize-lines: 3,12
+**Note:** Market data subscriptions may incur additional fees. Be sure to review the costs associated with each package before subscribing.
+
+Automating Two-Factor Authentication
+------------------------------------
+Currently, automating two-factor authentication (2FA) is not supported. For now, Lumibot will send you 2FA notifications through IBKey, and you will need to respond to them manually.
+
+Using a Secondary Account
+-------------------------
+IBKR only permits a single login at a time for any given set of credentials. Consequently, you can't use the IBKR website while the Gateway (or a sort of api connection) is running. If you try to anyway, the Gateway will be disconnected, which will trigger a re-authentication attempt, which will potentially disconnect your website session. If you then log in to the website again, you'll just start the loop all over again. An easy solution is to stop a sort of api connection before using the website, but that can be tedious.
+
+A more convenient solution is to create a second username for your account and use that for a sort of api connection. From the client portal API documentation section "Multiple usernames":
+
+Clients wishing to use multiple IBKR products at the same time (TWS, IBKR Mobile or Client Portal) can do so by creating a new username that can then be used to log into other services while using the Client Portal API. To create a second username please see the following IBKR Knowledge Base article.
+
+The Knowledge Base article linked above is slightly out of date with respect to the current layout of the IBKR Client Portal:
+
+- The "Manage Account" item in the User menu (upper right corner) is now called "Settings"
+- The "Users & Access Rights" panel is no longer on the Settings page, but you can find a similarly named link at the very bottom-left of the page (you may have to scroll to find it)
+- The resulting page doesn't have a "Configure (gear) icon". It does have a plus sign button in the "Users" header that does the same thing.
+
+From there you'll be shown a fairly standard-looking username and password dialog. Fill it out in the normal fashion, then click through the many many pages of boilerplate and settings. Feel free to disable many of the notification settings for the new username. Otherwise you'll start getting duplicate emails from IBKR.
 
-    from lumibot.traders import Trader
-    # Import interactive brokers
-    from lumibot.brokers import InteractiveBrokers
-    from lumibot.strategies.examples import Strangle
-    from credentials import INTERACTIVE_BROKERS_CONFIG
+At the end of the process, you'll get an email with a confirmation code, but no indication as to what to do with it. Log out and log back in using the new username and password that you just created. You'll then be prompted to enter the confirmation code. Note that you need to be quick about this part because the code expires fast. If you're not fast enough you'll get another automated email with a new code a few minutes after the previous one expired. In fact, you'll keep getting new codes until you get it right.
 
+After entering the confirmation code you'll likely be prompted to change the password for the secondary user.
+
+The final screen of the process says that the request for account creation will be reviewed on the next business day. Some time later you'll receive an email asking you to upload a "Proof of trader authority" for the new username. The verbiage implies that you need to generate a document to upload, however that isn't the case. Simply log in using your primary username (not the secondary!) and you'll be prompted to upload the requested document. At the bottom of the pop-up will be a button titled "Click Here to Upload Documents". Click that and you'll be shown a EULA and a signature block. Type your name (not the new user's name) and click Submit.
+
+After that you should be back at the normal dashboard page, with little indication that you're logged in as the second user. Log out and the new credentials will be ready to use with a sort of api connection. In the future be sure to only use your primary credentials on the website and the secondary credentials for a sort of api connection.
+
+**Source:** `IBeam GitHub <https://github.com/Voyz/ibeam/wiki/Runtime-environment>`_
+
+Strategy Setup
+--------------
+Add these variables to a `.env` file in the same directory as your strategy:
+
+.. list-table:: Interactive Brokers Configuration
+    :widths: 25 50 25
+    :header-rows: 1
+
+    * - **Secret**
+      - **Description**
+      - **Example**
+    * - IB_USERNAME
+      - Your Interactive Brokers username.
+      - `user123`
+    * - IB_PASSWORD
+      - Your Interactive Brokers password.
+      - `password123`
+    * - ACCOUNT_ID
+      - (Optional) An Interactive Brokers subaccount to trade on.
+      - `U17369206`
+    * - IB_API_URL
+      - (Optional) The URL of your self-hosted Interactive Brokers REST API. You likely don't need it.
+      - `https://localhost:8000`
+
+Example Strategy
+----------------
+
+.. code-block:: python
+     :emphasize-lines: 3,12
 
-    trader = Trader()
-    # Initialize interactive brokers
-    interactive_brokers = InteractiveBrokers(INTERACTIVE_BROKERS_CONFIG)
+     from lumibot.traders import Trader
+     # Import interactive brokers
+     from lumibot.strategies.examples import Strangle
 
-    strategy = Strangle(broker=interactive_brokers)
-    trader.add_strategy(strategy)
-    trader.run_all()
+     trader = Trader()
 
-You can also see the file simple_start_ib.py for a working bot here: https://github.com/Lumiwealth/lumibot/blob/master/getting_started/simple_start_ib.py
+     strategy = Strangle()
+     trader.add_strategy(strategy)
+     trader.run_all()

docs/_sources/brokers.interactive_brokers_legacy.rst.txt

@@ -0,0 +1,47 @@
+Interactive Brokers Legacy
+===================================
+
+.. note::
+    This method has been deprecated. Please use the other Interactive Brokers tutorial.
+
+Once installed, navigate in Trader Workstation to ``File -> Global Configuration -> API -> Settings`` The key settings required to trade using Lumibot are:
+
+#. Enable ActiveX and Socket Clients
+#. Disable Read-Only API
+#. Socket port ``7496`` for live trading, ``7497`` for paper account trading.
+    * NOTE: It is highly recommended to thoroughly test your algorithm in paper trading mode before trading live.
+#. Master API Client ID: You can find in the Trader Workstation by going to ``File -> Global Configurations -> API -> Settings``, then looking for "Master API client ID". This can be any number you choose up to ``999``. You will use this in your configuration file to log in.
+
+.. image:: interactive_brokers.png
+
+Set up your credentials as follows in ``credentials.py``:
+
+.. code-block:: python
+
+    INTERACTIVE_BROKERS_CONFIG = {
+        "SOCKET_PORT": 7497,
+        "CLIENT_ID": "your Master API Client ID three digit number",
+        "IP": "127.0.0.1",
+    }
+
+Set up your entry point file as above, except using Interactive Brokers. Here is an example of a completed file:
+
+.. code-block:: python
+    :emphasize-lines: 3,12
+
+    from lumibot.traders import Trader
+    # Import interactive brokers
+    from lumibot.brokers import InteractiveBrokers
+    from lumibot.strategies.examples import Strangle
+    from credentials import INTERACTIVE_BROKERS_CONFIG
+
+
+    trader = Trader()
+    # Initialize interactive brokers
+    interactive_brokers = InteractiveBrokers(INTERACTIVE_BROKERS_CONFIG)
+
+    strategy = Strangle(broker=interactive_brokers)
+    trader.add_strategy(strategy)
+    trader.run_all()
+
+You can also see the file simple_start_ib.py for a working bot here: https://github.com/Lumiwealth/lumibot/blob/master/getting_started/simple_start_ib.py

docs/_sources/brokers.rst.txt

@@ -9,5 +9,6 @@ There are several different brokers that you can use to trade with Lumibot, and
 
    brokers.alpaca
    brokers.interactive_brokers
+   brokers.interactive_brokers_legacy
    brokers.ccxt
    brokers.tradier

docs/_sources/vars.rst.txt

@@ -1,59 +1,85 @@
 Variable Backup & Restore
 =========================
 
-Every `Strategy` object includes a `vars` attribute (`self.vars`), which is an instance of the `Vars` class. It stores the strategy's runtime variables and is periodically backed up to the database provided with the `DB_CONNECTION_STR` environment variable.
+Every `Strategy` object has a `vars` attribute (`self.vars`), an instance of the `Vars` class. It stores runtime variables and is periodically backed up to the database specified by the `DB_CONNECTION_STR` environment variable.
 
-.. important::
+How It Works
+------------
 
-  We strongly recommend using `self.vars` for storing your strategy's runtime variables, as this ensures that if the strategy stops unexpectedly, it can safely resume from the backup state stored in `self.vars`.
+- **Loading Variables:** Before the first trading iteration, saved variables are loaded into `self.vars`.
 
-Class Definition
-----------------
+- **Backing Up Variables:** After each trading iteration and when the bot stops or crashes, `self.vars` is automatically backed up to the database.
 
-.. code-block:: python
+Benefits of `self.vars`
+-----------------------
+
+- **Persistence:** Resume from the last state after interruptions.
+- **Namespacing:** Prevent conflicts with other components or strategies.
+- **Scalability:** Efficient storage, especially when scaling or using multiple strategies.
 
-  class Vars:
-    def __init__(self):
-      super().__setattr__('_vars_dict', {})
+Usage Guide
+-----------
 
-    def __getattr__(self, name):
-      try:
-        return self._vars_dict[name]
-      except KeyError:
-        raise AttributeError(f"'Vars' object has no attribute '{name}'")
+**Setting Variables:**
 
-    def __setattr__(self, name, value):
-      self._vars_dict[name] = value
+Assign values using attribute notation:
 
-    def set(self, name, value):
-      self._vars_dict[name] = value
+.. code-block:: python
 
-    def all(self):
-      return self._vars_dict.copy()
+   def on_trading_iteration(self):
+       self.vars.trade_count = self.vars.get('trade_count', 0) + 1
 
-Usage Examples
---------------
+**Accessing Variables:**
 
-**Set attributes:**
+Retrieve variables using attribute notation:
 
 .. code-block:: python
 
-  self.vars.some_var = 10
-  self.vars.another_var = "Hello, World!"
+   def on_trading_iteration(self):
+       current_count = self.vars.trade_count
+       print(f"Current trade count: {current_count}")
+
+**Getting All Variables:**
 
-**Get attributes:**
+Use the `all()` method:
 
 .. code-block:: python
 
-  print(self.vars.some_var)    # Output: 10
-  print(self.vars.another_var) # Output: Hello, World!
+   all_variables = self.vars.all()
+   print(all_variables)
 
-**Get all attributes:**
+   # Output: {'trade_count': 5, 'last_price': 102.5}
 
-.. code-block:: python
+.. tip::
 
-  print(self.vars.all())  # Output: {'some_var': 10, 'another_var': 'Hello, World!'}
+   Use `all()` for debugging or processing multiple variables at once.
 
-.. tip::
+Database Configuration
+----------------------
+
+Set the `DB_CONNECTION_STR` environment variable in your .env file:
+
+.. code-block:: bash
+
+   DB_CONNECTION_STR="postgresql://user:password@localhost:5432/database_name"
+
+Database Storage Structure
+--------------------------
+
+Variables are stored in a PostgreSQL table:
+
+- **Table Name:** Defined by `self.backup_table_name` (default is `vars_backup`).
+
+- **Columns:**
+  - `id` (Primary Key)
+  - `last_updated`: Timestamp of the last backup.
+  - `variables`: JSON string of the strategy's variables (`self.vars.all()`).
+  - `strategy_id`: Strategy name (`self.name`).
+
+Example Database:
 
-   **Tip:** Use the `all()` method to get a copy of all stored variables in a dictionary format.
++------------+---------------------+---------------------------+-------------------------+
+| **id**     | **last_updated**    | **variables**             | **strategy_id**         |
++============+=====================+===========================+=========================+
+| 550e8400   | 2023-10-05 14:30:00 | {"var1": 10, "var2": "A"} | OptionsCondorMartingale |
++------------+---------------------+---------------------------+-------------------------+

docs/backtesting.backtesting_function.html

@@ -371,6 +371,7 @@ <h5>Need Extra Help?</h5>
 <li class="toctree-l1 has-children"><a class="reference internal" href="brokers.html">Brokers</a><input class="toctree-checkbox" id="toctree-checkbox-14" name="toctree-checkbox-14" role="switch" type="checkbox"/><label for="toctree-checkbox-14"><div class="visually-hidden">Toggle navigation of Brokers</div><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
 <li class="toctree-l2"><a class="reference internal" href="brokers.alpaca.html">Alpaca</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers.html">Interactive Brokers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers_legacy.html">Interactive Brokers Legacy</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.ccxt.html">Crypto Brokers (Using CCXT)</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.tradier.html">Tradier</a></li>
 </ul>

docs/backtesting.how_to_backtest.html

@@ -371,6 +371,7 @@ <h5>Need Extra Help?</h5>
 <li class="toctree-l1 has-children"><a class="reference internal" href="brokers.html">Brokers</a><input class="toctree-checkbox" id="toctree-checkbox-14" name="toctree-checkbox-14" role="switch" type="checkbox"/><label for="toctree-checkbox-14"><div class="visually-hidden">Toggle navigation of Brokers</div><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
 <li class="toctree-l2"><a class="reference internal" href="brokers.alpaca.html">Alpaca</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers.html">Interactive Brokers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers_legacy.html">Interactive Brokers Legacy</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.ccxt.html">Crypto Brokers (Using CCXT)</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.tradier.html">Tradier</a></li>
 </ul>

docs/backtesting.html

@@ -371,6 +371,7 @@ <h5>Need Extra Help?</h5>
 <li class="toctree-l1 has-children"><a class="reference internal" href="brokers.html">Brokers</a><input class="toctree-checkbox" id="toctree-checkbox-14" name="toctree-checkbox-14" role="switch" type="checkbox"/><label for="toctree-checkbox-14"><div class="visually-hidden">Toggle navigation of Brokers</div><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
 <li class="toctree-l2"><a class="reference internal" href="brokers.alpaca.html">Alpaca</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers.html">Interactive Brokers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers_legacy.html">Interactive Brokers Legacy</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.ccxt.html">Crypto Brokers (Using CCXT)</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.tradier.html">Tradier</a></li>
 </ul>

docs/backtesting.indicators_files.html

@@ -371,6 +371,7 @@ <h5>Need Extra Help?</h5>
 <li class="toctree-l1 has-children"><a class="reference internal" href="brokers.html">Brokers</a><input class="toctree-checkbox" id="toctree-checkbox-14" name="toctree-checkbox-14" role="switch" type="checkbox"/><label for="toctree-checkbox-14"><div class="visually-hidden">Toggle navigation of Brokers</div><i class="icon"><svg><use href="#svg-arrow-right"></use></svg></i></label><ul>
 <li class="toctree-l2"><a class="reference internal" href="brokers.alpaca.html">Alpaca</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers.html">Interactive Brokers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="brokers.interactive_brokers_legacy.html">Interactive Brokers Legacy</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.ccxt.html">Crypto Brokers (Using CCXT)</a></li>
 <li class="toctree-l2"><a class="reference internal" href="brokers.tradier.html">Tradier</a></li>
 </ul>