From ef471fcea1f3667442f5ecbf7b4c214610a5dd55 Mon Sep 17 00:00:00 2001 From: Krish Nandan Das Date: Wed, 1 Jan 2025 13:51:38 +0530 Subject: [PATCH] column name change from close to open (#1304) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * updated elegantrl agents for latest elegentrl release * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * time interal condition and sessions in range changed * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ib data downloadder added * Add GroupBy Scaler (#1212) * Add graph convolutional layers parameter to GPM * Add groupby scaler * Format code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change groupby scaler default scaler * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update portfolio optimization example --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Bump pydantic from 1.10.7 to 1.10.13 (#1216) Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.10.7 to 1.10.13. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md) - [Commits](https://github.com/pydantic/pydantic/compare/v1.10.7...v1.10.13) --- updated-dependencies: - dependency-name: pydantic dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update README.md * Update README.md * Update README.md * [pre-commit.ci] pre-commit autoupdate (#1219) updates: - [github.com/psf/black: 24.4.0 → 24.4.2](https://github.com/psf/black/compare/24.4.0...24.4.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Bump tqdm from 4.65.0 to 4.66.3 (#1222) Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.65.0 to 4.66.3. - [Release notes](https://github.com/tqdm/tqdm/releases) - [Commits](https://github.com/tqdm/tqdm/compare/v4.65.0...v4.66.3) --- updated-dependencies: - dependency-name: tqdm dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump werkzeug from 3.0.1 to 3.0.3 (#1224) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/3.0.1...3.0.3) --- updated-dependencies: - dependency-name: werkzeug dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump jinja2 from 3.1.3 to 3.1.4 (#1225) Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/jinja/compare/3.1.3...3.1.4) --- updated-dependencies: - dependency-name: jinja2 dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update requirements.txt * --- (#1234) updated-dependencies: - dependency-name: requests dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * --- (#1235) updated-dependencies: - dependency-name: pymysql dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1240) updates: - [github.com/asottile/reorder-python-imports: v3.12.0 → v3.13.0](https://github.com/asottile/reorder-python-imports/compare/v3.12.0...v3.13.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix in_channels to use input_features instead of hard-coded 3. (#1239) * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sahil Jain Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Bump tornado from 6.3.3 to 6.4.1 (#1241) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.3 to 6.4.1. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.3...v6.4.1) --- updated-dependencies: - dependency-name: tornado dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix Bug on YahooFinanceProcessor (#1242) * [pre-commit.ci] pre-commit autoupdate (#1243) updates: - [github.com/asottile/pyupgrade: v3.15.2 → v3.16.0](https://github.com/asottile/pyupgrade/compare/v3.15.2...v3.16.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1244) updates: - [github.com/PyCQA/flake8: 7.0.0 → 7.1.0](https://github.com/PyCQA/flake8/compare/7.0.0...7.1.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Bump urllib3 from 1.26.18 to 1.26.19 (#1247) Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.18 to 1.26.19. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst) - [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump scikit-learn from 1.2.2 to 1.5.0 (#1246) Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.2.2 to 1.5.0. - [Release notes](https://github.com/scikit-learn/scikit-learn/releases) - [Commits](https://github.com/scikit-learn/scikit-learn/compare/1.2.2...1.5.0) --- updated-dependencies: - dependency-name: scikit-learn dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update README.md * Update README.md * Bump certifi from 2023.7.22 to 2024.7.4 (#1256) Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.7.22 to 2024.7.4. - [Commits](https://github.com/certifi/python-certifi/compare/2023.07.22...2024.07.04) --- updated-dependencies: - dependency-name: certifi dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump setuptools from 67.7.2 to 70.0.0 (#1260) Bumps [setuptools](https://github.com/pypa/setuptools) from 67.7.2 to 70.0.0. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](https://github.com/pypa/setuptools/compare/v67.7.2...v70.0.0) --- updated-dependencies: - dependency-name: setuptools dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * sinopac_datasource (#1261) * Add files via upload Data source from sinopac which is a taiwan stock platflom. Need API KEY !!! If something go wrong please contact me directly or common below * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Bump torch from 2.0.0 to 2.2.0 (#1262) Bumps [torch](https://github.com/pytorch/pytorch) from 2.0.0 to 2.2.0. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](https://github.com/pytorch/pytorch/compare/v2.0.0...v2.2.0) --- updated-dependencies: - dependency-name: torch dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1263) updates: - [github.com/asottile/pyupgrade: v3.16.0 → v3.17.0](https://github.com/asottile/pyupgrade/compare/v3.16.0...v3.17.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add example of using shioajidownloader from sinopac data source (#1266) * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1267) updates: - [github.com/psf/black: 24.4.2 → 24.8.0](https://github.com/psf/black/compare/24.4.2...24.8.0) - [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](https://github.com/PyCQA/flake8/compare/7.1.0...7.1.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Bump opencv-python from 4.7.0.72 to 4.8.1.78 (#1274) Bumps [opencv-python](https://github.com/opencv/opencv-python) from 4.7.0.72 to 4.8.1.78. - [Release notes](https://github.com/opencv/opencv-python/releases) - [Commits](https://github.com/opencv/opencv-python/commits) --- updated-dependencies: - dependency-name: opencv-python dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 42.0.4 to 43.0.1 (#1275) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.4 to 43.0.1. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/42.0.4...43.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update README.md * fix: Moved device parameter from DRLAgent initialization to get_model method (#1279) * fix: Moved device parameter from DRLAgent initialization to get_model method * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: Moved device parameter from DRLAgent initialization to get_model method * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: add progress bar when downloading data from yfinance * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add ensemble reinforcement learning example with performance metrics analysis --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Revert "fix: Moved device parameter from DRLAgent initialization to get_model…" (#1280) This reverts commit e11f7297c6bd6440bcd7305906157392a22c1a71. * [pre-commit.ci] pre-commit autoupdate (#1281) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.6.0...v5.0.0) - [github.com/psf/black: 24.8.0 → 24.10.0](https://github.com/psf/black/compare/24.8.0...24.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1283) updates: - [github.com/asottile/reorder-python-imports: v3.13.0 → v3.14.0](https://github.com/asottile/reorder-python-imports/compare/v3.13.0...v3.14.0) - [github.com/asottile/pyupgrade: v3.17.0 → v3.18.0](https://github.com/asottile/pyupgrade/compare/v3.17.0...v3.18.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Bump werkzeug from 3.0.3 to 3.0.6 (#1287) Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.3 to 3.0.6. - [Release notes](https://github.com/pallets/werkzeug/releases) - [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/werkzeug/compare/3.0.3...3.0.6) --- updated-dependencies: - dependency-name: werkzeug dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (#1288) updates: - [github.com/asottile/pyupgrade: v3.18.0 → v3.19.0](https://github.com/asottile/pyupgrade/compare/v3.18.0...v3.19.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update yahoodownloader.py (#1290) * Bump tornado from 6.4.1 to 6.4.2 (#1296) Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.4.1 to 6.4.2. - [Changelog](https://github.com/tornadoweb/tornado/blob/v6.4.2/docs/releases.rst) - [Commits](https://github.com/tornadoweb/tornado/compare/v6.4.1...v6.4.2) --- updated-dependencies: - dependency-name: tornado dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update processor_yahoofinance.py (#1299) Fix Multi-Level Column Names in DataFrame Downloaded using processor_yahoofinance.py * Update README.md * [pre-commit.ci] pre-commit autoupdate (#1302) updates: - [github.com/asottile/pyupgrade: v3.19.0 → v3.19.1](https://github.com/asottile/pyupgrade/compare/v3.19.0...v3.19.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update renaming of columns from yfinance and passing auto_adjust as optional parameter (#1301) * Update renaming of columns from yfinance * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix spelling with volume column --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * change open to close * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: dependabot[bot] Co-authored-by: krish-athenasoft Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Caio Souza <48871523+C4i0kun@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: ByFinTech Co-authored-by: Ming Zhu Co-authored-by: Sahil Jain Co-authored-by: Sahil Jain Co-authored-by: Tsou, Dong-You <118954765+dytsou@users.noreply.github.com> Co-authored-by: Charliesj0129 Co-authored-by: Steven Chen <117523987+StevenChen16@users.noreply.github.com> Co-authored-by: makari <46679773+kano5266@users.noreply.github.com> Co-authored-by: Nitasu <92837902+Nitasurin@users.noreply.github.com> Co-authored-by: kuds --- .../data_processors/processor_yahoofinance.py | 4 +- finrl/meta/preprocessor/ibkrdownloader.py | 144 ++++++++++++++++++ 2 files changed, 146 insertions(+), 2 deletions(-) create mode 100644 finrl/meta/preprocessor/ibkrdownloader.py diff --git a/finrl/meta/data_processors/processor_yahoofinance.py b/finrl/meta/data_processors/processor_yahoofinance.py index b1abdc3cc..11c4c2da0 100644 --- a/finrl/meta/data_processors/processor_yahoofinance.py +++ b/finrl/meta/data_processors/processor_yahoofinance.py @@ -121,10 +121,10 @@ def download_data( # convert the column names to match processor_alpaca.py as far as poss data_df.columns = [ "timestamp", - "open", + "close", "high", "low", - "close", + "open", "volume", "tic", ] diff --git a/finrl/meta/preprocessor/ibkrdownloader.py b/finrl/meta/preprocessor/ibkrdownloader.py new file mode 100644 index 000000000..0bee18ae1 --- /dev/null +++ b/finrl/meta/preprocessor/ibkrdownloader.py @@ -0,0 +1,144 @@ +"""Contains methods and classes to collect data from +interactive broker API +""" + +from __future__ import annotations + +import math +from datetime import datetime + +import pandas as pd +from ib_insync import * + + +class ibkrdownloader: + """Provides methods for retrieving daily stock data from + IBKR API + + Attributes + ---------- + start_date : str + start date of the data (modified from neofinrl_config.py) + end_date : str + end date of the data (modified from neofinrl_config.py) + ticker_list : list + a list of stock tickers (modified from neofinrl_config.py) + interval: str + ime period of one bar. Must be one of: + '1 secs', '5 secs', '10 secs' 15 secs', '30 secs', + '1 min', '2 mins', '3 mins', '5 mins', '10 mins', '15 mins', + '20 mins', '30 mins', + '1 hour', '2 hours', '3 hours', '4 hours', '8 hours', + '1 day', '1 week', '1 month'. + + Methods + ------- + fetch_data() + Fetches data from IBKR API + """ + + def __init__( + self, + start_date: str, + end_date: str, + ticker_list: list, + interval="1 day", + host="127.0.0.1", + port=7497, + clientId=999, + ) -> None: + self.start_date = start_date + self.end_date = end_date + self.ticker_list = ticker_list + self.interval = interval + self.host = host + self.port = port + self.clientId = clientId + + def createContract(self, ticker): + contract = Stock(ticker, "SMART", "USD") + return contract + + def connect2ibkr(self): + self.ib = IB() + self.ib.connect(self.host, self.port, clientId=self.clientId) + + def calculate_duration(self, start_date_str, end_date_str): + start_date = datetime.strptime(start_date_str, "%Y-%m-%d") + end_date = datetime.strptime(end_date_str, "%Y-%m-%d") + + duration_days = (end_date - start_date).days + + return duration_days + + def fetch_data(self) -> pd.DataFrame: + """Fetches data from IBKR API + Parameters + ---------- + + Returns + ------- + `pd.DataFrame` + 7 columns: A date, open, high, low, close, volume and tick symbol + for the specified stock ticker + """ + # Download and save the data in a pandas DataFrame: + self.connect2ibkr() + data_df = pd.DataFrame() + duration = self.calculate_duration(self.start_date, self.end_date) + if duration > 365: + duration = f"{math.ceil(duration / 365)} Y" + else: + duration = f"{duration} D" + + end_date_time = datetime.strptime(self.end_date, "%Y-%m-%d") + for tic in self.ticker_list: + contract = self.createContract(tic) + self.ib.qualifyContracts(contract) + bars = self.ib.reqHistoricalData( + contract, + endDateTime=end_date_time, + durationStr=duration, + barSizeSetting=self.interval, + whatToShow="TRADES", + useRTH=False, + formatDate=1, + ) + temp_df = util.df(bars) + print(f"************* {tic} *************") + temp_df["tic"] = [tic] * len(temp_df) + data_df = pd.concat([data_df, temp_df], ignore_index=True) + data_df = data_df[["date", "open", "high", "low", "close", "volume", "tic"]] + filter_df = data_df[data_df["date"] > pd.Timestamp(self.start_date).date()] + filter_df = filter_df.sort_values(by=["date", "tic"]) + filter_df.index = filter_df["date"].factorize()[0] + + self.disconnect() + return filter_df + + def disconnect(self): + self.ib.disconnect() + + def select_equal_rows_stock(self, df): + df_check = df.tic.value_counts() + df_check = pd.DataFrame(df_check).reset_index() + df_check.columns = ["tic", "counts"] + mean_df = df_check.counts.mean() + equal_list = list(df.tic.value_counts() >= mean_df) + names = df.tic.value_counts().index + select_stocks_list = list(names[equal_list]) + df = df[df.tic.isin(select_stocks_list)] + return df + + +if __name__ == "__main__": + intr = ibkrdownloader( + "2023-01-01", "2023-04-12", ["AAPL", "MSFT", "CSCO", "WMT", "TSLA"] + ) + try: + df = intr.fetch_data() + df.to_csv("data.csv", index=False) + except: + intr.disconnect() + + intr.disconnect()