Skip to content

Commit

Permalink
column name change from close to open (#1304)
Browse files Browse the repository at this point in the history 
* updated elegantrl agents for latest elegentrl release

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* time interal condition and sessions in range changed

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ib data downloadder added

* Add GroupBy Scaler (#1212)

* Add graph convolutional layers parameter to GPM

* Add groupby scaler

* Format code

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change groupby scaler default scaler

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update portfolio optimization example

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump pydantic from 1.10.7 to 1.10.13 (#1216)

Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.10.7 to 1.10.13.
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/main/HISTORY.md)
- [Commits](pydantic/pydantic@v1.10.7...v1.10.13)

---
updated-dependencies:
- dependency-name: pydantic
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update README.md

* Update README.md

* Update README.md

* [pre-commit.ci] pre-commit autoupdate (#1219)

updates:
- [github.com/psf/black: 24.4.0 → 24.4.2](psf/black@24.4.0...24.4.2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump tqdm from 4.65.0 to 4.66.3 (#1222)

Bumps [tqdm](https://github.com/tqdm/tqdm) from 4.65.0 to 4.66.3.
- [Release notes](https://github.com/tqdm/tqdm/releases)
- [Commits](tqdm/tqdm@v4.65.0...v4.66.3)

---
updated-dependencies:
- dependency-name: tqdm
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump werkzeug from 3.0.1 to 3.0.3 (#1224)

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.1 to 3.0.3.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.0.1...3.0.3)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump jinja2 from 3.1.3 to 3.1.4 (#1225)

Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4.
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](pallets/jinja@3.1.3...3.1.4)

---
updated-dependencies:
- dependency-name: jinja2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update requirements.txt

* --- (#1234)

updated-dependencies:
- dependency-name: requests
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* --- (#1235)

updated-dependencies:
- dependency-name: pymysql
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1240)

updates:
- [github.com/asottile/reorder-python-imports: v3.12.0 → v3.13.0](asottile/reorder-python-imports@v3.12.0...v3.13.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix in_channels to use input_features instead of hard-coded 3. (#1239)

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Sahil Jain <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump tornado from 6.3.3 to 6.4.1 (#1241)

Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.3 to 6.4.1.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](tornadoweb/tornado@v6.3.3...v6.4.1)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix Bug on YahooFinanceProcessor (#1242)

* [pre-commit.ci] pre-commit autoupdate (#1243)

updates:
- [github.com/asottile/pyupgrade: v3.15.2 → v3.16.0](asottile/pyupgrade@v3.15.2...v3.16.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1244)

updates:
- [github.com/PyCQA/flake8: 7.0.0 → 7.1.0](PyCQA/flake8@7.0.0...7.1.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump urllib3 from 1.26.18 to 1.26.19 (#1247)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.26.18 to 1.26.19.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.18...1.26.19)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump scikit-learn from 1.2.2 to 1.5.0 (#1246)

Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.2.2 to 1.5.0.
- [Release notes](https://github.com/scikit-learn/scikit-learn/releases)
- [Commits](scikit-learn/scikit-learn@1.2.2...1.5.0)

---
updated-dependencies:
- dependency-name: scikit-learn
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update README.md

* Update README.md

* Bump certifi from 2023.7.22 to 2024.7.4 (#1256)

Bumps [certifi](https://github.com/certifi/python-certifi) from 2023.7.22 to 2024.7.4.
- [Commits](certifi/python-certifi@2023.07.22...2024.07.04)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump setuptools from 67.7.2 to 70.0.0 (#1260)

Bumps [setuptools](https://github.com/pypa/setuptools) from 67.7.2 to 70.0.0.
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v67.7.2...v70.0.0)

---
updated-dependencies:
- dependency-name: setuptools
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* sinopac_datasource (#1261)

* Add files via upload

Data source from sinopac which is a taiwan stock platflom. Need API KEY !!! If something go wrong please contact me directly or common below

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Bump torch from 2.0.0 to 2.2.0 (#1262)

Bumps [torch](https://github.com/pytorch/pytorch) from 2.0.0 to 2.2.0.
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](pytorch/pytorch@v2.0.0...v2.2.0)

---
updated-dependencies:
- dependency-name: torch
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1263)

updates:
- [github.com/asottile/pyupgrade: v3.16.0 → v3.17.0](asottile/pyupgrade@v3.16.0...v3.17.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add example of using shioajidownloader from sinopac data source (#1266)

* Add files via upload

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1267)

updates:
- [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0)
- [github.com/PyCQA/flake8: 7.1.0 → 7.1.1](PyCQA/flake8@7.1.0...7.1.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump opencv-python from 4.7.0.72 to 4.8.1.78 (#1274)

Bumps [opencv-python](https://github.com/opencv/opencv-python) from 4.7.0.72 to 4.8.1.78.
- [Release notes](https://github.com/opencv/opencv-python/releases)
- [Commits](https://github.com/opencv/opencv-python/commits)

---
updated-dependencies:
- dependency-name: opencv-python
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump cryptography from 42.0.4 to 43.0.1 (#1275)

Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.4 to 43.0.1.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](pyca/cryptography@42.0.4...43.0.1)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update README.md

* fix: Moved device parameter from DRLAgent initialization to get_model method (#1279)

* fix: Moved device parameter from DRLAgent initialization to get_model method

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Moved device parameter from DRLAgent initialization to get_model method

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: add progress bar when downloading data from yfinance

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add ensemble reinforcement learning example with performance metrics analysis

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Revert "fix: Moved device parameter from DRLAgent initialization to get_model…" (#1280)

This reverts commit e11f729.

* [pre-commit.ci] pre-commit autoupdate (#1281)

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0)
- [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1283)

updates:
- [github.com/asottile/reorder-python-imports: v3.13.0 → v3.14.0](asottile/reorder-python-imports@v3.13.0...v3.14.0)
- [github.com/asottile/pyupgrade: v3.17.0 → v3.18.0](asottile/pyupgrade@v3.17.0...v3.18.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Bump werkzeug from 3.0.3 to 3.0.6 (#1287)

Bumps [werkzeug](https://github.com/pallets/werkzeug) from 3.0.3 to 3.0.6.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](pallets/werkzeug@3.0.3...3.0.6)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit autoupdate (#1288)

updates:
- [github.com/asottile/pyupgrade: v3.18.0 → v3.19.0](asottile/pyupgrade@v3.18.0...v3.19.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update yahoodownloader.py (#1290)

* Bump tornado from 6.4.1 to 6.4.2 (#1296)

Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.4.1 to 6.4.2.
- [Changelog](https://github.com/tornadoweb/tornado/blob/v6.4.2/docs/releases.rst)
- [Commits](tornadoweb/tornado@v6.4.1...v6.4.2)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update processor_yahoofinance.py (#1299)

Fix Multi-Level Column Names in DataFrame Downloaded using processor_yahoofinance.py

* Update README.md

* [pre-commit.ci] pre-commit autoupdate (#1302)

updates:
- [github.com/asottile/pyupgrade: v3.19.0 → v3.19.1](asottile/pyupgrade@v3.19.0...v3.19.1)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update renaming of columns from yfinance and passing auto_adjust as optional parameter (#1301)

* Update renaming of columns from yfinance

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix spelling with volume column

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* change open to close

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: krish-athenasoft <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Caio Souza <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: ByFinTech <[email protected]>
Co-authored-by: Ming Zhu <[email protected]>
Co-authored-by: Sahil Jain <[email protected]>
Co-authored-by: Sahil Jain <[email protected]>
Co-authored-by: Tsou, Dong-You <[email protected]>
Co-authored-by: Charliesj0129 <[email protected]>
Co-authored-by: Steven Chen <[email protected]>
Co-authored-by: makari <[email protected]>
Co-authored-by: Nitasu <[email protected]>
Co-authored-by: kuds <[email protected]>
  • Loading branch information
@krishdotn1 @krish-athenasoft
@pre-commit-ci @CaioSBC @dependabot @BruceYanghy @zhumingpassional @sahiljain01 @dytsou @Charliesj0129 @StevenChen16 @kano5266 @Nitasurin @kuds
15 people authored Jan 1, 2025
1 parent b632d6e commit ef471fc
Show file tree
Hide file tree
Showing 2 changed files with 146 additions and 2 deletions.
4 changes: 2 additions & 2 deletions finrl/meta/data_processors/processor_yahoofinance.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,10 +121,10 @@ def download_data(
# convert the column names to match processor_alpaca.py as far as poss
data_df.columns = [
"timestamp",
"open",
"close",
"high",
"low",
"close",
"open",
"volume",
"tic",
]
Expand Down
144 changes: 144 additions & 0 deletions finrl/meta/preprocessor/ibkrdownloader.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
"""Contains methods and classes to collect data from
interactive broker API
"""

from __future__ import annotations

import math
from datetime import datetime

import pandas as pd
from ib_insync import *


class ibkrdownloader:
"""Provides methods for retrieving daily stock data from
IBKR API
Attributes
----------
start_date : str
start date of the data (modified from neofinrl_config.py)
end_date : str
end date of the data (modified from neofinrl_config.py)
ticker_list : list
a list of stock tickers (modified from neofinrl_config.py)
interval: str
ime period of one bar. Must be one of:
'1 secs', '5 secs', '10 secs' 15 secs', '30 secs',
'1 min', '2 mins', '3 mins', '5 mins', '10 mins', '15 mins',
'20 mins', '30 mins',
'1 hour', '2 hours', '3 hours', '4 hours', '8 hours',
'1 day', '1 week', '1 month'.
Methods
-------
fetch_data()
Fetches data from IBKR API
"""

def __init__(
self,
start_date: str,
end_date: str,
ticker_list: list,
interval="1 day",
host="127.0.0.1",
port=7497,
clientId=999,
) -> None:
self.start_date = start_date
self.end_date = end_date
self.ticker_list = ticker_list
self.interval = interval
self.host = host
self.port = port
self.clientId = clientId

def createContract(self, ticker):
contract = Stock(ticker, "SMART", "USD")
return contract

def connect2ibkr(self):
self.ib = IB()
self.ib.connect(self.host, self.port, clientId=self.clientId)

def calculate_duration(self, start_date_str, end_date_str):
start_date = datetime.strptime(start_date_str, "%Y-%m-%d")
end_date = datetime.strptime(end_date_str, "%Y-%m-%d")

duration_days = (end_date - start_date).days

return duration_days

def fetch_data(self) -> pd.DataFrame:
"""Fetches data from IBKR API
Parameters
----------
Returns
-------
`pd.DataFrame`
7 columns: A date, open, high, low, close, volume and tick symbol
for the specified stock ticker
"""
# Download and save the data in a pandas DataFrame:
self.connect2ibkr()
data_df = pd.DataFrame()
duration = self.calculate_duration(self.start_date, self.end_date)
if duration > 365:
duration = f"{math.ceil(duration / 365)} Y"
else:
duration = f"{duration} D"

end_date_time = datetime.strptime(self.end_date, "%Y-%m-%d")
for tic in self.ticker_list:
contract = self.createContract(tic)
self.ib.qualifyContracts(contract)
bars = self.ib.reqHistoricalData(
contract,
endDateTime=end_date_time,
durationStr=duration,
barSizeSetting=self.interval,
whatToShow="TRADES",
useRTH=False,
formatDate=1,
)
temp_df = util.df(bars)
print(f"************* {tic} *************")
temp_df["tic"] = [tic] * len(temp_df)
data_df = pd.concat([data_df, temp_df], ignore_index=True)
data_df = data_df[["date", "open", "high", "low", "close", "volume", "tic"]]
filter_df = data_df[data_df["date"] > pd.Timestamp(self.start_date).date()]
filter_df = filter_df.sort_values(by=["date", "tic"])
filter_df.index = filter_df["date"].factorize()[0]

self.disconnect()
return filter_df

def disconnect(self):
self.ib.disconnect()

def select_equal_rows_stock(self, df):
df_check = df.tic.value_counts()
df_check = pd.DataFrame(df_check).reset_index()
df_check.columns = ["tic", "counts"]
mean_df = df_check.counts.mean()
equal_list = list(df.tic.value_counts() >= mean_df)
names = df.tic.value_counts().index
select_stocks_list = list(names[equal_list])
df = df[df.tic.isin(select_stocks_list)]
return df


if __name__ == "__main__":
intr = ibkrdownloader(
"2023-01-01", "2023-04-12", ["AAPL", "MSFT", "CSCO", "WMT", "TSLA"]
)
try:
df = intr.fetch_data()
df.to_csv("data.csv", index=False)
except:
intr.disconnect()

intr.disconnect()

0 comments on commit ef471fc

Please sign in to comment.