AlgoTraders/stock-analysis-engine issues and pull requests

#376 - Removing pinance, yahoo tests, finviz integration test, relaxing flake8 + pycodestyle + numpy, fix for lib2to3 errors on python 3.8

Pull Request - State: closed - Opened by jay-johnson about 4 years ago - 1 comment

#375 - KeyError("No metadata except PKG-INFO is available")

Issue - State: open - Opened by vmajor almost 5 years ago - 1 comment

#374 - creating stock_analysis_engine.egg-info error: could not create 'stock_analysis_engine.egg-info': Permission denied

Issue - State: closed - Opened by vmajor almost 5 years ago - 1 comment

#373 - pkg_resources.ContextualVersionConflict: numpy 1.14.0

Issue - State: open - Opened by ugurtigli about 5 years ago - 4 comments

#372 - fixing location of keras dropout layer for dnn builder

Pull Request - State: closed - Opened by virdesai over 5 years ago

#371 - rook-ceph cluster osd pod in permanent crash loop CrashLoopBackOff with steps on how to fix without deleting the entire cluster

Issue - State: closed - Opened by jay-johnson over 5 years ago - 4 comments

#370 - trying travis using the docs example again

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#369 - trying this for travis docker push failures

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#368 - travis.yml fixes for docker push to docker hub failures in travis - trying the new docker password arg from https://docs.travis-ci.com/user/docker/#pushing-a-docker-image-to-a-registry

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#367 - fix for supporting engines running in non-UTC timezones by changing the env var on the container with: export EST_OFFSET_HOURS=NUMBER_OF_HOURS - this allows for auto-converting tradier epoch option date fields: ask_date/bid_date/trade_date (which are in the analysis_engine.td.consts.TD_EPOCH_COLUMNS list) when engines are running in EST timezones

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#366 - fix for extended backups starting from the original datasets back on 2018-11-01 and still configurable as an arg with -s YYYY-MM-DD as needed

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#365 - fixes for celery 4.3.0

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#364 - fixes for docker-compose pathing on fedora 29 that installs docker-compose as an rpm to /usr/bin

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#363 - grammar fix for comment in is_holidays docstring

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#362 - option expiration detection for holidays - using new analysis_engine.holidays.is_holiday module and detection function. should fix getting options for 2019-04-19 where the US Markets are closed so the option expiration is set to 2019-04-18 (or the last previous trading day with support for: Monday go back to Friday and if Friday is a holiday too go back to previous Thursday and then Wednesday)

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#361 - celery[redis] is limited on the redis pip's verion and failed the build on redis 3.2.1 pip... pinning the max pip redis version to redis<3 based off the build's error: https://readthedocs.org/projects/stock-analysis-engine/builds/8789547/

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#360 - adding links to new deployment tool: Metalnetes (https://github.com/jay-johnson/metalnetes) for running multiple Analysis Engine instances at the same time on the same bare-metal server (say one for prod and one for dev)

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#359 - helper for deploying AE without helm for the legacy k8 cluster using rook-ceph

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#358 - path fix for ae-restore + turning off prometheus-operator values for redis startup. fixes for rebooting from a clean k8 cluster - ceph looks like it fails the first helm install but passes the helm second install but that could be timing related or maybe a service account/rbac creation issue? either way the ceph cluster works this way and even starts prometheus and grafana

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#357 - adding readme link to ./helm/grafana/values.yaml for auto-importing your own custom dashboards when grafana starts up. also moved the Pods dashboard to the top because it is awesome (cpu + memory + nginx + net tx/rx)

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#356 - Fix minio persistence (pv) with helm deployment

Issue - State: closed - Opened by jay-johnson over 5 years ago - 1 comment
Labels: bug, medium priority

#355 - adding grafana dashboards for redis + ceph + minio + kubernetes with sections in the readme. minio is still not persisting beyond helm delete so be aware of s3 data getting deleted for now

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#354 - fix for showing multiple redis clusters in different namespaces with prometheus + grafana

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#353 - adding helm upgrade notes to the debugging helm guide and ceph grafana dashboard notes

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#352 - link fix for importing in grafana for the celery dashboard - this dashboard is still in testing

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#351 - added ./helm/monitor-start.sh to deploy Prometheus and Grafana with Redis metric collection enabled by default. Importing the Redis Dashboard https://grafana.com/dashboards/763 works in Grafana's dashboard importer: https://grafana.example.com/dashboard/import Also added docs on logging into Grafana with user credentials: trex / 123321 also included support for using ./helm/handle-reboot.sh for handling all reboot operations in a single script which bootstraps all helm deployments including monitoring (Prometheus + Grafana)

Pull Request - State: closed - Opened by jay-johnson over 5 years ago

#350 - Fix S3 feature flag publishing to S3 even if the intraday.s3.enabled: False and fetch -U 0 is used

Issue - State: open - Opened by jay-johnson almost 6 years ago
Labels: bug, low priority

#349 - turning s3 back on for intraday + daily + weekly jobs because it's causing a bug where these jobs do not respect this feature flag and try to publish s3 minio anyway with bad credentials and it causes timeouts that slow these jobs down too much for recording per minute.

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#348 - s3 publishing disabled by default for intraday, daily and weekly charts (the backup job only needs this to save on AWS s3 costs). adding cron job helper script and examples for cron-automating minute, daily, weekly pricing data collection, daily backups and restore-on-reboot that work with helm + kubernetes. added new ae-restore helm job chart for handling a cron-initiated @reboot restore job. on a server reboot this job will download the latest backed up pricing data from S3 (AWS or Minio) and deploy it to the kubernetes Redis cluster. Note: running the backup and restore cron jobs have cost implications with many tickers (AWS charges for usage).

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#347 - adding build.sh step to the asciinema video on the helm readme https://stock-analysis-engine.readthedocs.io/en/latest/deploy_on_kubernetes_using_helm.html

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#346 - reducing RUN directives in Dockerfile and uninstall previous builds from the base-1.9.3 image. fixing read the docs width sizing on mobile showing the jupyter notebook link.

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#345 - now using a base Dockerfile to decrease image build + download time

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#344 - travis warning on dockerfile copying vimrc

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#343 - indentation fix on private docker registry and add numpy for ta-lib install that failed on travis build https://travis-ci.org/AlgoTraders/stock-analysis-engine/builds/499665427

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#342 - attempt to reduce the docker image size by using the python:3.6-slim as the base image. locally the container reports 2.33 GB with the command: docker images

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#341 - spelling/grammar in readme

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#340 - helm guide documentation for setting IEX Cloud, Tradier, tickers to collect per minute, day and week, and using a private docker registry outside kubernetes, and jupyter login password. adding a vimrc for internal container editing

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#339 - adding asciinema video on how to deploy using helm https://asciinema.org/a/230411?autoplay=1 to the guide

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#338 - helm is now supported for deploying to kubernetes. helm guide added at: ./helm/README.rst and (hopefully) on read the docs: https://stock-analysis-engine.readthedocs.io/en/latest/deploy_on_kubernetes_using_helm.html to get started run: cd helm && ./build.sh && ./start.sh && ./show-pods.sh also includes support for using a private docker registry hosted outside of kubernetes to deploy using the values.yaml file.

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#337 - Paper Trading

Issue - State: open - Opened by lion7p almost 6 years ago - 3 comments
Labels: paper trading

#336 - comment fix for rtd

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#335 - easier fetch command line alias for pulling data from the command line. added verbose flag propagation to debug IEX Cloud response data + status codes with -d flag like: fetch -t QQQ -g min -d

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#334 - inspect_datasets.py examples and usage added to docs

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#333 - inspect_datasets.py - improved logging

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#332 - Spelling and RTD doc cleanup

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#331 - inspect datasets in redis tool added: analysis_engine/scripts/inspect_datasets.py to help find dates that do not make sense with the redis cached values. proposed fixes for backfilling missing intraday minute pricing data as well using: fetch -t TICKER -g iex_min -F DATE_TO_FIX

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#330 - adding install step for IEX Cloud API docs and renaming the extract doc title

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#329 - link fix attempt 3

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#328 - link fix attempt 2

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#327 - link fix for rtd

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#326 - cleaning up the function comments for fetch + extract. put ticker arg first to make fetch and extract consistent and easier to code for users

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#325 - moving fetch above extract on rtd

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#324 - fixes for function doc strings on rtd

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#323 - moving ticker + date as first two args on the IEX extraction calls to make it easier to use (without keyword args for ticker + date)

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#322 - more extract from Redis examples and fixes for extracting historical options. adding support for verbose logging arg propagation to see the redis keys in use. adding a sample fix IEX Cloud daily data tool for showing how to use the extract and publish API calls to fix cached data with a one-off tool

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#321 - migrating analysis_engine/extract.py to use the new analysis_engine/build_dataset_node.py which supports ticker + date + verbose arguments through both the IEX Cloud and Tradier API calls. adding docs for extracting stock minute + options data from redis

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#320 - more doc fixes for readmes

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#319 - fixing links for extraction api references in rtd

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#318 - IEX Cloud fetch + extraction source code examples added to docs. added links to the readme to find the api guides easier. need to verify links on rtd

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#317 - adding IEX Cloud extract api support for ticker + optional historical date arguments for pulling data out of redis. fix for backups to use all datasets in redis instead of just the empty algo's indicator datasets in the backup case

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#316 - spelling fix for rtd docs

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#315 - proposed option fetch from Tradier that is aware of the latest minute intraday pricing so that the in-the-money, at-the-money and out-of-the-money options are limited to [current close - 10 strikes, atm the money strike, current close + 10 strikes]. This helps to reduce the amount of data being stored in redis for intraday option chains.

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago - 1 comment

#314 - adding a vprof image to the readme and docs

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#313 - adding profiler performance test tool vprof to the pip. included a module to benchmark and profile algorithms using the AlgoRunner with an example: prof -c cm ./analysis_engine/perf/profile_algo_runner.py also added two perf fixes from testing with vprof

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#312 - some cleanup

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#311 - adding ./analysis_engine/scripts/backtest_with_runner.py to the installed scripts with the pip. changing algo_runner priority on read the docs to make it easier to find

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#310 - updating comments for algo.process and removing logger that is not used

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#309 - performance boost for extracting only the minimum pricing data from redis to run a backtest. this works by identifying the minimum set of distinct datasets in the algorithm config indicators[*].uses_data values. once the list of datasets to extract is determined, the new analysis_engine.build_dataset_node module will pull those compressed values from redis + decompress them and then run the algorithm + indicators. additional support for custom datasets per algorithm is stubbed out, but this is still a concept and would require analysis_engine.indicator_processor upgrades to support non-pricing data supported in the analysis_engine.indicators.base_indicator module.

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#308 - backfill previous 30 calendar days of intraday minute pricing data from IEX Cloud with: backfill-minute-data.sh TICKER and support for fetching an individual historical minute dataset for a single ticker and publish it to redis + s3 (if enabled) with: fetch -t TICKER -F YYYY-MM-DD -g iex_min

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#307 - fixes for no publish on history or report at the end of a backtest. better logging on redis and file analysis_engine.publish.publish calls

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#306 - allow algos to run minute timeseries data even if there are gaps in the cached datasets in redis - show a warning instead of failing to run. adding debug information for the redis cache location during fetch on the engines for tracking updates on k8

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#305 - fixing IEX minute dataset date column to include the minute value as well in the string before publishing to the redis cache

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#304 - f-string update

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#303 - k8 fixes for automating IEX and Tradier pricing intraday, daily and weekly with easier logging - also added to the docker container too

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#302 - adding mocks to unit tests for running without an internet connection log a warning on fetches that return empty data. adding a helper analysis_engine.iex.get_pricing_on_date for extracting the lastest (or historical with date_str arg) from cached IEX pricing data from daily or minute values in redis

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#301 - new fetch support for controlling which IEX data feed will be used to prevent charges on the on-demand usage model for the IEX Cloud support. includes support for using intraday + daily + weekly kubernetes jobs and by default avoids downloading the financials and earnings which are more 'expensive' at the moment

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#300 - use the last close date in case testing on the weekend

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#299 - indentation fix for rst in the readme docs

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#298 - more doc fixes for the getting started

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#297 - link fixes for IEX apis on read the docs and in code comments

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#296 - providing a warning if the IEX_TOKEN environment variable is not set before trying to fetch data

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#295 - Fetch calls support new IEX Cloud (https://iexcloud.io/docs/api/#stocks) with a valid beta account and Publishable Token set as the IEX_TOKEN environment variable - Removing pyEX dependency and using internal calls instead with publishable token authentication

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#294 - readme updates for less typing on the algo runner tutorial and fix for doc string example on read the docs

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#293 - reducing length of indicator names in the trading history dataframe. removed "<indicator>_obj" value that was None from the trading history. f-string updates for indicator processing

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#292 - link fix for code example in readme

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#291 - better algo runner examples and easier plotting with the plot_trade_history after using: df = AlgoRunner.latest() to get the latest trading history as a pandas.DataFrame based off the algorithm's backtesting of the last 200 minutes of the trading session

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#290 - moving the note to the correct section in the readme

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#289 - adding new analysis_engine.algo_runner.py module for making it easier to run an algo backtest and make predictions on live pricing data during the day. examples on using the AlgoRunner api and updates for f-strings in the algo.py

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#288 - adding buy and sell stats to the trading history for use with predictions

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#287 - fix for restoring compressed datasets after downloading from AWS S3

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#286 - installing the pip from pypi is adding a bad line that is not in the code https://github.com/AlgoTraders/stock-analysis-engine/blob/609c0f92d1da0bcd45f112e8c8ef66e413362417/analysis_engine/talib.py#L7: 'from . import talib' to the pip. this line causes collisions with analysis_engine.talib so renamed it to analysis_engine.ae_talib and adding 'import talib as ta' to the ae_talib.py module to see if installing from pypi adds it again - refactoring unittests to make the imports easier to read and maintain for tests/test_base_algo.py and fixes to the new ae_talib path for mocks

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#285 - k8 secretKeyRef fix for new SLACK image plot posting into channel(s)

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#284 - Slack Integration Updates

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#283 - integrate posting matlibplot pyplot plots to slack channels

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#282 - rebase fix attempt

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#281 - Slack plots

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#280 - fix for load_history_dataset log line showing PLOT_ACTION_SAVE_TO_S3 instead of SA_DATASET_TYPE_TRADING_HISTORY

Pull Request - State: closed - Opened by jay-johnson almost 6 years ago

#279 - Multidnn

Pull Request - State: closed - Opened by virdesai almost 6 years ago

#183 - reverting unwanted doc change

Pull Request - State: closed - Opened by virdesai about 6 years ago

#123 - Cohosting

Pull Request - State: closed - Opened by virdesai about 6 years ago

GitHub / AlgoTraders/stock-analysis-engine issues and pull requests