Install PIP & ML Packages
Machine Learning Setup:
===================
$ cat ml-setup.sh
sudo curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
sudo python get-pip.py
# resolve requests error
# pip list | grep request
sudo -H pip install --ignore-installed requests==2.7.0
sudo -H pip install quandl
sudo -H pip install sklearn
sudo -H pip install scipy
$
Box 1:
=====
# sudo sh
# curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
# python get-pip.py
# pip install quandl
...
Installing collected packages: numpy, more-itertools, pycparser, cffi, enum34, asn1crypto, idna, ipaddress, cryptography, pyOpenSSL, python-dateutil, pandas, ndg-httpsclient, inflection, certifi, chardet, urllib3, requests, quandl
/usr/lib/python2.7/site-packages
Installing collected packages: requests, quandl
Found existing installation: requests 2.6.0
Cannot uninstall 'requests'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
# pip list | grep request
# pip install --ignore-installed requests==2.7.0
# pip install quandl
Collecting quandl
Using cached https://files.pythonhosted.org/packages/47/8c/b61f5c1e9167f4c1c1c2b86991bbbac51a2bd937b36cbc4cc39248dfb2d1/Quandl-3.3.0-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.8 in /usr/lib64/python2.7/site-packages (from quandl) (1.14.3)
Requirement already satisfied: more-itertools in /usr/lib/python2.7/site-packages (from quandl) (4.1.0)
Requirement already satisfied: pyOpenSSL in /usr/lib/python2.7/site-packages (from quandl) (17.5.0)
Requirement already satisfied: pandas>=0.14 in /usr/lib64/python2.7/site-packages (from quandl) (0.22.0)
Requirement already satisfied: ndg-httpsclient in /usr/lib/python2.7/site-packages (from quandl) (0.5.0)
Requirement already satisfied: inflection>=0.3.1 in /usr/lib/python2.7/site-packages (from quandl) (0.3.1)
Requirement already satisfied: requests>=2.7.0 in /usr/lib/python2.7/site-packages (from quandl) (2.7.0)
Requirement already satisfied: python-dateutil in /usr/lib/python2.7/site-packages (from quandl) (2.7.2)
Requirement already satisfied: six in /usr/lib/python2.7/site-packages (from quandl) (1.11.0)
Requirement already satisfied: pyasn1 in /usr/lib/python2.7/site-packages (from quandl) (0.1.9)
Requirement already satisfied: cryptography>=2.1.4 in /usr/lib64/python2.7/site-packages (from pyOpenSSL->quandl) (2.2.2)
Requirement already satisfied: pytz>=2011k in /usr/lib/python2.7/site-packages (from pandas>=0.14->quandl) (2018.4)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != "PyPy" in /usr/lib64/python2.7/site-packages (from cryptography>=2.1.4->pyOpenSSL->quandl) (1.11.5)
Requirement already satisfied: enum34; python_version < "3" in /usr/lib/python2.7/site-packages (from cryptography>=2.1.4->pyOpenSSL->quandl) (1.1.6)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/lib/python2.7/site-packages (from cryptography>=2.1.4->pyOpenSSL->quandl) (0.24.0)
Requirement already satisfied: idna>=2.1 in /usr/lib/python2.7/site-packages (from cryptography>=2.1.4->pyOpenSSL->quandl) (2.6)
Requirement already satisfied: ipaddress; python_version < "3" in /usr/lib/python2.7/site-packages (from cryptography>=2.1.4->pyOpenSSL->quandl) (1.0.22)
Requirement already satisfied: pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->pyOpenSSL->quandl) (2.18)
Installing collected packages: quandl
Successfully installed quandl-3.3.0
sh-4.2#
sh-4.2# pip install sklearn
Collecting sklearn
Downloading https://files.pythonhosted.org/packages/1e/7a/dbb3be0ce9bd5c8b7e3d87328e79063f8b263b2b1bfa4774cb1147bfcd3f/sklearn-0.0.tar.gz
Collecting scikit-learn (from sklearn)
Downloading https://files.pythonhosted.org/packages/c4/b8/eb447f84e0012b0bce97d12d1bc6ea6882b4ed9eb7faaca00e8f627733fb/scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl (12.2MB)
100% |████████████████████████████████| 12.2MB 2.8MB/s
Building wheels for collected packages: sklearn
Running setup.py bdist_wheel for sklearn ... done
Stored in directory: /root/.cache/pip/wheels/76/03/bb/589d421d27431bcd2c6da284d5f2286c8e3b2ea3cf1594c074
Successfully built sklearn
Installing collected packages: scikit-learn, sklearn
Successfully installed scikit-learn-0.19.1 sklearn-0.0
sh-4.2#
sh-4.2# pip install scipy
Box 2:
=====
sh-4.2# pip install quandl
The directory '/home/surikuma/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/surikuma/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting quandl
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/home/surikuma/.cache/pip/wheels/7d/29/b6/60270405bfa9eaca54c2afc87b5501d0afb4906eba07a08bc1'
Consider using the `--user` option or check the permissions.
sh-4.2# exit
exit
$ sudo -H pip install quandl
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these two things:
#1) Respect the privacy of others.
#2) Think before you type.
Password:
Collecting quandl
Downloading https://files.pythonhosted.org/packages/47/8c/b61f5c1e9167f4c1c1c2b86991bbbac51a2bd937b36cbc4cc39248dfb2d1/Quandl-3.3.0-py2.py3-none-any.whl
Collecting numpy>=1.8 (from quandl)
Installing collected packages: numpy, more-itertools, python-dateutil, pytz, pandas, ndg-httpsclient, inflection, certifi, chardet, idna, urllib3, requests, quandl
Found existing installation: chardet 2.2.1
Uninstalling chardet-2.2.1:
Successfully uninstalled chardet-2.2.1
Found existing installation: idna 2.4
Uninstalling idna-2.4:
Successfully uninstalled idna-2.4
Found existing installation: urllib3 1.10.2
Uninstalling urllib3-1.10.2:
Successfully uninstalled urllib3-1.10.2
Found existing installation: requests 2.6.0
Cannot uninstall 'requests'. It is a distutils installed project
and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
$ sudo -H pip install --ignore-installed requests==2.7.0
Collecting requests==2.7.0
Using cached https://files.pythonhosted.org/packages/26/ff/c71b3943bebdd9f7ceb9e137296370587eb0b33fe2eb3732ae168bc45204/requests-2.7.0-py2.py3-none-any.whl
rtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you'll have pyudev 0.15 which is incompatible.
Installing collected packages: requests
Successfully installed requests-2.7.0
$
$ sudo -H pip install quandl
Collecting quandl
Using cached https://files.pythonhosted.org/packages/47/8c/b61f5c1e9167f4c1c1c2b86991bbbac51a2bd937b36cbc4cc39248dfb2d1/Quandl-3.3.0-py2.py3-none-any.whl
Requirement already satisfied: numpy>=1.8 in /usr/lib64/python2.7/site-packages (from quandl) (1.14.3)
Requirement already satisfied: more-itertools in /usr/lib/python2.7/site
Requirement already satisfied: pytz>=2011k in /usr/lib/python2.7/site-packages (from pandas>=0.14->quandl) (2018.4)
rtslib-fb 2.1.63 has requirement pyudev>=0.16.1, but you'll have pyudev 0.15 which is incompatible.
...
Installing collected packages: quandl
Successfully installed quandl-3.3.0
$
$ sudo -H pip install sklearn
$ sudo -H pip install scipy
[surinder_kumar_432@oracle-db-kafka-instance src]$ python ml-stock.py
/usr/lib64/python2.7/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
"This module will be removed in 0.20.", DeprecationWarning)
Open High Low Close Volume Ex-Dividend \
Date
2018-03-21 1586.45 1590.00 1563.17 1581.86 4667291.0 0.0
2018-03-22 1565.47 1573.85 1542.40 1544.10 6177737.0 0.0
2018-03-23 1539.01 1549.02 1495.36 1495.56 7843966.0 0.0
2018-03-26 1530.00 1556.99 1499.25 1555.86 5547618.0 0.0
2018-03-27 1572.40 1575.96 1482.32 1497.05 6793279.0 0.0
Split Ratio Adj. Open Adj. High Adj. Low Adj. Close \
Date
2018-03-21 1.0 1586.45 1590.00 1563.17 1581.86
2018-03-22 1.0 1565.47 1573.85 1542.40 1544.10
2018-03-23 1.0 1539.01 1549.02 1495.36 1495.56
2018-03-26 1.0 1530.00 1556.99 1499.25 1555.86
2018-03-27 1.0 1572.40 1575.96 1482.32 1497.05
Adj. Volume
Date
2018-03-21 4667291.0
2018-03-22 6177737.0
2018-03-23 7843966.0
2018-03-26 5547618.0
2018-03-27 6793279.0
Adj. Close Prediction
Date
2018-03-21 1581.86 NaN
2018-03-22 1544.10 NaN
2018-03-23 1495.56 NaN
2018-03-26 1555.86 NaN
2018-03-27 1497.05 NaN
('confidence: ', '0.9892798601739748')
[1497.72077245 1536.49472298 1547.85950159 1533.99043937 1554.85239524
1570.31316314 1572.26565545 1588.43738523 1611.72934512 1601.14980799
1601.64854243 1581.48693707 1588.70266951 1613.49083275 1628.3785866
1636.18855584 1643.46795652 1672.150493 1692.84266693 1682.00845688
1685.00086357 1675.79019333 1664.49969433 1636.11427624 1680.23635788
1675.30207026 1635.23353243 1583.7259364 1647.71250501 1585.30703072]
[surinder_kumar_432@oracle-db-kafka-instance src]$
[surinder_kumar_432@oracle-db-kafka-instance src]$ cat ml-stock.py
#!/bin/python
import quandl
import pandas as pd
import numpy as np
import datetime
from sklearn.linear_model import LinearRegression
from sklearn import preprocessing, cross_validation, svm
df = quandl.get("WIKI/AMZN")
print 'dataframe'
print(df.tail(60))
df = df[['Adj. Close']]
print 'dataframe : only Adj.close'
print(df.tail(60))
forecast_out = int(30) # predicting 30 days into future
# label column with data shifted 30 units up
df['Prediction'] = df[['Adj. Close']].shift(-forecast_out)
print 'dataframe : with prediction label'
print(df)
# It should be same as
# drop (index or column, 0-index | 1-column))
# X = np.array(df.drop(['Prediction'], 1))
X = np.array(df[['Adj. Close']])
# print 'X : np array after dropping prediction'
print 'X : np array of Adj. Close'
print X
X = preprocessing.scale(X)
print 'X : after preprocessing scale'
print X
X = X[:-forecast_out] # remove last 30 from X
print 'X : remove last 30 from forecast'
print X
y = np.array(df['Prediction'])
print 'y : np array of dataframe prediction'
print y
y = y[:-forecast_out]
print 'y : remove last 30 days from y for forecst'
print y
print 'splited training and testing : data : cross_validation.train_test_split '
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size = 0.2)
print 'X_train'
print X_train
print 'y_train'
print y_train
print 'X_test'
print X_test
print 'y_test'
print y_test
# Training : clf - confidence level functionreplaced by model
model = LinearRegression()
model.fit(X_train,y_train)
# Testing
confidence = model.score(X_test, y_test)
print("confidence: ", confidence)
X_forecast = X[-forecast_out:] # set X_forecast equal to last 30
print 'X : forecast to last 30'
print X
forecast_prediction = model.predict(X_forecast)
print(forecast_prediction)
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.pycurl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
