Raspberry pi Stock data collection

I made a stock collection program on my raspberry pi that web scrapes data from a website to give me stock updates every 5 minutes. All the details are stored as a file system and it is managed solely by a python program and crontab.

The script is in python 3 and it is used to track down prices of BSE (Bombay stock exchange) in India.

The whole script is available on my GitHub repository. Link:

To run it just place the script in a folder and add the cron entry as stated in the README.md

python3 stock.py ^BSESN

The data is first extracted via BeautifulSoup and the following tags are

title = ['h1', {'data-reactid': '7'}]
quote_header = ['div', {'id': 'quote-header-info'}]
present_price = ['span', {'class': 'Trsdu(0.3s)'}]
opening_price = ['td', {'data-test': 'OPEN-value'}]
previous_close_price = ['td', {'data-test': 'PREV_CLOSE-value'}]
day_range_price = ['td', {'data-test': 'DAYS_RANGE-value'}]
price_change = ['span', {'data-reactid': '17'}]

title : gives the title of the Quote
quote_header : It is complicated to extract present_price at once, so this is used
present_price : The current price of the stock
opening_price : The price at which the stock had opened on that day
previous_close : The price at which the market was shut
day_range : It give the range of maximum and minimum on that day
price_change : the percent rise or fall of the stock rate

The following directory tree is produced by the code

├── 10_11_2017
│   ├── graph_output.txt
│   ├── logfile.txt
│   └── output.txt
├── 10_1_2018
│   ├── graph_output.txt
│   ├── logfile.txt
│   └── output.txt

graph_output.txt : contains time series data which can be used for plotting
logfile.txt : contains the appication log for that day
output.txt : contains the latest extracted data

I've put the code to run automatically by using a crontab entry :

cron : */5 8-16 * * 1-5 python3 stock.py ^BSESN

 This makes it run every 5 minutes from Monday to Friday

 You can find my code on my github link:
https://github.com/divyaksh-shukla/Yahoo_stock_data_scraper

Comments

  1. NandhaKumar22 December 2020 at 17:59


    Very Nice Article:
    Hi Readers, If you wanna track your expenses and save money online or offline, Just Download Timelybills.app. It will show you all the ways to manage your expenditure without consuming your whole day. Please feel free to check out here. Thank you!

    Timelybills.app

    ReplyDelete

Post a Comment

Popular posts from this blog

Running Oracle Outsidein Technology (OIT) in Docker

Oracle Outsidein Technology provides a set of tools and SDKs to convert many forms of data to readable documents. It also holds data extraction and reduction capabilities. Used by multiple firms and tech fronts for more than 3 decades, OIT is a vast product here to stay for a long time. Here, I made a starting attempt on using OIT's Image export using docker. Created a small docker file with the oracle-java8 base image, loaded the image export jars and dependencies, created a few mounting volumes and ran it on a single sample pdf file. An enriching and learning experience for me and my father all the same. This blog shows the steps I took to get image export working on my machine. Prerequisites Docker (to be installed on your machine, you can go to docker's website and download it for your OS windows/linux/mac) OIT Image Export SDK (Get it here ) The image export SDK also contains some sam...
Read more

Arduino with 7-Segment Display (LT542)

Image
A 7-segment display is a LED-LCD display with 8 LCD cells are controlled by 8 pins. Usually a 7-segment display has 10 pins, 2 are common pins and the rest 8 control each LCD cell. Now, a 7-segment display is of 2 types, common cathodic and common anodic display. While the common cathodic display has its common pins hooked up to the ground(GND), the common anodic display has its common pins hooked up to high voltage(+5). A diagram explaining this is given. I have used a display numbered LT542 which is a common anodic display. This project is aimed to control the LT542 to display each number from 0 to 9 at a second's gap. MATERIALS REQUIRED: Arduino Uno LT542 Jumper wires (male-to-male) Breadboard STEPS: Wire the setup as shown in the schematic and pictures. Copy the code given and paste it into the arduino IDE. Plug in your board tho the computer using a USB cable. Upload the code. CODE:  /**   PINOUT DIAGRAM FOR THIS CODE   ...
Read more

NAS (National Achivement Survey) data extraction

I had recently been to a datathon (A hackathon related to data science) in PES University, Bangalore. There my team was given a task to extract data from the National Achievement Survey - 2017 conducted by NCERT. NAS collects data about CBSE schools across states and districts of India to collect data about student achievements and their overall reports. This data is present in PDF formats. We were tasked to extract data from PDF and tabulate it. $ pdftotext is a linux utility to convert pdf to text. By supplying a -layout option the default layout of the data is mostly preserved. I made a python script (pdf_convert.py) to convert the pdf data to text files sequentially. Next I made a script to convert the text files to csv data. So each text file was turned to a record (row) in the csv file. Here is a snapshot of the directory structure of PDF file that we got. . ├── Andaman & Nicobar Islands │   ├── Andaman │   │   ├── Andamans Class - ...
Read more