Newsscraper - A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

Overview

NewsScraper

A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

🔧 Installation

  1. Clone the repo locally.
  2. Use the package manager pip to install the requirements.
pip install -r requirements.txt

Basic Usage

import NewsScraper

all_data = NewsScraper.fetch_all()
news_data = NewsScraper.fetch_news_data()
crypto_data = NewsScraper.fetch_crypto_data()

fetch_all()

Returns a set of NewsScraper.Result containing fetched results from all available RSS feeds

Can include categories: GLOBAL, US, EU, CRYPTO, BLOCKCHAIN, BTC, ETH, LTC.

fetch_news_data()

Returns a set of NewsScraper.Result containing fetched results from CNN, ABC News, Yahoo News, Fox News RSS feeds

Can include categories: GLOBAL, US, EU.

fetch_crypto_data()

Returns a set of NewsScraper.Result containing fetched results from CoinJournal, Crypto Currency News RSS feeds.

Can include categories: CRYPTO, BLOCKCHAIN, BTC, ETH, LTC.

🔨 Advanced Usage

NewsScraper.Result class

A class used to represent a returned article.

Attributes
  • context : str

    A string describing the category of the article.

    ex. "GLOBAL", "US", "BLOCKCHAIN", "BTC".

  • title : str

    A string containing the name of the article.

  • summary : str

    A string containing the summary of the article.

    NOTE: sometimes it can have the value of "", because the RSS feed didn't provide a summary.

  • content : str

    A string containing the content of the article.

Methods
  • Result.json()

    Returns a dictionary with the attributes of the class formatted in JSON.

    ex.

{
  "context": "global",
  "title": "title of the article",
  "summary": "summary of the article",
  "content": "content of the article"
}

News RSS Feeds

All of these functions return a set of NewsScraper.Result containing fetched results of the described RSS feeds.

fetch_abc()
fetch_cnn()
fetch_yahoo()
fetch_fox_news()

Can include categories: GLOBAL, US, EU.

Alternatively, you can use fetch_news_data() to receive results from all of them.


Crypto RSS Feeds

All of these functions return a set of NewsScraper.Result containing fetched results of the described RSS feeds.

fetch_coinjournal()
fetch_cryptocurrencynews()

Can include categories: CRYPTO, BLOCKCHAIN, BTC, ETH, LTC.

Alternatively, you can use fetch_news_data() to receive results from all of them.

🤝 Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

📝 License

This project is licensed under the MIT license.

Owner
Rokas
Rokas
Raspi-scraper is a configurable python webscraper that checks raspberry pi stocks from verified sellers

Raspi-scraper is a configurable python webscraper that checks raspberry pi stocks from verified sellers.

Louie Cai 13 Oct 15, 2022
Parse feeds in Python

feedparser - Parse Atom and RSS feeds in Python. Copyright 2010-2020 Kurt McKee Kurt McKee 1.5k Dec 30, 2022

An utility library to scrape data from TikTok, Instagram, Twitch, Youtube, Twitter or Reddit in one line!

Social Media Scraper An utility library to scrape data from TikTok, Instagram, Twitch, Youtube, Twitter or Reddit in one line! Go to the website » Vie

2 Aug 03, 2022
a Scrapy spider that utilizes Postgres as a DB, Squid as a proxy server, Redis for de-duplication and Splash to render JavaScript. All in a microservices architecture utilizing Docker and Docker Compose

This is George's Scraping Project To get started cd into the theZoo file and run: chmod +x script.sh then: ./script.sh This will spin up a Postgres co

George Reyes 7 Nov 27, 2022
This is my CS 20 final assesment.

eeeeeSpider This is my CS 20 final assesment. How to use: Open program Run to your hearts content! There are no external dependancies that you will ha

1 Jan 17, 2022
Web Content Retrieval for Humans™

Lassie Lassie is a Python library for retrieving basic content from websites. Usage import lassie lassie.fetch('http://www.youtube.com/watch?v

Mike Helmick 570 Dec 19, 2022
Use Flask API to wrap Facebook data. Grab the wapper of Facebook public pages without an API key.

Facebook Scraper Use Flask API to wrap Facebook data. Grab the wapper of Facebook public pages without an API key. (Currently working 2021) Setup Befo

Encore Shao 2 Dec 27, 2021
Google Maps crawler using Selenium

Google Maps Crawler using Selenium Built as part of the Antifragile Dev Project Selenium crawler that browses Google Maps as a regular user and stores

Guilherme Latrova 46 Dec 16, 2022
Get paper names from dblp.org

scraper-dblp Get paper names from dblp.org and store them in a .txt file Useful for a related literature :) Install libraries pip3 install -r requirem

Daisy Lab 1 Dec 07, 2021
Consulta de CPF e CNPJ na Receita Federal com Web-Scraping

Repositório contendo scripts Python que realizam a consulta de CPF e CNPJ diretamente no site da Receita Federal.

Josué Campos 5 Nov 29, 2021
一款利用Python来自动获取QQ音乐上某个歌手所有歌曲歌词的爬虫软件

QQ音乐歌词爬虫 一款利用Python来自动获取QQ音乐上某个歌手所有歌曲歌词的爬虫软件,默认去除了所有演唱会(Live)版本的歌曲。 使用方法 直接运行python run.py即可,然后输入你想获取的歌手名字,然后静静等待片刻。 output目录下保存生成的歌词和歌名文件。以周杰伦为例,会生成两

Yang Wei 11 Jul 27, 2022
A Scrapper with python

Scrapper-en-python Scrapper des données signifie récuperer des données pour les traiter ou les analyser. En python, il y'a 2 grands moyens de scrapper

Lun4rIum 1 Dec 05, 2021
Amazon scraper using scrapy, a python framework for crawling websites.

#Amazon-web-scraper This is a python program, which use scrapy python framework to crawl all pages of the product and scrap products data. This progra

Akash Das 1 Dec 26, 2021
:arrow_double_down: Dumb downloader that scrapes the web

You-Get NOTICE: Read this if you are looking for the conventional "Issues" tab. You-Get is a tiny command-line utility to download media contents (vid

Mort Yao 46.4k Jan 03, 2023
Searching info from Google using Python Scrapy

Python-Search-Engine-Scrapy || Python-爬虫-索引/利用爬虫获取谷歌信息**/ Searching info from Google using Python Scrapy /* 利用 PYTHON 爬虫获取天气信息,以及城市信息和资料**/ translatio

HONGVVENG 1 Jan 06, 2022
联通手机营业厅自动做任务、签到、领流量、领积分等。

联通手机营业厅自动完成每日任务,领流量、签到获取积分等,月底流量不发愁。 功能 沃之树领流量、浇水(12M日流量) 每日签到(1积分+翻倍4积分+第七天1G流量日包) 天天抽奖,每天三次免费机会(随机奖励) 游戏中心每日打卡(连续打卡,积分递增至最高

2k May 06, 2021
crypto currency scraping

SCRYPTO What ? Crypto currencies scraping (At the moment, only bitcoin and ethereum crypto currencies are supported) How ? A python script is running

15 Sep 01, 2022
Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

Parsel Parsel is a BSD-licensed Python library to extract and remove data from HTML and XML using XPath and CSS selectors, optionally combined with re

Scrapy project 859 Dec 29, 2022
Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data.

Web scraped S&P 500 Data from Wikipedia using Pandas and performed Exploratory Data Analysis on the data. Then used Yahoo Finance to get the related stock data and displayed them in the form of chart

Samrat Mitra 3 Sep 09, 2022
An experiment to deploy a serverless infrastructure for a scrapy project.

Serverless Scrapy project This project aims to evaluate the feasibility of an architecture based on serverless technology for a web crawler using scra

José Ferraz Neto 5 Jul 08, 2022