SCRAPEGOAT
Scrapegoat is a python library that can be used to scrape the websites from internet based on the relevance of the given topic irrespective of language using Natural Language Processing. It can be mainly used for non-English language to get accurate and relevant scraped text.
Concept
Initially the data is scraped from a website and processed ( to remove English words if the data required is in other language). The BERT model is feed with processed data and topic to compute the cosine similarity of the given topic with each word of the scraped data then mean of cosine similarity scores of is computed. If the mean is greater than threshold then scraped data is generated as output. Also there is a section where we are using Adaptive threshold.
BERT Model
BERT, which stands for Bidirectional Encoder Representations from Transformers, is based on Transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection. The BERT framework was pre-trained using text from Wikipedia. The transformer is the part of the model that gives BERT its increased capacity for understanding context and ambiguity in language. The transformer does this by processing any given word in relation to all other words in a sentence, rather than processing them one at a time. By looking at all surrounding words, the Transformer allows the BERT model to understand the full context of the word, and therefore better understand searcher intent.
Cosine Similarity
Cosine similarity is one of the metrics to measure the text-similarity between two documents irrespective of their size in Natural language Processing. A word can be represented in the vector form, therefore the text documents are represented in n-dimensional vector space. If the Cosine similarity score is 1, it means two vectors have the same orientation. The value closer to 0 indicates that the two documents have less similarity. The Cosine similarity of two documents will range from 0 to 1.
Multi Processing
The multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. The basic ideology of Multi-Processing is that if you have an algorithm that can be divided into different workers (small processors/cores), then you can speed up the program. Machines nowadays come with 4,6,8 and 16 cores, therefore parts of the code can be deployed in parallel.
Using Scrapegoat
The examples/test.py file contains these
url = "https://hindi.newslaundry.com/2021/01/22/loan-developing-countries-and-epidemics#:~:text=120%20%E0%A4%A8%E0%A4%BF%E0%A4%AE%E0%A5%8D%E0%A4%A8%20%E0%A4%94%E0%A4%B0%20%E0%A4%AE%E0%A4%A7%E0%A5%8D%E0%A4%AF%E0%A4%AE%20%E0%A4%86%E0%A4%AF,8.1%20%E0%A4%85%E0%A4%B0%E0%A4%AC%20%E0%A4%A1%E0%A5%89%E0%A4%B2%E0%A4%B0%20%E0%A4%B9%E0%A5%8B%20%E0%A4%97%E0%A4%AF%E0%A4%BE."
topic = "Debt of developing countries"
language = 'hi'
if __name__=="__main__":
from scrapegoat.utils import automate
from scrapegoat.main import getLinkData
text,score = getLinkData(url, topic, language=language, tag='p')
print(text, score)
3 Feb 13, 2022
7 Dec 3, 2022
7 Sep 30, 2022
1 Jan 12, 2022
2 Jul 20, 2022
5 Jun 23, 2022
3 Feb 18, 2022
15 Jul 16, 2022
105 Dec 29, 2022
0 Jan 22, 2022
47 Nov 09, 2022
1 Nov 05, 2021
5 Nov 29, 2022
2 Jul 22, 2022
45 Dec 30, 2022
4 Feb 13, 2022
45 Dec 14, 2022
4 Aug 30, 2022
4 Aug 29, 2022
35 Aug 29, 2022
4 Oct 25, 2022
2k Jan 05, 2023
16 Aug 02, 2022
16 Sep 17, 2022
3 Oct 14, 2022
1 Jan 12, 2022
2 Jan 28, 2022
11 Jul 27, 2022
2k May 06, 2021