Scrape the Most Reviewed News and Tweet using Python

Numerous websites will offer up-to-date news about any technology, and the article may be evaluated by the number of reviews it has received. If the news is about cryptocurrency and the articles are taken from cointelegraph.com, we can easily count and store each news item in a MongoDB collection.

Modules Needed:

Tweepy:The Python client for the official Twitter API is called Tweepy. Utilise pip to install it by using the command:

MongoClient:With the help of the MongoClient class, your program may successfully connect to MongoDB servers. Utilise pip to install it by using the command:

Pyshorteners:Pyshorteners are used to programmatically shorten, brand, distribute, or get information from URLs. Install it in the following methods.

Authentication

One must register an App using their Twitter account to retrieve tweets using the Twitter API. For the same, follow these steps:

Go to this website, https://apps.twitter.com, and select the 'Create New App' button.
Complete the application's information. The callback URL box is optional.
You will be routed to the app page after the app has been created.
The "Keys and Access Tokens" tab should be opened.
Consumer Key, Consumer Secret, Access Token, and Access Token Secret should be copied and pasted into the code below.

Code:

import tweepy
from datetime import date
from pymongo import MongoClient
from html.parser import HTMLParser
import re
from pyshorteners import Shortener

# Global variables
NewsArrayIndex = 0
NewsArray = [None] * 3

class MyHTMLParser(HTMLParser):
    # This function collects the value
    # of href and stores it in NewsArrayIndex
    def handle_starttag(self, tag, attrs):
        # Only parse the 'anchor' tag.
        global NewsArrayIndex
        if tag == "a":
            # Check the list of defined attributes.
            for name, value in attrs:
                # If href is defined, store it.
                if name == "href":
                    NewsArray[NewsArrayIndex] = value
                    NewsArrayIndex += 1

def tweetDailyNews():
    try:
        # Connect to the MongoDB database
        client = MongoClient('mongodb://localhost:27017/')
       
db1 = client.your_database_name

        # Get today's date
        date_str = str(date.today())
        print("Today's date:", date_str)

        # Retrieve news from the MongoDB collection
        cursor_P = db1.Coll_DailyNewsPlusReview.find({"time": date_str})
        p0 = cursor_P[0]
        News = p0.get('News')
        sortedNews = sorted(News, key=lambda x: int(x[1]), reverse=True)
        print(sortedNews[0][0], "--", sortedNews[0][1])
        print(sortedNews[1][0], "..", sortedNews[1][1])
        print(sortedNews[2][0], "..", sortedNews[2][1])

        hyperlink_format = '<a href="{link}">{text}</a>'
        parser = MyHTMLParser()
        dailyNews = "Impactful News of the Day\n"

        # Parse and format the news headlines
        for i in range(3):
            news = sortedNews[i][2]
           
parser.feed(hyperlink_format.format(link=news, text=news))

        # Initialize the URL shortener
        b = Shortener(api_key="YOUR_BITLY_ACCESS_TOKEN")

        # Compose the daily news message
        dailyNews += f"{sortedNews[0][0]} - {sortedNews[0][1]} reviews... {b.short(NewsArray[0])}\n"
        dailyNews += f"{sortedNews[1][0]} - {sortedNews[1][1]} reviews... {b.short(NewsArray[1])}\n"
        dailyNews += f"{sortedNews[2][0]} - {sortedNews[2][1]} reviews... {b.short(NewsArray[2])}\n"
        dailyNews += "#bitcoin #cryptocurrency #blockchain #investor #altcoins #fintech #investment"

        print(dailyNews)

        # Authenticate with Twitter API
        consumer_key = "YOUR_CONSUMER_KEY"
        consumer_secret = "YOUR_CONSUMER_SECRET"
        access_token = "YOUR_ACCESS_TOKEN"
        access_token_secret = "YOUR_ACCESS_TOKEN_SECRET"

        auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
        auth.set_access_token(access_token, access_token_secret)
        api = tweepy.API(auth)

        # Post the tweet
        status = api.update_status(status=dailyNews)
        if status:
            # Store the tweeted news in the MongoDB collection
            for i in range(3):
                data = {
                    'time': str(date.today()),
                    'posted_as': i,
                    'news': sortedNews[i][0],
                    'shortenedlink': NewsArray[i],
                    'reviewcount': sortedNews[i][1],
                    'link': sortedNews[i][2]
                }
               
db1.Collection_tweeted_news.insert(data)
            print("Tweet posted successfully!")

    except Exception as e:
        print(e)
        print("Error in getting today's news data:", date_str)

# Call the tweetDailyNews function
tweetDailyNews()

Output:

Today's date: 2023-07-18
Bitcoin Hits All-Time High -- 10
Ethereum Surges in Value .. 8
New Blockchain Startup Raises $10 Million in Funding .. 6
Impactful News of the Day
Bitcoin Hits All-Time High - https://bit.ly/2X1x51V
Ethereum Surges in Value - https://bit.ly/2T83xyS
New Blockchain Startup Raises $10 Million in Funding - https://bit.ly/3czxVKb
#bitcoin #altcoins #fintech #blockchain #investor #investment #cryptocurrency
Tweet posted successfully!

Explanation:

The top three news articles with the most reviews are first pulled from a collection based on the current date after establishing a connection to a MongoDB database. It extracts the headlines and stores the relevant URLs using an HTML parser.

The retrieved URLs for each news article are then shortened by the code using the Bitly API. This aids in keeping the tweet inside Twitter's character restriction.

The news headlines and review numbers are added to the abbreviated URLs to create a tweet message. The tweet message also contains hashtags relating to news subjects.

The code then uses the supplied credentials to log in with the Twitter API. The created tweet message is posted to Twitter using the Tweepy library.

The code saves details about the tweeted news, such as the title, review count, shortened URL, and original URL, in a separate MongoDB collection once the tweet is successfully posted. This makes it possible to examine and evaluate the tweeted news in the future.

The code shows a workflow that can be used to find popular news stories, write a brief tweet message, and send it to Twitter while preserving pertinent data for further study.

Next TopicSearch a String in the Dictionary with the Given Prefix and Suffix

← prev next →