Python Requests Module - HTTP Request

In this tutorial, we will learn about the Python request module or how we can handle the requests using the Python requests library. We will also discuss the features of the request. Finally, we will learn how to optimize and customize those features for different situations.

Introduction

Generally, we look for some information from the website. Or we may want to get access to social media accounts, checking mails, watching online videos. We likely send the request using the web browser, and the webserver returns the desired result from the webserver. To get information from the Internet, we need to send the request through the web browser.

For example - We search the Python tutorial on Google. Then the request sends an HTTP request to the Google server, and the server returns the searched result with the status code.

There are two main components to initiate the communication - Client and Server. We use HTTP (Hypertext Transfer Protocol), which enables the communication between a client and server over the Internet.

Python provides the requests module, which allows us to make these requests using the Python script.

Python Request Module

The request module is a standard way for making HTTP requests in Python. It is the most powerful tool of Python that allows us to send requests on the web. It includes many features and methods to send HTTP requests. The system that sends requests is known as the client, and the system that holds the webserver is known as a server.

While working with the requests, we will come across the following methods.

• GET - It is used to request data from a server.
• POST - It is used to submit some data to the server for processing it.

We will discuss more about them in the article later.

The Python request module consists of many simple API that is helpful to handle those requests. These API has many features such as adding headers, sending custom headers, passing parameters with URLs, and many others.

Installation

To start working with the requests, the first step is to install the request module in Python using the following command.

Syntax:

Or

We can also use the Pipenv (Python packaging tool) to install the request module. Type the following command.

Import the requests module in the file to check whether it is successfully installed or not. To do this type the following command.

The GET Request

The GET request is one of the most general-purpose methods of the Python requests module. It specifies that the user is attempting to retrieve data from a specified resource. In other words, it is used to send a request to a URL. To invoke the GET request, we use the following syntax.

Syntax:

Explanation:

In the above method, the url argument is the URL of the particular website where the user sends the request. The param argument is used to send a query string in the dictionary and the args is one of the multiple named arguments.

When the get request is successfully sent, the method will return a requests.Response object. This object saves the response received from the server. We can assign get() request result in variable.

This object saves the response received from the server. Therefore, we can assign get() request results in a variable.

Making a GET Requests

It is quite an easy way to make an HTTP request using the requests module.

Parameters:

• content - It returns the content of the data for the responses.
• status_code - It returns the status of our request. For example - 200 OK means you made successful request, 404 NOT FOUND means resource is not found.
• cookies - It uses to obtaining a CookieJar object having all cookies we get from the server.

Following is the code for making the GET request.

Example -

Output:

ISO-8859-1
200
0:00:01.007097
https://www.javatpoint.com/
[]
text/html;charset=ISO-8859-1

The POST Request

The POST method is used to send information using the post() method. The basic syntax is given below.

Syntax:

Parameters:

• url - The url is a mandatory parameter that indicates URL where we want to send some data.
• data - It specifies a dictionary, file object, or tuple that we want to send to the URL. It is an optional parameter.
• json - This is the JSON object to be sent to the URL.

The POST method returns the requests.Response object.

Status Code

Status code is a response that we get after making the GET or POST request. A status code informs us of the status of the request.

There are many status codes available for the responses. For example, when our request was successful the status code will be 200 or 201, 404 status code is the bad request or the page we are looking for was not found.

We can also use this information to make the decision in our code.

One point should never forget that this method does not verify that the status code is equal to 200. This is because other status codes within the 200 to 400 range, such as 204 NO CONTENT, are also considered successful in the sense.

For example - The 204 will indicate the successful request, but there's no content to return in the message body.

JSON Response

JSON represents the JavaScript Object Notation which is a most popular way to transmitting the data format. JSON data can be easily readable by the web-browser. The data is stored in the dictionary form (key, value pair).

How to convert JSON to Python Dictionary?

We use the r.json() method to create a Python dictionary from the JSON response. Let's see the following example.

Code

Output:

{'args': {}, 'data': '', 'files': {}, 'form': {'password': '1234', 'username': 'mathew'}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Content-Length': '29', 'Content-Type': 'application/x-www-form-urlencoded', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.25.1', 'X-Amzn-Trace-Id': 'Root=1-60b711e8-60f535db7df2f6a61f710e29'}, 'json': None, 'origin': '132.154.100.245', 'url': 'https://httpbin.org/post'}

Storing in a variable

The JSON data can be converted to be Python dictionary and can also store in a variable. Let's understand the following example.

Code

Output:

{'password': '1234', 'username': 'mathew'}

Request Header

We can create a custom header using the get() method. The dictionary of HTTP header is passed in the get() using the headers parameter.

Let's understand the following example.

Code

Output:

Text matches: [{'object_url': 'https://api.github.com/repositories/4290214', 'object_type': 'Repository', 'property': 'description', 'fragment': 'Requests + Gevent = <3', 'matches': [{'text': 'Requests', 'indices': [0, 8]}]}]

Explanation -

In the above code, the Accepts header is a content type that the application can handle. For example, we have used the header value application/vnd.github.v3.text-match+json, a proprietary GitHub Accept header. The content of the header is in JSON format.

Session Management

Requests session management is the process of maintain the session state and persisting the parameters for HTTP Requests. The ability of storing Authentication and credentials, cookies, etc.

Cookies Persistence

Sessions empower the programmed taking care of and the executives of cookies. While making various requests inside a session, the cookies got in the underlying reaction are put away and consequently remembered for ensuing requests.

Connection Persistence

Sessions additionally empower the reuse of the hidden TCP association while making various requests to a similar server. This further develops execution by staying away from the above of laying out another association for each request.

Shared Arguments

Session objects permit you to set normal boundaries or headers that ought to be applied to all requests inside that session. For instance, you can set custom headers like client specialist, acknowledge encoding, or content-type, which will be naturally remembered for all resulting requests made through the session.

Authentication

On the off chance that a site requires validation, session objects give a helpful method for verifying once and keep up with the confirmed state all through the session.

Other Important HTTP Methods

Apart from GET and POST methods, there are few essential methods in HTTP, including PUT, DELETE, HEAD, PATCH, and OPTIONS. These methods are used to create, read, update, and delete (or CRUD) operations. Below is a summarize table of HTTP methods.

Best Practices

• Security contemplations, like SSL/TLS declaration confirmation and dealing with delicate information
• Proficient treatment of enormous responses utilizing streaming strategies
• Streamlining demand execution, for example, utilizing association pooling or tenacious associations
• Taking care of rate restricts and executing backoff procedures
• Composing perfect and viable code by using capabilities, classes, and modules really
• Keeping awake to-date with the most recent adaptation of the Solicitations library and its conditions.

Conclusion

We have discussed the basic details about the Python request module which will be very helpful to make basic server request.

The main point is the get() method is less secure than the post() method because the request can only be passed through the URL. So, the sensitive password can be breached by the hackers.

We have also mentioned the important HTTP methods and how we can send request using the custom header. If you want learn more about the request module then you can visit its well-written request documentation.