Tokens and Character Set in Python

Python is a high-level, all-purpose coding language. Its syntax enables developers to articulate their notions in minimal lines of code, referred to as scripts. It was built with a focus on script readability. Character sets and tokens are all included in these scripts. We shall discover more about various character sets and tokens in this tutorial.

Character Set

A character set in Python is a collection of legal characters that a scripting language will recognize when writing a script. We are referring to the Python coding language in this instance. Therefore, the character set in Python is a legitimate collection of characters that the Python language can recognize. These represent the Python scripting language's supported characters. Python is compatible with all ASCII and Unicode characters, including:

Alphabets: These include all the small (a-z) and capital (A-Z) alphabets.
Digits: It includes all the single digits 0-9.
Special Symbols: It includes all the types of special characters," 'l ; : ! ~ @ # $ % ^ ` & * ( ) _ + - = { } [ ] \ .
White Spaces: White spaces are also a part of the character set. These are tab space, newline, blank space, and carriage return.
Other: Python supports all the types of ASCII and UNICODE characters that constitute the Python character set.

Tokens

The smallest distinct element in a Python program is called a token. Tokens are used to construct each phrase and command in a program. The different Python tokens include:

Keywords

In a computer language, keywords are English words with particular importance or meaning. We cannot utilize them for any arbitrary reason, including as names for functions or variables. They are utilized for their unique qualities. The 33 keywords in Python include, among others: try, break, continue, except, or, False, True, class, and, as, assert, print, import, raise, while, for, in, not, if, elif, etc.

Code

# Python program to show how keywords work

string = "This is Python Tutorial for character sets"
# using the keyword for to run a loop
# We will calculate the length of the string using the len keyword
for x in range(len(string)):
    
    # Printing the index of a and o in the string
    # We will do this by using the if, or and print keyword
    if string[x] == 'a' or string[x] == 'o':
        print(x)

Output:

Identifiers

The names assigned to any variable, class, function, list, method, etc., for their recognition are known as identifiers. Python has certain guidelines for naming identifiers and is a case-sensitive programming language. To create an identifier, follow these guidelines: -

Python is case-sensitive, as was already mentioned. So, when defining identifiers, the case is important. Thus, the terms "python" and "Python" are not interchangeable.
A capital alphabet (A-Z), a small alphabet (a-z), or an underscore (_) precedes the word "identifier." No other character can be used as the starting point.
Digits are also allowed to be a member of the identifier, along with the letters and underscores, but they cannot be the initial character.
Any additional special symbols or whitespace must be avoided when creating an identity.
A keyword cannot be an identifier.

Code

# Python program to show how to create identifiers

# Here, String and s are identifiers
String = 'Hello'
s = "Javatpoint"

print(String)
print(s)

# These are also identifiers having digit and underscore
String1 = "World"
String_1 = "!!"

print(String1)
print(String_1)

Output:

Hello
Javatpoint
World
!!

Literals or Values

The constant values or data objects utilized in a source program are known as literals. Various types of literal are supported by Python, including:

String Literals: The Python string literals are represented by text enclosed in solo, double, or triple quotations. Examples include "Computer Programming," "sigma," etc. Multi-line phrases can also be written using triple quotations.

Code

# Python program to show how to create string literals

# Examples of string literals
string = 'Hello'
s = "World"
A = "'Python is a
    high-level and
    general purpose language'"

print(string)
print(s)
print(A)

Output:

Hello
World
Python is a
    high-level and
    general purpose language

Character Literals: Another string literal is called the character literal, which encloses the letter in single or double quotes.

Numeric Literals: These are literals that have been written as numbers. Python supports the following three numerical literals:

Integer Literal: Contains 0 as well as positive and negative values. There are no fractional pieces included. In addition, literals in binary, decimal, octal, and hexadecimal are permitted.
Float Literal: It contains real numbers in the positive and negative ranges. Fractional portions are also a part of it.
Complex Literal: This contains the numerals a+bi, where a stands for the fundamental component and b for the complex component.

Code

# Python program to show how numeric literals work

# Examples of numeric literals
n1 = 50
n2 = 13.3
n3 = -14

# We will print the literal and its type
print(f"literal {n1} is of type {type(n1)}")
print(f"literal {n2} is of type {type(n2)}")
print(f"literal {n3} is of type {type(n3)}")

Output:

literal 50 is of type <class 'int'>
literal 13.3 is of type <class 'float'>
literal -14 is of type <class 'int'>

Boolean Literals

In Python, Boolean tokens only have two possible values. Every one of these is true or false.

Code

# Python program to show how the Boolean literals work

# Examples of the Boolean Literals
v = True
w = False
x = 3
y = (x == 3)
z = True + 10

# Printing the above literals i.e., their boolean value
print(f"The Boolean value of v is {v}")
print(f"The Boolean value of w is {w}")
print(f"The Boolean value of y is {y}")
print(f"The value of z is {z}")

Output:

The Boolean value of v is True
The Boolean value of w is False
The Boolean value of y is True
The value of z is 11

Special Literals

The special literal "None" in Python exists. It is employed to signify emptiness, the lack of values, or nothingness.

Code

# Python program to show how the special literals work

# Examples of the special literal
var = None
print(var)

Output:

None

Literals Collections

List, dictionary, tuple, and sets are examples of Python literal collections.

List: It is a comma-separated list of components enclosed in square brackets. These elements can be of any data type and can be changed.
Tuple: In round brackets, this is similar to a list having comma-separated items or values. The values are fixed and can have any Python data type.
Dictionary: This data structure is an unordered set of key-value combinations.
Set: It is the group of components enclosed in curly braces, "{}"

Code

# Python program to show how the literal collections work

# Examples of literal collections
# A list literal collection
my_list = [23, "Python", 1.2, 'Character']

# A tuple literal collection
my_tuple = (1, 3, 6, 'world')  

# A dictionary literal collection
my_dict = {'a': 1, 'b': 2, 'c': 3}  

# A set literal collection
my_set = {5, 3, 7, 9}

# Printing the literals along with their data structure type
print(f"Literal {my_list} is of type {type(my_list)}")
print(f"Literal {my_tuple} is of type {type(my_tuple)}")
print(f"Literal {my_dict} is of type {type(my_dict)}")
print(f"Literal {my_set} is of type {type(my_set)}")

Output:

Literal [23, 'Python', 1.2, 'Character'] is of type <class 'list'>
Literal (1, 3, 6, 'world') is of type <class 'tuple'>
Literal {'a': 1, 'b': 2, 'c': 3} is of type <class 'dict'>
Literal {9, 3, 5, 7} is of type <class 'set'>

Operators

These represent the tokens in an expression in charge of carrying out an operation. Operands are the elements to which an operation is executed. Operators come in binary and unary forms. Unary operators operate on a single argument, such as complementing and others. At the same time, the operands for binary operators require two.

Code

# Python program to show different types of operators

x = 5
y = 2

# equal to operator
print('x == y =', x == y)

# multiplication operator
print('x * y =', x * y)

# Bitwise & operator
print('x & y =', x & y)

# Bitwise OR operator
print('x | y =', x | y)

# add and assign operator
x += y
print('x += y =', x)

# divide and assign operator
x /= y
print('x /= y =', x)

Output:

x == y = False
x * y = 10
x & y = 0
x | y = 7
x += y = 7
x /= y = 3.5

Next TopicWeb Development Project in Python

← prev next →

Latest Courses

We provides tutorials and interview questions of all technology like java tutorial, android, java frameworks

Contact info

G-13, 2nd Floor, Sec-3, Noida, UP, 201301, India

[email protected].

Python

AI, ML and Data Science

Java

B.Tech and MCA

Web Technology

Software Testing

Technical Interview

Java Interview

Web Interview

Database Interview

Company Interviews

Python Tutorial

Python OOPs

Python MySQL

Python MongoDB

Python SQLite

Python Questions

Plotly

Python Tkinter (GUI)

Python Web Blocker

Python MCQ

Related Tutorials

Python Programs

Python String functions

Python Functions

Tokens and Character Set in Python

Character Set

Tokens

Keywords

Identifiers

Literals or Values

Boolean Literals

Special Literals

Literals Collections

Operators

Latest Courses

Contact info

Follow us

Tutorials

Interview Questions

Online Compiler