+4 votes
in Programming Languages by (17.9k points)
How can I check the original encoding of a webpage using the Python BeautifulSoup package?

1 Answer

+2 votes
by (28.5k points)

You can use the original_encoding attribute of the BeautifulSoup object to fetch the original encoding of a web page.

Here is an example:

from bs4 import BeautifulSoup

import urllib.request


# user agent so that request is not declined

user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/2009021910 Firefox/3.0.7'

headers = {'User-Agent': user_agent, }

url = your_url_here


# open the page

request = urllib.request.Request(url, None, headers)

response = urllib.request.urlopen(request)

soup = BeautifulSoup(response, 'html.parser')