+4 votes
in Programming Languages by (14.6k points)

When I run the following 'BeautifulSoup' code, I get a warning message. How can I fix the warning message?

request = urllib.request.Request(url, None, headers)
response = urllib.request.urlopen(request)
soup = BeautifulSoup(response)

The UserWarning is as follows:

UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 33 of the file test.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

1 Answer

0 votes
by (25.6k points)

You have to mention an HTML parser with BeautifulSoup. You can use either 'lxml' or 'html5lib'.

Make the following change in your code and it should work.

request = urllib.request.Request(url, None, headers)
response = urllib.request.urlopen(request)
soup = BeautifulSoup(response, 'html5lib')

If 'lxml' or 'html5lib' is not installed on your system, you can install using the following commands:

$ pip install html5lib
$ pip install lxml

You can find details about BeautifulSoup here.

...