How to handle response encoding from urllib.request.urlopen()


Question

I'm trying to search a webpage using regular expressions, but I'm getting the following error:

TypeError: can't use a string pattern on a bytes-like object

I understand why, urllib.request.urlopen() returns a bytestream and so, at least I'm guessing, re doesn't know the encoding to use. What am I supposed to do in this situation? Is there a way to specify the encoding method in a urlrequest maybe or will I need to re-encode the string myself? If so what am I looking to do, I assume I should read the encoding from the header info or the encoding type if specified in the html and then re-encode it to that?

1
40
2/13/2011 2:08:22 AM

Accepted Answer

You just need to decode the response, using the Content-Type header typically the last value. There is an example given in the tutorial too.

output = response.decode('utf-8')
40
2/13/2011 2:09:29 AM

As for me, the solution is as following (python3):

resource = urllib.request.urlopen(an_url)
content =  resource.read().decode(resource.headers.get_content_charset())

Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow
Icon