How to replace unicode characters by ascii characters in Python (perl script given)?


I am trying to learn python and couldn't figure out how to translate the following perl script to python:

#!/usr/bin/perl -w                     

use open qw(:std :utf8);

while(<>) {

The script just changes unicode umlauts to alternative ascii output. (So the complete output is in ascii.) I would be grateful for any hints. Thanks!

4/23/2010 6:01:36 PM

Accepted Answer

  • Use the fileinput module to loop over standard input or a list of files,
  • decode the lines you read from UTF-8 to unicode objects
  • then map any unicode characters you desire with the translate method would look like this:

#!/usr/bin/env python2.6
# -*- coding: utf-8 -*-

import fileinput

table = {
          0xe4: u'ae',
          ord(u'ö'): u'oe',
          ord(u'ü'): u'ue',
          ord(u'ß'): None,

for line in fileinput.input():
    s = line.decode('utf8')
    print s.translate(table), 

And you could use it like this:

$ cat utf8.txt 
sömé täßt
sömé täßt
sömé täßt

$ ./ utf8.txt 
soemé taet
soemé taet
soemé taet
  • Update:

In case you are using python 3 strings are by default unicode and you dont' need to encode it if it contains non-ASCII characters or even a non-Latin characters. So the solution will look as follow:

line = 'Verhältnismäßigkeit, Möglichkeit'

table = {
         ord('ä'): 'ae',
         ord('ö'): 'oe',
         ord('ü'): 'ue',
         ord('ß'): 'ss',


>>> 'Verhaeltnismaessigkeit, Moeglichkeit'
5/4/2016 2:15:12 PM

For converting to ASCII you might want to try ASCII, Dammit or this recipe, which boils down to:

>>> title = u"Klüft skräms inför på fédéral électoral große"
>>> import unicodedata
>>> unicodedata.normalize('NFKD', title).encode('ascii','ignore')
'Kluft skrams infor pa federal electoral groe'

Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow