C++ - How to read Unicode characters( Hindi Script for e.g. ) using C++ or is there a better Way through some other programming language?


I have a hindi script file like this:

3.  भारत का इतिहास काफी समृद्ध एवं विस्तृत है।

I have to write a program which adds a position to each and every word in each sentence. Thus the numbering for every line for a particular word position should start off with 1 in parentheses. The output should be something like this.

3.  भारत(1) का(2) इतिहास(3) काफी(4) समृद्ध(5) एवं(6) विस्तृत(7) है(8) ।(9)

The meaning of the above sentence is:

3.  India has a long and rich history.

If you observe the '।'( which is a full stop in hindi equivalent to a '.' in English ) also has a word position and similarly other special symbols would also have as I am trying to go about English-Hindi Word alignment( a part of Natural Language Processing ( NLP ) ) so the full stop in english '.' should map to '।' in Hindi. Serial nos remain as it is untouched. I thought reading character by character could be a solution. Could you please help me with how to go about in C++ if its easy or if easier could you suggest some other way through some other programming language may like Python/Perl..?

The thing is I am able to get word positions for my English text using C++ as I was able to read character by character using ASCII values in C++ but I don't have a clue to how to go about the same for the hindi text.

The final aim of all this is to see which word position of the English text maps to which postion in Hindi. This way I can achieve bidirectional alignment.

Thank you for your time...:)

2/18/2010 5:39:16 PM

Accepted Answer

I would seriously suggest that you'd use Python for an applicatin like this. It will lift the burden of decoding the strigns (not to mention allocating memory for them and the like). You will be free to concentrate on your problem, instead of problems of the language.

For example, if the sentence above is contained in an utf-8 file, and you are uisng python2.x. If you use python 3.x it is even more readible, as you don't have to prefix the unicode strings with 'u" ', as in this example (but you will be missing a lot of 3rd party libraries:

separators = [u"।", u",", u"."]
text = open("indiantext.txt").read()
#This converts the encoded text to an internal unicode object, where
# all characters are properly recognized as an entity:
text = text.decode("utf-8")

#this breaks the text on the white spaces, yielding a list of words:
words = text.split()

counter = 1

output = ""
for word in words:
    #if the last char is a separator, and is joined to the word:
    if word[-1] in separators and len(word) > 1:
        #word up to the second to last char:
        output += word[:-1] + u"(%d) " % counter
        counter += 1
        #last char
        output += word[-1] +  u"(%d) " % counter
        output += word + u"(%d) " % counter
    counter += 1

print output

This is an "unfolded" example, As you get more used to Python there are shorer ways to express this. You can learn the basics of teh language in just a couple of hours, following a tutorial. (for example, the one at http://python.org itself)

2/18/2010 1:33:38 PM

Wow, already 6 answers and not a single one actually does what mgj wanted. jkp comes close, but then drops the ball by deleting the daṇḍa.

Perl to the rescue. Less code, fewer bugs.

use utf8; use strict; use warnings;
use Encode qw(decode);
my $index;
join ' ', map { $index++; "$_($index)" } split /\s+|(?=।)/, decode 'UTF-8', <>;
# returns भारत(1) का(2) इतिहास(3) काफी(4) समदध(5) एव(6) विसतत(7) ह(8) ।(9)

edit: changed to read from STDIN as per comment, added best practices pragmas

Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow