Globalization, CultureInfo and Regional settings

[moster67]

Problems when writing software for the international market taking into account Globalization, CultureInfo and Regional settings - and a possible solution how to resolve said problems.

I have searched the forum for this topic but I have found very little information about it. This is strange since Basic4PPC is a forum with many international users/developers. Maybe this is because Erel was already aware about these kind of problems and somehow he has already adapted Basic4PPC to handle them all. I admit that what I am describing below has not been tested in Basic4PPC since my library is being written in VB.Net but nevertheless Basic4PPC is based upon the CompactNet.framework and maybe these problems occur in Basic4PPC as well.

While adding support for other languages than English in my SpellChecking-library, I was informed by some beta-testers about some strange problems. For instance, when running the spellchecker on a Swedish device and using spellchecking in English then certain words, for instance "was", would be indicated by the spellchecker as not correct i.e. not found in the English dictionary. I decided to look further into this issue and at the end I narrowed it down to the regional settings of the device which in above case was set to Swedish. The problem is due to sorting.

The English dictionary furnished with my spellchecking-library has been sorted using English regional settings but when used on a Swedish device with Swedish regional settings, then the binarysearch would not work as expected and the word "was" was not found in the dictionary. This is because the binarysearch takes into consideration the "sorting-order" of the loaded regional-settings and since the Swedish language consider the letter "w" the same as "v", somehow the binarysearch would not work. In the Swedish alphabet, "w" is mixed with "v" as follows:

var
wassenius
vem
........

So binarysearch using Swedish regional settings does not distinguish between "w" and "v". When it read the supplied English dictionary (pre-sorted using English sorting-rules), it considered the following sort-order:

vase
was
verify
when
......

This is the reason, I believe, binarysearch would not find "was".

Now, how can I solve this problem? Well, I added simply an Array.sort(Dict) when loading the English dictionary and now the English dictionary was sorted using Swedish sorting-rules and out of sudden binarysearch worked and the word "was" was considered as a correct word i.e. it was found in the dictionary. However, Array.sort(Dict) on the device using a dictionary with more than 70'000 words took ages to load and I could not consider this is as an acceptable "solution".

I could of course supply the end user with an external program to be run on a desktop which would sort the English dictionary in accordance with his/her regional settings and which most likely would correspond with his/her device Regional settings. Then the user copied the sorted dictionary to his/her device and the problem was resolved. Also in this case, I did not consider this an acceptable solution.

Of course, I could tell the end-user to change the Regional settings into English or I could change the Registry-settings but this would mean that the device would need to be reset (restarted). Also this was not an acceptable solution.

Well, I did some research to see if one could change the regional-settings programatically on the device. If this was possible, I could keep the English-dictionary sorted as it was and I could change the regional settings of the device into English momentarily and then binarysearch would work. I found out that one could use:

Thread.CurrentThread.CurrentCulture = CultureInfo.CreateSpecificCulture("en")
'en=English

only to find out later that this is not possible in the CF.NET.

I gave up but after a few days I began searching again for the solution and I found some code, considered actually a hack and risky but which resolved the problem. Here it is in VB.Net:

'I declare the following statement

Dim myCIintl As New Globalization.CultureInfo("en") 'where "en" is for English

'Then I call the following sub

SetDefaultLocale(myCIintl)

'Here is the code of the sub

Public Shared Sub SetDefaultLocale(ByVal locale As System.Globalization.CultureInfo)
        If Nothing Is locale Then
            Throw New ArgumentNullException("locale")
        End If

        Dim fi As System.Reflection.FieldInfo = GetType(System.Globalization.CultureInfo).GetField _
        ("m_userDefaultCulture", System.Reflection.BindingFlags.NonPublic Or System.Reflection.BindingFlags.Static)
        If Nothing Is fi Then
            Throw New NotSupportedException("Setting locale is not supported in this version of the framework.")
        End If
        fi.SetValue(Nothing, locale)
End Sub

Fortunately, most devices these days have most regional settings already preloaded and as long as the regional setting you wish to use is present on the device, then above code should work. To see which regional settings are supported by the device one could use the code (here in C#) which I found here Compact Framework - Retrieve a list of countries and regions - Stack Overflow.

For the time being, I have implemented the VB.Net-code shown above in my library and I have resolved my problem or at least so it seems. Of course, by inserting another language-code which corresponds to a regional setting supported by the device, you can for instance have Swedish-sorting ("se") on a device with Italian settings.

If above problems are present also in Basic4PPC (I have not tested as I mentioned earlier on), one could perhaps write a library of the cited code for use in Basic4PPC. Example:
1) a function to see if the desired language (Regional setting) is supported by the device
2) If the function returns true, then call the SetDefaultLocale-sub.
3) A sub to reset the original Regional Settings.

Agraham or Erel: If needed, could this work in Basic4PPC?

Above solution could also resolve other problems related to regional settings such a currencies, dates etc.

See also this very interesting article User Interface bug tests, especially the part at "6. Change regional and language settings". Of course, not everything mentioned in this article is applicable to Windows Mobile.

I know that I have written a lot, hopefully not rubbish, but maybe some of it was interesting for you and I hope that I haven't wasted your time.

Take care!

[Zenerdiode]

That's good, but does your 'hack' effect a permanent change to the settings - ultimately a registry change? I understand you change it back but I always think about the unforseen program crash or a forced termination of the process that leaves the registry in an undesirable state.

It would be quite acceptable for me to have the application discover that it is the first time run and re-index the dictionary file - with a .ini file, your own registry hive or even an Input Locale tag stored in the dictionary file, so the program could read that and check with the Device's setting to see if the file needs re-indexed.

[moster67]

First of all, it's not my "hack" - just some code I found googling

I don't think the Regional Settings are changed permanently. Running the code on the emulator, using it with Swedish Regional settings", then "forcing" it into English, ending my application, the screen where you set Regional Settings remained "Swedish".

I will try to experiment a bit more but so far, I haven't ran into any strange side-effects. Maybe before posting my "findings" I should have tried the "hack" a little bit more but as I said, "so far so good". I got so excited that I wanted to share it with you.

Originally Posted by Zenerdiode

That's good, but does your 'hack' effect a permanent change to the settings - ultimately a registry change? I understand you change it back but I always think about the unforseen program crash or a forced termination of the process that leaves the registry in an undesirable state.

It would be quite acceptable for me to have the application discover that it is the first time run and re-index the dictionary file - with a .ini file, your own registry hive or even an Input Locale tag stored in the dictionary file, so the program could read that and check with the Device's setting to see if the file needs re-indexed.

[Erel]

Since Basic4ppc v2.0 (after having a similar bug), all culture affected methods explicitly use the culture-info "en-US".
So internally users will not be affected by different regional settings. However external libraries should manage it.
Here is the code for example of the Decimal library:

    public class DecNumber
{
    public decimal Value = 0;
    private static CultureInfo cul = new CultureInfo("en-US",false);
    public override string ToString()
    {
        return Value.ToString(cul);
    }
    public string ToString(string format)
    {
        return Value.ToString(format,cul);
    }
    public int ToInt ()
    {
        return (int) Value;
    }
    public double ToDouble()
    {
        return (double) Value;
    }

}

I was not aware of your solution, though it may be broken if the private variable 'm_userDefaultCulture' will change its name in a future version.

[Erel]

Here is the code that deals with sorting:

                case 0: //case sensitive
    if (b4p.caseCompare == null)
        b4p.caseCompare = new Comparer(b4p.cul);
    return b4p.caseCompare;
case 1: //not case sensitive
    if (b4p.caseNotCompare == null)
        b4p.caseNotCompare = new CaseInsensitiveComparer(b4p.cul);
    return b4p.caseNotCompare;
default: //numbers
    if (b4p.numbersCompare == null)
        b4p.numbersCompare = new CCompareNumbers();
    return b4p.numbersCompare;

b4p.cul is the same CultureInfo as above.

[moster67]

OK Erel:

Let's say that I use no external libraries - everything is done in Basic4PPC. Then the sorting would be done using "en-US" as mentioned in your previous post which is default for Basic4PPC.

If I wrote an application for the Lithuanian-market and for devices using Lithuanian Regional settings, which perhaps needed to sort a list of words according to the Lithuanian alphabet and which has the following peculiarity:

the letter Y: In most languages Y is sorted like X<Y<Z. In Lithuanian, the sort order is I<Y<J.

Then for instance "ArrayList1.Sort" would sort it according to American English because Basic4PPC uses "en-US". Is this correct?

Just being curious...

[Erel]

You are correct. There are disadvantages for ignoring the user's regional settings.
However as I see it, most of the time it is a source of hard to find bugs.

Here is an interesting related bug. A Turkish user had a problem with using certain controls.
He sent me the source code file which looked fine at first glance.
After a more thorough check I found out that several controls had an unusual 'i' in their name (dotless).
Apparently when you do

"SOME STRING".ToLower()

with a Turkish OS you get "some strıng" which is different than "some string".
This latter caused havoc in other parts of the system.

If someone wants an ArrayList that has a custom cultural sorting it can easily be done with an external library. But it is much safer to use one default culture.

[moster67]

Erel,

During my search, I came across the "Turkish" bug as well. It was mentioned in a MSDN-article if I recall correctly. Edit: Here it is: New Recommendations for Using Strings in .NET 2.0

I think you have adapted the best solution possible because at the end it's not Basic4PPC's fault - it's actually the CF.Net (and Windows Mobile) which is responsible for this since changing Regional Settings programatically is not supported unless using "hacks" like the one I found.

[moster67]

At the end, I decided not to use the so called "hack" mentioned earlier in this thread even though I did not run into any problems despite numerous sessions of testing.

I resolved my problem by using an overload of Array.Binarysearch namely using IComparer of System.Collections.

If anyone is interested (although this is not needed in Basic4PPC where CultureInfo defaults to "en-US") and if you wish to implement it in a library of yours, here is the code:

Create a Class:

Imports System.Globalization
Imports System.Collections

Public Class SortPerGlobalInfo

    Implements IComparer

    Private _culture As CultureInfo

    Public Sub New(ByVal WhichCultureInfo As String)
        _culture = New CultureInfo(WhichCultureInfo)
    End Sub

    Public Function Compare(ByVal x As Object, ByVal y As Object) As Integer _
    Implements System.Collections.IComparer.Compare
        Dim src As String = x.ToString()
        Dim trg As String = y.ToString()
        Dim result As Integer = String.Compare(src, trg, False, _culture)
        Return result
    End Function
End Class

Then you can execute the Array.BinarySearch as follows:

SearchIx = Array.BinarySearch(ArrayName, 0, ArrayName.Length,  _ 
 ValueToSearch, New SortPerGlobalInfo("en-US")) 

'replace "en-US" with country-code needed

This should work with Array.Sort as well.

[doctor]

Hi everybody.
Is there anyway to avoid external libraries? I need to be able to sort in Danish but have no tools for making libraries.
In advance thanks.