Archive for July, 2009

Android MediaScanner Cannot Display Chinese Characters

Tuesday, July 28th, 2009

Problem Description

On Android Develop Phone 1 (ADP1) running HTC-provided system image of Android 1.5, most of the Chinese characters cannot be diplayed in the library of the music player. This is actually due to the media scanner in OpenCore fails to resolve the proper character encoding for Chinese characters in ID3 tags of MP3 files. The problem is also described at http://code.google.com/p/android/issues/detail?id=2777.

In the parseMP3() function of “external/opencore/android/mediascanner.cpp”, PVID3ParCom from “external/opencore/fileformats/id3parcom/src/pv_id3_parcom.cpp” is used to parse ID3 tag of MP3 files to get frames information. Then the key string is read from each frame. The key string may look like “artist;valtype=char*;char-encoding=UTF8” if the frame contains the name of the artist and the characters are encoded as UTF8. It may also look like “album;valtype=char*” if the frame contains the name of the album and the characters are encoded as one of the native charsets, which can be ISO 8859-1 — a standard character encoding for the Latin alphabet, GBK for simplified Chinese, Big5 for traditional Chiense, etc.

However, in the implementation of parseMP3() function, all the non-UTF8 native charsets are treated as ISO 8859-1 and converted to UTF8 as if they are ISO 8859-1. Then in the endFile() function, possibleEncodings() function is used to “compute a bit mask containing all possible encodings” (quoted from the comment in the original source file, so are the following two quotations). “If the locale encoding matches”, then convertValues() function is called to “untangle the utf8 and convert it back to the original bytes”, which was mistaken to be treated as ISO 8859-1 before. Then the original bytes are converted to UTF8 again by using the charset converter from ICU library. This time the conversion is based on more proper esitimation of the charset.

Solution

One obvious solution is to change the locale of your ADP1. For me, most of my music collection use simplified Chinese in the ID3 tags, and most of them are using GBK encodings instead of UTF8. But HTC-provided system image comes with en-US as the only available locale. In order to add more locales, it’s required to build the source code yourself and flash the images to ADP1. For getting and building the source code, flashing the phone, and including Google applications, please refer to Johan de Koning’s 5-post series of “Building Android 1.5”, which starts from http://www.johandekoning.nl/index.php/2009/06/07/building-android-15-build-environment/.

After booting up the ADP1 with the system image you build, change your locale in the Settings. The mediascanner service might not be able to update the ID3 tag information yet because it will skip the media files that haven’t been modified since last time. One way is to force the “scanAlways” parameter in doScanFile() function from “frameworks/base/media/java/android/media/MediaScanner.java” to be “True” in the code. However, it’s better to only use this trick temporarily, otherwise it is too resource consuming to re-scan all media files every time.

In case you don’t want to change your locale but still would like your media scanner to be able to read Chinese, Japanese or Korean, then an alternative way is to apply some patch to “external/opencore/android/mediascanner.cpp” like this:


diff -Naur src-1.5-git-orig/external/opencore/android/mediascanner.cpp src-1.5-git/external/opencore/android/mediascanner.cpp
--- src-1.5-git-orig/external/opencore/android/mediascanner.cpp	2009-07-24 13:28:09.000000000 +0200
+++ src-1.5-git/external/opencore/android/mediascanner.cpp	2009-07-24 13:30:02.000000000 +0200
@@ -1008,10 +1008,16 @@
         // compute a bit mask containing all possible encodings
         for (int i = 0; i < mNames->size(); i++)
             encoding &amp;= possibleEncodings(mValues->getEntry(i));
-        
-        // if the locale encoding matches, then assume we have a native encoding.
-        if (encoding &amp; mLocaleEncoding)
-            convertValues(mLocaleEncoding);
+            
+        /* FIX: convert to utf8 accordingly disregard of the current locale */
+        if (encoding &amp; kEncodingGBK)
+            convertValues(kEncodingGBK);
+        else if (encoding &amp; kEncodingBig5)
+            convertValues(kEncodingBig5);
+        else if (encoding &amp; kEncodingEUCKR)
+            convertValues(kEncodingEUCKR);
+        else if (encoding &amp; kEncodingShiftJIS)
+            convertValues(kEncodingShiftJIS);
         
         // finally, push all name/value pairs to the client
         for (int i = 0; i < mNames->size(); i++) {

Note this is not a proper bug fix, because charset can never be easily estimated. It is not uncommon that the same string can be estimated as GBK, Big5, EUCKR or ShiftJIS at the same time. Depending on the proportion of your music, the sequence can be adjusted in if-statement in the patch above to reflect the charset priorities.

Nevertheless the best way to solve such problems is to convert ID3 tags of all your media files to UTF8 beforehand. Unicode or UTF8 is really the way to go.

Android on the Beagle Board Gains Internet Access by Connecting with the Host Machine Acting as NAT through Ethernet over USB

Tuesday, July 14th, 2009

Brief introduction

There are already solutions to share the mobile network connections of Android phone by the host PC via USB cable, so that browser in the host PC can surf Internet by using Android’s mobile network subscription. This is called tethering. However, today I’m going to present a complete solution to connect Android with the host PC by using ethernet via USB and enable Andriod to access the Internet through the Internet connection of the host PC.

I’m using the Beagle Board (http://www.beagleboard.org) as the platform to run Android (http://labs.embinux.org/index.php/Android_Porting_Guide_to_Beagle_Board).

Enable ethernet over USB in the linux kernel

– go to the “kernel” directory in the android source, use menuconfig (CC_PATH is mentioned in the link above)

make CROSS_COMPILE=$CC_PATH menuconfig

– enable usb-eth by choosing “Ethernet Gadget (with CDC Ethernet support)” for “Device Drivers -> USB support -> USB Gadget Support -> USB Gadget Drivers”, exit and save
– re-build the uImage

make CROSS_COMPILE=$CC_PATH uImage

the previous built uImage might need to be removed first

rm arch/arm/boot/uImage

– copy the uImage to the first partition of the sdcard
– make sure the kernel on the host machine is built with usbnet (“Device Drivers -> Network device support -> USB Network Adapters -> Multi-purpose USB Networking Framework” in the menuconfig). Most GNU Linux distributions already come with usbnet built-in, including Ubuntu in my case

Config ethernet over USB

– refer to http://benno.id.au/blog/2007/11/14/android-busybox to download and install the static-linked busybox built for ARM. It’ll be assumed busybox is installed at “/data/busybox” below
– in minicom connected with Beagle Board, wait until Android is fully booted up
– use ifconfig from busybox mentioned above

/data/busybox/ifconfig usb0 192.168.1.2 up

– config the gateway as well

route add default gw 192.168.1.1 dev usb0

– similarly, type the following command in the shell of the host machine

sudo ifconfig usb0 192.168.1.1 up

Use ADB

– the ADB daemon may require to be terminated first if already running:

ADBHOST=192.168.1.2 adb kill-server

– ADBHOST requires to be set if ADB works over ethernet

ADBHOST=192.168.1.2 adb shell

Config the host machine to act as NAT

– load necessary kernel modules

sudo modprobe ip_tables
sudo modprobe iptable_nat
sudo modprobe ip_nat_ftp
sudo modprobe ip_nat_irc

– enable IP forwarding

echo "1" | sudo dd of=/proc/sys/net/ipv4/ip_forward

– in case the IP address is assigned dynamically

echo "1" | sudo dd of=/proc/sys/net/ipv4/ip_dynaddr

– enable SNAT functionality on the corresponding network interface

sudo iptables -t nat -A POSTROUTING -o [NETWORK_INTERFACE] -j MASQUERADE

Note: [NETWORK_INTERFACE] is the corresponding network interface your host is using to access the Internet.

Proxy

It’s not uncommon that computer access Internet via network proxy. But WIFI and other network connections in Android require direct network connection and network proxy is not supported.

The code handling network proxy in Android is mainly in “frameworks/base/core/java/android/net/http/RequestQueue.java”. In the function setProxyConfig(), getHost() from “frameworks/base/core/java/android/net/Proxy.java is invoked to get the proxy host. However, as illustrated in “frameworks/base/core/java/android/net/ConnectivityManager.java”, Android seems to only support WIFI and mobile network as network connection type. WIFI happens to be the default connectivity type, the reason of which is shown below taken from the file “[ANDROID_SRC_DIR]/frameworks/base/core/java/android/net/http/RequestQueue.java”:


                /*
                 * Initializing the network type is really unnecessary,
                 * since as soon as we register with the NCL, we'll
                 * get a CONNECTED event for the active network, and
                 * we'll configure the HTTP proxy accordingly. However,
                 * as a fallback in case that doesn't happen for some
                 * reason, initializing to type WIFI would mean that
                 * we'd start out without a proxy. This seems better
                 * than thinking we have a proxy (which is probably
                 * private to the carrier network and therefore
                 * unreachable outside of that network) when we really
                 * shouldn't.
                 */

Then no proxy is set in case of WIFI network, which means WIFI requires direct Internet connection as shown below also taken from the file “frameworks/base/core/java/android/net/http/RequestQueue.java”. The way to solve it is to comment out the “if (mNetworkStateTracker.getCurrentNetworkType() == ConnectivityManager.TYPE_WIFI)” statement:


    private synchronized void setProxyConfig() {
        /* get rid of this branch */
        if (mNetworkStateTracker.getCurrentNetworkType() == ConnectivityManager.TYPE_WIFI) {
            mProxyHost = null;
	    } else {
            String host = Proxy.getHost(mContext);
            if (HttpLog.LOGV) HttpLog.v("RequestQueue.setProxyConfig " + host);
            if (host == null) {
                mProxyHost = null;
            } else {
                mActivePool.disablePersistence();
                mProxyHost = new HttpHost(host, Proxy.getPort(mContext), "http");
	        }
	     }
     }

Still in “frameworks/base/core/java/android/net/http/RequestQueue.java”, setProxyConfig() is only invoked when the NetworkStateTracker sends out message that the network state is changed. And setProxyConfig() is not called during initialization because as soon as Android registers with the NCL, a CONNECTED event is received for the active network. However, this is only true to mobile network, not for WIFI or wired networks. The way to fix this is to invoke setProxyConfig() at the end of enablePlatformNotifications(), so that proxy is properly configured at initialization regardless of what network type it is. The corresonding code is illustrated below:


    public synchronized void enablePlatformNotifications() {
        if (HttpLog.LOGV) HttpLog.v("RequestQueue.enablePlatformNotifications() network");

        if (mProxyChangeReceiver == null) {
            mProxyChangeReceiver =
                    new BroadcastReceiver() {
                        @Override
                        public void onReceive(Context ctx, Intent intent) {
                            setProxyConfig();
                        }
                    };
            mContext.registerReceiver(mProxyChangeReceiver,
                                      new IntentFilter(Proxy.PROXY_CHANGE_ACTION));
        }

        /* Network state notification is broken on the simulator
           don't register for notifications on SIM */
        String device = SystemProperties.get("ro.product.device");
        boolean simulation = TextUtils.isEmpty(device);

        if (!simulation) {
            if (mNetworkStateTracker == null) {
                mNetworkStateTracker = new NetworkStateTracker(mContext);
            }
            mNetworkStateTracker.enable();
        }
        
        /* call setProxyConfig() as part of the initialization.
          Otherwise, setProxyConfig() will only be invoked in case of
          network state change, and only mobile network get CONNECTED
          event after registration with NCL */
        setProxyConfig();
    }

Then in Proxy.java, it uses “frameworks/base/core/java/android/provider/Settings.java”, which is actually an utility wrapper of sqlite database to get the proxy and port settings. The corresponding table is “secure” from “/data/data/com.android.providers.settings/databases/settings.db”. The “name”, “value” pair should be “http_proxy”, “[HTTP_PROXY_HOST]:[HTTP_PROXY_PORT]”. The way to append the record is:

adb shell sqlite3 /data/data/com.android.providers.settings/databases/settings.db "INSERT INTO secure VALUES(99,'http_proxy','[HTTP_PROXY_HOST]:[HTTP_PROXY_PORT]');"

If nothing is got from the above database, then the system property “net.gprs.http-proxy” is used instead. The way to put it is to add the following command in “on boot” trigger in “init.rc”:

setprop net.gprs.http-proxy [HTTP_PROXY_HOST]:[HTTP_PROXY_PORT]

One more thing to mention is DNS has to be configured to resolve domain names. The way to do that is to add the following commands in “on boot” trigger in “init.rc” as well:

setprop net.dns1 [DNS_SERVER_1_IP]
setprop net.dns2 [DNS_SERVER_2_IP]