USB OTG code issues solved

Just solved the QTD_STS_HALT issues with the USB OTG port. Turned out that this was related to OTG shutting down the USB port too fast, while there was still pending I/O going over the bus. The fixed has already been checked in the repo.

I hope that this also fixes the memory corruption issues. While I cannot tell for sure, I assume that the aborted transactions(the transactions that raised the QTD_STS_HALT messages) have been resumed by the hardware after the bus has become active again, thereby writing(via DMA) to memory blocks that were already assigned for other purposes because they had already been freed by the kernel.

Next thing on the list is going to be to make this custom Linux on the Kindle usable(for example by installing QT Embedded) and of course to test the kernel with the standard Amazon distribution such that normal users can also make use of this.

USB Host on the Kindle 4

Yes, it has been done! In the past days I spent a lot of energy trying to get the USB host to work on the Kindle. But now it works and I am able to attach both hosts(PCs) and peripherial devices(USB hubs etc.) to the Kindle. Below is a picture of the Kindle being connected to a USB hub, to which a USB network adapter and a sound card are attached:

Kindle 4 with USB network adapter and sound card

Kindle 4 with USB network adapter and sound card

This was done using the custom kernel that I finished earlier and that is available from here

So, is this ready for production use?

Unfortunately not, no. I’m working towards this point, but at the moment this is still a highly experimental driver. Basic stuff should work, but there are still some issues at this point:

  • During plug and unplug of USB devices, various QTD_STS_HALT messages are displayed on the screen. This is due to data transmission errors. The source of this problem is currently unclear and is pending further investigation.
  • There seem to be some memory corruption issues. I assume those are not in my code, but were already present in the code base provided by Amazon. So, at times some messages such as “illegal instruction: 0″ appear on the screen, which is a good indication of SW memory corruption issues. Since those messages only appear when the gadget driver is active and the device is operating in host mode, I assume that the gadget driver is responsible for this.

Is this really one of those drivers that are going to stay beta for ever?

I don’t think so. Basically everything else already works, including:

  • Switch between host and gadget mode depending on whether a host(PC) or peripherial device(USB hub) has been attached
  • Custom implementation of low power idle mode for USB OTG that will completly turn off the USB port and stop all USB clocks if no device is attached.
  • Charging both in host and device mode via the new yoshi_charger driver. Charge current can be controlled via sysfs attribute when in host mode

How to test the USB host devices?

Well, this is a problem because the Kindle image has no software support for USB keyboard or anything. If you only want to test a USB network adapter or USB stick, then you should still be able to go with the standard Kindle image and replace the kernel with the patched one. However, things such as USB sound card require additional software support(ALSA) that the Kindle image does not have. In those cases it is generally best to install a custom Linux distro for testing.

The screenshot above was taken when running with a Debian Squeeze image. I have not yet set up any GUI on this system, but for basic testing of the sound card it works just fine. And we’ll see about the UI later :-)

Oh, I want this! How can I test this?

As stated, this is highly experimental. If you nevertheless want to test, use the following instructions to compile the kernel:

1
2
3
4
git clone "git://christian-hoff.com/kindle-4-kernel.git"
cd kindle-4-kernel
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- imx50_tequila_usbotg_defconfig
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j 2 uImage modules

See this page for complete compilation instructions. Beware that the config is now called imx50_tequila_usbotg_defconfig.

On the target device, you will have to load a few modules to take advantage of USB OTG support:

1
2
3
4
5
6
7
8
9
10
# Load the new charger driver for host and device mode USB charging
modprobe yoshi_charger
# Then the OTG state machine. This one supervises the port and controls whether to activate host or device mode.
modprobe fsl_otg_fsm
# OTG gadget driver
modprobe arcotg_udc
# OTG ethernet gadget driver(can load any other gadget driver here as well)
modprobe g_ether
# And finally the OTG host:
modprobe ehci-hcd

Charge current can be adjusted via a sysfs attribute. Normally this should be correct, but when in host mode you will have to do this manually as the USB standard does not permit charging when in host mode.

chgcurr=500 # charge 500 Milliamps from OTG port
echo "$chgcurr" > /sys/devices/platform/fsl-usb2-otg/charge_current

ID pin detection(host/device) can be overidden if for some case the HW pin detection is not correct:

override="0" # -1: no override, 0=force host, 1=force gadget
echo "$override" > /sys/devices/platform/fsl-usb2-otg/id_pin_override

First peek at Kindle OTG support

This post is there to kind of explain the high-level design for the Kindle USB OTG(On-the-Go) support. This series is to be continued with other more low-level design documents.

OTG basic design

The basic design is as follows: We have one USB gadget and one USB host driver. The gadget driver is responsible for the USB port when it is attached to a host(such as a PC or netbook). The host driver will take control of the bus whenever USB slave devices such as keyboards, mice or an USB hub are attached.

Then we need some additional logic to determine if the gadget or the host driver should become active at a given moment. This is done by means of an OTG transceiver driver. The transceiver driver monitors the port to see when any devices are attached or detached. It then tries to determine, by means of the ID pin, to determine if a slave or host device has been attached. Based on this information, it will wake up either the host or gadget driver. This is done by means of a suspend()/resume() call that both drivers export to the OTG transceiver. The whole layout looks as follows:

So essentially the OTG transceiver(fsl_otg_arc) decides based on the ID pin if a host(PC) or peripheral device(mouse, hub etc.) has been attached. If the ID pin is 1, then a peripherial device was attached to the bus and the USB host driver will be started. Otherwise the gadget driver will become active. This typically means that the Kindle will be put in mass storage mode.

Port suspend/resume

Some additional complexity is added to the problem by the fact that we need to support USB port suspend. This means that we should turn off the USB port whenever no slave or host device is attached to the bus. This helps increase the battery lifespan in an order of magnitude. If we continue to power the port all the time, the battery is going to be dead at least after 2 days without charging, which is probably not what any Kindle user would want.

So port suspend also has to be implemented. The problem is, where do we put that functionality? It seems logical that all this code should live in the transceiver(since this is the only entity that knows when none of the host/gadget drivers are active), but this has some implementation issues:

  • Both the host and gadget drivers may not be “alive” at the beginning when the transceiver is loaded. Host and gadget drivers may be loaded later by using the “modprobe” command. So far this is no problem, but the issue is here: At the beginning the transceiver does not know of the existence of the host or gadget drivers that are suddenly trying to access the USB port registers. However, the transceiver has put the USB port into idle state, so USB register access is not possible.
  • It should also be possible to load the gadget/host drivers without the transceiver controlling the port. This is a problem because the drivers may also want to be able to suspend the ports on their own.

To overcome these issues, the suspend code was moved to architecture level and is accessible by all of the drivers. It maintains a use counter that is incremented or decremented whenever one of the drivers acquire or release control of the USB port. Whenever the use counter reaches the value 0, the port will be suspended. This covers even the most complex usage scenarios. Updated UML looks as follows:

Charger support

Charging shall be possible both when we are in host or gadget mode. Otherwise the Kindle’s battery might not survive for a long time when it is attached to a peripherial such as an USB hub.

Charging will be controlled by the OTG transceiver, which then communicates with the charger in order to start charging with the requested charge current(typically 100mA or 500mA when attached to a USB host).

What is already there

In the offical Kindle kernel sources, support for the following is already implemented:

  • Support for USB gadget mode
  • Support for charging over USB
  • Low power suspend – clock disable

The following items are needed for host support, but not implemented:

  • The OTG transceiver code is completely unusable for our purposes and has to be rewritten from scratch. It currently does not support things such as low power suspend-clock disable

TODO: To be continued..

Getting Gaphor to run

I just spent a frustrating hour trying to get the UML modelling tool Gaphor to run. There appears to be some change inside the Zope Component Architecture that caused it to break.

I always got the following error:

    load_entry_point('gaphor==0.17.1', 'console_scripts', 'gaphor')()
  File "/home/hoffch/DH/Studienarbeit/tools/gaphor/lib/python2.7/site-packages/gaphor-0.17.1-py2.7.egg/gaphor/__init__.py", line 97, in main
    launch(model)
  File "/home/hoffch/DH/Studienarbeit/tools/gaphor/lib/python2.7/site-packages/gaphor-0.17.1-py2.7.egg/gaphor/__init__.py", line 33, in launch
    Application.init()
  File "/home/hoffch/DH/Studienarbeit/tools/gaphor/lib/python2.7/site-packages/gaphor-0.17.1-py2.7.egg/gaphor/application.py", line 52, in init
    self.init_all_services()
  File "/home/hoffch/DH/Studienarbeit/tools/gaphor/lib/python2.7/site-packages/gaphor-0.17.1-py2.7.egg/gaphor/application.py", line 83, in init_all_services
    self.init_service(name)
  File "/home/hoffch/DH/Studienarbeit/tools/gaphor/lib/python2.7/site-packages/gaphor-0.17.1-py2.7.egg/gaphor/application.py", line 99, in init_service
    srv.init(self)
  File "/home/hoffch/DH/Studienarbeit/tools/gaphor/lib/python2.7/site-packages/gaphor-0.17.1-py2.7.egg/gaphor/services/componentregistry.py", line 30, in init
    self._components = component.registry.Components(
AttributeError: 'module' object has no attribute 'registry'

However, the fix is really, really simple:

diff --git a/gaphor/services/componentregistry.py b/gaphor/services/componentregistry.py
index 333c8e8..07ee5f0 100644
--- a/gaphor/services/componentregistry.py
+++ b/gaphor/services/componentregistry.py
@@ -7,6 +7,7 @@ unregister_handler, handle), a AdapterRegistry and a Subscription registry.
 """
 
 from zope import interface, component
+from zope.component import registry
 from gaphor.core import inject
 from gaphor.interfaces import IService, IEventFilter
 
@@ -27,7 +28,7 @@ class ZopeComponentRegistry(object):
         pass
 
     def init(self, app):
-        self._components = component.registry.Components(
+        self._components = registry.Components(
                                name='component_registry',
                                bases=(component.getGlobalSiteManager(),))

This follows the way the imports are defined in the zope.component docs. Hope this helps!

UPDATE: I have submitted a pull request to Gaphor the Gaphor GIT tree which got accepted today. The fix is not in 0.17.1, but any later version should also have the fix preinstalled.

Compiling Kindle 4 kernel

I have spent a frustrating couple of days trying to get the Kindle 4 Non-Touch kernel to compile. In this post I will post the instructions that finally made the whole thing compile.

The problem is that a number of files are missing in the Kindle 4 Non-Touch kernel source code, namely:

  • include/linux/einkfb.h
  • include/battery_id.h
  • include/boardid.h
  • include/llog.h
  • include/wan_types.h

I first tried to replace all these files with stubs, and excluded the E-Ink frame buffer from the build(because of the missing include/linux/einkfb.h). This worked, however most developers also want to have a frame buffer. Fortunately it turned out that the files that are listed as missing are provided in the source code of the Kindle Touch. And as it turns out the kernel diff between the Non-Touch and the Touch Kindle is minimal, so you are able to simply copy the missing files from the Kindle Touch source code to the kernel code of the Non-Touch edition.

Afterwards the compilation is pretty straightforward. I don’t recommend the CodeSourcery toolchain because even now they haven’t managed to compile a working set of compilers for 64-bit hosts. But you can use the compilers from the Ubuntu package archives:

apt-get install gcc-4.6-arm-linux-gnueabihf

This installs the hard float cross compiler for gcc 4.6. That is, the compiler itself can produce both soft and hard float code, but the libc toolchain it comes with contains hard float code. The Kindle’s sysroot appears to have been compiled using soft float, but it is safe to use the hard float compiler for the kernel compilation because we are not linking any libc source code and(as noted) the compiler is capable of producing soft and hard float binaries. I am using the hard float compiler because the soft float compiler did not work for the kernel compilation. Why? I can’t tell. But somehow all the compilers with a soft float libc didn’t work for me. This includes the CodeSourcery Toolchain and the Ubuntu soft float compilers.

Then we need to install some symlinks:

for i in gcc gcov cpp; do ln -sf arm-linux-gnueabihf-${i}-4.6 /usr/bin/arm-linux-gnueabihf-${i}; done

Now we are ready to compile the kernel. But we have to do one additional step. For a complete kernel image, a so-called “ramdisk” is usually needed. This ramdisk is a complete file system tree that gets loaded during the early stages of the boot process. The tasks of the “ramdisk” can vary widely, but usually include, among others, two main items:

  1. Load additional kernel modules
  2. Switch to the actual root file system.

    On the kindle, the actual root file system is stored on /dev/mmcblk0p1.

The ramdisk is needed for the boot process and it is impossible to boot a kernel without a ramdisk. There are two kinds of ramdisks, initrd and initramfs. Only initramfs is interesting for us.

There are 2 ways to get an initramfs:

  • Compile your own.

    Unfortunately, this process is very complicated, so it will not be elaborated here.

  • Use the one from the Amazon stock kernel.

    This is the most promising approach because it will save you a lot of time and frustration. And as you know, time is money :-) .

    Unfortunately, this task too can become complex and time consuming.

    But I have already written a guide on how to extract the initramfs here, in order to save you most of the hard work.

    You can use this tutorial in order to extract the initramfs from the Amazon “normal” kernel image. Just follow the instructions step by step until you have an extracted initramfs somewhere on your file system.

    Aftwards compress the initramfs to a .cpio file:

    1
    2
    
    cd /path/to/extracted/initramfs
    find . | cpio -o -H newc > ../initramfs.cpio

Then we can proceed with the kernel compilation:

1
2
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- imx50_tequila_defconfig
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig

A menuconfig should shop up. You need this menuconfig in order to tell the compiler where your initramfs is located. To do this, navigate into the “General setup” menu and change the value of “Initramfs source file(s)” to the location of your initramfs.cpio that you created in the previous step. Save your kernel config and exit.

1
2
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j 2 uImage
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j 2 targz-pkg

Before we finally flash the kernel to the device, it is important to create a backup copy of the Kindle kernel since the default kernel will be overridden in the process. If you have used the initramfs from Amazon, then you should already have a copy of the uImage. Otherwise just run the following commands to get a full backup of both the “normal” and the “diagnostic” kernel of you device.

1
2
dd if=/dev/mmcblk0 bs=1024 skip=260 count=$((14*1024)) of=/mnt/us/uImage
dd if=/dev/mmcblk0 bs=1024 skip=14596 count=$((14*1024)) of=/mnt/us/uImage-diags

When you really need the backup, there’s still time to truncate the uImage and then flash back again via the fastboot utility as described here.

Afterwards you can transfer all the kernel modules to your rootfs. You can do this by copying the generated linux-*.tar.gz file(from make targz-pkg target) to the Kindle and extracting it there.

Finally it is time transfer the image to the device using fastboot. Download and install fastboot as described here. Afterwards you can transfer the image to your device the commands below. ATTENTION: This is going to override the default kernel image of the device. Before you do this, make sure that you have created a backup of the Kindle kernel as described above.

sudo fastboot download ./arch/arm/boot/uImage
sudo fastboot flash kernel
sudo fastboot boot

That’s it! Obviously, in order to see kernel messages, you will need serial console access.

Back up Kindle 4 kernel and extract initramfs

In this part of the series I will describe how I backed up the kernel of the Kindle 4 non-touch and how you can use the kernel to extract the initramfs. History has proven that if things can go wrong, they will probably also go wrong. So before you start working on the device, you should create a full back up of everything that you may or may not modify.

Extracting the kernel is not that difficult after all. You have two options for this: If you want, you can use the boot loader and transfer the kernel using a serial connection. Alternatively the kernel can also be obtained by running “dd if=/dev/mmcblk0″ with the right offsets and lengths. This is generally the less painful method, but it only works if you still have a Linux that boots. Note that downloading kernel images via fastboot does not work. Amazon only provided a way to download kernels to the device, but has no options for uploading a kernel image to the host PC.

General stuff

There are two kernel images on the Kindle 4:

  • The “normal” kernel image that is used to boot the device
  • The diagnostics kernel image that is used e.g. when an ENABLE_DIAGS file has been created on the USB partition

Looking at the bootloader reveals the memory addresses of the images:

uboot > printenv
[...]
bootcmd=bootm 0x41000
bootcmd_diags=run bootargs_diags ; bootm 0xE41000

So the address of the standard kernel image is at 0×41000 on the flash storage, whereas the recovery image is at 0xE41000.These numbers are also defined in include/configs/imx50_yoshi.h in Amazon’s u-boot source code:

98
99
100
#define CONFIG_MMC_BOOTFLASH_ADDR       0x41000
#define CONFIG_MMC_BOOTFLASH_SIZE       (14*1024*1024) /* 14 MiB */
#define CONFIG_MMC_BOOTDIAGS_ADDR       0xE41000

Getting the kernel image – the easy way

If you still have a working Linux system on the Kindle, you can use the following process to get the kernel image:

  1. The kernel image can be obtained by running a simple dd command. The parameters for dd are:
    • block size=1024 (just my convention)
    • skip=260(normal kernel) or 14596 for recovery image. This can be calculated by dividing 0×41000/0xE41000 through the block size
    • count=14*1024. This was obtained by dividing 14Mb through the block size

    The dd command then goes like this:

    dd if=/dev/mmcblk0 bs=1024 skip=260 count=$((14*1024)) of=/mnt/us/uImage
    dd if=/dev/mmcblk0 bs=1024 skip=14596 count=$((14*1024)) of=/mnt/us/uImage-diags

    We now have two U-Boot images that contain the kernel images for normal operation and recovery, respectively.

  2. Now we need to strip the unused bytes from the uImage. We do this by looking at the U-Boot header of the downloaded files with a hex editor like hexedit. The size of the image can be obtained by looking at bytes 12-15. This offset can be explained by looking at “struct image_header” in file include/image.h in the Amazon U-Boot source code:
    174
    175
    176
    177
    178
    179
    180
    
    typedef struct image_header {
            uint32_t        ih_magic;       /* Image Header Magic Number    */
            uint32_t        ih_hcrc;        /* Image Header CRC Checksum    */
            uint32_t        ih_time;        /* Image Creation Timestamp     */
            uint32_t        ih_size;        /* Image Data Size              */
            [...]
    } image_header_t;

    In my example file, the image data size is 0x0048A540, which amounts to something like 4.5 Mb. However, the 64 bytes of the uImage header are not included in this size, so we need to add these 64 bytes in order to get the final size of the uImage. The file can then be truncated as follows:

    truncate -s $(( 0x0048A540+64 )) ./uImage

    Replace the number with the size of your kernel image.

  3. Test your image using mkimage:
    $ mkimage -l uImage
    Image Name:   Linux-2.6.31-rt11-lab126
    Created:      Fri Sep 23 06:34:55 2011
    Image Type:   ARM Linux Kernel Image (uncompressed)
    Data Size:    4760896 Bytes = 4649.31 kB = 4.54 MB
    Load Address: 70008000
    Entry Point:  70008000
  4. Now we can extract the kernel image file from the uImage by stripping the 64 bytes of U-Boot header information:
    dd if=uImage bs=64 skip=1 of=Image

Extract the initramfs

The initramfs can be extracted using the unpack.sh script from here:

git clone "https://github.com/choff/galaxys2_kernel_repack.git" kindle_kernel_repack
cd kindle_kernel_repack
sudo ./unpack.sh /path/to/Image ./initramfs

The extracted initramfs will be placed in the “initramfs” directory.

Remember to always run the “unpack” command as root. This is critical because you need root priviliges in order to extract the device nodes in the “/dev” directory of your initramfs. If you are not root, you get a full initramfs, but without any device nodes. But usually one wants to have the device nodes as well, especially when one plans to reuse the initramfs for a new kernel.

Getting the kernel image – the painful way

This information is here just for completeness. This is probably not what you want. Only use this method if you cannot boot your kindle any more and need to extract the initramfs from the device. The other method above is much simpler and in most cases preferable. You will also need a serial console for this experiment since we need to access the boot loader directly.

To be able to do this, we first have to enter the “built-in self-test”(bist) mode of the bootloader. We this this by typing “bist” on the U-Boot command line(accessible only via serial console):

bist > mmcinfo 0
Device: FSL_ESDHC
[...]
Rd Block Len: 512
[...]

This gave us the information that mmc #0 uses a block size of 512 bytes. Now, the whole space for both the recovery and “normal” kernel is 14 Mb, so we will have to read a total of 14Mb/512B = 28672 = 0×7000 blocks from the internal MMC into RAM:

uboot > mmc read 0 $(loadaddr) 0x41000 7000 0

This loads the uImage from position 0×41000 to the RAM address “loadaddr” of the device. “loadaddr” is a defined position in the RAM where the kernel images are written to. The number of blocks(7000) is interpreted as hex number and was calculated as above.

Let’s see if we did everything correct by displaying some details about our image:

bist > imi $(loadaddr)
 
## Checking Image at 7a000000 ...
   Legacy image found
   Image Name:   Linux-2.6.31-rt11-chris
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    4890016 Bytes =  4.7 MB
   Load Address: 70008000
   Entry Point:  70008000
   Verifying Checksum ... OK

We can also obtain this header information in hexadecimal by issuing

bist > md.b $(loadaddr) 40
7a000000: 27 05 19 56 3e 26 3f 1a 4f 78 57 62 00 4a 9d a0    '..V>&?.OxWb.J..
7a000010: 70 00 80 00 70 00 80 00 1e f0 e6 b4 05 02 02 00    p...p...........
7a000020: 4c 69 6e 75 78 2d 32 2e 36 2e 33 31 2d 72 74 31    Linux-2.6.31-rt1
7a000030: 31 2d 63 68 72 69 73 00 00 00 00 00 00 00 00 00    1-chris.........

Which displays us the first 64 bytes(the uImage header). By looking at bytes 12-15 again, we find the data size of the kernel image which is 0x004a9da0 in our case(decimal 4890016, just like in the output of the “imi”(image info) command.

Now comes the tricky part: We are going to dump the whole kernel using md.b. I will not go into the details here since they have already been discussed this post. The rest of the instructions are identical with the “easy” method, so you can proceed with step 2.

Soldering a serial console to the Kindle 4

This is the first part of a series of blog posts in which I will describe my progress in implementing USB-On-The-Go functionality on the Kindle 4. The goal is to be able to connect USB slave devices such as keyboards and USB network adaptors to the kindle. By connecting an USB hub, it would also be possible to have many of these devices attached to the Kindle at the same time.

This first blog post is about getting the required serial console access to the Kindle 4 device. I needed this in order to be able to control the boot loader, flash custom kernels and get debugging messages for my USB On-The-Go testing.

Here is a list of things that might be useful:

Be careful when soldering the 3 serial pads of the Kindle! The contacts are very close and it is best to have this work done by an expert. After spending a frustrating Friday night trying to do all this myself, I finally got some help from some guys from my local hackerspace. So be warned: This procedure is nothing for noobs! You are likely going to end up bricking your device.

After soldering the 3 pins, we applied some hot glue above the contacts and at the chassis so that the contacts would stay in place for a long time. Below is a picture of the result:

Kindle with soldered serial console

Kindle with serial console

With all this in place I was successfully able to obtain boot messages from the Kindle:

Kindle serial console output

Kindle serial console output