EuroPython Bag

Python Environment

At our company, we deploy non-interactive Python tools to our internal users. So far, we deploy these tools into the users’ local Python installations. Problems arise when the package versions that our tools require are different from the versions that the user has installed, or when updates to our tools require new dependencies to be installed. To avoid such problems, the “virtualenv” package allows you to keep separate Python environments for different tools. The “pip” package can then be used to automatically install dependencies into these environments. Alex Clemesha has an article up on his blog about the Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip.

Another useful addition to the Python toolbox is the “nose” unit-testing package, which extends the Python “unittest” module. It has advanced support for test fixtures, generated test cases, running test batteries, and more. More at the “nose” project homepage.

JavaScript

As a language, I find JavaScript pretty…awful. But it can be used to build pretty cool stuff for the web, so it’s worth learning. That’s why I attended Jonathan Fine’s “JavaScript for Python Programmers” training.

One of the learning resources that Jonathan recommended was “JavaScript: The Good Parts” (on YouTube):

Network Programming

For the grid computing system at our company, we rely mostly on the standard XML-RPC modules in Python. All the high-level logic, such as message queues, failure handling, authentication, etc. are custom-built on top of that.

Unsurprisingly, there is a wealth of existing libraries that might make our lives easier by not having to reinvent the wheel. (While adding their own layers of complexity, I’m sure.)

• Django REST Framework—A library for adding a RESTful API to a Django application.
• Celery—An asynchronous task queue based on distributed message passing.
• ZeroMQ—A “socket library that acts as a concurrency framework.”
• RabbitMQ—Message-oriented middleware. Jason Williams describes what it is and what it can be used for.
• AMQP (Advanced Message Queuing Protocol)—The protocol that RabbitMQ implements.
• gevent—”A coroutine-based Python networking library that uses greenlet to provide a high-level synchronous API on top of the libevent event loop.” Geez, I don’t even begin to understand what this means…
• Twisted—An event-driven network engine. (Twisted training by Orestis Markou.)

Alex Clemesha’s article “Real-world real-time web apps with Python equals Django + Orbited + Twisted” describes how some of these fit in the big picture—and adds even more buzzwords and acronyms for good measure.

Some of these libraries come with implementations for the architectural patterns described in the book “Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Objects.”

Little Tidbits

• Paramiko—An SSH client/server library written in Python (using PyCrypto). Jesse Noller has an introduction on his site.
• Kivy—A “library for rapid development of applications that make use of innovative user interfaces, such as multi-touch apps.” For his lightning talk, Mathieu Virbel used a really cool presentation tool, PreseMT, that is itself based on Kivy. (Watch on YouTube)
  
• scrypt key derivation function—This can be used to calculate password hashes in a way that makes brute-force attacks way harder than using traditional MD5 or SHA hashes. Python bindings exist.
• blockdiag—A block diagram image generator. It uses an input file format that’s similar to GraphViz.

See you at EuroPython 2012!

Continued fractions are a representation of real numbers that allows for arbitrary-precision arithmetic.

If you’ve worked with floating-point numbers in Python (or most any other programming language, for that matter), you are aware of precision problems like this:

x = 1.0 / 3.0
total = 0.0
for i in range(300):
    total += x
print repr(total)
print total == 100.0

This prints 99.99999999999966, not 100.0. The reason is that the IEEE 754 floating-point representation of 1/3 isn’t exact, and this (initially small) error accumulates 300 times.

There is a different representation for real numbers: continued fractions.

• Consider the number PI = 3.141592653589793.
• This can also be written as: 3 + 1 / 7.062513305931052
• This can be written as: 3 + 1 / (7 + 1 / 15.996594406684103)
• This can be written as: 3 + 1 / (7 + 1 / (15 + 1 / 1.0034172310150002))
• This can be written as: 3 + 1 / (7 + 1 / (15 + 1 / (1 + 1 / 292.6345908750162)))
• This can be written as: 3 + 1 / (7 + 1 / (15 + 1 / (1 + 1 / (292 + 1 / 1.5758184357354204))))

This can be stored efficiently in some form of list: PI = [3, 7, 15, 1, 292, 1, ...]. The “cf” module is a Python implementation of continued fractions by Marcin Ciura that uses lazy evaluation. Here’s the previous example rewritten using continued fractions:

from cf import cf
x = cf(1.0) / cf(3.0)
total = cf(0.0)
for i in range(300):
    total += x
print repr(float(total))    # prints 100.0
print total == 100.0

The mathematical background is explained on Wikipedia and in this presentation by Mark Jason Dominus.

TWiT and the Ubuntu terminal on a TV set via DLNA

Many modern TVs (and set-top boxes, gaming consoles, etc.) support DLNA streaming. Suppose you have a PC that stores all your music, downloaded podcasts, video podcasts, photos, and so on. You can run some DLNA media server software on your PC and stream your entire media collection to your TV over your home network. No more carrying around USB sticks, it’s all in your home cloud.

On GNU/Linux, I am using MediaTomb as my DLNA server. It’s nothing fancy (it’s a file server, after all), and it just works.

Okay, this takes care of media files stored on your PC. But can we do more? Is it possible to stream a live capture of your desktop to the TV?

Let’s say you’re watching a Flash video in your browser, and there’s no way to download the video file. Or, you’re watching a live event being streamed via Flash or whatever. It would be kinda cool to be able to stream that to your TV via DLNA. And it’s possible—not trivial, mind you, but I’ve seen it working at least once…

The same approach that’s taken here for live streaming might also be useful for on-the-fly transcoding (e.g., an .ogg file needs to be transcoded to .vob for the player to be able to read it).

(I should mention that something like this isn’t unheard of. In fact, my Philips TV came with the Windows-only(!) WiFi MediaConnect software to do desktop streaming. I have never seen it in action, because I don’t use Windows. Of course, Philips based the TV’s firmware on the Linux kernel, like so many other manufacturers do. But, also unsurprisingly, they use it because it’s “free as in beer”, not because they care about users’ freedom. The fact that I could still get all this to work on GNU/Linux is thanks to the invaluable work of many free software projects: MediaTomb, FFmpeg, Matroska, FUSE, Python, to name just a few.)

What you can find here

I wrote a couple of scripts that allow you to capture your desktop and stream it to a DLNA-capable player. To use the code as-is, your player must support the Matroska (.mkv) file format.

I used the scripts in conjunction with MediaTomb, but other media servers should work just as well.

The scripts are very rough at the edges, and if you are afraid of the command line or of reading Python code, you shouldn’t attempt to use them.

Download the scripts

See Usage Instructions below.

What is missing / Invitation to contributors

As I mentioned, the scripts only work with devices that can play Matroska files via DLNA right now. The basic ideas and concepts should apply equally as well to MPEG-2 and other formats. I’m not sure about MPEG-4 files, though, and I would appreciate feedback from someone more familiar with the format.

The scripts could definitely use better error checking, a nicer command-line interface, and lots of testing. If anyone’s interested in helping with that, please contact me.

Usage Instructions

• Make some changes to dlna_fuse.py to specify the temporary file, control the captured display region, set the output format, etc.
• Configure MediaTomb, as described below.
• Mount dlna_fuse.py with “python dlna_fuse.py -s -f fuse_mnt”. This automatically starts the capture.
• At this point, add the file “fuse_mnt/a/fuse_live.mkv” to MediaTomb’s database. (You only have to do this once.)
• Start playback.
• As this is more of a proof-of-concept than a polished tool, please read on below and let me know if you have any feedback.

The Basic Idea

First off, how do we capture the desktop? Someone named Verb3k explains this in “How to do Proper Screencasts on Linux Using FFmpeg“. Here’s an example:

ffmpeg -f alsa -ac 2 -i pulse -f x11grab -r 20 -s 1024×576 -i :0.0+128,224 -acodec ac3 -ac 1 -vcodec libx264 -vpre fast -threads 0 -f matroska ~/Videos/capture.mkv

This command line takes sound from PulseAudio and screen images from X11 (at 20 fps) and combines them into a Matroska file using the H.264 codec for video and AC3 for audio. It grabs a rectangular area of 1024×576 pixels, 128 pixels from the left edge of the screen and 224 pixels from the top edge of the screen.

Now, what happens when we have MediaTomb serve up the file capture.mkv to the player while the file is still being captured? If you are luckier than I was, it might just work, and you’re done. Maybe you can find some other combination of video codec, audio codec, and container file format that your player likes better. (Before attempting to do live streaming, you should have ffmpeg convert an existing video file, and finish the conversion before starting playback, in order to find a format that your player understands.)

Starting the playback while the capture was still in progress didn’t work for me. Instead, when starting playback too soon after capturing had begun, the player would simply tell me the file was unplayable. When waiting for a few minutes, the player would play up until the point where I had started playback. That is, when I started playback after having captured for five minutes, playback would stop after five minutes, even if the captured file contained 10 minutes at that point.

You probably have an idea already why this might have failed. Let’s take a look at what the file contents might look like while the capture is in progress.

A video file after 2 minutes of capturing, after 5 minutes, after it is complete

In the figure, you can see a hypothetical file format that stores the data length and video duration in the front, then the video data, then a table containing seek information and other stuff that’s only known after encoding has finished. This particular encoder seems to update the duration field periodically while encoding is in progress. On the other hand, it leaves the data size field blank until it has finished.

This hypothetical case shows many things that can go wrong when playback starts in the middle of encoding (which, in the case of live streaming, basically means at any time):

• The player might encounter the “unknown” size field and decide that the file is broken.
• Or, it takes the “unknown” size as an indication that it should seek to the end of the file and determine the size itself (which breaks, because the file has no end yet).
• The player might read the duration of 2:00 min. and never look at it again—playback will simply stop after 2:00 min., no matter what happens to the file in the meantime.
• The player might know that there’s supposed to be a seek table, a list of keyframes, a checksum, etc., at the end and fail when it tries to read it.
• <endless other complications>

The scripts that I wrote try to circumvent these issues in two steps:

• Modify the file that ffmpeg produces during the capture so that the file appears to be a regular, albeit very, very long, video. Give the player all the information that it needs right away, so that it does not try to seek (through the media server) to various places in the file, searching for the information.
  • For Matroska files, this is what “matroska_live_filter.py” does. Hopefully, it will be possible to write filters for other container file formats in the future.
• Intercept calls to the filesystem, so that when the player (through the media server) tries to access parts of the file that don’t exist yet, we can wait for ffmpeg to produce them (if it’s just video data for 5 seconds in the future, for example), or come up with fake data.
  • This is what “dlna_fuse.py” does. This is a virtual (FUSE) filesystem that simply waits for ffmpeg to produce more data when the media server tries to prefetch more data than is available in the file.

Matroska is a container format that lends itself well to this kind of interception. Matroska has streaming support (i.e., it defines what should go into the headers so that players know it’s a live stream), but unfortunately, my particular player didn’t care much.

There are other container formats where I’m not sure that such a thing is possible. In MP4, for example, there are atoms like “stsz” (“sample table sizes”) and “stss” (“sample table sync samples”) that seem to go before the video stream and that contain information about the encoded sizes of frames—I’m not sure there’s a way to fake this data without waiting for the encoding to finish. If you are familiar with the MP4 or QuickTime formats and have an idea, please leave a comment!

Filtering Matroska for Live Streaming

The Matroska specification points out that a live stream is designated by setting the “Segment” size to “unknown”. ffmpeg does this, but it didn’t convince my TV to treat the file as a live stream. Instead, I ended up simply setting the size to a very large value and setting the duration of the video to 100 hours.

In addition, I suppress “SeekHead” elements (pointers to other sections of the file) and “Cues” elements (pointers to keyframes for fast-forwarding). This isn’t strictly necessary; ffmpeg only produces these elements when encoding finishes (which it never does with a live capture). However, this functionality was quite handy when testing out the DLNA streaming with existing .mkv files.

As a final hack, I output 128 KB of “Void” data after each “Cluster” (which appears to be a block of ~5 seconds of audio/video data). The “Void” data doesn’t serve any purpose other than being able to send data to the player when it requests some. The player pre-fetches data. Without the “Void” blocks, there is sometimes not as much data as it requests, because ffmpeg hasn’t produced it yet. If the requested data can’t be delivered fast enough, though, the player appears to give up. By producing the “Void” data, there is always enough data to satisfy the player, even though the data doesn’t contain anything useful.—At least, that’s what I think is happening…

All this is done in the Python script “matroska_live_filter.py”.

Usage:

• python matroska_live_filter.py <mkv-filename> : Output a pretty-printed tree of the Matroska file structure. (Similar to the mkvinfo tool from the mkvtoolnix package.)
• python matroska_live_filter.py - : Read a Matroska file from stdin (maybe the output of ffmpeg writing to stdout). Writes the modified Matroska file to stdout on-the-fly, i.e., it writes data as soon as it becomes available and doesn’t wait for the input to end.

As an example, to filter an ffmpeg-produced live stream, write this:

ffmpeg -f alsa -ac 2 -i pulse -f x11grab -r 30 -s 1024×768 -i :0.0 -acodec pcm_s16le -vcodec libx264 -vpre fast -threads 0 – | python matroska_live_filter.py – >~/Videos/filtered_live.mkv

(I tried to use libebml and libmatroska in C++ first. However, documentation was hard to come by, and the code wasn’t quite self-explanatory. I found a Matroska tag reader written in Python by Johannes Sasongko and built the filter based on that.)

FUSE Filesystem to Fool the Media Server

When you’re using GNOME, chances are you’re using FUSE filesystems already. When you use “Places”→”Connect to Server” to connect to an FTP server, for example, the remote server appears as a local folder in ~/.gvfs. This is a virtual filesystem that uses FUSE.

For the DLNA streaming, I decided to write a FUSE filesystem in Python. This filesystem would appear to MediaTomb as a regular directory containing a Matroska video file. Whenever MediaTomb would access the file or read parts of it, my Python code could intercept these calls and do its magic.

When the filesystem is mounted (i.e., when the Python script is started), the desktop capture is started and redirected to a temporary file. When MediaTomb (or any other program) requests some part of the file, the script can check whether there’s enough data in the file. If yes, it simply returns the data. If not, it blocks briefly until ffmpeg has written enough data. If the player tries to read too far ahead, this might indicate that the file isn’t suitable for live streaming yet, and the script will log an error. (This shouldn’t happen for Matroska files anymore, but it will be useful when trying to add support for more container formats later.)

The FUSE filesystem is in “dlna_fuse.py”. It requires the Python bindings for FUSE (package “python-fuse” on Ubuntu).

You should make a few changes to the file to adapt it to your needs:

• Change the variable TEMP_FILE. While ffmpeg captures the desktop, the resulting video is not kept in memory but written to this file. This means, you need some free memory on your hard disk while watching live streams. Of course, the whole purpose of the FUSE filesystem is that the file doesn’t need to exist physically. At a later point, I will change the FUSE script to keep only a part of the ffmpeg output in memory, and discard older parts once the player has read them. For now, the temporary file is used as a buffer.
• Change the ffmpeg command line. The current command corresponds roughly to this:
  • MONITOR=$(pactl list | grep -A2 '^Source #' | grep 'Name: .*\.monitor$' | awk '{print $NF}' | tail -n1)
parec -d "$MONITOR" | ffmpeg -f s16le -ac 2 -ar 44100 -i - -f x11grab -r 20 -s 1024x576 -i :0.0+128,224 -acodec ac3 -ac 1 -vcodec libx264 -vpre medium -threads 0 -f matroska - | python matroska_live_filter - > ~/Videos/live.mkv
• Instead of using “ffmpeg -f alsa -i pulse”, which produced crackling noises every now and then, I use “parec” (PulseAudio recorder) to capture the audio. The part “-f s16le -ac 2 -ar 44100″ is the format that parec produces (at least for me): 44 kHz, 16-bit stereo. “-r 20″ instructs ffmpeg to capture at 20 fps. I chose “-s 1024×576 -i :0.0+128,224″ to capture a 1024-pixel-wide rectangle with an aspect ratio of 16:9 at the center of my screen, which is 1280×1024. You can change this to whatever suits you (as long as your computer can encode it fast enough). “-acodec ac3 -ac1″ converts the audio to the AC3 codec in mono (the TV had problems with stereo AC3 streams). “-vcodec libx264 -vpre medium” uses the “medium” profile for the H.264 encoding. “-vpre” can also be “fast”, “ultrafast”, “lossless_ultrafast” and lots of others—you need to experiment to find an encoding profile that provides good quality, yet doesn’t overwhelm your CPU or network.
• Note: If “parec” doesn’t record anything, open the “PulseAudio Volume Control” (installed with “sudo apt-get install pavucontrol”) and make sure that on the “Input Devices” tab, the device named “Monitor of Internal Audio Analog Stereo” isn’t muted.
• Make a directory for the mount point and mount the filesystem:
  • mkdir fuse_mnt
python dlna_fuse.py -s -f fuse_mnt
  • “-s” means single-threaded (just in case my implementation isn’t entirely thread-safe), “-f” means foreground (so that you can see log output on stdout).
• To test it, you can point Nautilus (assuming you use GNOME) at the fuse_mnt directory and play the file fuse_live.mkv that you find in there using a player of your choice.
• Add the file “fuse_mnt/a/fuse_live.mkv” to MediaTomb’s database.
• Note: I start “mediatomb” manually from a terminal, which works just fine. The MediaTomb service that’s started automatically during boot, on the other hand, can’t see the file “fuse_live.mkv” due to permission problems–I’m not sure why.
• To stop the FUSE filesystem, run “sudo umount fuse_mnt”. If this doesn’t work, you can also kill the dlna_fuse.py process:
  • ps aux | grep dlna_fuse.py
kill -9 <pid>
• If starting dlna_fuse.py fails with “bad mount point: Transport endpoint is not connected”, make sure the process has been killed and run “sudo umount fuse_mnt” again.

Configuring MediaTomb

MediaTomb web UI: Ugly but functional. And ugly.

Install MediaTomb via your package manager (package “mediatomb” in Ubuntu). In Ubuntu, MediaTomb is started automatically as a service. The configuration file is in /etc/mediatomb/config.xml.

I prefer to start mediatomb manually whenever I need it and place the configuration file in ~/.mediatomb/config.xml.

Provided that the config.xml contains “<ui enabled=”yes”>”, you can open the MediaTomb GUI in a web browser at http://localhost:49152/ (or a subsequent port number). Once the FUSE filesystem is up and running, add the file fuse_mnt/a/fuse_live.mkv to the database. At this point, you should be able to find it and play it back on your DLNA player. (Of course, it can’t hurt to try a regular file first to check whether it works at all.)

How well does it work?

The image quality is astonishing, even with the “fast” encoding profile that I am using. The fonts and window details are very crisp. You have to look very closely to see the typical MPEG compression artifacts, if you can see them at all.

I am not trying to watch HD movies this way. I mostly use this for the TWiT live stream and similar talking-heads programs, so the 20 fps that my PC can deliver are good enough.

The latency is currently whatever it takes me to start the FUSE filesystem (which automatically starts the capture), walk over to the TV, and start playback there. I think if I would delay the capture until the file is actually accessed, I could reduce latency to just a few seconds (although I do think it’s a good idea to give the encoder a head start). Maybe it will be possible at some point to reduce the latency far enough to be able to remote-control the PC and get feedback almost instantly. We’ll see.

If the video hangs, try the obvious things: Reduce the frame rate, make the captured screen area smaller, lower the bitrate, etc.

Tested Hardware and Software

I tried all this on Ubuntu 10.10 (Maverick) with MediaTomb 0.12.1, FFmpeg 0.6, and Python 2.6.

The TV is a Philips PFL 7605H/12 with firmware 000.140.025.000. (As far as I can tell, models 8605 and 9705 use the same firmware, so they might work as well.)

If any of you can successfully replicate this on other player models and brands (or even if you can’t), please leave a comment.

Request for Comments

Again, this is a proof-of-concept. I hope to expand and improve the scripts in the future. If you have any questions, suggestions, or comments, please leave a comment, contact me via identi.ca/mfoetsch, twitter.com/mfoetsch, or e-mail.

Figure source

Timothy Wu describes how, historically, information industries have tended to cycle from the freedom and openness that follows a disruptive invention to monopoly. About two thirds of the book are devoted to retelling the history of four such industries: Telephone, radio, movies, television. These are intriguing and eye-opening stories: How Hollywood started out as a bunch of “IP pirates”; how the FCC came about and what they were up to; how the Bell System delayed technologies like magnetic storage media and the answering machine for decades; etc.

Then he continues with the Internet, from its decentralized beginnings as ARPANET, to the AOL era, all the way to Apple, Google, and Net Neutrality (and why it is that the iPad fits the monopolistic agenda so well).
Having learned about the monopolistic powers of the Bell System (AT&T and Verizon rising from the ashes), of cable companies, and of media conglomerates, it’s hard to see a bright future for freedom and openness on the Internet.

Wu makes the case for Net Neutrality, and with all the context that the book provides, it is a compelling argument.

Borrowing from what Jonathan Lethem said about Larry Lessig: Timothy Wu is “the best source if you want to get radicalized in a hurry.”

Longer review at Salon.com (by Laura Miller)

I ran into this problem today where Gwibber (the micro-blogging client) on Ubuntu would not work with identi.ca anymore. The bug report referred to the log file ~/.xsession-errors, which contained this message:

Traceback (most recent call last):
File “/usr/lib/python2.6/dist-packages/gwibber/microblog/network.py”, line 53, in __init__
self.curl.perform()
error: (60, ‘server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none’)

This is actually a cURL error, so I tried to reproduce the issue directly with curl on the command line:

$ curl https://identi.ca/api/statuses/friends_timeline.json?count=200&since_id=61361942
curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a “bundle”
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn’t adequate, you can specify an alternate file
using the –cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you’d like to turn off curl’s verification of the certificate, use
the -k (or –insecure) option.

After searching for this error on the web, I found a cURL page about SSL certificates. This page talks about importing SSL certificates for cURL to use. After some trial and error, I came up with these steps to import the certificate and solve the issue:

1. I went to https://identi.ca/ in Firefox and inspected the certificate. This pointed to the certificate download URL: http://crt.comodoca.com/COMODOHigh-AssuranceSecureServerCA.crt

2. I downloaded the .crt file from that URL.

3. The downloaded file is in some binary format. I converted the .crt file using this openssl command:

$ openssl x509 -inform DER -in COMODOHigh-AssuranceSecureServerCA.crt -out outcert.crt -text

4. I copied the resulting file, outcert.crt, to /usr/share/ca-certificates/COMODO_High_Assurance_Secure_Server_CA.crt.

5. From there, the new certificate can be installed system-wide by running dpkg-reconfigure:

$ sudo dpkg-reconfigure ca-certificates

This last step updates the files /etc/ca-certificates.conf and the directory /etc/ssl/certs.

The same curl command that didn’t work initially should work now (at least, it won’t complain about incorrect certificates.)

I’m sure there will be an “official” fix soon, but at least this provided for some geeky evening entertainment tonight. (Did I mention my TV is broken… )

"Chimpzilla Attacks!" mockup (no, I'm not serious)

Let’s use the game “Chimpzilla Attacks!” as an example. I made up this game specifically for the purpose of this tutorial—and already spent way too much time on the mockup. I have no idea what the game mechanics are, but judging from the cheesy graphics, it’s got to be some kind of “Punch The Monkey” knock-off. Anyway, back to the tutorial…

The skyline backdrop has a resolution of 800×600 pixels. In the following image, you can see its size and aspect ratio relative to various common screen sizes, from a QVGA screen to a 1024×600 screen.

Background image relative to various screen sizes and aspect ratios

The goal here is to scale the game screen so that its width fits the display. In the following image, you can see what the scaled image will look like on various screens.

Background image relative to various aspect ratios

On a 320×240 screen, all of the backdrop is visible (at 40% of the size, of course). This is because both have an aspect ratio of 4:3 (or 1.33). Most of the common screen sizes are wider, though. For example, a 1024×600 screen is 5:3 (or 1.66). The wider the screen, the higher the aspect ratio. On any screen with an aspect ratio higher than 1.33, we’ll have to cut off some of the backdrop’s height, but those parts are not used for the gameplay anyway.

Having to cut off some of the backdrop on wide screens sounds like the game isn’t designed for wide screens. It’s quite the opposite, though. The part of the game screen that’s used for actual gameplay is 800×400, with an aspect ratio of 2 (which is not quite Cinemascope, but close). With such a wide picture, we’d have to display black bars above and below the picture on all screens. Instead of black bars, I can just as well show more of the photo as a filler.

There are still aspect ratios where we have to pad the screen with black bars, namely ratios smaller than 1.33 and higher than 2, as shown in the following image.

Image relative to extremely narrow and extremely wide screens

The 800×640 screen has an aspect ratio of 1.25. There, we’ll display black bars above and below the picture. The second screen is a hypothetical 800×340 Cinemascope (2.35:1) screen. While it is wide enough to show the entire game screen, it is not high enough. We’ll have to scale the game screen down to 680×340 so that it fits the height of the display. At that scale, we’ll have to display black bars (or whatever) to the left and to the right.

Okay, let’s create a “Hello, Chimpzilla!” app that demonstrates all of this.

(I will explain the steps based on the Eclipse Android plug-in. If you haven’t set it up yet, see my first-steps article. If you prefer to work without Eclipse, there’s also an article for that.)

New Android Project wizard

1. In Eclipse, click “File” → “New” → “Project…” and select “Android Project”. Specify the project name, location, package name, etc. As the Build Target, select at least API Level 7 (Android 2.1), and specify 7 as the Min SDK Version. Click Finish.

2. Place the file “processing-core.jar” in the libs/ sub-folder of the project folder. Press F5 in the Package Explorer. The file should now be visible in your project tree.

3. Add a reference to “processing-core.jar” to the project: Click “Project” → “Properties”, select “Java Build Path” and go to the “Libraries” tab. Click the “Add JARs…” button and select the file “libs/processing-core.jar” from your project.

4. Double-click the file AndroidManifest.xml. On the Manifest tab, set the “Target SDK version” of the “Uses Sdk” element to 7 or higher. On the Application tab, set “Debuggable” to “true” (needed to debug the app on a physical device).

When you edit the file AndroidManifest.xml directly, it should look something like this:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
 package="org.realmike.chimpzilla"
 android:versionCode="1"
 android:versionName="1.0">
 <application android:icon="@drawable/icon" android:label="@string/app_name"
     android:debuggable="true">
   <activity android:name=".HelloChimpzilla" android:label="@string/app_name">
     <intent-filter>
       <action android:name="android.intent.action.MAIN" />
       <category android:name="android.intent.category.LAUNCHER" />
     </intent-filter>
   </activity>
 </application>
 <uses-sdk android:minSdkVersion="7" android:targetSdkVersion="7"/>
</manifest>

5. Place the file “chimpzilla.png” in the assets/ sub-folder.

6. Replace the code in HelloChimpzilla.java with this:

package org.realmike.chimpzilla;

import processing.core.*;

public class HelloChimpzilla extends PApplet
{
    PImage backdropImage;

    public void setup()
    {
    	orientation(LANDSCAPE);
    	background(39, 70, 134);
        backdropImage = loadImage("chimpzilla.png");
    }

    public void draw()
    {
        image(backdropImage, 0, 0);
    }

    public static void main(String[] args)
    {
        PApplet.main(new String[] { "HelloChimpzilla" });
    }
}

7. To debug the app, click the “Debug As…” button and select “Android Application” as the target. If this doesn’t appear, go to “Run” → “Debug Configurations…” instead and create a new launch configuration below “Android Application”. Click Debug.

Note: If the debugger stops because of a NullPointerException, the app probably can’t load the image file. Note that Eclipse does not rebuild the app package automatically when you change the files in the assets/ folder. If you placed the file chimpzilla.png in there after you edited the source code, the file might not be in the package yet. Make another source code change and save the file, and Eclipse should rebuild the app package the next time you start debugging.

8. Let’s add the code to scale the image:

    public void draw()
    {
        int newWidth = screenWidth;
        float scaleFactor = (float)screenWidth / (float)backdropImage.width;
        int newHeight = (int)(backdropImage.height * scaleFactor);
        int destX = (width - newWidth) / 2;
        int destY = (height - newHeight) / 2;
        image(backdropImage, destX, destY, newWidth, newHeight);
    }

Basically, we always scale the image to fill the width of the screen and draw it centered. This also works on aspect ratios smaller than 1.33 (where the background color will be visible above and below the scaled image). But we still have to add an “if” for extremely wide screens. On those screens, we will scale the image so that the height of the playfield (400 pixels) fits the height of the screen.

    public void draw()
    {
        float scaleFactor;
        float screenAspect = (float)screenWidth / (float)screenHeight;
        if (screenAspect <= 2)
        {
            scaleFactor = (float)screenWidth / (float)backdropImage.width;
        }
        else
        {
            final int PLAYFIELD_HEIGHT = 400;
            scaleFactor = (float)screenHeight / (float)PLAYFIELD_HEIGHT;
        }
        int newWidth = (int)(backdropImage.width * scaleFactor);
        int newHeight = (int)(backdropImage.height * scaleFactor);
        int destX = (width - newWidth) / 2;
        int destY = (height - newHeight) / 2;
        image(backdropImage, destX, destY, newWidth, newHeight);
    }

That’s it.

You might have noticed that we didn’t take Android’s screen classification (“small”, “normal”, etc.) or the screen density (“ldpi”, “mdpi”, etc.) into account when scaling the graphics. It simply wasn’t necessary to use different resources or a different formula based on this information. Once we add UI elements to the screen, we’ll have to take this into account. For example, when loading graphics for touch buttons, we might want to scale them according to the density, so that they end up in a touch-friendly size on the screen. (See my previous article for more about size classifications and densities.)

You can download a ZIP file containing the source code and the graphics.

Photo of the Boston Skyline is © 2007 Rene Schwietzke, licensed under a Creative Commons Attribution 2.0 Generic license. Boat and Chimpzilla graphics are from openclipart.org and have been dedicated to the public domain.

Some combination of adding and deleting images too fast broke it...

While editing my previous post, my WordPress installation suddenly started acting up on me. The problems had to do with uploading and deleting images using the post editor. I think what I did was upload an image, then click the “Delete” link right in the upload dialog—probably while WordPress was still updating the tables or something.

Anyway, suddenly the Media Library would not show any items anymore, and soon after that, the Posts were gone as well (both in the Admin interface and on the real site).

I checked the server logs* and found a number of relevant error messages:

*) I’m on Yahoo hosting. The relevant file was scripts.log in the logs directory of my site, accessible either through FTP or at http://yourdomain/logs.

[21-Dec-2010 11:26:48] WordPress database error Duplicate entry ’289′ for key 1 for query INSERT INTO `wp_postmeta` …
[21-Dec-2010 11:35:34] WordPress database error Incorrect key file for table ‘wp_posts’; try to repair it for query UPDATE `wp_posts` …
[21-Dec-2010 11:42:30] WordPress database error Can’t open file: ‘wp_posts.MYI’ (errno: 145) for query SELECT post_modified_gmt FROM wp_posts …

I did a Google search, which lead me to believe it was most likely some kind of database corruption. I logged into phpMyAdmin* and executed the following SQL command:

*) phpMyAdmin is a MySQL admin interface. It can be easily installed on Yahoo hosting.

CHECK TABLE wp_postmeta;

The check revealed that the table was broken in some way:

Table     Op     Msg_type     Msg_text
blog.wp_postmeta     check     warning     Table is marked as crashed
blog.wp_postmeta     check     warning     Size of datafile is: 53116       Should be: 51228
blog.wp_postmeta     check     error     Record at pos: 43796 is not remove-marked
blog.wp_postmeta     check     error     record delete-link-chain corrupted
blog.wp_postmeta     check     error     Corrupt

To fix the problem, I repaired the tables via SQL:

REPAIR TABLE wp_postmeta;
REPAIR TABLE wp_posts;

Repairing the tables apparently worked:

Table     Op     Msg_type     Msg_text
blog.wp_posts     repair     info     Wrong bytesec: 225- 36- 46 at 1431684; Skipped
blog.wp_posts     repair     info     Found block that points outside data file at 14317…
blog.wp_posts     repair     info     Wrong bytesec: 111-115-115 at 1451636; Skipped
blog.wp_posts     repair     info     Found block that points outside data file at 14516…
blog.wp_posts     repair     info     Found block that points outside data file at 14525…
blog.wp_posts     repair     warning     Number of rows changed from 286 to 288
blog.wp_posts     repair     status     OK

The blog appears to work again. All posts are back and the Media Library works as well. This was the first real WordPress issue I had—let’s hope it will remain the last one for a while.

Samsung Galaxy Tab (Android Virtual Device)

In my ongoing effort to port a desktop Processing application to Android, I am now trying to add support for multiple screen sizes. The goal is that the same .apk works on, say, a 3.7″ Milestone equally well as on a 7″ Galaxy Tab and on a multitude of other screens. The Android Dev Guide has a comprehensive page that explains the platform features for supporting different screens. I did my own tests and experiments to better understand the concepts. This article explains my findings and hopefully saves you some time and work.

Note: Android 3.2 introduces “numeric selectors” to manage different screen sizes, which I don’t explain here.

Android abstracts different screens into 4 generalized sizes and 4 generalized densities (as of version 2.3 of the SDK):

Screen sizes: small, normal, large, xlarge

Densities: ldpi (120 dpi), mdpi (160 dpi), hdpi (240 dpi), xhdpi (320 dpi)

Apps can provide layouts and resources for all the 16 different combinations (one set for “normal-hdpi” screens, one set for “large-ldpi” sreens, etc.). If the app does not provide resources for a specific combination, the platform will scale images automatically and employ other tricks to make the app still work. We’ll take a look at this later.

The pixel resolution, as in “800×400″, is not that important to supporting different screens. In fact, the Dev Guide notes that “in Android, applications do not work directly with resolution.” Apps do have access to the actual width and height in pixels of the screen. For normal Android GUI apps, however, the widget classes are going to handle the resolution transparently.

So, what exactly are size and density and how do they correlate to the resolution in pixels?

The density is a function of the screen size and the resolution. A 3.7-inch screen with a resolution of 854×480 pixels has a density of 265 dots per inch (dpi).

Size of a 854×480 screen at 240 dpi

Android collapses all physical densities into the four predefined categories. A density of 265 dpi will be reported as 240 dpi (“hdpi”).

The density tells you how large a bitmap with a specific resolution will appear on the screen. For example, on an “ldpi” screen (120 dpi), a 120×120 bitmap occupies a 1-inch square. (This is only a rough estimate, as Android does not report the exact physical density.)

In your app, you can provide different versions of images for various densities. For example, you can provide a 100×100-pixel bitmap for an “mpdi” screen and a 150×150-pixel bitmap for an “hdpi” screen, which has 1.5 times the density. Both images will appear in roughly the same physical size on the two screens. So, a higher density allows you to provide images that contain more detail.

A higher density does not necessarily mean that you can cram more widgets or more lines of text onto the screen. A 3″ screen is still a 3″ screen, whether it has 160 dpi or 240 dpi. On the “hdpi” screen, you might simply want to scale all images and widgets by a factor of 1.5, so that they are still readable and easily touchable.

To provide density- or size-specific resources, you add a suffix to the sub-folders of your project’s res/ folder. The right resources will automatically be loaded by the Android platform.

res/
    drawable-mdpi/
        icon.png
    drawable-hdpi/
        icon.png
    values/
        strings.xml
    values-large-hdpi/
        strings.xml

If you don’t provide resources for a specific size or density, the platform will load the default resources, which are assumed to be for a “normal-mdpi” screen. If, on an “hdpi” screen, the platform can only find “mdpi” images, it will scale them up for you. This is all transparent to the app.

The Dev Guide describes the rules of loading alternative resources in detail.

FroYo on a 1024×600 160-dpi screen

FroYo on a 1024×600 240-dpi screen

As an example, let’s take a look at two hypothetical 1024×600 screens. One is a 7.4″ screen, so it has a density of 160 dpi. The other one is a 4.9″ screen with a density of 240 dpi. For the first device, the platform uses the 48×48-pixel “mdpi” versions of the app icons, which are .3″ wide on the screen. For the second device, the 72×72-pixel “hdpi” versions are used (exactly 1.5 times as large), which is also .3″ on the screen. (In the screenshots, you can clearly see that the Alarm Clock icon is a different resource.)

Both screens have the same resolution, and the icons appear at the same physical size. But as the first screen uses smaller bitmaps, there is much more empty space between the icons. One could argue that the layout of the second screen looks prettier.

Apparently, that’s also what Samsung found when they made the Galaxy Tab. The Galaxy Tab has a 7″, 1024×600 screen with 170 dpi. Yet, the Tab does not report its density as “mdpi” but as “hdpi”, so the layout looks exactly as in the second screenshot. If they used “mdpi”, the icons would be .28″ wide, with “hdpi”, they are .42″ wide—not a big deal, and I must admit, the layout does look prettier this way.

(Talking about the Galaxy Tab, Samsung offers an add-on for the Android SDK that contains a virtual device skin that you can use for testing your app.)

Okay, how does all this apply to Processing apps?

First and foremost, make sure that your AndroidManifest.xml specifies the correct target SDK. For example, if you are developing for Android 2.2, the target SDK level is 8. As Processing requires at least level 7 to run at all, you’d specify 7 as the minimum SDK level:

<uses-sdk android:minSdkVersion=”7″ android:targetSdkVersion=”8″></uses-sdk>

The same Processing app without the <uses-sdk> element (left) and with targetSdkVersion=

If you don’t have a <uses-sdk> element at all, the platform assumes that your app was written for Android 1.5, for a “normal-mdpi” screen. When your app actually runs on an “hdpi” screen, the platform will scale all loaded images by the factor 1.5, and the width() and height() Processing functions will return the resolution divided by 1.5. For example, on a Milestone with its 854×480 screen, width() and height() would return 569×320. To avoid this, add the <uses-sdk> element.

A Processing app doesn’t use regular widgets or any of the Android layout managers. Therefore, you’ll have to calculate your own layout based on the reported resolution and density. To query the resolution and density, use the DisplayMetrics class:

DisplayMetrics dm = new DisplayMetrics();
getWindowManager().getDefaultDisplay().getMetrics(metrics);

For a WVGA854, “normal-hdpi” screen, the values of the DisplayMetrics object are as follows:

DisplayMetrics{density=1.5, width=854, height=480, scaledDensity=1.5, xdpi=240.0, ydpi=240.0}

If your graphics were designed with a medium-density screen in mind, you can scale all graphics by the factor given in the “density” field to match the current screen—if you want to retain the same physical size, that is.

If you need the size classification—”small”, “normal”, “large”, “xlarge”—use the Configuration class:

Configuration config = getResources().getConfiguration();
int size = config.screenLayout & config.SCREENLAYOUT_SIZE_MASK;
if (size == config.SCREENLAYOUT_SIZE_SMALL)
{
    // use special layout for small screens
}

In my Processing app, I store all graphics in the assets/ directory. The screen-specific suffixes do not work there in the same way they do in the res/ directory. Apart from that, for reasons specific to my app, I do not want to access the graphics by the auto-generated constants like “R.drawable.my_image” (if that’s even an option using Processing). Therefore, I just do my own scaling, just like I would for a desktop application where the user can resize the window.

Talking about the auto-generated constants: There is a field “R” in the interface processing.core.PConstants, which conflicts with the namespace for the resource constants. When accessing any Android resources through these constants, you get an error like this:

The primitive type int of R does not have a field string
The primitive type int of R does not have a field drawable
The primitive type int of R does not have a field layout
The primitive type int of R does not have a field id

The solution is to qualify “R” with the package namespace when accessing the resource constants:

String someText = (String) getResources().getText(org.realmike.helloandroid.R.string.someText);

Conclusion

When scaling a desktop application according to the client size of the window, there’s usually no need to worry about the physical size that the UI elements will end up at. It’s all different on a touch screen device, where you want to keep UI elements legible even on small screens, and you want users to interact with the app in a comfortable way. Thinking in terms of densities and screen sizes is a useful abstraction in this scenario.

For multimedia apps (say, games with pre-rendered graphics), there are unique challenges to supporting multiple resolutions and aspect ratios (I haven’t talked about aspect ratios at all). These challenges are not unique to Android, and you can apply the same techniques as to a desktop application.

If Angry Birds looks great on a multitude of devices, screen-independence should be possible for the rest of us to achieve as well.

The android.media.MediaPlayer class is just fine for that.

In general, a sound can be played using the following code:

snd = new MediaPlayer();
snd.setDataSource(path_or_url_of_sound_to_play);
snd.prepare();
snd.start()

My own app is a game that contains only a handful of .ogg files. I keep a HashMap of the MediaPlayer instances for all sounds in the game, with prepare() already called on them. Then I simply call start() whenever an event triggers the sound in the game.

I store all sounds in the assets/ sub-folder of the project, so that they are automatically included in the app package. However, this does not mean that these files will exist in the current working directory when the app is run.

Loading the files as if they existed in the current working directory does not work:

snd = new MediaPlayer();
snd.setDataSource("./Sounds/win.ogg");
snd.prepare();  // <- raises an exception because the file does not exist

This results in an error similar to this:

PlayerDriver: Command PLAYER_SET_DATA_SOURCE completed with an error
or info PVMFErrNotSupported
MediaPlayer: error (1, -4)
SoundTest: Error! java.io.IOException: Prepare failed.: status=0×1
at android.media.MediaPlayer.prepare(Native Method)

In Processing, you can normally use the createInput() method to open a file in the assets folder. This method returns an InputStream, but there is no way to initialize a MediaPlayer from an InputStream.

I did try to use the dataPath() method of the sketch to retrieve the absolute path to the file, but as we’ll see, this doesn’t work either:

snd = new MediaPlayer();
snd.setDataSource(theSketch.dataPath("Sounds/win.ogg"));
snd.prepare();  // <- raises an exception because the file does not exist

The dataPath() method returns a path like “/data/data/org.realmike.myapp/files/data/Sounds/win.ogg”, but no such file exists. All files in the assets/ sub-folder are actually inside the .apk (which I suppose is some kind of .jar file). They do not exist as separate files in the filesystem.

So, to load sound files stored in the assets folder, you should retrieve the AssetManager instance from the sketch and initialize the MediaPlayer with a file descriptor:

snd = new MediaPlayer();
AssetManager assets = theSketch.getAssets();
AssetFileDescriptor fd = assets.openFd("Sounds/win.ogg");
snd.setDataSource(fd.getFileDescriptor(), fd.getStartOffset(), fd.getLength());
snd.prepare();

Make sure to pass fd.getStartOffset() and fd.getLength() to setDataSource(). The file descriptor returned by fd.getFileDescriptor() refers to the entire app package. Without the offset and the length, the MediaPlayer will try to interpret the .apk file as a sound file—and fail.

There is a different way to load sounds: As described in the Android Dev Guide, you can also store sounds in the res/raw/ folder of the project and use MediaPlayer.create() to load them:

MediaPlayer mp = MediaPlayer.create(context, R.raw.sound_file_1);
mp.start();

As far as I can tell, this approach requires you to hard-code the integer constants that refer to the sounds (“R.raw.sound_file_1″) in the source code*. As these constants are auto-generated during the build, I did not find a way to store these constants in a data file that my app could read, so that I could associate in-game events with sounds at runtime.

*) I don’t know enough about Java introspection to turn a string such as “sound_file_1″ into the corresponding constant from R.raw.

Portions of this page are reproduced from work created and shared by Google and used according to terms described in the Creative Commons 3.0 Attribution License.

I wanted to look at the data graphically, so I created two OpenOffice.org charts.

Operating System Shares

OS Platform Statistics - Yearly averages

Not surprisingly, around 90% of visitors run Windows. But Windows usage is declining slowly, from 93% in 2003 down to 88% in 2010. Mac and Linux* are both growing slowly but steadily (Mac faster than Linux). All other operating systems account for only 0.67%.

*) I assume most of the Linux usage is from GNU/Linux, i.e., your average desktop Linux.

The w3schools.com statistics are obviously biased towards tech-savvy users (who else would read w3schools.com?). It almost certainly doesn’t mean that GNU/Linux runs on 5% of computers worldwide. The Wikipedia page on operating system usage share comes up with much lower numbers for desktop GNU/Linux.

My personal (biased) interpretation is this: The more serious the computing gets that you are doing, the more likely you are to encounter Linux and GNU, from 2% in the general public, to 5% among web developers, to 43% on servers, to 92% on supercomputers*.

Chart source (OpenOffice.org spreadsheet)