subogero linux blog

I’m through my most swearword-intensive development ever: creating a Debian package from the omxd Raspberry Pi media-player daemon. Debian developers take note: if strange freak-accidents happen to you and your families, it’s my fault. I’m currently being treated with post-traumatic stress disorder and want to forget the whole experience.

Except, I decided to write down the most interesting stations of my journey through the Valley of the Shadow of De… Debian. Documentation is available but hard to find, as most Google hits are about using deb packages, not creating them.

If you are an experienced Debian developer and find that I’m a noob, an idiot and a luser, you’re right. But then, why does Linus Torvalds refuse to provide SubSurface binaries for Linux? Exactly. Because he did not let his brain rot by thinking that creating deb or rpm packages is something sane. And no, the problem is not fragmentation. Once you’ve created you deb package, alien will turn it into and rpm in no time. It’s creating the deb that’s so difficult.

Init Scripts

Originally make install just inserted an omxd call into /etc/rc.local and started omxd. But better don’t mess with existing files, so I changed it to create /etc/init.d/omxd with start/stop/restart verbs and registered it for auto-start by

update-rc.d omxd defaults

and finally doing a

service omxd start

How naive I was…

Automated Releases

I copied the make tag rule from szg, which creates releases, where the tag message is a commit list in Debian changelog format.

$(DESTDIR)

In your Makefile, use the $(DESTDIR) prefix whereever you install something.

cp omxd $(DESTDIR)/usr/bin

debian/omxd.dirs

I did not remember this, but this file is needed with a list of directories you install files into. Note the lack of leading slash:

usr/bin
usr/share/man/man1
usr/share/doc/omxd
etc/init.d

force-reload

Lintian throws an error that you need a force-reload verb in your init script. So I gave in.

Auto start/restart/stop omxd – The Real Troubles Begin

Having make install starting and registering your daemon does not help. dpkg-buildpackage just uses this rule to collect the installed files. Later when the deb is installed, it’s not run. You need a debian/postinst maintainer script to do a

service omxd start

But during upgrades you need to stop it first, so add prerm and preinst scripts too with

service omxd stop

Idempotency

But a reinstall/upgrade will fail if your init script returns an error when attempting starting an already started daemon or stopping an already stopped one or trying to do anything with a non-existing one. So make sure they are idempotent: running them repeatedly has no effect at all, not even an error. In case of preinst:

service omxd stop || true

Also make sure the init scripts start/stop verbs don’t fail for these cases.

debian/conffiles, Bloody conffiles

dpkg-buildpackage considers anything you install into the /etc directory a configuration file and adds them automatically to the debian/conffiles file. Result: if you develop something using make install/uninstall, then install the deb package again, dpkg will refuse to install the those files, because their lack is a modification by the sysadmin. Unbelievable but true. And there is no obvious workaround, either. All you can do, and this is confirmed by the Debian docs, is to create anything in /etc by the debian/postinst script.

There are no words for the idiocy of the Debian conffiles policy. How often have you been prompted by Ubuntu/Debian package upgrades about mysterious config file changes you’ve never ever touched? It’s all because it’s so difficult to get it right.

And why is anything in /etc added to conffiles with no possibility to override? Sure, /etc/init.d/omxd is a config file. But it’s the init system’s config file, not mine!

And Finally, lintian

I used lintian. Did it complain about the lacking debian/omxd.dirs file? No. Did it warn me about the auto-start idempotency problems? No. Did it complain about a completely useless force-reload initscript verb? Yes. Did it complain about the lack of all the useless and paranoid GCC warning options in my makefile? Yes. And finally did it throw an error upon the only existing workaround for the conffile-hell? Yes.

Lintian is a useless piece of shite. I threw it out.

make utag

During packaging I had to undo release commits and tags countless times. I even automated it with make utag.

Conclusion

The Debian packages for omxd and the latest stable omxplayer are available in my repo. Just add it to your software sources by adding the file /etc/apt/sources.list.d/subogero.list containing this:

deb http://linux.subogero.com/deb/ /

The install

apt-get update
apt-get install omxd

I’m also accepting donations for my post-traumatic stress disorder treatment on my PayPal account.

Just a few photos of this stunning December moning with my new S5 Mini.

It’s been four long years since Ubuntu 10.04 Lucid Lynx. It was the perfect desktop OS. It was extremely easy to use for both mouse and keyboard junkies. And I remember showing off wobbling transparent terminal windows on a rotating desktop cube to baffled Windows and Mac fans.

And then Red Hat’s and Canonical’s brain-dead corporate fighters just had to destroy everything. Yes, step forward, William Jon McCann, project lead of the tasteless and dysfunctional Gnome 3. You don’t even know what XFCE is. If you’re interested, it is a means of survival in the dystopian world you created.

Fortunately, there were always silver linings in the clouds. Mainly thanks to Linux Mint. And now, with Linux MInt 17.1 Rebecca MATE, Compiz, our favourite compositing window manager, is back.

And what I’m most happy about is not Wobbly Windows. Not the Desktop Cube. Nor the Ring Switcher.

It’s the Place Windows plugin.

The world’s one and only sane window placement algorithm. It does not pile new windows on top of the old ones, but places them in the furthest corner of the screen to utilize real estate and keep many windows visible. Bliss.

I’m constantly bombarded with RESTful stuff at work. So much so, that I’ve started rewriting RemotePi as a REST API. Lots of learning and reading, then.

As usual most of the literature was written by people who studied computer science. They just love abstractions. Solving problems is OK, as long as they get their daily hit of abstractions. So there are resources, verbs and routers. Very nice.

You’re supposed to refer to a resource without ugly queries in the URL. Instead, it should be just http://pi/remotepi/home/ for the media root directory. But how the hell does such an URL generate a query that can be processed by a CGI or FastCGI app?

The computer scientists are silent on the issue.

So I had to dig deeper and discover mod_rewrite of Apache2. There is a lot in a name. It rewrites incoming URL requests and is able to turn them into proper queries. Thank you Ludovico Fischer.

So here is my “resource router”.

First enabled mod_rewrite:

a2enmod rewrite
service apache2 restart

And the router config in /etc/apache/sites-available/default

RewriteEngine On
RewriteRule /remotepi/(.+) /remotepi/api.pl?$1

Very Perly, ain’t it? It’s because it uses the Perl regex engine.

RemotePi, the remote-control webb-app of my Raspberry Pi media center, felt sluggish. Until I measured the response time with Firefox WebDeveloper/Network. It’s now official: it was sluggish: it took 800 ms to respond. That’s nearly one second.

So it was time to turn the original CGI solution into a FastCGI one. Instead of firing up a new Perl process upon each request, the app keeps running in the background and replies requests in a main loop. I’m using Apache2 mod_fcgid.

The improvement is shocking: File browsing is about 25 ms, requests involving an omxd call take 80 ms. And that’s basically the pure runtime of the omxd command.

FastCGI is fast indeed.

To cut a long story short: YouTube finally works in HD on my TV with RemotePi. I can’t stop watching LaFerrari and McLaren P1 videos in HD with great sound!

Why was it so difficult?

In the old times, youtube-dl -g spat out YouTube stream URLs, and omxplayer could play them straight away. The excellent ncurses based yt worked this way. But things have changed. YouTube now only streams the video if a session cookie is presented. But omxplayer can’t use cookie jars.

But curl can! We tell youtube-dl to save the cookie, and let curl save/stream the video using the cookie.

curl -b jar `youtube-dl -g --cookies jar --write-thumbnail`

will spit out the video stream.

1st Attempt – Save to File

Let curl save to a file, and start omxplayer a few seconds later to play it back. Keep fingers crossed that curl saves faster than omxplayer reads.

It does not. This solution is so disk/sdcard IO intensive, that omxplayer will reach the EOF too soon and exit.

2nd Attempt – Stream via FIFO

Let’s create a FIFO file, and let curl write the video stream into that. And start omxplayer immediately to read the video stream from the FIFO. It works beautifully:

• there is no disk IO at all, so curl writes faster than omxplayer reads
• even if it’s not the case due to a slow network, the player does not exit, just blocks until curl catches up. Bacause the FIFO is a character device!
• if omxplayer exits for any reason, curl gets a SIGPIPE and exits too

The Result – rpyt

It’s all wrapped up in a new script packaged with omxd: rpyt.

And the RemotePi remote-control web-app also uses that from now. See the project homepage and the code on Github.

This is the current state of RemotePi. Note the progress bar for the current track (fully text based), and the cover art. Involving no MP3 tags, no 3rd party databases, just good old directories and files.

Also note my current taste in music. Most say it’s crap. Others say it’s fun. It’s definitely controversial and definitely not mainstream.

Quiz: What model is the phone used as remote control?

Perl always does what you want, except if you want consistency.

I remember those “Wouldn’t it be great?” moments throughout my Perl hacking years. The answer was invariably “Yes, you can”.

Wouldn’t it be great if you could interpolate variables into regular expressions? Yes. you can.
Wouldn’t it be great if you could address a hash with a subset of keys and get a subset of values? Yes, you can.

Which, in turn, brings me to hash slices and a few related tricks.

Slicing Hashrefs

This is how you slice a hash, building an array of values from an array of corresponding keys. Pythonistas may now notice the beautiful simplicity compared to their silly list comprehensions…

%hash = (one => 1, two => 2, three => 3);
@one_three = @hash{'one', 'three'};

But how to slice a hashref?

$hashref = {one => 1, two => 2, three => 3};
@one_three = @{$hashref}{'one', 'three'};

The rule is that if you have a $hashref instead of a %hash, you can do everything the same way, just replace the bare sigil-less name of the hash with {$hashref}. Pythonistas are at this moment enlightened about the usefulness of sigils.

Merging Hashes

Hash slices can be lvalues too, so you can assign an array of values to an array of keys.

%numbers = (one => 1, two => 2);
%more = (three => 3, four => 4);
@numbers{keys %more} = values %more;

Note the @ sigil in front of all hash slices. They are arrays!

How can a cursing, politically incorrect, rude man lead a project of Linux’ size. This is how:

On Wed, Apr 2, 2014 at 4:28 PM, Andrew Morton <akpm@linux-foundation.org> wrote:
>
> Could be done per-fd: put a struct ratelimit_state into struct
> devkmsg_user.

Yeah, what Andrew said. My suggestion of per-task or per-cred is
obviously moronic in comparison.

                Linus "hangs head in shame" Torvalds

YouTube parties have a few flaws. Five to ten people gather around a small laptop, an even smaller tablet or a tiny smartphone. Nobody can see the screen really well. It’s no biggie, actually, as nobody cares for videos someone else wants to show. You only want to show off yours. And here is the problem:

You have to wait until the end of that lame boring clip, just to start the search for that brilliant masterpiece you’ve come across last night.

And that’s where remotepi and youtube-dl come for the help of the party, to fix the small problem and the big problem. First, the actual video is played on your huge flatscreen TV by the Raspberry Pi. Second, everyone can search for the next clip on their mobile phones in the meantime!

Head to the homepage or github.com/subogero/remotepi to solve your YouTube problems.

And why semi-ultimate? Erm, a few small problems. The omxd playlist daemon may or may not work properly with the freshest omxplayer. Also, a few video URLs can’t be played by the latter, probably due to authentication problems. But give it a try, anyway.