subogero linux blog

One may wonder what was happening in the last few months. Everyone knows about the Euro-crisis. A certain Mr Erdogan has been trying to revive the Ottoman Empire, sharpening his Unix-skills, especially bashing, on Israel.  Mahmoud failed to turn Judea and Samaria into a Judenrein state at the UN. It also turned out that a single Israeli soldier is worth 1027 heroic Palestinian freedom fighters. What’s more, Muammar Kadhafi, in a markedly Perl-like fashion, said:

keep $power or die $!

Yes, but what about the real important stuff, you may ask. Oh, I see what you mean! Here it is.

I needed a recursive git-grep function to scan submodules as well. So I wrote one. Along with a vim-plugin, which is a beautiful piece of plagiarism, but that’s open source for you!

http://github.com/subogero/git-grep-recursive-vim

I got acquainted with gdb, the GNU Debugger, courtesy of Richard Stallman. Typical of him, it prints 2 pages of licensing info and political agenda when started up. But after you get rid of that,

alias gdb='gdb -q'

what a brilliant piece of software it is! Easy, comfortable, fast, debugging on the command line. It’s just incredible. Gdb is a bit better explained here:

http://betterexplained.com/articles/debugging-with-gdb/

I rewrote my calculator, because one had to type too many uppercase letters for hex numbers. It’s all lowercase now. And it’s called “szg”, and no, not after my initials. It means SZámolóGép, which is Hungarian for calculator. Nothing to do with Hungarian notation, though. Which one of my colleagues turned out to actually like. Strange taste. It must give a deep satisfaction, when after 2 years of maintenance “u8Foo” has become a signed long.

Anyway the new calculator is available on github:

http://github.com/subogero/szg

At the office, I’m now co-developing a rather large project in git with a colleague. Lots of fun…

Having mentioned Muammar Kadhafi, there was a lot of hype about the death of Steve Jobs recently. A much more important person has passed away too: Dennis Ritchie, creator of the C programming language, co-creator of Unix with Ken Thompson. As a tribute, some quotes from the two heroes:

Obviously, the person who had most influence on my career was Ken Thompson.
Dennis Ritchie

That brings me to Dennis Ritchie. Our collaboration has been a thing of beauty.
Ken Thompson 

ogc 6.0 rc2 is out for the general public to enjoy. It now supports any number of user defined variables in the form of $FOO or $SPAM. Dollar sign plus all-caps. I know, it’s somewhat LarryWallesque. The dollar-sign, not the all-caps, you understand.

Just to show the severity of my mental state, I’ve implemented “hash-buckets” in C. With vi.

I’ve just reached a very severe stage of the disease called Unix. I’m learning vi.

I still vividly remember the shock when a command-line Linux expected me to edit something and kindly opened up vi for me. I did not know how to exit, so I had to shut down the computer.

After using pagers (more or less) and manual pages for some time, I slowly learned a few important hotkeys like “/pattern” to search forward for something, “n” to find the next match and “q” to quit. Playing the command-line version of robots, one quickly memorizes “h”, “j”, “k” and “l” for navigation. It turns out they all come from vi. Or vice versa.

But before going on, I’d like to enlighten my readers about quitting vi:

:q<Enter>   Quit
:q!<Enter>  Quit and discard changes
:wq<Enter>  Save and quit
ZZ          Save and quit

Now that quitting is out of the way, let me tell you about a few highlights of my steepish vi learning curve. It seems the usefulness of Unix tools is proportional with their scariness.  Same as with git.

Those 1-2-letter commands are quite logical, actually, usually consisting of a command and a move, eg “dw” stands for delete word.

Let’s take moving around in the window. After scrolling down a few pages, you find the interesting line, but it’s in the bottom of the screen along with the cursor. Command “z.” will put it in the middle.

When you try to select something with a Windows-style editor, you press Shift at the beginning, keep it pressed and move around with the arrows. Then you realize the beginning is at the wrong position. You can start over.

In vi, on the other hand, you press “v” for visual selection, move around to find the end of the highlighted block. Then you notice the beginning is wrong. Press “o” for opposite, your block’s end will get fixed and the cursor jumps to the beginning, which you can now move around. Astonishing!

And now to something completely different: the vi Cheat Sheet!

vi Cheat Sheet

Bonus: I was always wondering what text editor I should install onto my Android phone. I’ve just realized that busybox-vi has been there all the time. It works perfectly with the crude virtual keyboard.

Bonus 2: my ~/.vimrc file with syntax highlight, line numbers, autoindent, highlight search results and, most importantly, brighter foreground colours.

syntax on
set number
set autoindent
set hlsearch
set background=dark

I’m always very proud how incredibly easily my Linux systems cooperate with the external world, including Windows machines.

For instance, just now, after one day of searching, sweating and swearing, I was able to mount some Windows shares onto a Debian virtual server.

For those of you who boast that one can do this with two clicks in Windows, let me tell you that

1. it’s actually 6 clicks
2. I also could have used Nautilus and just connect to a server

But that would have been too easy and, at the same time, too difficult to use. Have you ever seen such a Gnome gvfs mount point?

~/.gvfs/My bloody space-separated sharename on Foo

Brrrr. One wants something nice like

~/Foo

One also wants to be able to mount and unmount it as a normal user. And to have write access. And not having to enter a password, but also not to reveal it to others. One is very picky.

Before all, don’t forget to create the mount point directory manually. It shall be world-executable as well.

$ mkdir ~/Foo
$ chmod 775 ~/Foo

Now let’s see the /etc/fstab entry. We’ll explain every option later.

//srv/share /home/foo/Foo cifs rw,user,credentials=/home/foo/.foo,uid=1000,gid=1000 0 0

The entries are share UNC path, mount point, file system type, options, etc.

If you miss anything from the dreaded options, you’ll get the following highly informative error message when trying the mount:

mount error(13): Permission denied

Option “rw” means read-write. Easy.

Option “user” means not only root is allowed to mount or unmount the share.

Server “srv” is in domain “bar” and shares “share” for user “baz” whose password is “spam”. We could add these options directly to fstab (username=bar,password=spam,domain=baz), but that would make it world readable. Instead the info is stored in a credentials file in Linux user foo’s home. Make sure this file is readable by foo only. Let’s see its contents too:

$ chmod 600 ~/.foo
$ ls -l ~/.foo
-rw------- 1 foo foo size date .foo
$ cat ~/.foo
username=bar
domain=baz
password=spam

Don’t put any whitespace around the “=” signs. There is a school of thought that there should not be any character after the end of the password, not even a newline.

And last but not least, the Linux user’s user and group IDs should be specified as well. In our case “foo” is the first normal user on this machine, that’s why it’s 1000.

Now user foo should be able to mount and unmount the above share:

$ mount ~/Foo
$ umount ~/Foo

The new version of the Lazy Man’s calculator maximizes laziness by taking your expressions directly from the command line. No pipes, no interactivity, no interpreter-scripts. Just what the customer wants.

$ ogc -e 2*21
42

The clever customer can actually take ogc’s laziness to even greater extremes. Just add the following line to ~/.bashrc

alias og='ogc -e'

And voilá:

$ og xFFd
255

You can download sources and Linux/Win32 binaries from the ogc page.

Geeks can clone the git repo as well:

git clone https://github.com/subogero/ogc.git
git clone http://subogero.dyndns.org/git/ogc.git

My Moto V3i began to give up. As usual, the flip bearings became loose and, the phone reset itself ever more often, usually while receiving calls. Slightly disturbing.

So, eventually, I also gave up resisting the smartphone wave and started looking for incrementing my Linux kernel-count. The main candidate was the HTC Hero with its sturdy aluminium housing. That is, until I found the strange Motorola Defy on the Hungarian Price Machine. Slightly cheaper, with an even more sturdy shock- and waterproof housing and scratch-proof glass. Hungarians don’t like stuff if it is scratched. The gadget also has a 0.8 GHz ARM processor, accelerometers, n-type WiFi, GPS and a Full Wide touchscreen with 854 x 480 resolution.

True to form I did two things with it on the first day:

• rooted it
• poured water on it

Both with excellent results. It also got a Terminal Emulator and Busybox installed on it. The reason? It turned out that Android’s underlying Linux is an extremely crude environment. First, it has no GNU tools. And they did not even bother with installing Busybox, the usual commands are links to either Google’s toolbox or Motorola’s motobox. Both very poor. My 8 MB OpenWrt router’s command line environment is a luxurious place compared to this phone (half gig internal memory, remember).

Oh, and how does it work as a phone, you may ask? Quite brilliantly, actually.

I no more need a camera. It takes brilliant pictures in daylight.
I no more need to carry a private laptop while travelling, as it’s got WiFi, Email and Web.
I no more need maps or a GPS device. i just GO.
I no more need those bloody 20th century CDs in the car. Just play music from the phone.

One Gadget To Rule Them All.

I’m not referring to the most boring sport EVER. Instead, it’s a command line tool I’ve discovered recently: curl. You specify a URL on the command line (e.g. curl http://napirajz.hu), and it prints the contents to standard output.

Time to write my most popular website ever:

The business case:
Bloody multinational corporations filter the internet. One cannot access inspiring content like http://napirajz.hu. Let’s make it accessible.

The tools:
My OpenWrt router hanging on the internet. I’ll create a mirror site on it, refreshed every day at 5:00 (before breakfast), 9:00 (arriving at the office), 13:00 (after lunch) and 18:00 (before I get home).

The beloved cron daemon will do the timing. And the mirroring? On OpenWrt, one lacks the luxuries or Perl and even the cosy comfort of Bash. One has to make do with the BusyBox versions of the Almquist Shell (ash), curl, sed and wget.

#!/bin/ash
PICS=`curl 'http://napirajz.hu' 2>/dev/null \
| sed -nr 's,^[\t ]*<p><img src=\"([a-z0-9/_.]+)\".+$,http://napirajz.hu\1,p'`
rm -f *.jp*
echo '<html><head><title>subogero napi</title></head><body>'
echo '<h1 align="center">subogero napi</h1>'
echo '<hr>'
for i in $PICS; do
 wget $i
 echo $i \
 | sed -r 's,^.+/([a-z0-9_]+)(\.[jpeg]+)$,<p><a href=\"napi/\1\2\">\1</a>,'
done
echo '<hr>'
echo '(c) CC - Mer&eacute;nyi D&aacute;niel'
echo '<br>Eredeti: h t t p : / / n a p i r a j z . h u'
echo '<p>Friss&iacute;tve:'
date "+%Y.%m.%d %H:%M"
echo '</body></html>'

See the result at http://subogero.dyndns.org/git/napi/

An OpenWrt router needs to be extended a bit to run big stuff. For instance to build a git-hub. The installed size of git is 91MB. One has an external USB HDD, of course, but what is the right way to do it? The forums are full of extroots and opts and similar stuff. I think they are all evil.

Extroot seems to involve compiling custom firmware images, while /opt brakes the nice Unixy directory structure (standard locations of programs, libs and daemons). There must be a better way:

Mount a partition of an external USB HDD to /usr

Just think about it. The really big files go to /usr/bin /usr/sbin and /usr/lib almost exclusively. Only small stuff goes to /etc, like config files and daemon startup scripts.

Today my theory was confirmed by practice. It’s very simple. These are the important steps:

1. Create an ext3 partition on your HDD just for this purpose

I actually did the partitioning before, on my normal Ubuntu laptop. It’s 2 GB and it’s called /dev/sda2 on the router.

2. Mount it to /opt temporarily

Create the /opt directory (or whatever else) temporarily and mount your partition into it using LUCI/Administration/System/Mount Points.

3. Copy the entire contents of /usr to /opt

The goal is that, initially, /usr looks the same with or without mounting something into it.

$ cp -dpr /usr/* /opt # preserve symlinks

4. Mount partition to /usr

Unmount /dev/sda2 from /opt, mount it into /usr using LUCI. You can also delete the temporary /opt directory.

5. Update /etc/opkg.conf

This involves OpenWrt’s most mysterious concept, overlay. Changing the line below has no effect except to calm down the worrying router about available storage-size for new packages:

option overlay_root /usr

You’re ready to install anything. As for me, I’ve since reinstalled mc and git. No dirty hacks this time with inserting /opt into PATH and manually creating some symlinks from /usr to /opt/usr. Just installed them like a breeze. My github is up and running again.

During my exploits to spread the Gospel of git-managed websites, once a great crisis broke out. A user threatened me he would return to MKS Source Integrity.

Now, such an event would shake the foundations of the entire Open Source Free Software movement. Richard Stallman would put sackcloth upon his loins. Linus Torvalds would rend his clothes. I would mourn for many days and my soul would refuse to be comforted.

Time to do something. Or rather, time to think.

I had actually been wondering, why Git’s submodule interface is so inconsistent. There is a –recursive clone, but there is no recursive option for any other remote operation. And then I was enlightened. Of course

There Is No Recursive Push

How could there be? The entire concept of submodules is based on universally read-accessible subprojects inserted into your superproject. You reuse one from gnu.org, another from the Moon, and some from Alpha Centauri. Write access for these is not guaranteed, it’s the privilege of a chosen few.

So what to do with your git-managed remote website? There are two options.

1. The remote repo has a detached worktree (repo in ~, website in /var/www)

Set up remotes and server-side post-receive hooks to check out for the subprojects.
You may want to add a pre-commit hook to the superproject pushing the submodules first.

2. The remote repo is in the website worktree

Add “git submodule update” to the post-update hook on the server side.
This assumes the server is able to access the submodule URLs.

Being a bloody webmaster has taught me a few hard lessons about git submodules.

The reason is that my CGI scripts managing the mailing lists and adding the navigation to the web pages proved to be quite reusable and I’ve moved them into their own git project. They are now used at at least 3 websites at the office as git submodules.

When Mr Noob first starts playing with them, usually this happens:

You have a project in ~/foo and another in ~/bar. Foo wants to reuse Bar as a subproject. So you call

$ cd foo
$ git submodule add ~/bar

It’s all very nice. Until, as usual, your Foo project becomes very popular and the whole Galaxy starts clonig it. At this moment great panic of cosmic proportions breaks out, as they cannot obtain the Bar submodule’s files from

/home/Noob/bar

Of course not. You get it? The Tao of git submodules is this:

Think Big.

In other words specify a path to your submodule in a universal way. With a Universal Resource Locator.

The used protocol is also of extreme importance. It should allow unlimited read access for your entire target audience. For write-access they can always add another remote to the subproject later. If they get write-access at all.

/somewhere/in/your/filesystem

We have already discussed this.
Bad.

smb://

Some will have access, but most not. Samba is a LAN thingy. And it’s bloody slow.
Bad.

ssh://

It’s a URL, but it’s not universal enough, as your users won’t have access.
Bad.

http://

Put your subproject on a webserver, allow directory listing: universal read access obtained.
Good. If a bit slow with git.

git://

Guarantees universal read access too, and it’s fast as well.
Good. The only problem is some corporate proxies are watching port 9418 with raised eyebrows.

Bottom line: use the http:// or git:// protocols.