Archive

Posts Tagged ‘LMDE’

Lured into Linux Real Time

April 13th, 2013 subogero Comments off

The support period of the Lucid Lynx on the Asus UL20A is about to end in May, so I was looking for a new OS. I gave a try to Debian Testing, or Linux Mint Debian Edition, to be precise. But not Pangolin. Sorry.

It was my smoothest Linux install ever. All hardware worked out of the box. Volume and brightness buttons? Check. Wifi? Check. Two-finger scrolling? Check. I also applied the usual MATE fixes for mutt and the power-button. Debian comes with a 486-kernel by default, so it handles only one processor core. One can upgrade manually to a multi-core 686-kernel with apt-get.

aptitude search linux-image

And what do I find?

linux-image-rt-686-pae

A PREEMPT_RT patched kernel, kindly compiled by Debian. Time to plunge into real-time again!

I put together a Linux syscall tutorial project a year ago (see github) which contained a small program performing a periodic real-time task and printing statistics like average/min/max dt and its standard deviation. It’s called rt and uses the setitimer() system call to generate periodic SIGALRM signals. Time to measure latencies of a 1ms periodic task on a RT kernel!

To my utter disappointment, performance was fine, until the tapeta daemon changed my wallpaper, that is. At which point a 300% latency appeared. That’s right, period time 4ms instead of 1ms. Same as earlier, without an RT kernel. What’s wrong?

I was not using real-time scheduling, that’s what. It turned out Linux processes run at static priority zero by default, where the actual scheduling priorities are dynamic, depending on interactivity, nice level, sleep/runtime, etc. This is called the SCHED_OTHER policy. In other words, one is at the mercy of the Completely Fair Scheduler.

On the other hand, a process’ static priority can be increased, and the scheduling policy can be changed to SCHED_FIFO with the sched_setscheduler() system call. In this case the only one whose mercy we are at, is another SCHED_FIFO or SCHED_RR (Round Robin) scheduled process with equal or higher static priority, lurking in the background somewhere. Important to note, though, a process needs to run as root to be able to change its own or another process’ scheduling. There is also a command line utility called chrt, to change the scheduling policy of processes.

By the way. I’m not using threads. Threads are evil. People who use them mostly condemn themselves to a thousand hells of corrupted data, deadlocks and, at the end of the day, slower performance, thanks to cache misses and dirty cache lines. And the uClibc library in many embedded systems does not support POSIX threads. Thank goodness. I’m particularly angry with threads, as every damn RT-Linux tutorial you look at, spends 75% with setting up the bloody things. And you even have to worry about which thread your signals will be delivered to. What about learning fork() and pipe() instead? KEEP IT SIMPLE STUPID!

But enough of my rants and dodgy theories, let’s see the practice. First let’s see, how the skeleton of a periodic task looks like.

#include <unistd.h>
#include <signal.h>
#include <sys/time.h>

/* Periodic SIGALRM handler routine: everything happens here */
void periodic(int signal)
{
    ... /* do your periodic stuff */
    if (continue_running)
        signal(SIGALRM, periodic);
    else
        exit(0);
}

int main(int argc, char *argv[])
{
    ... /* initialize your stuff */
    struct itimerval period = {
        { 0, 1000, }, /* 1st signal in [s], [us] */
        { 0, 1000, }, /* period time   [s], [us] */
    };
    signal(SIGALRM, periodic);       /* install periodic() to handle SIGALRM */
    setitimer(ITIMER_REAL, &period, NULL); /* start periodic SIGALRM signals */
    /* Main idle loop: everything done by the signal handler */
    while (1)
        pause();
    return 0;
}

Next, how to set up real-time scheduling. Add this code to main(), before the setup of the periodic stuff. Remember to run your program as root for the below code to take effect.

#include <sched.h>
...
int main(int argc, char *argv[])
{
    ...
    struct sched_param schedp;
    schedp.sched_priority = 1;
    sched_setscheduler(0, SCHED_FIFO, &schedp);
    ...
}

And now let’s see what happens when a process becomes real-time. I’m measuring latencies of a 1ms task on Raspberry Pi, actually without a PREEMPT_RT kernel. I start rt as a normal process, then change its scheduling to SCHED_FIFO with chrt. See the code of rt on github.

szg@og314 ~/syscalls $ ./rt 1000
Real time child process PID 3368
------------------------------------------------------------------------
     n  Mean [us]       [us] SD  [%]      [us] Min [%]      [us] Max [%]
------------------------------------------------------------------------
   982   1017.502    246.148  24.615        67  93.300      4449 344.900
   985   1015.249    247.086  24.709        71  92.900      4865 386.500
   975   1025.625    300.905  30.091        63  93.700      4669 366.900
   989   1011.116    162.667  16.267       181  81.900      2682 168.200
   986   1014.260    155.867  15.587       259  74.100      2506 150.600
   989   1011.115    163.982  16.398        51  94.900      2674 167.400
   983   1017.492    189.123  18.912        59  94.100      2803 180.300
   983   1017.102    199.678  19.968        74  92.600      3985 298.500
   985   1015.169    214.607  21.461        62  93.800      3938 293.800
   978   1022.632    267.480  26.748        83  91.700      4355 335.500
   988   1012.008    165.071  16.507        84  91.600      2707 170.700
   991   1009.079    222.581  22.258        72  92.800      6762 576.200
  1000   1000.001     18.526   1.853       769  23.100      1218  21.800
  1000   1000.040     16.464   1.646       834  16.600      1163  16.300
  1000    999.958     12.861   1.286       924   7.600      1086   8.600
  1000   1000.002     17.503   1.750       809  19.100      1223  22.300
  1000   1000.014     20.984   2.098       793  20.700      1212  21.200
  1000   1000.035     24.159   2.416       827  17.300      1175  17.500
  1000    999.998     26.596   2.660       795  20.500      1246  24.600
  1000    999.962     18.433   1.843       844  15.600      1152  15.200
  1000    999.989     34.235   3.424       716  28.400      1314  31.400
  1000   1000.003     18.630   1.863       849  15.100      1214  21.400
  1000   1000.001     22.666   2.267       717  28.300      1268  26.800
  1000    999.996     20.419   2.042       799  20.100      1196  19.600
  1000   1000.051     15.729   1.573       831  16.900      1159  15.900
  1000    999.951     13.385   1.339       920   8.000      1109  10.900
  1000   1000.000     18.562   1.856       830  17.000      1210  21.000
  1000   1000.002     28.782   2.878       812  18.800      1222  22.200
  1000   1000.039     19.713   1.971       763  23.700      1239  23.900
  1000    999.964     25.291   2.529       907   9.300      1118  11.800
     1   1035.000     35.000   3.500      1035  -3.500      1035   3.500
------------------------------------------------------------------------
     n  Mean [us]       [us] SD  [%]      [us] Min [%]      [us] Max [%]
------------------------------------------------------------------------
 29815   1006.212    136.875  13.687        51  94.900      6762 576.200

A line of statistics is printed after every second with the following columns: n (number of events), Mean dt, SD (standard deviation), and Min/Max dt. Whilst the program was running, after the 12th line of data above, I changed rt’s scheduling with the command below.

root@og314 ~ # chrt -f -p 1 3368  # -f : SCHED_FIFO, 1 : static priority 1

The effect was truly dramatic. With normal scheduling I was missing about 2% of the events, with an average latency of 20% (0.2ms), in the worst case up to 600% (6ms). After changing to real-time scheduling, not a single event went lost, average latency dropped to 2% (0.02ms!), and the worst case delay dropped to 31% (0.31ms).

All this is causing 7-8 % processor load on a Raspberry Pi running two ssh sessions and omxplayer playing an internet radio. Single core, no PREEMPT_RT kernel. You probably need the latter in a production system which must not miss a beat for weeks.

So Linux is definitely able to run an ECU. That’s where my new ARIA25 board will excel. After someone designs an ECU for me…