Lured into Linux Real Time
The support period of the Lucid Lynx on the Asus UL20A is about to end in May, so I was looking for a new OS. I gave a try to Debian Testing, or Linux Mint Debian Edition, to be precise. But not Pangolin. Sorry.
It was my smoothest Linux install ever. All hardware worked out of the box. Volume and brightness buttons? Check. Wifi? Check. Two-finger scrolling? Check. I also applied the usual MATE fixes for mutt and the power-button. Debian comes with a 486-kernel by default, so it handles only one processor core. One can upgrade manually to a multi-core 686-kernel with apt-get.
aptitude search linux-image
And what do I find?
linux-image-rt-686-pae
A PREEMPT_RT patched kernel, kindly compiled by Debian. Time to plunge into real-time again!
I put together a Linux syscall tutorial project a year ago (see github) which contained a small program performing a periodic real-time task and printing statistics like average/min/max dt and its standard deviation. It’s called rt and uses the setitimer() system call to generate periodic SIGALRM signals. Time to measure latencies of a 1ms periodic task on a RT kernel!
To my utter disappointment, performance was fine, until the tapeta daemon changed my wallpaper, that is. At which point a 300% latency appeared. That’s right, period time 4ms instead of 1ms. Same as earlier, without an RT kernel. What’s wrong?
I was not using real-time scheduling, that’s what. It turned out Linux processes run at static priority zero by default, where the actual scheduling priorities are dynamic, depending on interactivity, nice level, sleep/runtime, etc. This is called the SCHED_OTHER policy. In other words, one is at the mercy of the Completely Fair Scheduler.
On the other hand, a process’ static priority can be increased, and the scheduling policy can be changed to SCHED_FIFO with the sched_setscheduler() system call. In this case the only one whose mercy we are at, is another SCHED_FIFO or SCHED_RR (Round Robin) scheduled process with equal or higher static priority, lurking in the background somewhere. Important to note, though, a process needs to run as root to be able to change its own or another process’ scheduling. There is also a command line utility called chrt, to change the scheduling policy of processes.
By the way. I’m not using threads. Threads are evil. People who use them mostly condemn themselves to a thousand hells of corrupted data, deadlocks and, at the end of the day, slower performance, thanks to cache misses and dirty cache lines. And the uClibc library in many embedded systems does not support POSIX threads. Thank goodness. I’m particularly angry with threads, as every damn RT-Linux tutorial you look at, spends 75% with setting up the bloody things. And you even have to worry about which thread your signals will be delivered to. What about learning fork() and pipe() instead? KEEP IT SIMPLE STUPID!
But enough of my rants and dodgy theories, let’s see the practice. First let’s see, how the skeleton of a periodic task looks like.
#include <unistd.h>
#include <signal.h>
#include <sys/time.h>
/* Periodic SIGALRM handler routine: everything happens here */
void periodic(int signal)
{
... /* do your periodic stuff */
if (continue_running)
signal(SIGALRM, periodic);
else
exit(0);
}
int main(int argc, char *argv[])
{
... /* initialize your stuff */
struct itimerval period = {
{ 0, 1000, }, /* 1st signal in [s], [us] */
{ 0, 1000, }, /* period time [s], [us] */
};
signal(SIGALRM, periodic); /* install periodic() to handle SIGALRM */
setitimer(ITIMER_REAL, &period, NULL); /* start periodic SIGALRM signals */
/* Main idle loop: everything done by the signal handler */
while (1)
pause();
return 0;
}
Next, how to set up real-time scheduling. Add this code to main(), before the setup of the periodic stuff. Remember to run your program as root for the below code to take effect.
#include <sched.h>
...
int main(int argc, char *argv[])
{
...
struct sched_param schedp;
schedp.sched_priority = 1;
sched_setscheduler(0, SCHED_FIFO, &schedp);
...
}
And now let’s see what happens when a process becomes real-time. I’m measuring latencies of a 1ms task on Raspberry Pi, actually without a PREEMPT_RT kernel. I start rt as a normal process, then change its scheduling to SCHED_FIFO with chrt. See the code of rt on github.
szg@og314 ~/syscalls $ ./rt 1000
Real time child process PID 3368
------------------------------------------------------------------------
n Mean [us] [us] SD [%] [us] Min [%] [us] Max [%]
------------------------------------------------------------------------
982 1017.502 246.148 24.615 67 93.300 4449 344.900
985 1015.249 247.086 24.709 71 92.900 4865 386.500
975 1025.625 300.905 30.091 63 93.700 4669 366.900
989 1011.116 162.667 16.267 181 81.900 2682 168.200
986 1014.260 155.867 15.587 259 74.100 2506 150.600
989 1011.115 163.982 16.398 51 94.900 2674 167.400
983 1017.492 189.123 18.912 59 94.100 2803 180.300
983 1017.102 199.678 19.968 74 92.600 3985 298.500
985 1015.169 214.607 21.461 62 93.800 3938 293.800
978 1022.632 267.480 26.748 83 91.700 4355 335.500
988 1012.008 165.071 16.507 84 91.600 2707 170.700
991 1009.079 222.581 22.258 72 92.800 6762 576.200
1000 1000.001 18.526 1.853 769 23.100 1218 21.800
1000 1000.040 16.464 1.646 834 16.600 1163 16.300
1000 999.958 12.861 1.286 924 7.600 1086 8.600
1000 1000.002 17.503 1.750 809 19.100 1223 22.300
1000 1000.014 20.984 2.098 793 20.700 1212 21.200
1000 1000.035 24.159 2.416 827 17.300 1175 17.500
1000 999.998 26.596 2.660 795 20.500 1246 24.600
1000 999.962 18.433 1.843 844 15.600 1152 15.200
1000 999.989 34.235 3.424 716 28.400 1314 31.400
1000 1000.003 18.630 1.863 849 15.100 1214 21.400
1000 1000.001 22.666 2.267 717 28.300 1268 26.800
1000 999.996 20.419 2.042 799 20.100 1196 19.600
1000 1000.051 15.729 1.573 831 16.900 1159 15.900
1000 999.951 13.385 1.339 920 8.000 1109 10.900
1000 1000.000 18.562 1.856 830 17.000 1210 21.000
1000 1000.002 28.782 2.878 812 18.800 1222 22.200
1000 1000.039 19.713 1.971 763 23.700 1239 23.900
1000 999.964 25.291 2.529 907 9.300 1118 11.800
1 1035.000 35.000 3.500 1035 -3.500 1035 3.500
------------------------------------------------------------------------
n Mean [us] [us] SD [%] [us] Min [%] [us] Max [%]
------------------------------------------------------------------------
29815 1006.212 136.875 13.687 51 94.900 6762 576.200
A line of statistics is printed after every second with the following columns: n (number of events), Mean dt, SD (standard deviation), and Min/Max dt. Whilst the program was running, after the 12th line of data above, I changed rt’s scheduling with the command below.
root@og314 ~ # chrt -f -p 1 3368 # -f : SCHED_FIFO, 1 : static priority 1
The effect was truly dramatic. With normal scheduling I was missing about 2% of the events, with an average latency of 20% (0.2ms), in the worst case up to 600% (6ms). After changing to real-time scheduling, not a single event went lost, average latency dropped to 2% (0.02ms!), and the worst case delay dropped to 31% (0.31ms).
All this is causing 7-8 % processor load on a Raspberry Pi running two ssh sessions and omxplayer playing an internet radio. Single core, no PREEMPT_RT kernel. You probably need the latter in a production system which must not miss a beat for weeks.
So Linux is definitely able to run an ECU. That’s where my new ARIA25 board will excel. After someone designs an ECU for me…
