Collecting distributed variables with linker sets in C

I wondered how FreeBSD knows about drivers, that we include into the image. The only thing a developer does, is declare DRIVER_MODULE macro, which expands to some variables and structs, but newbus somehow have knowledge about those variables.

The solution is linker sets! It is the ability of the linker, to move variable/function to the other section. It is done by attribute “section” :

int __section("test section") variable = 666;

So, every DRIVER_MODULE macro creates a pointer to the existing driver structure with ‘sectio’ attribute, so all variables are collected in one place – and yes, we have array of pointers.

But how does the kernel know where is the beggining and the end of the section? It turns out, that linker provides __start_SECNAME and __stop_SECNAME symbols. Lets put this together, and make some code:

#include <stdio.h>

#define __section(x) __attribute__((__section__(x)))

int __section( "test_section" ) var1 = 1;
int __section( "test_section" ) var2 = 2;
int __section( "test_section" ) var3 = 3;
int __section( "test_section" ) var4 = 4;
int __section( "test_section" ) var5 = 5;

extern int __start_test_section[];
extern int __stop_test_section[];

int main(int argc, const char *argv[])
{
 int *a = (int *)__start_test_section;
 int b;
 int size;
 int count;

 printf( "Start: %p\n", (void *)a );
 while( 1 )
 {
   if( a == (int *)__stop_test_section )
   break;
   b = *a;
   printf( "%d\n", b );
   a++;
 }

 count = __stop_test_section - __start_test_section; 
 size = count * sizeof( int );

 printf( "Size of section: %d;\n" "number of variables: %d\n", size, count );

 return 0;
}
20:25:51 alek@alek-laptop in ~/tmp 
./sec 
Start: 0x601020
1
2
3
4
5
Size of section: 20;
number of variables: 5

Let’s see dump of the sections:

20:50:54 alek@alek-laptop in ~/tmp 
readelf -S sec 
There are 39 section headers, starting at offset 0x1638:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
[...]
  [25] test_section      PROGBITS         0000000000601020  00001020
       0000000000000014  0000000000000000  WA       0     0     4
[...]

We can see, that value of __start_test_section, and start of the section from readelf are the same. We can also count size and number of variables. Notice, that variables __start_test_section and __stop_test_section are declared extern (because they aren’t in the code. They will be available in the linking process) and mark as array.

We can of course, put functions in separate sections, but then, you can iterate them, because of the unknown size. It is better to use i.e. pointers to functions, which are fixed size, and interate them just like in good old arrays.

‘External Non-Linefetch Abort’ on FreeBSD/arm box

Last night while testing USB ehci driver I ran into an exception:

Fatal kernel mode data abort: 'External Non-Linefetch Abort (S)'
trapframe: 0xc0566ac4
FSR=00000008, FAR=c2de6848, spsr=400000d3
r0 =c053d0e0, r1 =c2de6848, r2 =00000004, r3 =c053d0e0
r4 =c2de6848, r5 =00000004, r6 =c2de6800, r7 =00000302
r8 =c2deac80, r9 =c025e550, r10=0000000c, r11=c0566b58
r12=c2deac96, ssp=c0566b10, slr=00000100, pc =c02e1680

[ thread pid 0 tid 100000 ]
Stopped at      intr_event_add_handler+0xbc:    ldrb    r12, [r1, r15]
db>

Quick investigation showed, that LDREX (Load Register Exclusive) from function atomic_cmpsel_32 is generating this exception:

(gdb) list *( intr_event_add_handler+0xbc)
0xc02c9d64 is in intr_event_add_handler (./machine/atomic.h:168).
163	
164	static __inline u_int32_t
165	atomic_cmpset_32(volatile u_int32_t *p, volatile u_int32_t cmpval, volatile u_int32_t newval)
166	{
167		uint32_t ret;
168		
169		__asm __volatile("1: ldrex %0, [%1]\n"
170		                 "cmp %0, %2\n"
171				 "movne %0, #0\n"
172				 "bne 2f\n"
(gdb) disassemble *( intr_event_add_handler+0xbc)
Dump of assembler code for function intr_event_add_handler:
[...]
   0xc02c9d5c :	mov	r0, r12
   0xc02c9d60 :	ldrex	r14, [r3]
   0xc02c9d64 :	cmp	lr, r1
   0xc02c9d68 :	movne	lr, #0
   0xc02c9d6c :	bne	0xc02c9d80 
   0xc02c9d70 :	strex	lr, r0, [r3]
   0xc02c9d74 :	cmp	lr, #0
[...]

Exclusive load (LDREX) reads data from memory, tagging the memory address at the same time. Exclusive store (STREX) stores data to memory, but only if the tag is still valid. Otherwise memory will not be modified.

My bug is somewhere in the USB driver, but besides my board (BeagleBoard-xM), this problem shows also on Raspberry-Pi.

There is no solution for this now (but we are working on it ;) ), but if you have only one core, you can try workaround – edit sys/arm/include/atomic.h and replace

#if ARM_ARCH_6 || ARM_ARCH_7A

with

#if 0

In that way, you will use standard atomic functions instead of the SMP-ready ones.

Thanks to ray and theraven for help!

Sources:
http://www.doulos.com/knowhow/arm/Hints_and_Tips/Implementing_Semaphores/