Evolution of the ePalm Build Process: Compiling Code

Evolution of the ePalm Build Process: Compiling Code

Adapting the SmartEiffel compiler to the PalmOS platform involves a great many tradeoffs and a special toolchain. In practice, once makefiles are set up, the process is mostly transparent... but because a great deal is going on "under the hood", it behooves the ePalm developer to understand the evolution of the tangled ePalm toolchain.

One reason that SmartEiffel was an easy choice as an Eiffel compiler for ePalm is that it is easy to modify its toolchain, runtime, and compilation process. We'll begin at the beginning: modifying the SmartEiffel runtime to produce a C file which could be legally compiled by the PalmOS toolchain.

A new SmartEiffel target : redefining the basic toolchain

SmartEiffel, by default, uses the system's traditional compilation toolchain-- gcc, gdb, and friends.

Of course, this generates executables for the development platform, not for a PalmOS target. An entire new C compilation toolchain has to be installed. The ePalm project uses the "prc-tools" toolchain, a set of free software modifications to the GCC compiler toolchain which allows cross-platform development for the m68k compilation target (see the prc-tools homepage for more information on the prc-tools project).

Because it was beneficial to do some other processing, the ePalm project uses its own version of the system.se (located in $EPALM/smalleiffel_sys/system.se) which defines two new targets:


[palmos]
c_compiler_type: gcc
c_compiler_path: ${EPALM}/tools/epalm_gcc.rb
c_compiler_options: -pipe -O2
c_linker_path: ${EPALM}/tools/epalm_link.rb
cpp_compiler_type: g++
cpp_compiler_options: -pipe -O2
smarteiffel_options: -no_strip

[palmos_debug]
c_compiler_type: gcc
c_compiler_path: ${EPALM}/tools/epalm_gcc.rb
c_compiler_options: -pipe -g
c_linker_path: ${EPALM}/tools/epalm_link.rb
cpp_compiler_type: g++
cpp_compiler_options: -pipe -g
smarteiffel_options: -no_strip


An eiffel .ace file can then be created for each ePalm project which uses the palmos or palmos_debug targets by including the following line under the generate section:


c_mode: palmos


This compiles the Eiffel code into C code and transfers control of the C compilation to the $EPALM/tools/epalm_gcc.rb Ruby script. A number of special-case tools written in the Ruby scripting language are used to massage the generated C code into a compilable form, which is then compiled and linked by the script using the prc-tools toolchain.

So now, it seems, we can generate C code which can be compiled into Palm-hardware-compatible binary executables. But there is much, much more to be done.

Cleaning the Code: Stuff We Can't Use

As mentioned earlier, the PalmOS environment is not a traditional desktop development environment, and the prc-tools package shows it by balking at many traditional unix includes. To get away with this, ePalm uses a substitute base.c (and many other base packages) from the $EPALM/smalleiffel_sys subdirectory, which has many of the offending includes removed. Some of the offenders include

There are other minor problems which may or may not be a factor (for example, PalmOS defines type Boolean as 'unsigned char'; Eiffel defines BOOLEAN as signed char. In practice, many of these make little difference.

One which makes a large difference is that an Eiffel INTEGER is 32-bit, while a PalmOS int is 16-bit. It is highly recommended that the ePalm developer use INTEGER_16 for most calculations.

To handle the lack of standard file system I/O combined with SmartEiffel's absolute need for it (SmartEiffel generates many calls to fprintf), a simple empty stub function named fprintf was created and put into the base.c file. It does absolutely nothing. This means that the developer currently has no access to any diagnostic information that the SmartEiffel-generated program may produce, but at this time there is no workaround.

A Default PilotMain

Unlike traditional console programs, the entry point for a PalmOS program is not the venerable main() function, but a more complicated entry:


DWord PilotMain( Word launchCode, Ptr cmdPBP, Word launchFlags );


The PilotMain function serves as the entry point into a PalmOS program, and thus into an ePalm program. It accepts a launch code (which indicates the way the program is opened), a pointer cmdPBP to any launch-code-specific data, and a set of launchFlags which indicade whether global variables are available, etc.

This actually presents something of a problem to running a program under SmartEiffel, although it's not immediately obvious. The Palm operating system can launch an application many different ways; although users are most commonly familiar with a "normal" launch, there are many ways in which a program can be launched, such as sysAppLaunchCmdFind, in which the program is launched in order to find a particular piece of data, or sysAppLaunchCmdGoTo which directs an application to open and then go to a particular piece of data.

What we will do with SmartEiffel is hijack the operation of PilotMain, and immediately transfer control of the program to the SmartEiffel runtime:


UInt32
PilotMain (UInt16 cmd, MemPtr cmdPBP, UInt16 launchFlags) {
  int Result = 0;
    if (cmd == sysAppLaunchCmdNormalLaunch) {
        Result = main( 0, NULL );
    }
  return Result;
}


When the program is executed, control passes through PilotMain, which checks the launch code and, if the launch code indicates a normal launch, passes control to the main() function of the SmartEiffel compiler

The problem isn't the different launch codes, but rather the flags associated with each. With many launch codes, the application's global variables are not available.

This situation is untenable with SmartEiffel, because a few global variables are vital to the operation of a SmartEiffel program. So many of the additional launch codes involve this case that _ePalm currently only handles normal launches_; hopefully a workaround can be found in the future.

Passing Control to main(): A Default Application Class

Once PilotMain hands control off to main, the standard SmartEiffel execution takes place: the stack is initialized, an instance of the Eiffel root object is created, and control passes to the default creation feature of the root class.

The root class, then, must be an instance of a class which can handle all the interaction and responsibilities of a PalmOS program. ePalm comes with one such class, the root of all ePalm applications, PALM_APPLICATION. The PALM_APPLICATION class handles basic setup and teardown of an application and has as its creation feature its own version of 'pilot_main':


pilot_main is
  do
   create last_event.as_new_memory
   create system_event_handler
   create menu_event_handler
   if launch_code = Sys_app_launch_cmd_normal_launch then
      start_application
      event_loop
      stop_application
      form_manager.close_all_forms
   end
end


Look primarily to the inner loop, ignoring the outside event handling. The pilot_main feature calls a deferred start_application feature, and begins the 'event_loop'; when event_loop finishes, the deferred feature stop_application is called, after which the form manager closes all forms and control passes back to main, which handles shutdown tasks.

So now we have C code which should call Eiffel code and return. Unfortunately, this works only for very small programs, so more has to be done.

"Sectionizing" the code

When programs got to be any reasonable size, I quickly discovered a new and different type of error that I had not been aware of before:


/usr/bin/m68k-palmos-gcc -g -o controls controls1.c controls2.c controls3.c
/usr/m68k-palmos/bin/ld: region coderes is full (controls section .text)
/usr/m68k-palmos/lib/crt0.o: In function `start':
crt0.c:62: relocation truncated to fit: DISP16 _GccRelocateData
crt0.c:64: relocation truncated to fit: DISP16 __do_bhook
crt0.c:67: relocation truncated to fit: DISP16 __do_ctors
crt0.c:72: relocation truncated to fit: DISP16 __do_dtors
crt0.c:74: relocation truncated to fit: DISP16 __do_ehook
/tmp/ccKWcOSW.o: In function `se_malloc':
controls1.c:53: relocation truncated to fit: DISP16 malloc
controls1.c:62: relocation truncated to fit: DISP16 exit
/tmp/ccKWcOSW.o: In function `se_calloc':
controls1.c:73: relocation truncated to fit: DISP16 calloc
controls1.c:82: relocation truncated to fit: DISP16 exit


The motorola dragonball processor is a curious beast. While it can handle some 32-bit values (and sizeof pointer shows that pointers are 32-bit), local jumps are signed 16-bit jumps, which means that local jumps must be very close together. When a program reaches a particular size so that these jumps can't be made, the compilation process will generate a large number of these relocation truncated to fit errors.

To work around this, it's possible to compile code into various sections, each of which makes local jumps into code contained within its own section and longer jumps "between sections".

It turns out that most of the offending code comes from the generated garbage collection routines. SmartEiffel has a very clever garbage collector that generates custom C code for every collected class in the program--which invariably means that more code is generated than can fit into one section. The solution is to take a great deal of the boilerplate generated garbage collection code and code for basic types in SmartEiffel and define their own sections, so the following code is inserted into the generated header file by the epalm_gcc.rb script prior to compilation:


#define BUILTIN_TYPES_SECTION __attribute__ ((section ("btypes")))
#define GC1_SECTION __attribute__ ((section ("gc1")))
#define GC2_SECTION __attribute__ ((section ("gc2")))
#define GC3_SECTION __attribute__ ((section ("gc3")))
#define GC4_SECTION __attribute__ ((section ("gc4")))


These lines define sections into which various code will be mapped during the compilation process. Then the epalm_gcc.rb script scans the generated header file and, for each function which belongs in a code section, places the name of the section after the function declaration:


...
void gc_sweep32(fsoc*c) GC1_SECTION;
void gc_mark32(T32*o) GC2_SECTION;
void gc_align_mark32(fsoc*c,gc32*p) GC4_SECTION;
...


The current script places sweep functions in GC1, mark functions in GC2, new functions in GC3, align_mark functions in GC4, and builtin type code in the BUILTIN_TYPES section. This works for now, but is a fairly mechanical and primitive way of doing it; there should be some way to identify which functions call each other and maximize the number of local jumps made (thus minimizing expensive jumps between sections) but so far that is impractical.

Unfortunately, there is one more catch regarding memory, SmartEiffel, and PalmOS

Using Less Memory by Default: Resetting GC Chunk Size

The SmartEiffel garbage collector is truly a remarkable thing. For more full details on this section, I highly recommend Colnet, Coucaud, and Zendra's paper, "Compiler Support to Customize the Mark and Sweep Algorithm, available from the SmartEiffel site here, which outlines more fully the concepts in the next few paragraphs.

Basically, the SmartEiffel garbage collection mechanism separates objects into "fixed size" objects and "resizeable" objects. In the case of the fixed size objects, each live object type is given its owns Linear Allocation Chunk (LAC) which is just a chunk of memory with a Free-Space Pointer (FSP) pointing to the beginning of free space for similarly-typed objects. This allows the allocator to allocate blocks of a known size very efficiently.

The problem is that, as the paper says, "we found that fixed-size chunks of 8kb were a good tradeoff between fast allocation and tight memory footprint". To make things worse, it takes a bit before garbage is actually collected: "In order not to trigger a full GC cycle whenever no free room can be found either in the LAC or in the type list of free objects... the memory ceiling is considered. It represents... the ammount of allocated memory under which no garbage collection is requested, but a new chunk is malloc'd instead."

In other words, SE-generated code begins by allocating an 8k chunk for each live type. When those chunks are full, instead of triggering a GC cycle, the offending type gets a new 8k chunk until a ceiling is reached.

For a desktop, this situation is quite acceptable. For PalmOS, it is horrifying. PalmOS has extremely tight memory requirements--in PalmOS version 3.0 (under which ePalm development began), executables had a mere 96k of dynamic memory available for programs, and for application globals and dynamic allocation (in other words, the pile of memory we'd be using up) only 36k was allowed. Obviously, leaving each fixed-size class with an 8k block is prohibitive. The memory requirements have been relaxed considerably with further releases of the OS, but it was still necessary to reduce the fixed and relocateable object chunk sizes. This is handled by once again massaging the header file (the chunk sizes are defined during compilation, so it was not possible to make a one-time modification to the SmartEiffel base.h file):


#define RSOC_SIZE 4096
#define FSOC_SIZE 1024


This sets more reasonable demands on the memory allocation algorithm. We also define the default memory allocation scheme to use as little memory as possible (triggering GCs earlier).

Currently these chunk sizes are not modifyable by the developer (they are hard-coded in the replace_gc_sizes.rb script). Future versions of the tools will likely include more configurability.

An INTEGER by any other name...

One more modification generated some difficulties. I had intended to redefine the Eiffel INTEGER type to be 16-bit (the default size of an int under PalmOS). By redefining it in the base C files, things appeared to work well for some time.

Unfortunately, there was a snag. I had redefined the SmartEiffel INTEGER type to generate internal type T10 (the internal type for INTEGER_16). But since the Eiffel standard says that INTEGER is 32-bit, the SmartEiffel compiler generates internal type T2 for some declarations of INTEGER -- and T2 is type INTEGER_32.

This presents real problems when linking with external C code. If the developer declares an external C type to have INTEGER arguments when he means 16-bit integers, SmartEiffel will happily generate stub code declaring the arguments to be 32-bit integers. Because each version of the header happily seems to be self-consistent, no error messages appear and the code will link fine. However, since SmartEiffel is generating an argument stack with 32-bit integer arguments and the C code expects an argument stack with 16-bit integer arguments, the code will fail spectacularly for no obvious reason. For this reason it is highly encouraged to not use the standard Eiffel INTEGER type in ePalm programs! Always use INTEGER_16 or INTEGER_32!

At Last... Linking

At this point, the code should generate and compile. To finish making an executable file, the epalm_link.rb script takes over the compilation process in a few short steps.

  1. First, an application definition file (.def file) is processed using the prc-tools multigen tool to create stubs (.s and .ld files) that define the sections for proper linking.

  2. Next, the assembly-code .s file is compiled into an object (.o) file for linking.

  3. Finally, the produced .o file and the .ld file are inserted into the argument string for the linking command, and the files are linked by the m68k prc-tools toolchain to produce an executable.

Building a PalmOS installable package

Suppose that now our application (which does absolutely nothing!) compiles successfully. We now have a binary executable which should execute.

But there is much more to it than that. PalmOS programs are not viewed as islands unto themselves; like Windows programs, they are packaged together with resources that are available to the program at runtime (such as forms, strings, bitmaps, etc). It is impossible to upload a solitary binary program to a Palm device; instead it must be packaged into a .prc file--a Palm resource database.

Currently this is handled by further processing in the epalm_link.rb script, which uses the pilrc tool to compile a textual description of resources (a .rcp file; see the pilrc documentation here for more information). The pilrc tool compiles the .rcp file into a series of binary resources (.bin extension).

Finally, the build-prc tool is invoked to assemble the project definition (.def) file, the executable file, and all the generated binary resources (.bin) files into a .prc resource database file. Finally, the file can be uploaded to the Pilot.

Of course, without any support classes or program code, it won't do a great deal. We now have the ability to cause SmartEiffel to generate compilable PalmOS code... but it remains to create the support structure and libraries to actually develop proper software. We'll cover that in the next section.

Powered by Zope