Transterpreter Summer '09 - Day 19 - Code Uploading

As the Transterpreter on the Arduino is getting more capable we are starting to write more and more occam programs. As these occam programs do not change the runtime itself, our current strategy of compiling the program bytecode into the runtime is becoming boring. Uploading the 16 KB interpreter together with a few hundred bytes of bytecode wastes quite a bit of time per upload. Thus, our next step is to separate the bytecode from the interpreter so that either can be uploaded independently of the other.

Currently we are linking the Transterpreter bytecode together with the interpreter without any of the metadata normally contained in a .tbc file. This metadata consists of information such as memory requirements for the bytecode, debugging, symbol information, and other things. We really don't want to be uploading much more than the bytecode to the Arduino though, as it has a limited amount of flash. If we are interested in using the debugging information, we are probably better off sending the relevant state back from the Arduino and making the host computer perform a mapping from the current program state to useful debugging information, using the debugging symbols found in the .tbc file.

Fortunately Carl thought of this while we were sleeping and introduced various changes (changeset: 5880, 5881, 5882, and 5883) to the plinker in order to be able to exclude certain sections of the bytecode file.

Uploading code

The Surveyor port of the Transterpreter uses a virtual machine with two contexts. One context executes the firmware code (occam code statically compiled into the Transterpreter) and the other context executes user code, which is only uploaded at runtime. In order to upload user code, the firmware is able to read bytecode sent over the network, store this into RAM, and later execute it.

On the AVR chips we don't have enough RAM available to store bytecode in RAM (we only have a few kilobytes). Instead it must be stored in flash (we generally have tens of kilobytes of flash). Currently we are storing bytecode in flash, but only as a side effect of compiling the bytecode directly into the interpreter. Instead we want to be able to upload either the interpreter or the bytecode into flash memory, without disturbing the other.

AVRDude

Avrdude is the tool which we use to upload the firmware. It reads a number of formats, including srec and ihex (amongst others). We are using ihex¹ as this is what the Arduino makefiles generate by default and it seems like a reasonable format.

Before going further we decided to start by establishing that we can in fact upload bytecode to flash at a location after the Transterpreter VM, ie without any undesirable side effects such as the entire flash being erased. After satisfying that we are indeed able upload 'stuff', without erasing the VM, given the right flags and the appropriate addresses² written in the ihex file, we proceed to look for an easy way to convert a bytecode file to ihex.

Our first idea was to use objcopy in order to generate an ihex file. This almost works, except that when passing a start address (the location where we want our bytecode written in flash) to objcopy it is not used in the address portion of the ihex data record, but instead added on as a special command at the end of the ihex file. avrdude unfortunately seems to ignore the start address command, and only obey the addresses that are part of each data record. As the data records written by objcopy always start at address zero, this unfortunately overwrites our virtual machine. After some poking around we decided that was no other way forward than to write a small tool that can generate an ihex file given any data and a start address: binary-to-ihex.

TEncode on the Arduino

We now have the ability to upload any data we like to the Arduino, which enables us to upload the .tbc (Transputer bytecode) files generated by our tools. We are careful (I think) to strip information out of the file that we don't need, such as debugging information and the symbol table. Other information is necessary for the correct execution of our programs though, such as memory requirements. In order to obtain this information and to correctly locate the bytecode within the uploaded data, we need to be able to correctly parse TEncode, the format of the .tbc files, based on the IFF format³.

The libtvm source code already contains a decoder for TEncode, as the posix interface to the Transterpreter already loads .tbc bytecode files. Unfortunately we cannot use this decoder due to the 'Harvard Machine' "problem". Since we are storing the bytecode in flash on the AVR we need to use special instructions to access the bytecode. The TEncode decoding routines in libtvm are not aware of the program vs. data memory distinction and only currently work with data memory. Instead of trying to generalise these routines, which are able to decode more of the TEncode format than we currently care about, we instead wrote a few AVR specific decoding functions which work with the AVRs flash memory.

As of changeset 5884 we can upload occam programs (as TEncode data) to flash independently of the interpreter, which, upon reset, will load the uploaded code out of flash and execute it.

Aside: Blinkenlights speed

At this point Matt decided that we should eyeball how fast the interpreter is on the Arduino. Given our current setup of 12 LEDs in a circle, turning these on and off as fast as possible would identify if we are running very slowly (ie we see the lights blinking).

After we had identified that there was no visible blinking going on with the LEDs, we decided to try to see if we could guage the blink speed with our Saleae USB logic analyser. Unfortunately we were unable to test the blinkenlights speed using the logic analyser. The software is currently Windows only and Linux and OS X drivers and software have been promised, but are not currently available. Our only option currently is therefore to run the device in a virtual machine. While we have been told by the manufacturer that this should not be a problem, we have been unable to sample at data rates above one megahertz, presumably due to the virtualised USB interface. The software also crashes a lot, but again, this might be due the fact that the analyser is being run through a virtual machine. We have tried both Virtual Box 3.0 and VMWare Fusion 2.0.X.

So we were not able to identify exactly how quickly the LEDs were blinking, but our visual inspection assured us that it is plenty fast! Since the Arduino is running at 16MHz and has a rating of 1 MIPS per 1MHz our virtual machine would have to be pretty inefficient in order to blink the LEDs slow enough that the delay would be visible. We decided however that we had more important things than to sit and count instructions or go over to the physics department to borrow an oscilloscope.

Other changes

Adam also fixed the compilation of the forall library (occam's standard library) on 16 bit machines in changeset 5885. We will need use forall in most interesting programs so this fix had to happen sooner or later. Additionally occbuild is now more aware of what architecture it is compiling for in changeset 5886, which ensures that we build 16 bit bytecode for 16 bit targets. Various less interesting changes happened in 5888, 5889, 5891.

5890 fixed a problem where we were not clearing memory too late, which meant that some state which had already been set up in the memory allocated for the soon to be run occam program would be overwritten. This only manifested itself if a program actually ran to completion and quit, which is not actually something that occam programs tend to do a lot⁴.

The Sanguino

In addition to our Arduino boards we also have a Sanguino. This is a board much like the Arduino, but made specifically for use in the RepRap project. Since we have it, and they are so similar, we thought we should demonstrate the Transterpreter running on it as well. Changeset 5887 commits some (for now, commented out) makefile magic to do this.

---