Transterpreter The Official Transterpreter Blog en-uk Sun, 09 Dec 2018 06:58:14 +0000 Liquorice 0.0.1 Transterpreter Summer '09 - Day 20 - 172us /blog/2009/06/day-twenty /blog/2009/06/day-twenty Christian Jacobsen Tue, 30 Jun 2009 22:23:23 +0000 <p>The underlying theme of the last few days has been <code>commstime</code>, a rather large program (in comparison to our earlier Arduino test cases) which also contains some calculations which use integers that are double the wordsize of the machine (32 bit calculations on the Arduino). While the last few days hacking has made it possible to run <code>commstime</code>, we have had to comment out the double word calculations in the program as the virtual machine did not support double length arithmetic on anything but 32 bit platforms. </p> <p>The reason for this goes back to the compiler we initially used on Windows: the free Borland C compiler (<code>BCC</code>). It put some limitations on how we could implement the virtual machines <em>long instructions</em>, used in arithmetic on numbers large than the machines wordsize (ie <code>long long int</code>s). <code>BCC</code> only supports the C89 standard (which does not include support for <code>long long int</code>s). While we later moved onto MinGW which gives us a <code>GCC</code> compiler with C99 support, our initial implementations of the virtual machines long instructions were done in terms of single length words only. Fortunately the excellent book "<a href="">Hackers Delight</a>" by Henry S. Warren, contains algorithms for doing just this sort of thing. However, they assume 32 bit words and we have never, it seems, had a compelling reason to get these instructions working on 16 bit machines.</p> <p><code>GCC</code>'s C compiler supports C99 as well as extensions to C89 that enables the use of <code>long long int</code>s. This allows us to implement the long instructions in terms of much more straightforward C compared to the algorithms from Hackers Delight. Changeset <a href="">5892</a> introduces C99 (<code>long long int</code>) versions of the long instructions, which are enabled if the compiler supports <code>long long int</code>s for the particular target architecture.</p> <p>While implementing the long instructions were perhaps not the hardest thing in the world, there were some subtleties. Notably some casting issues where we have to be careful that <code>signed int</code>s did not get sign extended when promoted to <code>unsigned long long int</code>s, as the long instructions strictly work on unsigned values.</p> <h3>Commstime</h3> <p>After fixing the long instructions we finally got to run <a href=""><code>commstime</code></a> (committed as changeset <a href="">5894</a>). The context switch time (that is the time it takes the runtime to switch between two processes) for the Transterpreter running on the Arduino, ie an ATmega328P clocked at 16 Mhz, is 172 us.</p> <h3>Looking forward</h3> <p>Our proress so far looks roughly like this: we have the ability run occam programs on the ATmegas (with parallel processes); blink lights (or more generally set the value of an output pin); put occam programs in the AVR's flash (to run large programs); do 32 bit maths (which we fixed today). In light of our achievements we spent the rest of the day pondering out next steps. Given that we have done various kinds of output, lights, printing to the serial port, it now seems natural to get some input going.</p> <p>While we perhaps don't strictly need to deal with interrupts in order to do input on the AVR chips, we decided that this would be our next task to poke at. We decided on using interrupts as we almost certainly do not want to poll the input pins from within occam. It would be possible to poll the pins from the runtime itself, keeping a state of the values last time round and checking for change. For example, when the runtime goes through the scheduler to run a new process (something that usually happens fairly often in occam programs) it would poll the various input pins on which the user is currently listening. If a pin had changed, the runtime would send a message to any process which was waiting on that pin. This should be easy to implement, but has a few drawbacks: we would not be able to use any of the power saving features of the chip, as we would need to constantly spin looking for pin change events in order to generate 'soft' interrupts for the running occam program; and, since occam is co-operatively scheduled, there is a danger that a process could hog the processor for long enough that a pin change event would be missed.</p> <p>Regardless of how we do implement the interception of external events, the interface we ultimately want to see most programmers that use occam on the Arduino to use is that of a channel. The channel abstraction nicely mirrors the interrupt mechanism<sup class="footnote-ref" id="fnref-1"><a href="#fn-1">1</a></sup>, where you declare that you want to wait for a particular event and are only woken when that event has actually occurred. We want our occam programmers on the Arduino to think about what they are going to do once something has happened and not have to bother with thinking about the, probably rather complex, machinery used to detect that the something has actually happened.</p> <div class="footnotes"> <hr /> <ol> <li id="fn-1"> <p>Actually the channel communication is a rendezvous, where when the two parties that communicate are made runnable again, they both know that the other party has engaged in the communication. Ie, it is a synchronisation point in the code where we know something about the state of the two processes in relation to each other. This is not necessarily the same in interrupt driven programming as we are not necessarily able to acknowledge the interrupt (in the case of an input only device, such as a button for example). Some hardware, such as a UART, might work differently and require us to acknowledge the interrupt in order to stop it from firing again immediately after re-enabling interrupts.&nbsp;<a href="#fnref-1" class="footnoteBackLink" title="Jump back to footnote 1 in the text.">&#8617;</a></p> </li> </ol> </div> Transterpreter Summer '09 - Day 19 - Code Uploading /blog/2009/06/day-nineteen /blog/2009/06/day-nineteen Christian Jacobsen Mon, 29 Jun 2009 23:01:22 +0000 <p>As the Transterpreter on the Arduino is getting more capable we are starting to write more and more occam programs. As these occam programs do not change the runtime itself, our current strategy of compiling the program bytecode into the runtime is becoming boring. Uploading the 16 KB interpreter together with a few hundred bytes of bytecode wastes quite a bit of time per upload. Thus, our next step is to separate the bytecode from the interpreter so that either can be uploaded independently of the other.</p> <p>Currently we are linking the Transterpreter bytecode together with the interpreter without any of the metadata normally contained in a <code>.tbc</code> file. This metadata consists of information such as memory requirements for the bytecode, debugging, symbol information, and other things. We really don't want to be uploading much more than the bytecode to the Arduino though, as it has a limited amount of flash. If we are interested in using the debugging information, we are probably better off sending the relevant state back from the Arduino and making the host computer perform a mapping from the current program state to useful debugging information, using the debugging symbols found in the <code>.tbc</code> file.</p> <p>Fortunately <a href="/people#carlritson">Carl</a> thought of this while we were sleeping and introduced various changes (changeset: <a href="">5880</a>, <a href="">5881</a>, <a href="">5882</a>, and <a href="">5883</a>) to the plinker in order to be able to exclude certain sections of the bytecode file.</p> <h3>Uploading code</h3> <p>The Surveyor port of the Transterpreter uses a virtual machine with two contexts. One context executes the firmware code (occam code statically compiled into the Transterpreter) and the other context executes user code, which is only uploaded at runtime. In order to upload user code, the firmware is able to read bytecode sent over the network, store this into RAM, and later execute it.</p> <p>On the AVR chips we don't have enough RAM available to store bytecode in RAM (we only have a few kilobytes). Instead it must be stored in flash (we generally have tens of kilobytes of flash). Currently we are storing bytecode in flash, but only as a side effect of compiling the bytecode directly into the interpreter. Instead we want to be able to upload either the interpreter or the bytecode into flash memory, without disturbing the other.</p> <h3>AVRDude</h3> <p><code>Avrdude</code> is the tool which we use to upload the firmware. It reads a number of formats, including <code>srec</code> and <code>ihex</code> (amongst others). We are using <code>ihex</code><sup class="footnote-ref" id="fnref-1"><a href="#fn-1">1</a></sup> as this is what the Arduino makefiles generate by default and it seems like a reasonable format.</p> <p>Before going further we decided to start by establishing that we can in fact upload bytecode to flash at a location after the Transterpreter VM, ie without any undesirable side effects such as the entire flash being erased. After satisfying that we are indeed able upload 'stuff', without erasing the VM, given the right flags and the appropriate addresses<sup class="footnote-ref" id="fnref-2"><a href="#fn-2">2</a></sup> written in the <code>ihex</code> file, we proceed to look for an easy way to convert a bytecode file to <code>ihex</code>.</p> <p>Our first idea was to use <code>objcopy</code> in order to generate an <code>ihex</code> file. This almost works, except that when passing a start address (the location where we want our bytecode written in flash) to <code>objcopy</code> it is not used in the address portion of the <code>ihex</code> data record, but instead added on as a special command at the end of the <code>ihex</code> file. <code>avrdude</code> unfortunately seems to ignore the start address command, and only obey the addresses that are part of each data record. As the data records written by <code>objcopy</code> always start at address zero, this unfortunately overwrites our virtual machine. After some poking around we decided that was no other way forward than to write a small tool that can generate an <code>ihex</code> file given any data and a start address: <a href=""><code>binary-to-ihex</code></a>.</p> <h3>TEncode on the Arduino</h3> <p>We now have the ability to upload any data we like to the Arduino, which enables us to upload the <code>.tbc</code> (Transputer bytecode) files generated by our tools. We are careful (I think) to strip information out of the file that we don't need, such as debugging information and the symbol table. Other information is necessary for the correct execution of our programs though, such as memory requirements. In order to obtain this information and to correctly locate the bytecode within the uploaded data, we need to be able to correctly parse <a href="">TEncode</a>, the format of the <code>.tbc</code> files, based on the IFF format<sup class="footnote-ref" id="fnref-3"><a href="#fn-3">3</a></sup>.</p> <p>The <code>libtvm</code> source code already contains a decoder for TEncode, as the posix interface to the Transterpreter already loads <code>.tbc</code> bytecode files. Unfortunately we cannot use this decoder due to the 'Harvard Machine' "problem". Since we are storing the bytecode in flash on the AVR we need to use special instructions to access the bytecode. The TEncode decoding routines in <code>libtvm</code> are not aware of the program vs. data memory distinction and only currently work with data memory. Instead of trying to generalise these routines, which are able to decode more of the TEncode format than we currently care about, we instead wrote a few AVR specific decoding functions which work with the AVRs flash memory.</p> <p>As of changeset <a href="">5884</a> we can upload occam programs (as TEncode data) to flash independently of the interpreter, which, upon reset, will load the uploaded code out of flash and execute it.</p> <h3>Aside: Blinkenlights speed</h3> <p>At this point <a href="/people#mattjadud" title="Matt">Matt</a> decided that we should eyeball how fast the interpreter is on the Arduino. Given our current setup of 12 LEDs in a circle, turning these on and off as fast as possible would identify if we are running very slowly (ie we see the lights blinking).</p> <p>After we had identified that there was no visible blinking going on with the LEDs, we decided to try to see if we could guage the blink speed with our <a href="">Saleae USB logic analyser</a>. Unfortunately we were unable to test the blinkenlights speed using the logic analyser. The software is currently Windows only and Linux and OS X drivers and software have been promised, but are not currently available. Our only option currently is therefore to run the device in a virtual machine. While we have been told by the manufacturer that this should not be a problem, we have been unable to sample at data rates above one megahertz, presumably due to the virtualised USB interface. The software also crashes <em>a lot</em>, but again, this might be due the fact that the analyser is being run through a virtual machine. We have tried both Virtual Box 3.0 and VMWare Fusion 2.0.X.</p> <p>So we were not able to identify exactly how quickly the LEDs were blinking, but our visual inspection assured us that it is plenty fast! Since the Arduino is running at 16MHz and has a rating of 1 MIPS per 1MHz our virtual machine would have to be pretty inefficient in order to blink the LEDs slow enough that the delay would be visible. We decided however that we had more important things than to sit and count instructions or go over to the physics department to borrow an oscilloscope.</p> <h3>Other changes</h3> <p>Adam also fixed the compilation of the <code>forall</code> library (occam's standard library) on 16 bit machines in changeset <a href="">5885</a>. We will need use <code>forall</code> in most interesting programs so this fix had to happen sooner or later. Additionally occbuild is now more aware of what architecture it is compiling for in changeset <a href="">5886</a>, which ensures that we build 16 bit bytecode for 16 bit targets. Various less interesting changes happened in <a href="">5888</a>, <a href="">5889</a>, <a href="">5891</a>. </p> <p><a href="">5890</a> fixed a problem where we were not clearing memory too late, which meant that some state which had already been set up in the memory allocated for the soon to be run occam program would be overwritten. This only manifested itself if a program actually ran to completion and quit, which is not actually something that occam programs tend to do a lot<sup class="footnote-ref" id="fnref-4"><a href="#fn-4">4</a></sup>.</p> <h3>The Sanguino</h3> <p>In addition to our Arduino boards we also have a <a href="">Sanguino</a>. This is a board much like the Arduino, but made specifically for use in the <a href="">RepRap</a> project. Since we have it, and they are so similar, we thought we should demonstrate the Transterpreter running on it as well. Changeset <a href="">5887</a> commits some (for now, commented out) makefile magic to do this.</p> <p><img src="/images/blog/2009/day-nineteen-sanguino.jpg" alt="The Transterpreter on the Sanguino" /></p> <div class="footnotes"> <hr /> <ol> <li id="fn-1"> <p>Intel HEX, see <a href=""></a> for further info.&nbsp;<a href="#fnref-1" class="footnoteBackLink" title="Jump back to footnote 1 in the text.">&#8617;</a></p> </li> <li id="fn-2"> <p>we used <code>objcopy</code> to generate some testing files. We were not able to get <code>objcopy</code> to output addresses in the <code>ihex</code> file in the way we wanted which initially made testing that we could write to arbitrary addresses in flash difficult. We decided manually edit the addresses in the file and see if <code>avrdude</code> would ignore the checksum (which include the address of course). <code>avrdude</code> does not ignore the checksum, <em>but</em> it helpfully calculates and displays the correct checksum for the line, making it easy to further patch the <code>ihex</code> file until all the checksums are correct.&nbsp;<a href="#fnref-2" class="footnoteBackLink" title="Jump back to footnote 2 in the text.">&#8617;</a></p> </li> <li id="fn-3"> <p>see: <a href=""></a> or <a href=""></a>&nbsp;<a href="#fnref-3" class="footnoteBackLink" title="Jump back to footnote 3 in the text.">&#8617;</a></p> </li> <li id="fn-4"> <p>occam programs are often made up of parallel processes. These parallel processes are most often written in a manner such that they will never exit, ie using a <code>WHILE TRUE</code> loop in their body.&nbsp;<a href="#fnref-4" class="footnoteBackLink" title="Jump back to footnote 4 in the text.">&#8617;</a></p> </li> </ol> </div> Transterpreter Summer '09 - Day 18 - Virtual Memory /blog/2009/06/day-eighteen /blog/2009/06/day-eighteen Christian Jacobsen Mon, 29 Jun 2009 03:01:22 +0000 <p>Today began with some more cleanup (as can be evidenced by commits <a href="">5874</a>, <a href="">5875</a>, <a href="">5876</a>, and <a href="">5878</a>, which introduced various changes). The main commit of the day <a href="">5877</a> introduced the new memory interface for the AVR.</p> <p>There is no real magic involved in the new memory interface thankfully. And as a bonus, our work was cut in half by the fact that we will never be writing data directly to flash from a running program. Thus all the write and pointer arithmetic functions required by the memory interface can be copied from the default interface. Only the four read functions require modification.</p> <p>The memory interface lays out SRAM starting at 0x000 and ending at 0x7FFF, for a total of 32 KB of addressable space for SRAM. On our current AVR we have 2 KB of SRAM so that is enough for now. The address space for flash starts at 0x8000 and goes to 0xFFFF, which fits the entirety of the 32 KB of flash on our AVR<sup class="footnote-ref" id="fnref-1"><a href="#fn-1">1</a></sup> has. Thus the read function inspects the requested address and if it is below 0x8000 it is read out of memory unchanged, otherwise the address is mapped into flash and then the appropriate magic is applied to read values out of flash.</p> <p>The only problem is with FFI code, which now needs to be modified to use the memory interface<sup class="footnote-ref" id="fnref-2"><a href="#fn-2">2</a></sup>. This makes working with the FFI a little bit more tedious unfortunately (but we are of course going to try to get rid of as much FFI anyway and implement things in occam :)</p> <p>The only other commit today is <a href="">5879</a> in which <a href="/people#carlritson">Carl</a> fixed the plinker to output 16 bit bytecode.</p> <h3>Commstime</h3> <p>With our new memory interface we are now able to run commstime. There is still a bit of work needed to get commstime to work fully as it does some 32 bit calculations for which we need to compile the occam standard libraries for 16 bit architectures. Something we have not done for a while.</p> <div class="footnotes"> <hr /> <ol> <li id="fn-1"> <p>the 328P by the way.&nbsp;<a href="#fnref-1" class="footnoteBackLink" title="Jump back to footnote 1 in the text.">&#8617;</a></p> </li> <li id="fn-2"> <p>in fact FFI functions currently use a function called memcpy<em>from</em>tvm, which is defined together with the FFI.&nbsp;<a href="#fnref-2" class="footnoteBackLink" title="Jump back to footnote 2 in the text.">&#8617;</a></p> </li> </ol> </div> Transterpreter Summer '09 - Day 17 - The Harvard machine /blog/2009/06/day-seventeen /blog/2009/06/day-seventeen Christian Jacobsen Sun, 28 Jun 2009 01:21:33 +0000 <p>Today was Saturday and we only worked half a day. We still got things done though.</p> <p>First off we did some cleanup and abstraction. In commits <a href="">5866</a>, <a href="">5867</a>, <a href="">5868</a>, <a href="">5869</a>, <a href="">5870</a>, <a href="">5871</a>, and <a href="">5872</a> we made various changes:</p> <ul> <li>Split the code into multiple files.</li> <li>Deal with return codes from running the interpreter, ie in order to print sensible deadlock, error, and status messages.</li> <li>Make an occam wiring module (<a href=""><code>wiring.module</code></a>), containing the bindings to the Wiring functions that we're using.</li> <li>Bind the serial output functions from Wiring so that we can output characters to the serial port from occam.</li> </ul> <p>Once we had printing from occam working the next step for any hotheaded occam programmer is to run commstime. Since everything seems to have gone great so far, it should be absolutely no problem running a slightly more complex program. Two days worth of porting yielded <a href="day-sixteen">parallel blinkenlights</a> so we were confident that commstime should 'just work'.</p> <h3>What is Commstime?</h3> <p>Commstime is a small benchmark used to measure the cost of process startup and shutdown in occam. It also provides a measure of the context switch time, which is probably the figure that is used more often. (Peeking into the future, the source of commstime we use can be found here in the repository: <a href="">commstime.occ</a>.</p> <p>Commstime represents a much larger program than anything we have run on the Arduino before. While the benchmark portion of commstime is only slightly larger than the programs we have been testing with so far, there also also a largeish number of support functions for printing the results of the benchmark. All in all, the bytecode for the commstime program comes to just over two kilobytes (incidentally just over the amount of available RAM on our AVR). This is compiled into the Transterpreter (as a static array of bytes) and the whole thing (interpreter + bytecode) is uploaded to flash.</p> <h3>So, Lets Run Commstime...</h3> <p>Bang! The machine crashes. Things that make you go mmm...</p> <p><img src="/images/blog/2009/day-seventeen-mmm.jpeg" alt="foo" /></p> <p>Now, we had noticed, during out initial debugging sessions on the Transterpreter, that the virtual machines workspace pointer (think it as the stack pointer) seemed to live at an address awfully close to the instruction pointer. We did not think too much of it at the time, perhaps the flash and RAM were simply mapped such that our pointers are very close together. Given that the values of both the workspace pointer and the instruction pointer were very low and that there is 32KB of flash on the AVR chip we are using we should probably have been more suspicious than we initially were.</p> <p>Since the commstime program is 'large' we did have a suspicion that it was perhaps being copied into memory. The bytecode for commstime is larger than the available memory on the chip and we are going to have a problem if the bytecode is in fact being copied into RAM. So, the first port of call is to check the declaration of the C file containing the bytecode (which is currently linked in directly with the Transterpreter) it should be <code>const</code>. It looks ok, but several failed attempts at running commstime convinces us that something must be up. We turn the debugging output of the instruction and workspace pointers on again and, as earlier, observe that they are closer than we'd like. It now seems prudent to look at the AVR manuals, you know, to figure out how memory is laid out on the chip, and stuff. According to the manual, flash and RAM are laid out as follows:</p> <ul> <li>Flash starts at <code>0x0000</code> and ends at <code>0x3FFF</code> with the bootloader living at the bottom of memory (close to <code>0x3FFF</code>).</li> <li>SRAM starts at <code>0x0000</code> and finishes at <code>0x08FF</code> <ul> <li>registers and I/O registers are mapped at <code>0x0000</code> to <code>0x00FF</code></li> <li>is mapped at <code>0x0100</code> to <code>0x08FF</code></li> </ul></li> </ul> <p>This is when it dawned on us that for both memories to start at <code>0x0000</code> the AVR must be based on the Harvard Architecture, which means that it has separate program and data memories.</p> <p>Armed with this new information, we quickly found the magic which would enable us to specify that we really really want the bytecode array to stay in flash (instead of being copied into RAM) in the <a href="">avr-libc documentation</a>. After adding the correct attribute to the declaration of the array, we naively compiled, uploaded, and ran the commstime program again to see if it now worked. It didn't, and crashed in a slightly different way this time.</p> <p>Reading just a little bit further on in the avr-libc documentation it quickly became clear that when reading constants out of flash, we must use a special macro to ensure that code is generated to read out of flash and not RAM.</p> <h3>The Problem and Potential Solutions</h3> <p>The Transterpreter is von Neumann architecture machine and it has one address space containing both code and data. Thus, inside the virtual machine there is no distinction between reading code and reading data. This makes it hard to apply a trivial fix to the codebase, ie we can not just replacing all reads out of program memory with a call that uses the appropriate avr-libc macro to correctly read out of flash. In the virtual machine we simply don't know if we are reading code or data.</p> <p>What further complicates matters is that code and constant data is interspersed in the bytecode files with no markers as to which is which. There are no sections in our bytecode files which mark a particular section as code and another as data. We are therefore also denied the opportunity to apply a simple fix which would check to see if we are reading out of a data segment of the bytecode file, as we simply don't have that information available. </p> <p>A third feature of our virtual machine which could have helped would be a specific constant loading instruction. This would make it easy to identify loading of constants out of program memory and hence flash. This would presumably make it easier to redefine the code in the load constant instruction to explicitly load out of flash. But we don't have a specific constant loading instruction, so this is not a quick fix either.</p> <h3>The Fix</h3> <p>Since we are trying to get up and running quickly we are not really interested in implementing support for any of the potential solutions above. They are all to invasive as they would require changes in the bytecode format (ie, introducing sections) and/or changes in the compiler (ie, introducing new instructions). We are really only interested in changing the virtual machine and at that, we'd like to change it as little as possible.</p> <p>Fortunately the virtual machine contain machinery to allow the user to specify a memory interface. This memory interface is used for all reading from and writing to memory. Thus it is possible to substitute the default memory interface (which is a set of macros which do a one-to-one mapping of addresses supplied by the virtual machine to the address space of the hosts memory), with a memory interface that is, for example, able to inspect the addresses and remap incoming requests to flash or RAM as appropriate.</p> <p>While we have not yet implemented this, the last commit today (changeset <a href="">5873</a>) is mostly a note to ourselves that we need to implement a new memory interface for the AVR which enables us to map both RAM and flash into the same address range.</p> Transterpreter Summer '09 - Day 16 - It works, huzzah! /blog/2009/06/day-sixteen /blog/2009/06/day-sixteen Christian Jacobsen Fri, 26 Jun 2009 21:32:12 +0000 <p>Yesterday left us with the ability to upload the Transterpreter runtime, but not the ability to use it. Today, with the first full day of work, we were hoping to end up with running code. Our very first attempt at this was the very exciting 'do nothing' program:</p> <pre><code>PROC main() SKIP : </code></pre> <p>which worked with out to much hassle (changeset <a href="">5862</a>). Good, but not quite cause for celebration yet. Next, we decided to move onto a more exciting program. One that never exits the interpreter and embodies a programming pattern often seen in occam: The <code>WHILE TRUE</code> loop. Most often there is code doing 'stuff' in the body of the loop, but for testing we sometimes see the following code, which is what we tried to run on the Arduino next:</p> <pre><code>PROC main() WHILE TRUE SKIP : </code></pre> <p>After uploading this program, it promptly crashed the Arduino. Mmmmmm.</p> <p>We then spent a considerable amount time digging around, during which <a href="/people#christianjacobsen" title="Christian">I</a> had to resurrect my Transputer bytecode reading abilities. Eventually we found that the <code>slinker</code><sup class="footnote-ref" id="fnref-1"><a href="#fn-1">1</a></sup> still had a bug which we are sure was fixed a long time ago. The bug in the <code>slinker</code> occurs only in <code>WHILE TRUE: SKIP</code> loops, which one would not expect to encounter in any serious program and thus do not see a lot of testing. The code which should be generated in the case of <code>WHILE TRUE: SKIP</code> is a backwards jump of four byte sized instructions (on a 16 bit machine)<sup class="footnote-ref" id="fnref-2"><a href="#fn-2">2</a></sup>. In the <code>slinker</code> however, the jump is represented as a jump to the absolute address of itself, but when we go to convert that to a relative jump, we fail to detect that we are jumping from the same address to the same address, and should therefore turn the absolute jump into a relative backwards jump (and not a relative forwards jump which is what currently happens). This failure is do to to the use of a <code>&lt;</code> where we should use a <code>&lt;=</code> in the backwards jump detection. Fixed (again! changeset <a href="">5863</a>).</p> <h3>Onwards; Towards Working Code</h3> <p>Next we increase the stakes and aim for running the following blinkenlights program (the code shown here has has some of the support code used in the checked in version stripped out, but this is ok will later be pushed into a library anyway. Full source code for <a href="">blink.occ</a> is in the repository):</p> <pre><code>PROC main () SEQ pinMode (LED.PIN, OUTPUT) WHILE TRUE SEQ digitalWrite (LED.PIN, HIGH) delay (100) digitalWrite (LED.PIN, LOW) delay (500) : </code></pre> <p>Getting this code to work was just an exercise in setting up the FFI functionality in the Transterpreter and wrapping the various functions required to poke the pins on the chip. Initial explorations looked using <code>PLACED</code> occam variables, which enables us to place a specific variable at a certain memory address. Reads and writes to that memory address should then poke the pins. This works great, on platforms where pins are mapped into memory. On the AVR chips however, you need to use a special IO instruction and direct memory access to the pins is therefore not currently possible from occam.</p> <p>We uploaded the new TVM and, as if by magic (ie occam code), the LED started blinking. Success!!! (there is a very exciting video at the end of the post showing the various blinkenlights).</p> <p><img src="/images/blog/2009/day-sixteen-working-hard.jpg" alt="Adam and myself working hard" /></p> <h3>A Note on Time</h3> <p>As we're currently running on top of the Arduino code, we get to use clock set up by that environment. The Arduino environment can give us time in both microseconds <code>micros()</code> and milliseconds <code>millis()</code>. Initially we used <code>micros()</code>, which returns long (32 bits on the Arduino), but occam only does time in the current word length, ie 16 bits on the Transterpreter Arduino port. When we using a long it is possible to get around 71 minutes out of the microsecond timer, but since we only use the lest significant 16 bits, we only get around 65 milliseconds before the counter wraps. Not that useful. So we have to switch to the millisecond timer (which also returns a long for around 50 days before rollover) but we only use the least significant 16 bits for just over one minute worth of counting before rollover. Better but still not great.</p> <p>The point of talking about this is to highlight a problem with occam that we really would like to fix. It is not possible to declare timers that return values in anything but the ports wordsize, which is unhelpful on small platforms where it is hard to get granularity without rolling over the clock very quickly. Also it is not possible to declare timers in occam with different granularities, ie I cannot say "give me a timer with millisecond and a timer with second granularity". This would certainly help alleviate the problem as it would be possible to declare timers with high granularity for short periods of waiting and low granularity timers for long periods of waiting<sup class="footnote-ref" id="fnref-4"><a href="#fn-4">3</a></sup>.</p> <p>I believe that on the Transputer chip there were two granularities of timer, but these were selected depending on whether a particular process was running at high or low priority. No syntax exists (or existed), nor did any instructions exist for selecting timer granularity.</p> <h3>Concurrent Printing</h3> <p>Timing aside, occam is a concurrent programming language and we really want to write concurrent programs. A single blinking LED is all fine and good, but it is multiple blinkenlights that we are interested in. The code to parallelise the blinkenlights introduced a parameterisation of the actual blinking of the LED. With this change the single LED blinking code from above would look like this:</p> <pre><code>PROC (VAL INT pin, on, off) SEQ pinMode (pin, OUTPUT) WHILE TRUE SEQ digitalWrite (pin, HIGH) delay (on) digitalWrite (pin, LOW) delay (off) : PROC main () (LED.1.PIN, 100, 500) : </code></pre> <p>The change to parallelise the blinkenlights is pretty simple, introducing a <code>PAR</code> and a replicated <code>PAR</code> block. The parallel blinkenlights is introduced in changeset <a href="">5865</a> which modifies the <a href="">blink.occ</a> program to look roughly like the following snippet:</p> <pre><code>PROC (VAL INT pin, on, off) ... as above ... : PROC main () PAR (LED.1.PIN, 200, 500) (LED.2.PIN, 300, 200) PAR i = 2 FOR 6 (i, i * 100, i * 230) : </code></pre> <p>flashing for <code>LED.1.PIN</code> and <code>LED.2.PIN</code> are set up individually and not in the replicated <code>PAR i = 2 FOR 6</code>, as the pin constants do not follow the sequence generated by the replicated <code>PAR</code>.</p> <p>This code makes the LEDs flash in pretty patterns. The code is also pretty clear (we think). We reuse the function that blinks a single LED (which uses the delay process to introduce delays between turning the LED on and off). To blink multiple LEDs with different delays, we simply run as many of these processes in parallel as we need. In fact the code above is simply a syntactic shorthand for this: </p> <pre><code>PROC main () PAR (LED.1.PIN, 200, 500) (LED.2.PIN, 300, 200) (2, 2 * 100, 2 * 230) (3, 3 * 100, 3 * 230) (4, 4 * 100, 4 * 230) (5, 5 * 100, 5 * 230) (6, 6 * 100, 6 * 230) (7, 7 * 100, 7 * 230) : </code></pre> <p>So, in occam, when we wish to start blinking multiple LEDs at with different delays at the same time, we simply run more of these processes in parallel, and we get the desired behaviour.</p> <p>This is in contrast with using a more traditional imperative language where one can not naively make the transition from a function which blinks one LED using provided delay function, to just using that function multiple times. Why? The delay function that is used in the example Arduino LED blinking code stops the world (adapted from the <a href="">Arduino Blink</a> program):</p> <pre><code>void flash_pin(int pin, int on, int off) { digitalWrite(ledPin, HIGH); delay(on); digitalWrite(ledPin, LOW); delay(off); } </code></pre> <p>That means, that when the delay is used, nothing else can run, and our LEDs will only blink one at a time. The Arduino tutorial proposes a solution for performing delays without using the delay function: <a href="">blink without delay</a> but this radically changes the structure of the program from the much simpler one using <code>delay()</code>. Using this method we also loose the ability to show users a natural progression from using a simple, reasonably intuitive parameterised pin flashing routine, to blink one LED, and then using it to blink multiple LEDs. Not so when we use a parallel language, such as occam.</p> <h3>Evidence (The Video)</h3> <div id="video"> <!-- Video for Everybody by Kroc Camen <> cc-by --> <video width="640" height="360" controls="controls"> <source src="/images/blog/2009/day-sixteen-blinkenlights.ogv" type="video/ogg" /> <source src="/images/blog/2009/day-sixteen-blinkenlights.mp4" type="video/mp4" /> <object width="640" height="400" type="application/x-shockwave-flash" data="/flowplayer/flowplayer-3.1.1.swf" flashvars="/images/blog/2009/day-sixteen-blinkenlights.mp4" > <param name="movie" value="/flowplayer/flowplayer-3.1.1.swf" /> <param name="allowfullscreen" value="true" /> <param name="quality" value="high" /> <param name="bgcolor" value="#000000" /> <param name="flashvars" value="config={'clip':{'url':'/images/blog/2009/day-sixteen-blinkenlights.mp4'}}" /> <!--[if gt IE 6]> <object width="640" height="495" classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B"> <param name="src" value="/images/blog/2009/day-sixteen-blinkenlights.mp4" /><! [endif]--><!--[if gt IE 6]><!--> <object width="640" height="495" type="video/quicktime" data="/images/blog/2009/day-sixteen-blinkenlights.mp4" > <param name="src" value="/images/blog/2009/day-sixteen-blinkenlights.mp4" /> <!--<![endif]--><p> <strong>No video playback capabilities detected.</strong> Why not try to download the file instead?<br /> <a href="/images/blog/2009/day-sixteen-blinkenlights.mp4">MPEG4 / H.264 .mp4 (Windows / Mac)</a> | <a href="/images/blog/2009/day-sixteen-blinkenlights.ogv">Ogg Theora &amp; Vorbis .ogv (Linux)</a> </p><!--[if gt IE 6]><!--></object><!--<![endif]--> <!--[if gt IE 6]></object><![endif]--> </object> </video> </div> <div class="footnotes"> <hr /> <ol> <li id="fn-1"> <p>The slinker is the bytecode scrubber and linker we use when compiling code for the Transterpreter. It is written in Scheme (<a href="">PLT Scheme</a> to be exact). There is also now a new linker, the <code>plinker</code> (written in Perl), but this does not currently support 16bit plinking. The <code>slinker</code> does support 16bit slinking, but is missing some other features, such as dead-code elimination.&nbsp;<a href="#fnref-1" class="footnoteBackLink" title="Jump back to footnote 1 in the text.">&#8617;</a></p> </li> <li id="fn-2"> <p>We jump four bytes back as the <code>slinker</code> does not optimally prefix the generated bytecode (optimal prefixing being the task of determining the minimum amount of bytes required to represent the loading of particular address offset, due to the Transputer bytecode format<sup class="footnote-ref" id="fnref-3"><a href="#fn-3">4</a></sup> which uses a variable number of bytes to encode operands).&nbsp;<a href="#fnref-2" class="footnoteBackLink" title="Jump back to footnote 2 in the text.">&#8617;</a></p> </li> <li id="fn-4"> <p>You might wonder why getting the time is not just a function call in occam and we should therefore be able to return whatever we like. While we can do this, there is a special syntax for timers which is used, especially in the ALTernation construct, where enables the reading from a number of channels or a timeout value. It probably is possible to fudge something, ie with a timeout process which uses a function to read time with any granularity and wordsize we like and then sends a signal when the timer fires, which can be used in the ALT. This is a kludge though.&nbsp;<a href="#fnref-4" class="footnoteBackLink" title="Jump back to footnote 3 in the text.">&#8617;</a></p> </li> <li id="fn-3"> <p>The Transputer bytecode, which the Transterpreter interprets, has an instruction length of one byte (eight bits). The most significant nibble of each instruction contains the function (fn) and the least significant nibble contains the operand (op). This only leaves 16 instructions, which is not very much, and arguments in the range of 0-15 (unsigned), which is not in itself enough to do very many interesting things. In order to overcome this operand size problem two of the 16 instructions are used for prefixing and negative prefixing, that is, building up a larger operand in the special operand register, on which the remaining 14 instructions operate. To be able to have more than 14 instructions, a further instruction is reserved as the <code>operator</code> and uses the current value in the operand register in order to jump to a secondary instruction contained in the secondary instruction table. None of these secondary instructions can take arguments (other than those passed on the stack).&nbsp;<a href="#fnref-3" class="footnoteBackLink" title="Jump back to footnote 4 in the text.">&#8617;</a></p> </li> </ol> </div> Transterpreter Summer '09 - Day 15 - Initial hacking /blog/2009/06/day-fifteen /blog/2009/06/day-fifteen Christian Jacobsen Thu, 25 Jun 2009 21:32:12 +0000 <p><a href="/people#adamsampson" title="Adam">Adam</a> has now arrived and after we all had a chance to sleep in for a bit, we went to the <a href=";source=s_q&amp;hl=en&amp;geocode=&amp;q=Alden+hall,+Allegheny+college,+Meadville,+PA&amp;sll=41.669927,-80.279068&amp;sspn=1.107878,2.466431&amp;ie=UTF8&amp;ll=41.655984,-80.14286&amp;spn=0.016513,0.038538&amp;t=h&amp;z=15&amp;iwloc=A">office</a>. Initial discussion mostly centred around ways in which we could do the port of occam to the <a href="">Arduino</a> platform. While we were probably all expecting to use the Transterpreter (the virtual machine for occam), we still discussed the merits of using a native solution where we might generate code. We dismissed this quickly as future work however, not least as we are looking to have a reasonably polished product finished at the end of three weeks when Adam goes back to the UK. And when we say finished product we don't just mean a bunch of commits in the repository, but a download link, documentation, and other infrastructure which actually lets other people use our work(!).</p> <p><img src="/images/blog/2009/day-fifteen-whiteboarding.jpeg" alt="Whiteboarding" /></p> <p>After the initial discussions in the conference room we moved into the coding plex (the lecture hall) which has a projector which serves nicely as our coding screen. The day then mostly proceeded with some infrastructure work, such as setting up our development environment and compiling libtvm for the AVR. The libtvm compile was fortunately easy (requiring only a Makefile fix) and enabled us to link an Arduino sketch with our code to get a rough estimate of the size of the virtual machine (<a href="">commit 5860</a>):</p> <ul> <li>~12KB - Without occam-pi features</li> <li>~24KB - With occam-pi features</li> </ul> <p><img src="/images/blog/2009/day-fifteen-working-hard.jpeg" alt="Working hard" /></p> Transterpreter Summer '09 - Day 9 /blog/2009/06/day-nine /blog/2009/06/day-nine Christian Jacobsen Mon, 22 Jun 2009 04:35:48 +0000 <p><a href="/people#mattjadud" title="Matt">Matt</a> and <a href="/people#christianjacobsen" title="Christian">I</a> are now back in Meadville after more travel than you could shake a stick at. USENIX was fun, but we now need to get ready to do some Ardoino haxing (as we foolishly promised): bits are are being purchased<sup class="footnote-ref" id="fnref-1"><a href="#fn-1">1</a></sup> and <a href="/people#adamsampson" title="Adam">Adam</a> will arrive in a few days<sup class="footnote-ref" id="fnref-2"><a href="#fn-2">2</a></sup>.</p> <p><em>In other news:</em> we managed to get our new website up during the USENIX conference. <a href="/people#jonsimpson">Jon</a> has been instrumental in making the site look good, getting old content pushed over, and much more. I've also done quite a bit of work, mostly on the server side. We might publish some very exciting details about the website and how we serve it up at a later date, but for now that will be shrouded in mystery. One of the most important things, perhaps, is that the blog is now functional again and the old content has been restored. We do have the old comments hanging around, but we have not had time to put those up. There is currently, to make our lives easier, no comment facility.</p> <p>While we're working on the Arduino, we are also going to work on some automated build scripts that will make it easier for us to roll binaries (for OS X and Windows especially). This should allow us to update the binaries on the <a href="/download">download page</a> in the near future.</p> <div class="footnotes"> <hr /> <ol> <li id="fn-1"> <p>although we will invariably forget to buy something important.&nbsp;<a href="#fnref-1" class="footnoteBackLink" title="Jump back to footnote 1 in the text.">&#8617;</a></p> </li> <li id="fn-2"> <p>which will result in another trip to Cleveland and back, yay!&nbsp;<a href="#fnref-2" class="footnoteBackLink" title="Jump back to footnote 2 in the text.">&#8617;</a></p> </li> </ol> </div> Transterpreter Summer '09 - Day 1 /blog/2009/06/day-one /blog/2009/06/day-one Christian Jacobsen Sat, 13 Jun 2009 04:03:38 +0000 <p>So, <a href="/people#christianjacobsen" title="Christian">I</a> have arrived in Meadville for what I have decided to dub the "Transterpreter Summer '09", a very exciting event where <a href="/people#mattjadud" title="Matt">Matt</a>, <a href="/people#adamsampson" title="Adam">Adam</a>, and <a href="/people#christianjacobsen" title="Christian">myself</a> will be in the same location to hax on various Transterpreter related things; and hopefully the rest of the group will be able to join in on some of the fun even though they are many miles away (mostly in the UK). Also, some of Matt's students will be dropping by to work on various interesting bits, including, but not limited to, the Transterpreter.</p> <p>First things first however, is preparation for the <a href="">USENIX talk</a>, which will be very exciting. I'm currently making builds of the TVM for OS X and Windows, which is not very exciting but must be done so that we tell people at USENIX that they can use our amazing software. Matt on the other hand has been setting up the Ubuntu box in our lab (next to the not quite finished RepRap).</p> <p><img src="/images/blog/2009/setting-up-ubuntu-uh.jpeg" alt="Confused?" /></p> An IDE for occam on the Surveyor /blog/2008/02/an-ide-for-occam-on-the-surveyor /blog/2008/02/an-ide-for-occam-on-the-surveyor Matthew Jadud Fri, 08 Feb 2008 11:01:25 +0000 <p>Jon and Carl have really pulled out all the stops on this one. It could make you weep it's so beautiful. What you see here is a screenshot of the typical jEdit interface. We use jEdit because it is open source, lightweight, and runs on Windows, Mac, and Linux without too much fuss. The code in the window is actually a near-copy of the original Surveyor firmware, translated into occam-pi. (Click on the images to see big versions...)</p> <p align="center"><a href="/images/blog/2008/srvtvm-01.jpg"><img src="/images/blog/2008/srvtvm-01-tm.jpg" width="400" height="256" alt="srvtvm-01" /></a></p> <p>The first thing we need to do is fire up the occPlug. This is our plugin for compiling and running occam programs on a number of different platforms. Currently, the Windows release supports the desktop, the Mindstorms RCX, and the Surveyor SRV-1.</p> <p align="center"><a href="/images/blog/2008/srvtvm-03.jpg"><img src="/images/blog/2008/srvtvm-03-tm.jpg" width="400" height="256" alt="srvtvm-03" /></a></p> <p>Once the occPlug is started, we can hit the compile button, and as you can see, our code is compiled for the Surveyor SRV-1 target. This isn't the real magic, though...</p> <p align="center"><br /> <a href="/images/blog/2008/srvtvm-04.jpg"><img src="/images/blog/2008/srvtvm-04-tm.jpg" width="400" height="256" alt="srvtvm-04" /></a></p> <p>When I run this code after compiling it, it is uploaded to the Surveyor and kicked off. Note that <strong>this is how all user programs are treated</strong>; we've turned what used to be the only firmware on the Surveyor into just another program. This is important, because it means it is easy for users to extend the default firmware (so to speak) to work with existing tools on the market, as well as incorporate their own algorithms for processing images, or navigation, and so on.</p> <p>When I hit run, I get connected up to the Surveyor automatically. You see that "Command" area down below? I can type messages in and send them to my SRV-1 with no fuss whatsoever. Type, hit return (or the "Send" button), and my characters are whisked away into the aether through the magic of 802.11g. One of the commands I can send is the <strong>I</strong> command (meaning, I typed the letter <strong>I</strong> and hit return). Why "I"? I don't know; we'd have to ask Howard. The important thing is that the <strong>I</strong> command sends me an <strong>image</strong>. And Jon and Carl have done a wonderful job here; when the terminal catches the header for the JPEG-compressed image coming back, it opens up an image pane, and displays what the Surveyor sees. We don't have to do anything!</p> <p align="center"><br /> <a href="/images/blog/2008/srvtvm-05.jpg"><img src="/images/blog/2008/srvtvm-05-tm.jpg" width="400" height="256" alt="srvtvm-05" /></a></p> <p>Here, I've sent the <strong>a</strong> command followed by <strong>I</strong>, and the image pane automatically resizes. Because the a command is interpreted to mean to set the image size to 160 x 128, I get a smaller image back. I then sent an <strong>A</strong> followed by an <strong>I</strong>...</p> <p align="center"><br /> <a href="/images/blog/2008/srvtvm-06.jpg"><img src="/images/blog/2008/srvtvm-06-tm.jpg" width="400" height="256" alt="srvtvm-06" /></a></p> <p>And you can't tell exactly, but that's a really big picture of my kitchen floor in the background. It took a little longer to send than the tiny image, but we love those big pictures for their general awesomeness. The last thing I did was dial the size of the image back, and fired up my laser cannons. Those things are awesome.</p> <p align="center"><br /> <a href="/images/blog/2008/srvtvm-07.jpg"><img src="/images/blog/2008/srvtvm-07-tm.jpg" width="400" height="256" alt="srvtvm-07" /></a></p> <p>How long does it take to compile that firmware? Let me see... I'm running Windows as a VMWare virtual machine on my MacBook Amateur (2x2GHz, 2GB RAM), and it takes roughly "one Mississippi." How long does it take to upload the program? About as long. That alone has made developing on the Surveyor under occam-pi an absolute joy: no JTAG, no long compilation, just edit, compile, run, explore, tweak, try again... nice and easy.<br /></p> <p>Now that things are falling into place, we can start doing some very cool things, I believe. Earlier today, I was exploring the YUV colorspace, and wondering how hard it was to see an orange. Well, my first attempt was pretty lame:</p> <p align="center"><br /> <a href="/images/blog/2008/srv2-yuv-filtering3.jpg"><img src="/images/blog/2008/srv2-yuv-filtering3-tm.jpg" width="400" height="347" alt="srv2-yuv-filtering3" /></a></p> <p>As you can see, I'm highlighting pixels that I believe should be orange-colored. Oops. Well, given that I was squinting at a two-dimensional YUV colorspace plot, it wasn't a bad guestimate. Note that I apparently have a better orange detector than a baseball detector:</p> <p align="center"><br /> <a href="/images/blog/2008/srv2-yuv-filtering2.jpg"><img src="/images/blog/2008/srv2-yuv-filtering2-tm.jpg" width="400" height="347" alt="srv2-yuv-filtering2" /></a></p> <p>Don't read too much into this, though. I mean, I was mostly figuring out what Carl and Jon had done, and taking the occasional screenshot.</p> <p>So, what's the punchline? Jon and Carl have assembled an excellent IDE for exploring occam-pi and robotics on the Surveyor SRV-1. We're going to be releasing this to students this weekend, and it goes live for laboratories in the Robotics class at Olin on Monday. We're using the Surveyor to explore challenges of vision-based navigation in conjunction with simple state machines and behavior engines. What's more, they'll be learning how to architect these solutions in parallel-safe ways from day one. I think this laboratory (this is only one 2-week lab of four) is part of what makes Olin's robotics class completely unique in the world today.</p> <p>And yeah, I'm pretty jazzed that the software that we've all worked on for so long is being put to use for the purposes it was intended. That, really, is the awesome part.</p> <p><strong>Update</strong>: Does it work over the wireless? Yes. This is part of what we like about the Blackfin Surveyor---no wires. You write your code, hit compile, hit run, and the code goes through the air to your robot. You can then talk back and forth completely over the WiFi. And to answer the second qusetion: does it work on the Mac?</p> <p align="center"><br /> <a href="/images/blog/2008/surveyor-ide-on-mac.jpg"><img src="/images/blog/2008/surveyor-ide-on-mac-tm.jpg" width="400" height="250" alt="surveyor-ide-on-mac" /></a></p> <p>You tell me? Because everything in the IDE is WiFi and Java, there's no real platform dependency. We've done a Windows build that will be distributed to Olin students momentarily, and want to discover any obvious problems. Once we've done that, we'll do Mac and Linux builds. Doing releases is not a process that we've sufficiently streamlined---something that perhaps needs to be improved.</p> <p>Regardless, yes, the Mac is a full-peer in this enterprise. Given that the Transterpreter was written entirely on Macs, we're happy to support it.</p> <p>(That's a shot from Jon's Surveyor, which lives in England and gets nice views out over Canterbury.)</p> Drawing on the SRV-1 /blog/2008/02/drawing-on-the-srv-1 /blog/2008/02/drawing-on-the-srv-1 Matthew Jadud Thu, 07 Feb 2008 14:27:23 +0000 <p>I've discovered that programming while helping students and working with colleagues (both in the room and across the ocean) is actually distracting enough to prevent you from writing good code.</p> <p>By coming in around 6AM this morning, I got some good quiet time that I was able to use productively, and in a short amount of time wrapped my head around YUV 4:22 (UYVY), and got some drawing routines ported into our Surveyor package. I have some questions for Carl and Jon about this, but for the moment, I'm glad my explorations are (mostly) over.</p> <p>Last night, I got as far as wondering why I was able to make my images green:</p> <p align="center"><br /> <img src="/images/blog/2008/srv2-camera-031.jpg" width="320" height="278" /></p> <p>This morning, I quickly isolated the luma component (Y) of the image:</p> <p align="center"><br /> <img src="/images/blog/2008/srv2-camera-09.jpg" width="320" height="278" /></p> <p>Then, it was on to drawing. I think he sub-sampling for JPEG conversion is confusing me, but we can reliably draw shapes over the image:</p> <p align="center"><br /> <img src="/images/blog/2008/srv2-moving-with-shapes.jpg" width="320" height="278" /></p> <p>Now, when students start doing image processing, they can circle or box in regions they are interested in, and see if their algorithms are returning values that make sense. (They can also print debugging information back to the PC, which is useful as well.) Between these two modes of interaction, I'd say we have a reasonably rich platform/API developing upon which students can develop parallel-safe robotics applications on the SRV-1 using occam-pi.</p> Firmware... as user code! /blog/2008/02/firmware-as-user-code /blog/2008/02/firmware-as-user-code Matthew Jadud Wed, 06 Feb 2008 06:18:46 +0000 <p><strong>NOTE</strong>: We will be unleashing the things you read about here on students at Olin College very soon. We'll bundle up a release for general consumption (as well as the documentation that we alpha test on the Olin students) Real Soon Now(TM).</p> <hr /> <p>I've been experimenting with the new firmware for the Surveyor SRV-1b that Carl and Jon have done a lot of excellent work on. Here's something wacky: what used to be the "firmware" can now be executed as "user code".</p> <p>I'll get there in a few steps. Consider this piece of <a href="">test code that Jon wrote</a>:</p> <pre> PROC tests (CHAN BYTE in?, out!, CHAN P.LASER lasers!, CHAN P.LED leds!, CHAN P.MOTOR motors!) SEQ out.string("SRV-1 Test Program (of Doom)*n", 0, out!) out.string("Testing death lasers*n", 0, out!) test.lasers(lasers!) out.string("Testing less deadly LED*'s*n", 0, out!) test.leds(leds!) out.string("Testing harmless motors*n", 0, out!) test.motors(motors!) : </pre> <p>This is is the main process from Jon's <a href="">test program</a>. The parameters coming into the process header are channels out to the environment. In this case, the environment is the Surveyor, so we have channels for the serial communications over the WiFi radio (those are the channels `in' and `out'), and there are output channels for the lasers, the LEDs, and the motors. Each of these are named as you might expect. To run this on the Surveyor, we compile it, upload the bytecode, and things just go.</p> <p>But what's great is that this program, although it doesn't do much, is not really different than the firmware that used to be written in C. The original firmware would handle commands from the SRV Console, and then spit images back, drive around, and do whatever else you commanded your mobile wireless camera platform to do. That is, all the default firmware did was respond to commands over a textual protocol. (There was a C interpreter, to. But my point is that you used a textual protocol to initiate all kinds of things.) Given that we have a channel of bytes representing the textual input coming over the radio, it seems like we could implement the old protocol completely in occam-pi.</p> <p>And that's what <a href="">Carl and Jon have done</a>. They've implemented what used to be firmware as a program that any user can now write and upload to the Surveyor. The program srv1.occ can be compiled, uploaded to the Surveyor, and executed as a user program, even though it is implementing the entire SRV-1 protocol. It is now a "user program" that implements what I've referred to as the "old firmware." If we want to kill this "new firmware," we issue a '!', and it is shut down cleanly. Now, we can upload a newer version, or perhaps a completely different program.</p> <p>This has its tradeoffs. For example, it means that my SRV-1 doesn't wake up ready to send me images. On the other hand, it does mean that I can easily modify and extend the firmware, including extending the protocol or (because occam-pi is a parallel-safe language) running my own additional code along with the original firmware. I can filter the channel carrying the commands (perhaps ignoring every other request for an image, or drawing on the images to add data to them), and so on. Over time, we'll probably end up refactoring the "firmware" into a bunch of reusable components that program authors can selectively include in their programs to get parts of the original firmware's behavior in their programs. (We'll see... I haven't given this much thought yet, but perhaps Carl and Jon have.)</p> <p>This is still evolving rapidly, but it is an absolute joy to be able to easily modify my occam-pi code, send it over the WiFi to the Surveyor, and get completely new, low-level behavior without having to go through a lengthy reflashing of the bot over JTAG (or, worse, WiFi).</p> <p align="center"><img src="/images/blog/2008/draw-on-image.jpg" width="341" height="480" /></p> <p>As can be seen above (sorta), I've begun to explore drawing on the images before shipping them from the SRV-1 back to the user. I believe this is important for students exploring robotic vision, as they need a way to indicate what they are looking for; drawing onto the image strikes me as a very simple way for their code to communicate that "this looks important!". It might be that they're doing edge detection, or blob finding, or any of a host of other things. My code doesn't do anything exciting yet, but tomorrow is another day; more excitement will ensue.</p> <p>Nicely done to the UK team! Wootness.</p> Roborealm and imminent releases /blog/2008/02/roborealm-and-imminent-releases /blog/2008/02/roborealm-and-imminent-releases Matthew Jadud Tue, 05 Feb 2008 16:56:51 +0000 <p>I just wanted to say that <a href="">Roborealm</a> looks like a very nicely done set of tools for exploring robotics and vision. I haven't had a chance to play with it yet, but it is something I'd like to do.</p> <p>The team is nearing completion of a new firmware for the SRV-1 that provides low-level, low-latency parallelism on the Blackfin-based Surveyor. Some excellent work has been done by Carl and Jon on this port, and I'm very excited to see it in the hands of students this semester.</p> <p>When we release that documentation, I'll also release the tutorial booklet that students used on the LEGO Mindstorms at the start of the term. It provided a questions-first introduction to occam-pi on the Mindstorms RCX. These documents, taken together (and iterated/revised a bit) will provide, I think, an excellent introduction to the design of concurrent and parallel algorithms for robotic control on some very cool platforms.</p> <p>Watch this space!</p> A body for a bot /blog/2008/01/a-body-for-a-bot /blog/2008/01/a-body-for-a-bot Matthew Jadud Thu, 17 Jan 2008 10:23:38 +0000 <p>The last post on this blog was in October. Since then, things have been busy. Over the next few posts, I'll try and catch up a bit.</p> <p>For the last three months, we've been working hard on the new <a href="">Blackfin Surveyor SRV-1</a>.</p> <p><img src="/images/blog/2008/SRV-1bf500.jpg" width="300" height="324" alt="Blackfin Surveyor SRV-1" /></p> <p>Blackfin SRV-1, image stolen from the Surveyor webpages</p> <p>We first encountered the Surveyor at AAAI 2007. When Howard "Surveyor Daddy" Gordon updated the Surveyor from a 60MHz ARM with a few KB of RAM to a 500MHz Blackfin with 32MB of RAM, we took notice. And, even better, when it sprouted that oh-so-excellent WiFi tail, we really took notice. (It is so much easier to work with WiFi than the ZigBee radio that was there. At least for us it is.)</p> <p>This semester, I'm co-teaching ENGR 3390: Robotics with <a href="">Dave Barrett</a> at <a href="">Olin College</a>, and we're using the Surveyor as the platform for the mobile portion of the course. We'll be using the Surveyor as a platform as a demonstrator where we can introduce students to architectures for cognition on robotic platforms in a parallel-safe way using the Transterpreter and occam-pi. And what we realized very quickly is that the SRV-1 looks like it is just begging to be beaten all to hell in the classroom.<br /></p> <p>See that processor on top, all nice and exposed? After a team of students shuffles in from the dry, Boston winter---BZZAPT!. No more Blackfin. One good static discharge, and the processor or RAM are gone. Or, those little laser diode wires? <strong>Oops! A careless gesture, and they're yanked out. Or what about a drop from the counter-top? I'm not confident the board would handle the fall very gracefully.</strong></p> <p>Dave threw together a quick Solidworks model, and dumped it out to Olin's rapid prototyping machine. This thing is incredible: it prints in 3D in ABS plastic. Send the model, step back, and 12 hours later you have this:</p> <p align="center"><img src="/images/blog/2008/Photo%203.jpg" width="400" height="300" alt="Photo 3" /></p> <p>That is a battle-ready SRV-1. The laser mount was removed, and the diodes inserted into the body, which was simply printed with the diode mounts in the right place. The camera hole is oversize, and easily gives the camera a full field of view.</p> <p align="center"><br /> <img src="/images/blog/2008/Photo%2051.jpg" width="400" height="300" alt="Photo 5" /></p> <p>As you can see, the rapid prototyping machine has enough resolution to render depressed text in the body of the bot. This one clearly declares itself as property of Olin's ENGR 3390: Robotics course.</p> <p align="center"><img src="/images/blog/2008/Photo%206.jpg" width="400" height="300" alt="Photo 6" /></p> <p>What you may or may not be able to tell is that the entire top of the box has mounting holes at 1/2" spacing perforating it; it is amazing what the RP machine can do. This means we can easily screw-mount sensor towers or anything else we might like to the top of the bot quickly and easily, running wires straight down through the body to the SRV-1 board.</p> <p>Generally, the pictures are low quality because they were taken with the webcam built into my Macbook. I'll get better pictures at a later point.</p> <p>The first thing we're going to have the students in the course do is take the basic Solidworks model and design bodies for their team's bots. This way, each bot will have a unique body, and the students will have some ownership of the process. However, we already have a few things we want to improve in the basic body before we turn them loose:</p> <ul> <li>The body doesn't have anywhere to grab on to in the back; as you can see, it is just resting on the SRV-1's aluminum body. Our plan is to wrap around inside of the treads, where there is just enough clearance for us to sneak the body down. This way, we provide more protection for the boards from grit and the like, and can attach across the base of the bot on the underside.</li> <li>We need blinkenlight mounting holes build into the body. This way, you can just push an LED up from underneath, and it will sit flush with the housing. We can wire them into the GPIO pins inside, and have all the wires hidden from view, and safe from snags on obstacles.</li> <li>A view hole needs to be left for viewing the state of the three LEDs on the Blackfin's motherboard.</li> <li>A mounting point for a header board could be built into the body, so sensors could be attached and detached outside; this depends on whether we expect to be doing much of this kind of prototyping with different sensors or not.</li> <li>Mount points flush with the body wall for IR and other sensors on the front/back/sides could be very useful. We have a number of interesting sensors for the students to characterize and use, and mounting them safely inside the body would be a good idea.</li> <li>The antenna should probably be above the ground plane; we could run a cable from the current point, under the body, and up to the top, where we could easily grow an antenna mount.</li> <li>We have blue and yellow plastic coming in, which will produce a lasting color. Otherwise, we have to paint, which is less durable and a mess to work with. (Well, as messy as spray paint ever is, anyway.)</li> <li>I want my robot's body covered in dimples (and maybe some through mounting holes) that are compatible with a certain plastic building toy that rhymes with Tierra del Fuego.</li> </ul>Those were the immediate improvements that we saw could be made. <p>We're really excited about the course, and I think both Dave and I are pretty excited about how cool the bodies for the Surveyorcould be. What's really awesome is that they are printed in ABS plastic, and have a wall thickness of roughly 3/16" of an inch. (I haven't measured; that's just my eyeball estimate from memory.) I pounded on it with my fist (off the Surveyor), and I hurt my knuckles. In short, the body is tough. The ultimate goal is to make it possible for a student, in their excitement, to give the Surveyor a kick on the field of combat (or drive it off a table), and have the bot be in a good condition to survive the fall.<br /></p> <p>Over the next two weeks, this body will evolve a lot. We're going to be playing with it, and then 24 of Olin's energetic, creative engineers-to-be are going to rapid-fire evolve it to their hearts content. I think we'll end up with something very, very cool. I'll post pictures and Solidworks models as the body evolves.</p> Compilers: Some reflection /blog/2007/10/compilers-some-reflection /blog/2007/10/compilers-some-reflection Matthew Jadud Sat, 13 Oct 2007 03:15:52 +0000 <p>Some time ago (but only a few posts ago), I ranted about the state of embedded systems development under Linux. If we break the rant down, it basically read like this: </p> <blockquote> <p>How come I don't have carrier-grade compilers for every single chip under the sun for free? This is ridiculous!</p> </blockquote> <p>The truth is, these tools have a cost. GCC is a great resource, but the number of platforms that can be realistically supported through gratis development is actually, quite likely, small. One way that it could be done is if every hardware manufacturer employed one or more GCC developers to maintain back-ends for their CPUs. Perhaps some manufacturers do just that. </p> <p>However, they'd also have to maintain drivers for the programmers that connect to their chips, and make sure that you could debug the hardware from GDB on Windows, Mac, and Linux---which is a non-trivial amount of work. So, now you're maintaining a toolchain that goes from the compiler, to the linker, to the driver for the programmer... and there's probably a standard library to maintain as well. </p> <p>What I'm starting to see (from being in an engineering school) is that people don't always have the mentality that they can build things themselves... when it comes to software. Because I wrote a linker for a weird little instruction set, and helped write a bytecode interpreter for that instruction set, II have a different perspective regarding the authoring of software than, say, your typical mechanical engineer. I have no real way to compare, but I suspect I sit down and think about the design and implementation of compilers or distributed systems in much the same way they think about... drivetrains and ... moving... things. Or something. </p> <p>I've been wrestling with getting the Transterpreter compiled for a particular MSP430 part. You see, there are a number of parts in the MSP430 family that GCC supports very well---and we have compiled and executed occam-pi programs on those CPUs just fine. But the development board I purchased for $120 has a CPU that is... less-well supported. As in, I need to build a patched version of binutils to get things working. I spent the better part of six hours last weekend debugging what was going wrong and attempting to get things running, and finally realized that my time was worth more than the cost of a tool. </p> <p><a href="">Rowley Associates</a> produce the Crossworks line of compilers for a number of embedded processors, <a href="">including the MSP430 family</a> of devices. Lets look at their prices: </p> <p align="center"> <table width="40%" align="center"> <tr><td><b>Version</b></td><td><b>Cost</b></td></tr> <tr><td>Commercial</td><td>$1495</td></tr> <tr><td>Educational</td><td>$299</td></tr> <tr><td>Personal</td><td>$149</td></tr> </table> </p> <p>Right now, I'm too busy to chase after broken builds of binutils. Six hours of my time, valued at $25/hour, is the cost of a compiler that handles the CPU I'm trying to build for, as well as many others. And not only that, but the IDE and tools will work on Windows and Linux... and soon my Mac. </p> <p>In other words, I get the IDE on Windows, Mac, and Linux for $149. My problems of getting my software to compile on the MSP430FG4619 simply go away. Does this help the next person who comes along and wants to run the Transterpreter on this chip? No. However, they'll have the same options I currently have: </p> <ol> <li>Get the GCC-based tools to work</li> <li>Buy the compiler</li> </ol> <p>If I had infinite time, I would get the GCC toolchain working. But I don't. I just want the devboard to work, so I can test and demo our VM on a small platform. And because other projects I'm involved in leverage the MSP430, the compiler won't hurt there, either.</p> <p>So, as soon as I get another hour or two free, I'm buying the Crossworks tools. I'm no longer a grad student with infinite time---and this is a battle I can afford to win with cash.</p> USBP and WinXP under VMWare Fusion /blog/2007/08/usbp-and-winxp-under-vmware-fusion /blog/2007/08/usbp-and-winxp-under-vmware-fusion Matthew Jadud Wed, 08 Aug 2007 04:28:10 +0000 <p>I thought it might be worth mentioning (in light of my rants) that it is possible to use the <a href="">Softbaugh USBP</a> in a Windows XP VM running under VMWare Fusion---that is, on a MacBook. </p> <p align="center"> <img src="/images/blog/2007/20070808-vmware-usbp-03.jpg" height="464" width="400" border="0" hspace="4" vspace="4" alt="20070808-Vmware-Usbp-03" /> </p> <p>I was running into problems with VMWare Fusion 1.0beta4. However, I missed the recent upgrade to v1.0 "Gold Master", or (if you prefer), the final release of version one. In upgrading, Windows XP (running in the VM) correctly found the USBP, and as a result, I can now get on with programming MSP430-based devices on my MacBook. (I think; I have't actually tried loading code to a device yet, but Windows sees the programming dongle, and I think that's the important step. I expect things to work at this point.)</p> <p>The picture above is the Softbaugh PrgUSBP application running in VMWare Fusion's "Unity" mode. In this mode, the Windows app plays along with OSX as if it were a full-fledged Mac application. This way, I can Apple-TAB back-and-forth between my editor and the programmer, and not have to keep flipping back to the full VM. </p> <p>As it stands, I think it is sad that embedded programming with chips like the MSP430, ARM9, and other interesting devices is so damn expensive. The programmer was $150, my dev board was $115, VMWare Fusion is $80, and XP... I can't remember. Granted, many of these things were (in my case) paid by grants, but for a hobbyist, $300+ is rather pricey. </p> <p>I'd love it if we had a way of programming these chips cheaply ($50 or less), and it worked on every platform. Perhaps, someday, I'll make that a mission. It would have to be a mission of love, because my guess is that there isn't enough money in these tools to make it up in volume.</p> Embedded programming rant continued /blog/2007/08/embedded-programming-rant-continued /blog/2007/08/embedded-programming-rant-continued Matthew Jadud Mon, 06 Aug 2007 15:36:28 +0000 <p>I continue to be frustrated by the state of embedded systems programming. </p> <p>If I want to program an ARM9, I need a special programmer. If I want to program an MSP430, it will require a special programmer. The list goes on and on, for each new chip that I think I'd like to develop code for; the cost is prohibitive, and in each case, it is likely that the tools: </p> <ol> <li>Only work under Windows, or</li> <li>Work under Linux <b>if</b> you have a parallel port.</li> </ol> <p>I don't know about you, but my MacBook doesn't have a parallel port. And Windows running under VMWare doesn't always recognize these little programming dongles. And really, I don't want to use Windows or Linux at all. I want to use my Mac. </p> <p>I'm thinking that I need a sub-$300 device (sub $100 would be ideal) that can program <b>any</b> small device, whether it is the new Blackfin processor or a crappy little PIC. This implies a software solution, with some mix-n-match cabling coming out of the box. </p> <p align="center"> <img src="/images/blog/2007/embedded-programmer.png" height="156" width="403" border="0" hspace="4" vspace="4" alt="Embedded-Programmer" /> </p> <p>I'm going to start thinking about this a bit more, because it is a genuine problem in the embedded space. It represents the worst kind of vendor lock-in, and makes my life as an educator and hobbyist a complete nightmare. I need a solution that: </p> <ol> <li> Works with any embedded device (via software update, if necessary)</li> <li> Works with any computing platform (Mac, Linux, Windows)</li> <li> Is cheap and portable</li> </ol> <p>In the end, the solution is probably called a "LEGO Mindstorms NXT", and I should just give up on embedded platforms. But I'm not prepared for that yet.</p> Wrappers, tagging, libraries and build systems... oh my! /blog/2007/07/wrappers-tagging-libraries-and-build-systems-oh-my /blog/2007/07/wrappers-tagging-libraries-and-build-systems-oh-my Matthew Jadud Tue, 31 Jul 2007 23:29:35 +0000 <p>Along with sketching out an explicit group development process, we discussed several other aspects of our project's organization. As these were relatively straight-forward things, I've grouped them all here.</p> <h3>Wrappers</h3> <p>A 'wrapper' is what makes it possible to run the Transterpreter on a new platform. That is, cross-compiling the interpreter for a new architecture is boring and easy. Connecting up the language to the hardware (or underlying operating system) is the interesting bit, and the wrapper is where we do that work. </p> <p>Up until now, we have typically done wrapper development in branches. However, this is difficult, as it means each wrapper developer must constantly be merging the trunk back into their branch. This is silly, as most of our wrappers we want to remain current, and take advantage of any improvements in the trunk.</p> <p>So, moving forward, we're going to try and keep wrapper development in the trunk. Because wrappers are mutually exclusive, we don't have to worry about one adversely influencing the other, and it will make it easier for us to make sure that changes that help one branch doesn't hurt another (eg. using buildbot and cross compilers). </p> <p>This requires us to be a bit more vigilant about...</p> <h3>Tagging</h3> <p>Up until now, we've been rather lazy about tagging. As a group, we need to start tagging more often. This might be by deciding on features that should go into milestones, or perhaps through an ad-hoc democratic process, where someone suggests that we tag, and the group agrees (or disagrees) that it is a good time to do so. </p> <p>This is an area that requires more discussion, but it is clear that the project could do with more versioning and tagging, providing snapshots of known good/stable points.</p> <h3>Libraries</h3> <p>Adam has been doing a lot of work at making libraries for occam-pi cross-runtime buildable. By this, I mean that we have a situation where we have two runtimes (CCSP and the Transterpreter), each which are capable of different things. For example, the interpreter will always be smaller, and therefore more portable to more (embedded) platforms. Unfortunately, if I had developed a C library with occam-pi bindings, it was virtually impossible to use it from both the Transterpreter and the CCSP (KRoC) runtime.</p> <p>Adam's work has brought us into a world where we can easily build such libraries for either runtime from the same source with minimal effort. This is excellent, and we'll be converting all of our libraries to use this new build system over time. Which, brings me to the last bit...</p> <h3>Build systems</h3> <p>Some time ago, I explored the use of Scons as a build system. It represents a spike, and can build the Transterpreter for Linux, Mac (PPC and Intel), the MSP430, and Windows. </p> <p>However, it's a bit ugly. It represents my first time working with Scons, and it could be better. Also, we discovered that mingw provides an environment that lets us easily build everything using our Autotools-based environment. So, whereas we thought the Scons build would improve things across the board, it's not clear that it's a 100% necessary improvement.</p> <p>Either way, we're going to bring Scons into the trunk, and play with it. Because it is completely orthogonal to the Autoconf/Automake build system, it can't hurt. However, we're going to be looking at this closely over the next few months, as one of them has to go, and we definitely want our build system to be cleaner/better organized/simplified greatly. If that means using Scons, great. If it means revamping our existing system, great.</p> <h3>Up next...</h3> <p>Those are reasonably straight-forward changes coming down the pipe; as a side effect of these things, we'll likely do some branch cleanup and repos reorganization. As the time comes for these things, they'll be discussed on the development list.</p> <p>The last thing I'll talk about is a project mission statement and project goals. That'll come later this week.</p> Explicit TVM Dev Process /blog/2007/07/explicit-tvm-dev-process /blog/2007/07/explicit-tvm-dev-process Matthew Jadud Mon, 30 Jul 2007 12:24:51 +0000 <p>So, the first of our diaspora updates involves making our development process more explicit. While our group has, largely, worked this way already, it hasn't been explicit. This had a lot to do with the fact that the entire team was housed in the same suite of offices at the University of Kent. Now, things have changed.</p> <p>The first document I want to introduce you to is our <a href="">development process</a>. We've attempted to keep this as short and simple as possible. We want just enough process to keep us moving forward harmoniously, but not so much that developing on the Transterpreter is notfun. Given that we hope that students interested in concurrency, embedded systems, and other cool stuff like that will want to get involved, we also wanted the process to be heavy enough to provide some structure and be pedagogically valuable, but light enough so as not to get in the way of (say) an enjoyable final-year project.</p> <p>So take a look, and <a href="">join the development mailing list</a> if you want to discuss it. </p> All quiet on the western front /blog/2007/07/all-quiet-on-the-western-front /blog/2007/07/all-quiet-on-the-western-front Matthew Jadud Sun, 29 Jul 2007 12:44:57 +0000 <p>Arguably, with respect to England, I'm now the Western Front. That is, last week, I moved from the UK back to my native land of the USA. Kinda sad, really; it's hard to find cask-conditioned, top-fermented ales in the USA.</p> <p>On Friday, July 20th, we went ahead and had a group meeting discussing many aspects of the project. Over the next few days, I'll describe the various parts of that discussion to inform the development list as well as provide a partial record of what we came up with. </p> <p>To me, the most important part of the discussion was our agreeing some working patterns that we can document and adhere to as a distributed group. As a result, this should make it much easier for new developers to get involved and feel "at home" in the project. This has been missing so far, largely because we were all in adjacent offices for the first few years of the project. That, now, has changed.</p> <p>I'll try and do one of these every morning. Admittedly, I'm on vacation right now, but I'm also jetlagged. Or, something... I'm still going to bed around 10PM and waking up around 6AM, which means that I have a few quiet hours in the morning to think and write things like this. We'll see. Either way, keep an eye on this space, or watch the mailing lists, as I'll announce the posts in this series to the list, as we (as a group) think they matter.</p> The Joel Test /blog/2007/05/the-joel-test /blog/2007/05/the-joel-test Matthew Jadud Wed, 30 May 2007 11:15:01 +0000 <p>Somewhere in the past, I took the <a href="">Joel Test</a>. I like this simple measure of a software project's health, because it hits on the really important things with respect to quality and maintainability. </p> <p><a href="">The Joel Test</a> is: </p><ol> <li>Do you use source control? <strong>Yes</strong></li> <li>Can you make a build in one step? <strong>Yes</strong></li> <li>Do you make daily builds? <strong>Yes</strong></li> <li>Do you have a bug database? <strong>Yes</strong></li> <li>Do you fix bugs before writing new code? No</li> <li>Do you have an up-to-date schedule? <strong>Yes</strong></li> <li>Do you have a spec? No</li> <li>Do programmers have quiet working conditions? No</li> <li>Do you use the best tools money can buy? <strong>Yes</strong>(ish)</li> <li>Do you have testers? No</li> <li>Do new candidates write code during their interview? N/A</li> <li>Do you do hallway usability testing? Yes</li> </ol> <p>This is better than a few years ago. On ever commit, buildbot runs and makes sure we didn't break anything (on multiple architectures). We don't fix all the bugs before moving on---but some of those tickets in the database are things like "Write a new linker." I've given us a "no", but for actual, build-breaking, test-breaking bugs, things get fixed before we move on. We don't work to a fixed schedule---instead, we tend to work towards features. So, I'll claim we do have an up-to-date schedule. We don't have a spec anymore, but we work to the best documentation available whenever possible; this may change in the future. We don't have the best tools that money can buy, but we do the best with the freely available tools we can cobble together. And, we tend to chat about new features before implementing, which is as close to hallway usability testing as we can get when working on a virtual machine.</p> <p>So, I'd give us a 7 out of 11; we don't interview, so I'll drop it from the list. I think that's not bad... quite good, in fact, for a small, unfunded open-source project.</p> Regarding Timers /blog/2007/05/regarding-timers /blog/2007/05/regarding-timers Matthew Jadud Sat, 05 May 2007 10:55:37 +0000 <p>We ran into something interesting the other day on the LEGO Mindstorms. In particular, something to do with <em>time</em>. </p> <p>You see, the LEGO is a 16-bit machine. This means the largest number it can represent is 2^16 - 1, or 65535. This is OK in most cases, but I want to note that the LEGO's clock runs at millisecond speed. That means that it ticks over once per millisecond. The number of milliseconds we can count before the clock rolls over is... 65,535.</p> <p>If there are 1000 milliseconds in a second, that means that we can only count up to 65 seconds before the world ends. Now, this is fine for James Bond movies (where the hero always has one minute to accomplish his death-defying task), but we kinda want to be able to measure longer amounts of time than that. </p> <p>In occam-pi, we can do this in some very cool ways. So, consider this a brief introduction to timers in occam-pi, and more specifically, how we can neatly create timers that 'tick' at arbitrary intervals without too much effort. I'll continue this later in another post that explores how I build up to having timers that let me measure hours, days, weeks, and even months on the LEGO Mindstorms. For the moment, I'll start with some basics about timers.</p> <p>To start, occam-pi has a notion of a TIMER built into the language. This is kinda cool. If I wanted to write a process that would delay for some number of milliseconds, it would look like this:</p> <pre> <span style="color: #0066FF; font-weight: bold;">PROC</span> delay (<span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span> timeout) <span style="color: #006600; font-weight: bold;">TIMER</span> t: <span style="color: #006600; font-weight: bold;">INT</span> the.time: <span style="color: #0066FF; font-weight: bold;">SEQ</span> t ? the.time t ? <span style="color: #006600; font-weight: bold;">AFTER</span> timeout : </pre> <p>The process takes one argument, which is a timeout value given in milliseconds. The timeout value has a type, which is <span style="color: #006600; font-weight: bold;">VAL INT</span>. This means that it is an integer, and furthermore, we cannot change it---any attempt to modify this value will result in a compile-time error. In other words, we're saying it is a <em>constant</em> within this PROC. </p> <p>We then declare two local variables: one of type <span style="color: #006600; font-weight: bold;">TIMER</span> and one of time <span style="color: #006600; font-weight: bold;">INT</span>. In sequence, we then read the current time into the variable 'the.time', and then use a funny bit of syntax to do our delay: </p> <pre> t ? <span style="color: #006600; font-weight: bold;">AFTER</span> timeout </pre> <p>This is some ugly stuff---something I wish they had done differently in the language. However, occam is over 20 years old, so we'll cut it some slack. This syntax says "delay until <em>timeout</em> milliseconds have passed." </p> <p>And that's it. What's nice about this PROC is that it does not block other processes from doing their stuff. So, it effectively says "sleep, but let other people run around and do things." Very cool. </p> <p>Now, we can't use this process to delay for more than 60 seconds or so, because the LEGO Mindstorms is limited by a maximum of roughly 65K milliseconds. Since I want to be able to do nifty things that involve timing out for more than 60 seconds, I want to start by writing a 'ticker' process that will generate clock ticks that I can listen for at intervals other than milliseconds.</p> <pre> <span style="color: #0066FF; font-weight: bold;">PROC</span> tick.generator (<span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span> delay, <span style="color: #006600; font-weight: bold;">CHAN</span> <span style="color: #006600; font-weight: bold;">BOOL</span> tick!) <span style="color: #006600; font-weight: bold;">TIMER</span> t: <span style="color: #006600; font-weight: bold;">INT</span> timeout: <span style="color: #0066FF; font-weight: bold;">SEQ</span> t ? timeout <span style="color: #0066FF; font-weight: bold;">WHILE</span> <span style="color: #9900FF;">TRUE</span> <span style="color: #0066FF; font-weight: bold;">SEQ</span> timeout := timeout <span style="color: #006600; font-weight: bold;">PLUS</span> delay t ? <span style="color: #006600; font-weight: bold;">AFTER</span> timeout tick ! <span style="color: #9900FF;">TRUE</span> : </pre> <p>This is only slightly bigger than the previous PROC. The tick.generator has two arguments---a constant (<span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span>), and a channel that carries booleans (<span style="color: #006600; font-weight: bold;">CHAN</span> <span style="color: #006600; font-weight: bold;">BOOL</span>). The channel is an <em>output</em> channel because it has been decorated with a <strong>!</strong>. (The compiler can figure this out for itself, but good style dictates that we put the <strong>!</strong> on the channel end in the PROC definition.) </p> <p>Again, I declare the local variables t and timeout, read the current time, and then drop into an infinite loop. This loop increments the timeout value, delays, and then outputs a signal on the boolean channel. Now, I can put this and a few other things together to create a complete program. This will execute just fine on your desktop, so if you paste it into your JEdit window, save it as "ticker.occ", compile, and run, you can see what it does. (Note that I've defined constants for SECONDS and MILLIS; on the LEGO, you would redefine MILLIS to be <strong>1</strong>.)</p> <pre> <span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span> MILLIS <span style="color: #006600; font-weight: bold;">IS</span> <span style="color: #FF0000;">1000</span>: <span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span> SECONDS <span style="color: #006600; font-weight: bold;">IS</span> <span style="color: #FF0000;">1000</span> * MILLIS: <span style="color: #0066FF; font-weight: bold;">PROC</span> delay (<span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span> timeout) <span style="color: #006600; font-weight: bold;">TIMER</span> t: <span style="color: #006600; font-weight: bold;">INT</span> the.time: <span style="color: #0066FF; font-weight: bold;">SEQ</span> t ? the.time t ? <span style="color: #006600; font-weight: bold;">AFTER</span> timeout : <span style="color: #0066FF; font-weight: bold;">PROC</span> tick.generator (<span style="color: #006600; font-weight: bold;">VAL</span> <span style="color: #006600; font-weight: bold;">INT</span> delay, <span style="color: #006600; font-weight: bold;">CHAN</span> <span style="color: #006600; font-weight: bold;">BOOL</span> tick!) <span style="color: #006600; font-weight: bold;">TIMER</span> t: <span style="color: #006600; font-weight: bold;">INT</span> timeout: <span style="color: #0066FF; font-weight: bold;">SEQ</span> t ? timeout <span style="color: #0066FF; font-weight: bold;">WHILE</span> <span style="color: #9900FF;">TRUE</span> <span style="color: #0066FF; font-weight: bold;">SEQ</span> timeout := timeout <span style="color: #006600; font-weight: bold;">PLUS</span> delay t ? <span style="color: #006600; font-weight: bold;">AFTER</span> timeout tick ! <span style="color: #9900FF;">TRUE</span> : <span style="color: #0066FF; font-weight: bold;">PROC</span> tick.seconds (<span style="color: #006600; font-weight: bold;">CHAN</span> <span style="color: #006600; font-weight: bold;">BOOL</span> tick!) tick.generator(<span style="color: #FF0000;">1</span> * SECONDS, tick!) : <span style="color: #006600; font-weight: bold;">VAL</span> BYTE flush <span style="color: #006600; font-weight: bold;">IS</span> <span style="color: #FF0000;">255</span>: <span style="color: #0066FF; font-weight: bold;">PROC</span> show.seconds.tick(<span style="color: #006600; font-weight: bold;">CHAN</span> BYTE kyb, scr, err) <span style="color: #006600; font-weight: bold;">CHAN</span> <span style="color: #006600; font-weight: bold;">BOOL</span> ticker: <span style="color: #0066FF; font-weight: bold;">PAR</span> tick.seconds(ticker!) <span style="color: #0066FF; font-weight: bold;">WHILE</span> <span style="color: #9900FF;">TRUE</span> <span style="color: #0066FF; font-weight: bold;">SEQ</span> <span style="color: #006600; font-weight: bold;">BOOL</span> tmp: ticker ? tmp scr ! <span style="color: #9900FF;">'x'</span> scr ! flush : </pre> <p>The last process, 'show.seconds.tick', demonstrates how we don't always have to name our processes to run them in parallel. First, I declare a channel to carry boolean values (remember, channels are like wires---channels carry information from one process to another.) Then, in parallel, I want to run the process 'tick.seconds' and an infinite loop that I've written. The process 'tick.seconds' gets one end of the channel 'ticker'---in particular, it gets the end of the channel that will be written to. </p> <p>In the infinite loop, I do a few things in sequence. First, I declare a temporary variable that is only valid for <em>one line of code</em>. That is, it exists just long enough to read from the ticker channel into the variable 'tmp'. This is because I don't actually care about keeping that value, but I have to read from the channel into something. After reading from the channel, I output an 'x' to the screen, and then I flush the screen, so that the output is actually shown.</p> <p>Remember that occam-pi channels are <em>blocking</em> channels. That means that my infinite loop will stop every time it hits the line 'ticker ? tmp'. It will wait until the process 'tick.seconds' has written a value, and then the loop can continue. Also, it is important to remember that every time you see a channel read (eg. 'ch ? var') or a channel write (eg. 'ch ! var'), the Transterpreter deschedules the currently executing process, and goes and runs something else. This way, everyone gets a turn to execute. This is why the delay process I introduced earlier does not block everything running on the system, and why we can write infinite loops as casually as I have here... because each one contains a channel communication.</p> <p>In my next post, I'll follow up with how to extend this into timers that can last for days or even weeks on something as small as the LEGO Mindstorms. Also, I still need to follow up on our subsumption code for the <a href="">Surveyor</a>---but that will come, perhaps, after this timer exploration. In fact, this came up because we were trying to run our code from the Surveyor on the LEGO, and we discovered that having a 32-bit timer (like on the <a href="">Surveyor SRV-1</a>) is very handy, while having a 16-bit timer (like on the LEGO Mindstorms RCX) is a bit annoying... where "annoying" means "Hey! Our code doesn't work!"</p> Subsumption on the SRV-1 /blog/2007/04/subsumption-on-the-srv-1 /blog/2007/04/subsumption-on-the-srv-1 Matthew Jadud Sun, 08 Apr 2007 16:16:31 +0000 <p>I recently caught sight of the article <a href="">Subsumption for the SR04 and jBot Robots</a> over at the Dallas Personal Robotics Group's webpage. What is sad is that the first thing that David Anderson goes into is technical details regarding their <em>timing loop</em>. Sadly, timing has nothing to do with implementing the subsumption architecture. Our <a href="">recent explorations on the SRV-1</a> demonstrate this nicely, I think. </p> <p>At AAAI, Jon and I implemented code for the mini-robotics-challenge that was hosted in our session. We had to program a robot that would: </p> <ol> <li>Wander as far from its start point as possible in 30 seconds, avoiding obstacles</li> <li>Return as close to home in the next 30 seconds.</li> </ol> <p>This is a trivial task, but all the participants were using robotics platforms they had never seen before, and had roughly 1.5 hours to implement their solutions. Having first seen the SRV-1 just a day before, we had ported the Transterpreter to it, and had basic motor control and access to the IR sensors. Using this, we looked at the problem, and tackled it by implementing a three-layer subsumption architecture. </p> <p>At the lowest level, we wanted our robot to go forward. We wanted a higher-level behavior that would avoid obstacles, and a third, even higher level behavior that would (after 30 seconds) replay the motor commands that had been generated in the previous 30 seconds (causing the robot to retrace its steps). This is a bit odd as subsumption architectures go (having a layer that suppresses everything below it simply to replay previous actions), but it yielded a very clean implementation. </p> <p>What follows is the network, and a step-by-step decomposition of how it worked. </p> <p align="center"> <img src="/images/blog/2007/20070408-srv1-subsumption-01-1.png" height="429" width="394" border="0" hspace="4" vspace="4" alt="20070408-Srv1-Subsumption-01-1" /> </p> <p>Each orange box represents an occam-pi <strong>process</strong>; in some languages, you might call these "threads". However, in occam-pi, processes have certain properties. First, they are completely sealed, and therefore, no other process can manipulate their state; for example, anything in the "forward" process is completely isolated from the "avoid" process, and visa versa. Also, occam-pi processes are incredibly light-weight: we can run hundreds of these processes on robots the size of a LEGO Mindstorms or the SRV-1, and therefore casually write highly concurrent programs for very small robots. </p> <p>The arrows in this diagram are <em>channels</em>; they carry data from one process to another. In fact, channels are the only way two processes can communicate with each-other. First, they are <strong>unidirectional</strong>; you can only talk in one direction down a channel. Second, they are <strong>blocking</strong>, meaning that once committed to a communication (either a send or a receive), the communicating process deschedules, and waits until the other half of the communication is carried out. This explicit synchronization makes it easy to reason about many different processes taking turns doing computation and communicating between each-other. </p> <h3>Level 0: Going Forward</h3> <p>With this brief background in occam-pi, you can begin to piece together the network. The <em>forward</em> process sends motor commands to a <em>suppressor</em> process (S1), which communicates on to another <em>suppressor</em> process (S2). This then communicates to a <em>delta</em> process (which copies its input to two outputs). One of those outputs flows to the motors, and the other to a process called <em>record</em>, which records all the motor commands sent to it. In the common case, both suppressors are inactive, and motor commands flow around the network as depicted below: </p> <p align="center"> <img src="/images/blog/2007/20070408-srv1-subsumption-02.png" height="429" width="394" border="0" hspace="4" vspace="4" alt="20070408-Srv1-Subsumption-02" /> </p> <h3>Level 1: An object detected</h3> <p>When an object is detected, the <em>avoid</em> process kicks in. Note that it does not "turn off" or "disable" the forward process---both are always running concurrently. However, suppressor <em>S1</em> becomes active when the <em>avoid</em> process goes active. It does this when it detects something in front of the robot using the IR sensor. It then begins sending commands to pivot the robot to the left. (This is a pretty dumb robot, really.) These new commands are routed through <em>S1</em> instead of the commands from <em>forward</em>, and the robot pivots away from the obstacle (and records the pivot as well). </p> <p align="center"> <img src="/images/blog/2007/20070408-srv1-subsumption-03.png" height="429" width="394" border="0" hspace="4" vspace="4" alt="20070408-Srv1-Subsumption-03" /> </p> <h3>Level 2: Replay!</h3> <p>The trickiest bit of work comes when we begin replaying our movements after 30 seconds have passed. The live portions of the network look like this: </p> <p align="center"> <img src="/images/blog/2007/20070408-srv1-subsumption-04.png" height="436" width="394" border="0" hspace="4" vspace="4" alt="20070408-Srv1-Subsumption-04" /> </p> <p>The <em>replay</em> module suppresses the output of both the <em>forward</em> and <em>avoid</em> processes. It also <em>inhibits</em> communications between the <em>delta</em> process and the <em>record</em> process; this is because we definitely do not want to be recording the playback! (We made this error, initially; it yielded very strange robot behaviors, and usually ended in both a crash of hardware and software.) </p> <p>The <em>replay</em> module requests actions from the <em>record</em> process, and then plays each of those actions down the channel it has to <em>S2</em>; these are routed through to the <em>run.motor</em> process. The end result is that our robot replays all of the actions that were recorded over the previous thirty seconds. </p> <h3>It seems so complex!</h3> <p>It may, or it may not---it depends on how you like to think about control systems for embedded devices (like little robots). I prefer this model, where nowhere in my code am I worrying about timing---instead, I'm just worrying about where my data is flowing. In this case, I'm flowing motor commands from one process to another, and I've written special processes (the suppressors and inhibitor) to control how that data flows through the network. The language and runtime are responsible for handling the concurrency---not me. </p> <p>For a C programmer, this is a completely foreign idea. However, it is (I believe) the case that C is generally useful only for very specific tasks, like hardware interfacing. Handling complex data structures, complex data flows, and any kind of concurrency or parallelism is something that should be left to the compiler, not the programmer. Put another way, we as programmers are too error prone to be left on our own writing complex control loops, and languages like occam-pi are a first step towards managing that complexity. </p> <p>As always, we encourage you to take a look at <a href="">RoboDeb</a> if you're interested in the intersection between robotics and concurrent programming languages. We're very, very close to releasing the newly revised LEGO Mindstorms runtime, which will let you explore occam-pi on your Mindstorms as well. (Our NXT port will follow that release.) </p> <h3>What about the code?</h3> <p>Most of the code for this was written in the twenty minutes before the robotics competition. I could go into the code here, but for the moment will beg off (it is a beautiful day, and I'd rather be outside). However, you <a href="">can see the code online in our Subversion repository</a>. </p> <p>If you want to know more about the code, drop me an email (matt at transterpreter dot org), and I'll put together a post that goes through it in more detail. You might start with Jon Simpson's paper <a href="">Mobile Robot Control: The Subsumption Architecture and occam-pi</a> (PDF), as it goes through a similar process network, and includes code for the inhibitor and suppressor processes. (It will be more readable than anything I hack into the 'blog, anyway.) </p> Our Google TechTalk /blog/2007/04/our-google-techtalk /blog/2007/04/our-google-techtalk Matthew Jadud Wed, 04 Apr 2007 02:19:46 +0000 <p>On March 29th, we gave our <a href=";esrc=sr1&amp;ev=v&amp;q=transterpreter&amp;vidurl=;usg=AL29H23BWGkZgrYs6F3TNiSvN9stONipWg">first Google TechTalk</a>!</p> <p>You can hit the link to go to the Google Video site. So far, we've had quite a few views, and apparently, people enjoy it (according to the ratings, anyway). We had a hard time targeting the talk; we rewrote it several times before giving it, as we couldn't decide how best to approach the work we've been doing for the last four years. Do we dive straight into how we can use a language like occam-pi for abstracting over clusters, or how we can build safer, concurrent software on embedded platforms?</p> <p>In the end, we gave an introduction to the language, and then focused a bit on how we can convert parallel models of software directly into code, and motivated this using <a href="">Jon Simpson's work with the subsumption architecture</a> on small robotics platforms. We received many positive comments on the talk, and one person did say they thought it was light on "hard-core" detail. But what can you do in 45 minutes with an unknown audience? </p> <p>In the end, we gave a good talk, and were excited to be able to share our work with the Googlers there and who are now catching the video on-line. (And, for that matter, anyone else who wants to see what we had to say.) </p> <p>Certainly, it was a lot of fun. And yes, the food in the cafeterias at Google really is <em>that</em> good.</p> AAAI Robotics and Education 2007 /blog/2007/04/aaai-robotics-and-education-2007 /blog/2007/04/aaai-robotics-and-education-2007 Matthew Jadud Mon, 02 Apr 2007 18:35:56 +0000 <p>We spent the first three days of last week taking part in the AAAI 2007 Spring Symposium. In particular, we joined the <a href="">Robotics and Education</a> track with about 50 others interested in using robotics to teach everything from introductory programming to vision, pathfinding/planning, decision making, and all the other tasty bits of artificial intelligence.</p> <table align="right"> <tr><td> <center> <img src="/images/blog/2007/FrontRight.jpg" height="125" width="180" border="0" hspace="4" vspace="4" alt="Frontright" /> <br /><em>Scribbler</em> </center> </td><td> <center> <img src="/images/blog/2007/HistoriaLogo01.jpg" height="116" width="180" border="0" align="right" hspace="4" vspace="4" alt="Historialogo01" /><br /><em>Turtle</em> </center> </td></tr></table> <p>We saw some very cool projects and platforms. I really liked Roomba Pac-Man (where students program Roombas to wander hallways and vacuum up messes in a pac-man like way), and the <a href="">Scribbler</a> (while not very aesthetically pleasing) is a cute re-creation of the LOGO turtle. (I've provided a side-by-side comparison of what these two robots look like; the original turtle was, I think, far more attractive.)</p> <p>In terms of platforms, many people are doing "tethered" robotics, where they either physically or wirelessly tether their robot to a PC. This is, I think, unfortunate, as I feel it takes something away from the process of programming the robot---it is a remote control process, as opposed to an autonomous one. However, like many things, it depends on what your pedagogic goals are. However, I do look forward to a few things: I think the <a href="">Qwerk is a neat</a> (part of the TeRK project) and we hope that we have a chance to work with it sometime in the future. It's a powerful computational platform with a lot of nice outputs for motor and servo control, as well as managing a host of sensor inputs. Likewise, the <a href="">Blackfin Handyboard</a> is a very powerful platform, and will also be great to begin exploring, as it provides some very flexible programming options in the form of two Xilinx FPGAs. Fred and Andrew did a very nice job with the board design, and I expect many cool things will be done with this platform.</p> <p>We also saw <a href="">David Miller's XBC/Gameboy combination</a>, which I now have a Gameboy to try this out with (once I have a budget to get the rest of the bits, which is actually the expensive part). However, the really fun new platform was the <a href="">Surveyor Robotics SRV-1</a>.</p> <p>Howard <a href="">wrote about our experiments with the SRV-1 on his weblog</a>, and I'll add a bit here, and followup in a later post with some more detail. We were given an SRV-1 to borrow on Monday evening, and did very little with it, as we were missing critical software. Tuesday, during coffee breaks and the like, Christian ported the Transterpreter to the SRV-1. In less than two days, we saw a new, ARM7-based robotics platform, and had the Transterpreter running subsumption code on it. In three days (that is, the day after the conference), Christian had vision working. </p> <p align="center"> <img src="/images/blog/2007/AAAI_the_winners.jpg" height="414" width="500" border="0" hspace="4" vspace="4" alt="Aaai The Winners" /> </p> <p>I went ahead and stole a picture from Howard; here, you can see the end-result of our hacking, which is that we won the AAAI Robotics and Education robotics programming challenge. We did this with the SRV-1 after having seen it for the first time on Monday, having ported our runtime to it on Tuesday, and having seen the challenge on Wednesday morning. We managed a (not-quite-working) 3-layer subsumption network that tackled the challenge in about 20 minutes of furious hacking; I'll write more in detail about our code (which we've subsequently cleaned up and corrected) in a followup post.</p> <p>For now, we're still kicking around San Francisco and the Bay area, enjoying two days off before flying back to England on Wednesday. </p> Is that a supercomputer in your pocket? /blog/2007/03/is-that-a-supercomputer-in-your-pocket /blog/2007/03/is-that-a-supercomputer-in-your-pocket Matthew Jadud Fri, 23 Mar 2007 03:27:16 +0000 <p>Not too long ago, we made a few interesting changes to the Transterpreter.</p> <p>The first was that we updated the Multiterpreter. Donkeys-months ago (like donkeys-years, but not quite as long) we modified the interpreter to map occam-pi processes to operating system processes. This meant that we could run code in true parallel on multiple CPU or multiple-core machines. Unfortunately, the cost for process switching was so high, we couldn't show this to anyone on the planet for fear of being shamed.</p> <p>About two months ago, Adam and I sat down (I rode copilot) and rewrote that code to use POSIX threads instead of OS processes. In fact, we didn't even do anything savvy, like use a thread-pool; instead, we just create operating system threads when they're needed, reclaim then when they're done. Not the most efficient way to parallelize the virtual machine, but we like to do things in the simplest way possible first. In this case, it had the smallest impact on the codebase, so we did it that way.</p> <p>On my laptop ("Lyra", a dual-core 2.0GHz Intel Core Duo MacBook), I ran some code that varies three things:</p> <ol> <li>The number of parallel processes being executed</li> <li>The amount of computation each process carries out</li> <li>The number of communications each process makes while computing (context switches)</li> </ol> <p>In a multi-threaded or multi-core system, this will stress our run-time and tell us at what workload our parallel virtual machine is more efficient than running a single-threaded runtime on a single core. The core of the code comes from Kevin Vela's doctoral thesis from the University of Kent (not available online). </p> <pre> PAR i = 0 FOR abyg CHAN INT chan: SEQ j = 0 FOR pleng PAR SEQ SEQ k = 0 FOR granu array[(granu * i) + k] := array[(granu * i) + k] + i chan ! i INT t: chan ? t </pre> <p>So, what did we find? </p> <p>(<em>Note that in each of these graphs I've varied both the y- and x-axis; this is bad reporting practice, and something I will correct when the tech report is put together---for now, these are the graphs I have, and I'm using them as-is. </em>)</p> <p align="center"> <img src="/images/blog/2007/20070202-lyra-parallel.jpg" height="369" width="400" border="1" hspace="4" vspace="4" alt="20070202-Lyra-Parallel" /> </p> <p>To be clear, the different glyphs represent the maximum number of POSIX threads that were allowed to be running at the same time to support a single virtual machine. On the y-axis is the time in seconds needed to compute the replicated PAR shown above (averaged over two runs), and the x-axis is the granularity of the work packets---larger granularity means more work gets done before a context switch is allowed to take place.</p> <p>On my MacBook (above), we had to have each thread do a significant amount of work (process a 65K element array in a non-trivial manner) with relatively small amounts of communication (long work periods) before we saw a speedup due to parallelism. My suspicion is that OS X is such a hungry operating system that it (and other application processes) were constantly asking for attention, and therefore, the Transterpreter threads rarely had the opportunity to run for a significant amount of time.</p> <p>So, to belabor this just a bit, we can see that with one thread of control (a single-threaded interpreter), we compute packets of granularity 1 approximately 20x faster than if we run a 2-threaded interpreter. On a two-core machine, this doesn't necessarily make sense... unless we consider the possibility that the two POSIX threads are spending all of their time contending for an opportunity to run, in which case it makes sense that we're not seeing any benefit of having a parallel runtime.</p> <p>As the granularity approaches 50 (which we can convert into a specific number of VM instruction cycles, if we wanted to), we see that the single-threaded and dual-threaded interpreter run at almost the same speed. This is because we are no longer seeing the contention at the operating system level for multiple threads to be executing. Or, perhaps it is because we are no longer fighting the operating system for a chance to execute. </p> <p>On Hadar, a SunFire 880 with four 750MHz UltraSparc III processors and 8GB of RAM, I found the results were... a bit odd:</p> <p align="center"> <img src="/images/blog/2007/20070202-hadar-parallel.jpg" height="394" width="400" border="1" hspace="4" vspace="4" alt="20070202-Hadar-Parallel" /> </p> <p>Ignoring a granularity of one for a moment (which is effectively measuring context switch time of the POSIX implementation on a given machine), we can see that a two- and four-threaded interpreter are always slower than a single-threaded interpreter. However, if we increase the number of threads the Transterpreter is allowed to spawn to 8 or 16, the Transterpreter is always faster for even small workloads than a single-threaded Transterpreter. </p> <p>I have not fully investigated this yet; I believe this is because Solaris handles pthreads differently than other operating systems. In particular, it has a notion of virtual CPUs, and I think (but am not sure) that with too few threads, it assigns them all to a single virtual CPU, which is assigned to a single physical CPU. Therefore, even though we have four cores (most of which are idle), there is massive contention on a single CPU. When the OS sees more threads in play, it actually farms them out to multiple virtual (and physical) CPUs. </p> <p>I have to read more on this (and have some excellent resources from Sun that I picked up while at SIGCSE), and do not doubt that this situation can be improved. Our initial testing was only over the course of a few days, and I need to investigate this further before turning the work into a tech report. (From exploration, to blog post, to tech report, to publication, I suppose.) In other words, I'm sure that Solaris isn't completely borked---instead, I assume I have to do some more work to make sure it does what I want, instead of whatever the current default is.</p> <p>The last system we did some tests on was Ninja, an Intel SR1200 server with two PIII 1.4GHz processors and 2GB of RAM running Debian GNU/Linux with a SMP kernel (rev. 2.6.17-2-686). </p> <p align="center"> <img src="/images/blog/2007/20070202-ninja-parallel.jpg" height="405" width="400" border="1" hspace="4" vspace="4" alt="20070202-Ninja-Parallel" /> </p> <p>Ninja produced the curves I expected from the other two machines, actually. We see that a single-threaded Transterpreter is faster than a multi-threaded interpreter for very small work packets. However, with a granularity of 5, we see that a single-threaded, dual-threaded, and quad-threaded interpreter all run at roughly the same speed. From that point forward, it is better to have multiple threads in the interpreter than a single thread, as more work gets done in unit time. And, I believe that this easily represents "real world" work packet sizes.</p> <p>What I don't quite understand is why four threads is better than two for very small granularities. Like the other platforms, there is a good deal more investigation to be done before I understand the implementation of POSIX threads in a given operating system (OSX/BSD vs. Solaris vs. Debian GNU/Linux 2.6) and how that effects the execution of a parallel Transterpreter.</p> <h3>Why do I think this is cool?</h3> <p>The Transterpreter runs on the Texas Instruments MSP430 (an 8MHz, 10KB RAM embedded processor), the LEGO Mindstorms (a 16MHz embedded system with 32KB of RAM for both the interpreter and code), as well as my MacBook, Christian's G4 Powerbook, our Linux dual-processor server, quad-processor Suns, and most likely any machine with a C compiler. The code I ran to test our "POSIXterpreter" does not exercise any features of the language that would not run on all of these platforms---put another way, the <em>bytecode</em> for my tests could be executed on the MSP430 without modification; we might blow the RAM if we make the replicated PAR too big, but that's about it. That's how portable the code is across Transterpreter instances right now.</p> <p>Tyan recently announced their <a href="">40-processor "desktop supercomputer."</a> This is a set of five, dual quad-core (8-processor) blades in a box on wheels. So, you get five computers in a box, each with eight processors, and a max of 60GB of RAM (12GB per blade). And the price? $20,000.</p> <p>Not bad.</p> <p>I haven't written it up yet, but the Transterpreter also does OpenMPI. That means that we can actually spread computation across machines in a cluster as well as across SMP cores. This is <em>way alpha</em> code, but I've demonstrated to the group that we can split things up on the Darwin H4 supercomputing cluster (a 2.4GHz Dell and a 3.2GHz Dell under adjacent desks on a 100 Mb network). Given a bit of time, this could become a first-class part of the Transterpreter release, and we'd have an excellent environment for controlling parallelism in a heterogeneous cluster environment. </p> <p>Or, as the case may be, an excellent way of exploiting the resources of a 40-processor supercomputer-in-a-box. With five POSIXterpreters running large thread pools, adding some intelligent (batch) scheduling, and a more fully-featured set of MPI bindings, we'd have a seriously smart setup for doing parallel computing. All the pieces are there, but we don't have the financial justification to dedicate the time to the development. So, we continue stealing a little bit of time here and there to see if we can demonstrate that this is a really powerful way to orchestrate the use of high-perf libraries (like BLAST, LAPACK, etc.) in a robust, safe, and semantically clear way in a truly parallel environment.</p> Assumptions in Wireless Sensor Networks (WSNs) /blog/2007/03/assumptions-in-wireless-sensor-networks-wsns /blog/2007/03/assumptions-in-wireless-sensor-networks-wsns Matthew Jadud Sun, 04 Mar 2007 02:45:18 +0000 <p>I have been doing a fair bit of reading in the area of wireless sensor networks (WSNs) as a part of my work as a research associate on the DIAS project. It is interesting to note how few WSN papers actually deal with <em>real problems</em>. I should do a survey of my current literature holdings and determine how many of them are grounded in an actual problem that the authors were trying to solve.</p> <p>For example, consider this snippet:</p> <p align="center"> <a href="/images/blog/2007/20070304-wsn-assumptions.png" onclick="'/images/blog/2007/20070304-wsn-assumptions.png','popup','width=427,height=110,scrollbars=no,resizable=yes,toolbar=no,directories=no,location=no,menubar=no,status=yes,left=0,top=0');return false"><img src="/images/blog/2007/20070304-wsn-assumptions-tm.jpg" height="100" width="388" border="1" hspace="4" vspace="4" alt="20070304-Wsn-Assumptions" /></a> </p> <p>The authors assume you have enough nodes in your network to form <em>clusters</em>. Most of the deployments I have to consider aren't going to have a budget large enough to deploy so many nodes that we can form <em>clusters</em>. So, what do I do if I'm developing a four-node network? What use is this algorithm to me? Do I let four nodes duke it out for the role of "cluster head", even though I already know the node with the GSM radio (read: mobile phone) is going to be in charge?</p> <p>Assumptions are dangerous things.</p> The sorry state of embedded development under Linux /blog/2007/02/the-sorry-state-of-embedded-development-under-linux /blog/2007/02/the-sorry-state-of-embedded-development-under-linux Matthew Jadud Wed, 28 Feb 2007 12:00:19 +0000 <p>This... this is a rant.</p> <p>It is incredibly frustrating being a Mac and Linux user if you're interested in embedded systems development. In particular, if you are working with the MSP430, one of the only viable parts if you're really, <em>really</em> concerned about power consumption. There simply aren't, at this time, any parts (that I know of) that come close in terms of power saving modes.</p> <p>On the Mac and under Linux, it is possible to build the MSP430 GCC toolchain. That's a big step, and it integrates nicely into build systems like make and Scons. However, that's only part of the problem. Once you've built the executable for your embedded platform, you need to make the code go from the PC to the embedded device.</p> <p>By far, the most common way for this to take place is via JTAG. This multi-wire protocol provides a mechanism by which the flash in these little processors can be rewritten from the host PC, as well as (if the tools are available) in-circuit emulation. Of course, both of these assume certain hardware and software connected to and on the host PC. The biggest problem I'm running into right now is that the JTAG programmer <strong>MUST</strong> be USB-based, or I can't use it---because my Mac doesn't have a parallel port, so even if I'm going to build a Linux virtual machine, it still needs to be a USB-based solution.</p> <p>For example, both Rowley and Softbaugh produce USB-based MSP430 JTAG programmers; neither provide support for Linux. They have Windows XP and 2000 drivers, but they don't support any open-source platforms; I guess embedded systems developers don't use Linux. (As long as you're paying &pound;150 for a programmer, you might as well pay for your OS as well.) I'm not having much luck discovering which platforms the MSP-FET430UIF does or does not work with; are there Linux drivers? Aren't there? </p> <p>The problem, quite simply, is that we are an open-source project that builds under Windows, Mac, and Linux; somehow, I'd like to easily build the MSP430 version of the Transterpreter and install code on a development board. Currently, I've been working on the Tmote Sky simply because it has USB---but that's not acceptable looking forward. And, more specifically, I travel a fair bit---I have to have a mobile solution that can work from my MacBook. </p> <p>It seems to me that most embedded systems developers must not travel much, and have a big workbench where everything is set up <em>just so</em> for the work that they do. These developers all use Windows, and they pay a lot for their tools---because they are not trying to do embedded development under the auspices of an open-source research project. They are, instead, charging what their time is worth, and they can (therefore) afford the tools and bench space to do what they need to do.</p> <p>I can run a Windows virtual machine on my Mac, and I'll have to; it will run the lite tools provided by Softbaugh, and I'll be able to program MSP430 boards using Tom's (Windows only) programmer over the Texas Instrument flavor of JTAG. (Why, oh why, do embedded developers put up with the lack of standards on flash programmers? Every single chip should be programmable and inspectable <strong>over the same protocol</strong>.) I can mount my MacBook via Samba, and my development process will look like this:</p> <ol> <li> Edit source on Mac. (Using freely available, open-source tools.) </li> <li> Compile on Mac. (Using freely available, open-source tools.) </li> <li> Flip to Windows, and upload the code to the MSP430. (Using expensive, closed-source tools.) </li> </ol> <p>Rinse and repeat.</p> <p>If anyone has any thoughts on how we can improve this situation, please let me know. It seems to be we should build a device that has USB on one side, a 20-pin header on the other, and we build cables for each chip family we want to support. There is one protocol on the USB side, and the MPU in the middle of those two connectors handles the JTAG translation for each device family on the other side. Then, we demand that TI, AVR, and every other embedded chip developer provide the driver for that device that sits in the middle---and nothing else. If they don't provide that software, we don't use their product. This kills the JTAG programmer market (proprietary hardware that is often tied not only to a single OS, but also a single <em>compiler</em>), but that's fine by me.</p> <p>This is the end of this rant... for now.</p> A DOSterpreter? /blog/2007/02/a-dosterpreter /blog/2007/02/a-dosterpreter Matthew Jadud Sat, 24 Feb 2007 00:14:41 +0000 <table align="right"><tr><td> <img src="/images/blog/2007/ohthehumanity-1.jpg" height="195" width="250" border="0" align="right" hspace="4" vspace="4" alt="Ohthehumanity-1" /> </td></tr></table> <p>Although it sounds odd, Damian built a <strong>DOS</strong> version of the Transterpreter last week. </p> <p>Oh, the humanity! </p> <p>This may sound wacky, but the reasoning is sound. We have a large suite of tests that we run the Transterpreter through on a regular basis. Actually, we use <a href="">buildbot</a> to run all of the tests anytime someone checks code into the trunk. However, this only runs the tests on "big" platforms---it runs them on a Linux box or two, and a Solaris/SPARC machine. We cannot, at this time, automatically run all of our tests on an embedded 16-bit target; while we do have a development board or two we could dedicate to this task, I'm not interested (right now) in doing the scripting to upload code over a JTAG port, run the tests, and check (on the device) to see if we are doing the right thing. It's a lot of scripting. </p> <p align="center"> <img src="/images/blog/2007/h8300.jpg" height="133" width="200" border="0" hspace="4" vspace="4" alt="H8300" /> <br /> <em>The H8/300, the processor at the heart of the LEGO RCX</em> </p> <p>So, when Jon was having a hard time with the LEGO RCX recently, it became clear that we really needed to run our tests on 16-bit platforms. This would help shake out whether or not the compiler, linker, or run-time were having a hard time when there were fewer ones and zeros running around. As a result, Damian setup the <a href="">Open Watcom 16-bit DOS compiler</a>, built the VM, booted up a <a href="">FreeDOS</a> image in <a href="">QEMU</a>, and ran a bunch of the tests through the VM. We passed all the tests we could easily run, and are working to get the rest compiling. It turns out we need to build some libraries that we've never needed before... </p> <p>So, yes, the Transterpreter can run in 16-bit mode in DOS. There's a number of features that are untested at this time, but hopefully we'll find a way to automate the testing of the VM under the FreeDOS image. </p> Floating point is in da house /blog/2007/02/floating-point-is-in-da-house /blog/2007/02/floating-point-is-in-da-house Matthew Jadud Wed, 21 Feb 2007 11:44:14 +0000 <p>I have a backlog of things I can write about in this space, so I'll do my best to do a bit of catching up.</p> <p>To start, Damian has been working on floating point for some time, and we now have full floating-point support in the runtime on large, little-endian targets. To be fair, the Transterpreter has always supported floating-point operations from day one. Unfortunately, it has carried out floating point operations in emulation. So, whenever the Transterpreter did floating point operations <em>in the past</em>, it would be emulating floating-point support and be doing all of the operations using integer maths at the interpreted level.</p> <p>This was slow.</p> <p>Words like <em>molasses</em>, <em>glacial</em>, and <em>election reform</em> come to mind as good words to describe the pace at which we used to carry out floating-point operations in the VM.</p> <p>Now, due to Damian's efforts, we have full floating-point support in the VM. This was one of the few remaining parts of the instruction set that had not been implemented yet. Now, the floating-point instructions that were introduced in later generations of the Transputer (T8 and above, I think?) are supported natively by the runtime, and we don't have to go out for coffee every time we touch a number with a decimal point.</p> An update for the year 2007 /blog/2007/01/an-update-for-the-year-2007 /blog/2007/01/an-update-for-the-year-2007 Matthew Jadud Mon, 08 Jan 2007 09:36:06 +0000 <p>The Transterpreter project was born three years ago around this time; it started life as a Scheme implementation of a Transputer bytecode interpreter, and has grown quite a bit since then. Given that we're starting into 2007, it seemed like a good time to give an update about what is going on in the group. In no particular order:</p> <ul> <li><p>Jon Simpson (third-year undergraduate at Kent) has been working on revitalizing the LEGO Mindstorms RCX port of the Transterpreter. The old version was built on top of BrickOS; we were, effectively, running two operating systems on the LEGO, one on top of the other! Jon is at the point where he has the Transterpreter running natively on the LEGO Mindstorms RCX, can handle button input (including a nifty, non-blocking debouncer), and is continuing to slowly build up to more useful end-user interface code. By March he expects to have a complete and usable (native) port of the Transterpreter on the RCX. (<a href="">/branches/new-lego</a>)</p> <p>Looking forward, Jon (and the rest of us) are excited about getting the NXT port of the Transterpreter rolling. We have already had some interest expressed from others about taking part in this effort; we'll be generating some mail on that shortly.</p> </li> <li><p>Damian ended the year cleaning up his Cell port of the Transterpreter (<a href="">/branches/cell</a>), with particular attention being paid to developing a native, big-endian port of the Transterpreter (<a href="">/branches/big-endian</a>). Currently, we pay a penalty on all big-endian hardware, and Damian (and other's) efforts in this regard will give us full, native performance on big-endian platforms; this effects the PowerPC (Mac, Cell) and Mindstorms RCX, among others. Looking forward, he is coming back to <strong>42</strong> (our experimental compiler), and wants to work out the critical points in generating little/big endian code as well as "fat binaries".</p></li> <li><p>Adam Sampson picked up a few Arduino boards (<a href=""></a>), which are driven by an Atmel ATmega8. This little chip has roughly 7KB of free flash and 1KB of RAM; it represents the smallest device the Transterpreter has been targeted at yet (<a href="">/branches/arduino</a>). While Adam's intent was to develop an Arduino port of the Transterpreter, he says that "in the process I've done a load of cleanups which are probably more useful than the work I was intending to do."</p> <p>The "cleanups" that Adam has made in the Arduino branch are (in his words):</p> <ol> <li><p>wrapper (currently untested) for the ATmega8</p></li> <li><p>instruction dispatching via switch, so all the instructions end up inlined, which cuts the code size considerably (and should make it a bit faster)</p></li> <li><p>some fixes to comments</p></li> <li><p>consistent ANSI C prototypes throughout</p></li> <li><p>add multiple-inclusion guards to all the headers</p></li> <li><p>make the word size code use inttypes.h and be configurable at compile time</p></li> </ol> <p>These are all nice cleanups that will ultimately be merged back into the trunk, and therefore benefit all of the Transterpreter ports.</p> <p>Adam's explorations also involved cleaning up the "memory array" code (<a href="">/branches/mem-array</a>). When originally developing the TVM, we had a "virtual memory", which made detecting a variety of faults easier; this was the original memory interface for the Transterpreter, but was dropped shortly after porting to C. Adam has cleaned this up as well, and (in his words, again):</p> <ol> <li><p>make the array memory backend actually work (mostly with the aim of being able to use a non-native word size, although I haven't yet tested that)</p></li> <li><p>do bounds checking in mem_array so that the TVM stops (rather than segfaults) in the event of a bad memory access</p></li> <li><p>fix a number of bugs related to string handling and memory allocation which were revealed by Valgrind; we need to do a Valgrind run with some proper code (using dynamic memory) at some point to try and shake out more</p></li> <li><p>completely rework the code in stiw that builds the memory map, since it had a number of bugs and didn't work with mem_array before</p></li> <li><p>fix a few bugs in interpreter revealed by the work above</p></li> </ol> <p>These are all excellent, and again will find their way back into the trunk. Indeed, as part of his explorations, Adam generated a number of tickets that I added to Trac (<a href=""></a>), and we can begin addressing in the interpreter, which is good.</p> </li> <li><p>Matt Jadud (that's me) has been focusing on the MSP430 port, living in the Tmote Sky branch (<a href="">/trunk/wrappers/tmotesky</a>). As a side effect of this work (along with Jon Simpson), we developed 'tinyswig' (<a href="">/trunk/scripts/tinyswig.scm</a>), a mini, SWIG-like script that lets us quickly and easily write C extensions to the VM, callable from occam-pi. After this, we worked out the details of interacting with hardware directly from occam-pi, which (in some cases) completely eliminates the need to call out to C from occam. For example, in '<a href="">/trunk/wrappers/tmotesky/Native</a>', you can find our initial explorations in this area, where we are bringing up the virtual machine with no connections to the hardware, and then configuring the MSP430 directly from occam-pi. We expect this to make implementing functionality for new embedded platforms much easier for the developer in some cases. </p> <p>Although it is unexciting infrastructure, I also began work on migration from Autotools to <a href="">Scons</a> (<a href="">/branches/scons</a>). Currently, it is possible to build the Transterpreter for Linux/Intel, MacOSX/Intel, and the MSP430 in this branch using Scons. The build scripts are, I believe, clearer, easier to extend, and will (I believe) give us a better cross-platform build experience (eg. Windows). Currently, we have to maintain a completely separate build system for Windows, which is untenable in the long run; with Scons, we have a chance of bringing things together nicely across all major development platforms. So, while unexciting, I consider this a rather important revision of tools and infrastructure. (Of course, I'm working on it, so you might expect I'd say that.) </p> <p>Along with the under-the-radar updates to the build system, I've nearly finished <a href="">refactoring the linker</a>; what had grown to 5000+ lines of Scheme has been reduced to roughly 1500. While the new slinker is not done yet, it is much simpler and much more modular. The new slinker was informed greatly by our explorations with <strong>42</strong>---the underlying data structures are far more intelligent than they used to be, and allow us to do complex things very simply. It is, in a nutshell, a joy to work with (as source code), which I can no longer say about the original slinker.</p> <p>Looking forward, I'll be working more with two new hardware development boards for the MSP430, exploring radio communications, analog-to-digital conversion, storage of data to external devices (SD, etc.), and most importantly, the integration of interrupts with the Transterpreter scheduler. The development boards are important, as they allow me to do in-system debugging via JTAG, which will be essential for the implementation of interrupt handling. This will, like many other things we do, impact all Transterpreter ports. </p></li> <li><p>Christian Jacobsen has recently been sighted trying to set up a "build bot" for the Transterpreter project; this will automatically check out our code, build it, and run the test suite on a regular basis. However, he should not be doing this, but should be submitting the final version of his thesis. We expect that to go out the door <strong>ANY DAY NOW</strong>. I read it on my flight over to the USA, and believe-you-me, it was very, very exciting. Very.</p></li> </ul> <p>If this sounds like we're busy, it's because we are. We've got a few other things up in the air related to our ongoing efforts, and will drop word of those as they see the light of day. If you have any questions about the project, or want to get involved, please feel free to drop me a note (matt at this domain). </p> Configuring hardware from the VM /blog/2006/12/configuring-hardware-from-the-vm /blog/2006/12/configuring-hardware-from-the-vm Matthew Jadud Sun, 17 Dec 2006 21:42:55 +0000 <p>In my last post, I talked about how we have some new documentation available for people interested in porting the Transterpreter. This documentation is, primarily, about how to use C and occam-pi together to configure hardware devices for working with the Transterpreter. </p> <p>However, writing programs that involve two languages does not necessarily make life easier. Which is why I'm glad to say that it is possible to talk to hardware directly from occam-pi. In our porting documentation, we demonstrate the use of C and occam-pi to control the LEDs on a Texas Instruments MSP430-based device. What's nice is that we don't need C at all! </p> <p>In occam-pi, we can declare a variable PLACED. Normally when you declare a variable the compiler decides on where, in memory, that variable will go. However, when working with memory-mapped I/O, you might want to place a variable <b>there</b>, and know that whenever you set that variable's value, or read from it, you are reading from a particular location in memory. The PLACED keyword allows you to choose exactly where in memory that you want a particular variable to live. </p> <p>You can do the same thing in C, of course... but you can't guarantee the safety of that memory-mapped location in the context of many threads. occam-pi, on the other hand, guarantees at compile time that your program is free from race hazards. </p> <p>Currently, you can <a href="">see my explorations with this</a> in the trunk of the source tree in the tmotesky wrapper. Things will move, eventually; for now, the <em>Native</em> subdirectory contains my explorations in configuring the Tmote Sky entirely from the interpreted language. Right now, I'm working on the SPI bus and radio configuration. </p> Porting the Transterpreter /blog/2006/10/porting-the-transterpreter /blog/2006/10/porting-the-transterpreter Matthew Jadud Mon, 23 Oct 2006 18:32:45 +0000 Until now, we <span style="font-style: italic">claimed</span> that porting the Transterpreter was easy. Now, we&#39;ve got <a href="">docs</a> that demonstrate just how easy it really is. OK, OK, so anything that involves setting up a cross-compiler, writing a bit of C, and interfacing your hardware from a high-level language through a low-level language isn&#39;t exactly easy. But, my suspicion is that the approach we&#39;ve taken isn&#39;t so ugly, either. Certainly, there&#39;s a lot of rough edges around the Transterpeter (as we learn by writing documentation like this), but overall, I think we&#39;re pretty pleased. Take a look at our <a href="">documentation regarding how to port the Transterpreter</a> to a new platform, and see what you think. If you&#39;re interested in using the Transterpreter to do something cool at the intersection of concurrency and ... well, anything, don&#39;t be afraid to <a href="">join one of our mailing lists</a> and shout, or just <a href="">drop us a line directly</a>. Cores, cores, and more damn cores /blog/2006/09/cores-cores-and-more-damn-cores /blog/2006/09/cores-cores-and-more-damn-cores Matthew Jadud Thu, 28 Sep 2006 10:07:02 +0000 <p>Lets look at two articles that recently hit the fan:</p> <ul> <li><a href="">Terrascale Processing brings 80 cores to your desktop</a></li> <li><a href="">Chip Wars go to the Core</a></li> </ul> <p>The first is a regurgitation of a presentation made by Intel to developers. In short, they're considering a move to processors with 10s, if not 100s, of discrete computational units on a single die. The second article is a cry for Intel to stop, or slow down, using the argument that we "have to get the architecture right first." </p> <p>OK, so the architecture can slow us down. But what slows us down even more is the fact that most programming languages have no support for concurrency, and therefore most programmers think "concurrency" means "threads and spinlocks". The two most interesting explorations we've made in the last few months are the Multiterpreter and the Cell Transterpreter. </p> <h2>The Multiterpreter</h2> <p>If you check out the full Transterpreter project from our <a href="">Subversion repository</a>, you'll see in the branches a <a href="">multiterpreter</a> branch. This exploration was a quick proof-of-concept that Christian and I poked at for a few days. While the threading is terribly inefficient, it demonstrated that we can spawn multiple OS threads on-demand from the Transterpreter runtime, and distribute computation across those threads without any modification in the source program. Put simply, if you write: </p> <code> PAR process1() process2() process3() process4() </code> <p>on a quad-processor machine, then all four processors will be utilized by Transterpreter instances. We haven't publicized this branch because the approach was the least efficient implementation possible, but simplest to implement. Our explorations into native code generation are, in part, an exploration of what parts of the code-base would need to be refactored to allow efficient multithreading on big machines. </p> <h2>The Cell Transterpreter</h2> <p>The other branch that might be of interest is the <a href="">Cell branch</a>. The work being carried out here is described fully in the paper <a href="">A Cell Transterpreter</a>. Damian Dimmich has successfully run 9 Transterpreters on a single Cell Broadband Engine. Although we only have the Cell simulator, he has demonstrated that you can have multiple Transterpreters running, in parallel, all over the device, and preserve CSP channel semantics across the cores. Put another way, his proof-of-concept demonstrates that we can write occam-pi programs that work seamlessly across multiple, heterogeneous cores. </p> <h2>Bring on the cores</h2> <p>As we look to unify various compiler explorations within the group, we expect to target these kinds of platforms more directly. If your language makes it easy to express ideas in parallel, it should be easy for the compiler to automatically distribute work units to many different processors. In the case of occam-pi, this is absolutely the case... so if we have a processor with 8, 80, or 800 cores, we should be able to take advantage of that power without significant effort. </p> <p>Certainly, we can take advantage of this kind of hardware far more easily than a programmer writing in a sequential language like C, C++, C#, or Java.</p> Erlang and RoboDeb /blog/2006/08/erlang-and-robodeb /blog/2006/08/erlang-and-robodeb Matthew Jadud Wed, 30 Aug 2006 09:37:11 +0000 <p>I would love to see <a href="">Erlang</a> bindings to the <a href="">Player/Stage API</a>. If they existed, we would include them in a heartbeat on the <a href="">RoboDeb robotics virtual appliance</a>. </p> <p>I don't think any Erlangistas hang out here, but you never know. If you're savvy with Erlang, and are interested in collaborating on binding Erlang to the Player/Stage API, I'll put it in an environment where you can double-click and play with robots. </p> Hardware bringup on the NXT /blog/2006/08/hardware-bringup-on-the-nxt /blog/2006/08/hardware-bringup-on-the-nxt Matthew Jadud Tue, 08 Aug 2006 21:47:10 +0000 <p>The porting of the Transterpreter is straight-foward. You check the sources out, configure your cross-compiler, and go. Some twiddling with 'make' is usually the biggest problem we run into. </p> <p>Porting to the NXT will be slightly less straight-forward.</p> <p align="center"> <img src="/images/blog/2006/NXT-block-diagram.png" height="235" width="278" border="0" hspace="4" vspace="4" alt="NXT-block-diagram" title="NXT-block-diagram" /> </p> <p>The core Transterpreter will cross-compile to the ARM7 without any problems. However, there's a little problem of bringing up the hardware on the NXT that must be addressed. From my experience working on the Tmote Sky, the OS developer needs to check and set a significant number of pins to bring up each piece of hardware. These bit-twiddling operations are tricky; for this reason, it is likely that we'll want to get a JTAG reader for the Mac and be able to inspect the hardware while developing. </p> <p>The Atmel AVR is likely to be the first device we bring up; it gives us access to the sensors, motors, and buttons on the device. Once we have rudimentary access to the NXT's sensors and motors, the simplest way to get feedback from our programs will be to implement sound. In increasing order of difficulty, I suspect the display, the USB port, and Bluetooth radio will follow. Once we get the low-level bindings in place, however, we should be able to write the majority of the NXT port in <a href="">occam-pi</a> or <a href="">42</a>, which will make development a good deal more pleasant. </p> Anonymous SVN available /blog/2006/08/anonymous-svn-available /blog/2006/08/anonymous-svn-available Matthew Jadud Tue, 08 Aug 2006 20:11:27 +0000 <p>We're not prepared to call things "version 1.0" yet, but we are prepared to open the source. A read-only, anonymous Subversion repository is now available: </p> <p align="center"> svn co </p> <p>If you're just interested in using the Transterpreter on <a href="">Mac OSX (PPC)</a>, <a href="">Windows</a>, or <a href="">Debian GNU/Linux (Intel)</a>, you can just download a pre-built binary. However, if you want to use the Transterpreter on Mac OS X (Intel), or some other platform for which we do not have binaries, you will need to check out and build from source. </p> <p>If you are really crazy (or are really curious, or know what you're doing), you're welcome to check <em>everything</em> out, including all our exploratory branches. Simply get rid of 'trunk' on that first URL. I'd like to point out that this is not, in the general case, necessary.</p> Lastly, if you'd just like to browse the source, you can take a look at <p align="center"> <a href=""></a> </p> If you have any questions, I'd encourage you to <a href="">join the tvm-discuss mailing list</a>, or <a href="">drop me a note</a>. <strong>Update 20060809:</strong> I've set up a new <a href="">download page</a> that provides more detail regarding the Subversion download. LEGO NXT developer kits /blog/2006/08/lego-nxt-developer-kits /blog/2006/08/lego-nxt-developer-kits Matthew Jadud Tue, 01 Aug 2006 21:52:38 +0000 <p>The <a href="">LEGO Mindstorms NXT developer kits</a> (software, bytecode/VM reference, hardware, and Bluetooth) have been released. </p> <p>This means the clock is officially ticking. Porting can begin.</p> About /blog/2006/06/about /blog/2006/06/about Matthew Jadud Fri, 23 Jun 2006 22:39:33 +0000 The Transterpreter weblog is where we try and make mention of the things we're working on in the course of the Transterpreter project. Sometimes, we wax lyrical, and dream of directions we'd like to go. Regardless of whether we're reporting or fantasizing, we're likely to write about virtual machines, programming languages, and robotics. Sometimes, but not very often, we might write about monkeys. But only a little bit. If you're interested in learning more about the Transterpreter project, you might check out our <a href="">Wiki</a>. Or, if you're particularly keen on robotics, you should see <a href="">RoboDeb</a>, our VMWare virtual machine for exploring concurrency and robotics! Hubris in the form of 42 /blog/2006/06/hubris-in-the-form-of-42 /blog/2006/06/hubris-in-the-form-of-42 Matthew Jadud Fri, 23 Jun 2006 19:48:53 +0000 <p>Who are we to implement a programming language?! Truly, there is barely a more offensive act of hubris in the world of computing. </p> <p>42 is intended to be a platform for exploring parallel code generation and distribution. Furthermore, it is an experiment in compiler implementation; we are attempting to implement a safe, yet extensible framework for extending and maintaining our language. So far, we are incredibly pleased with our efforts, but time will tell. </p> <p>More information will make its way here in due time; for now, you may be interested in our project pages in the Transterpreter wiki: </p> <a href=""></a> <p>As we're likely to take Monday off, it might be interesting to see the size of the project; the compiler remains delightfully small at 3123 lines of Scheme! What is also fun about <code>sloccount</code> is that it gives you estimated development times and costs. We began our work on July 6th, 2006, and sit at July 23rd, 2006. So, by some unknown metric, there has been over one year's worth of work carried out, worth roughly $150,000.</p> <p>If only.</p> <pre>SLOC Directory SLOC-by-Language (Sorted) 3123 top_dir lisp=3123 1289 tests soccam=1045,lisp=244 473 macros lisp=473 179 docu lisp=179 90 drscheme lisp=90 8 scripts lisp=8 Totals grouped by language (dominant language first): lisp: 4117 (79.76%) soccam: 1045 (20.24%) Total Source Lines of Code (SLOC) = 5,162 Dev Effort Estimate, Years = 1.12 (Basic COCOMO, Months = 2.4 * (KSLOC**1.05)) Schedule Estimate, Years = 0.56 (Basic COCOMO, Months = 2.5 * (mths**0.38)) Est. Num. of Developers = 2.00 (Effort/Schedule) Total Estimated Cost to Develop = $ 151,391 (average salary = $56,286/year, overhead = 2.40). SLOCCount, Copyright (C) 2001-2004 David A. Wheeler SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL. SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to redistribute it under certain conditions as specified by the GNU GPL license;see the documentation for details. Please credit this data as "generated using David A. Wheeler's 'SLOCCount'." </pre> Websites, source code, and compilers /blog/2006/06/websites-source-code-and-compilers /blog/2006/06/websites-source-code-and-compilers Matthew Jadud Thu, 22 Jun 2006 09:01:32 +0000 <p>I need to be better about posting regular updates. If we're going to have a blog, I should use it. </p> <h3>Website and source code</h3> <p>First, we've moved to MediaWiki for our main website. This was a large change, and hopefully it means more documentation will find its way more readily to the WWW. If nothing else, it means Damian will be able to edit pages related to projects he works on, since the old system was very, very scary, and very, very ugly. (I thought the old system would be a good idea! My bad!) </p> <p>Second, source will soon be available via anonymous SVN. We have one or two more things to put into place, but will be making source code available shortly. (If you can't wait, send me an email at matt at transterpreter dot org.) Tied into this is our switch from Mantis to Trac for project management. You can view </p> <a href=""></a> <p>which lets you view the source to the Transterpreter, skroc, and slinker, as well as submit bugs and whatnot. We'll make a Subversion checkout URL available shortly. </p> <h3>42, a compiler for ...</h3> <p>Third, we've begun work on a new compiler for an oocam-like language. The existing compiler is too large and monolithic to consider extending, and the up-and-coming <a href="">NOCC</a> is already 50,000 lines of C. I've already <a href="">said my peace on the foolishness of writing a compiler in C</a>, so I'll move along swiftly.</p> <p>Our goal is for our compiler to be extensible, first and foremost. To do interesting research in areas surrounding concurrent programming languages, we're going to need a platform upon which to experiment and test ideas. Furthermore, I want it to be a platform that undergraduates who are interested in doing research can contribute to in meaningful ways. So, we've started from scratch with a subset of occam; we're calling the compiler <em>42</em>. </p> <p>I'll probably show similar pictures again in the future, but I wanted to show a few syntax errors that the compiler generates so far. You see, I'm a <a href="">bit picky about syntax errors</a>, and if we do our job right, this compiler will have better syntax errors than any other compiler for any other language out there. </p> <img src="/images/blog/2006/42-instance-undefined-1.png" height="398" width="398" border="0" hspace="8" vspace="8" alt="42-instance-undefined" /> <br /> <em>An undefined process instance</em> <p>If you're trying to call a pre-defined process, the question is: is it defined? This is much like trying to invoke a method in Java that hasn't been defined, or calling a function in Haskell that you hadn't yet actually written the code for. Currently, we just say <code>'process-name' not defined</code>; in my book, this isn't enough. And since the compiler is written in a micro-pass style (very much like the <a href="">Indiana nanopass style (PDF)</a>), I can improve this error while writing this post. </p> <p>One file is responsible for this check, and it is called <code>check-instances-exist.scm</code>. I can open it up and improve the message very simply. I like the improved error a great deal more. </p> <img src="/images/blog/2006/42-instance-undefined-new.png" height="398" width="398" border="0" hspace="8" vspace="8" alt="42-instance-undefined-new" /><br /> <em>An improved, undefined process instance message</em> <p>That took one minute. Perhaps less. Now, instead of just saying <code>'bar' is undefined</code>, the message says </p> <pre>instance:not-defined: 'bar' is not defined. For it to be defined you must have something that looks like (proc bar (...) ...) in your code, where the '...'s might be filled in with code you write yourself. in: (seq (:= x 4) (foo x) (bar)) </pre> <p>Along with this message, the relevant code is highlighted, so the user can see exactly where their syntax error took place. (To do this, I'm leveraging the excellent and extensible <a href="">DrScheme programming environment</a>, which was designed to allow end-users the ability to add new programming languages to the environment.) </p> <p>Adding new syntax errors to the compiler takes only slightly more time than modifying an existing error message, but in general we have broken every check or test out into its own, separate unit of functionality---a separate compiler pass. This makes maintaining and extending the compiler a manageable, and even enjoyable, task. Finding code that fails unit tests also becomes a snap. </p> <p>Another error a programmer might make is to define a process more than once; again, this is like having two methods in Java with the same name and same arguments---and this isn't allowed. Likewise, in our little language, you can only define a named process once. </p> <img src="/images/blog/2006/42-duplicate-definition-names.png" height="398" width="398" border="0" hspace="4" vspace="4" alt="42-duplicate-definition-names" /><br /> <em>Duplicate definitions</em> <p>Here, the first instance is highlighted (I can't easily highlight all of them, unfortunately---but we'll work on that), and the error message reports all the lines in the code where the process is defined. In this example, we have three definitions of <code>foo</code> in the source code. The compiler catches all three, tells me where they are, and highlights the first offending instance. </p> <p>My personal goal is for every error message in the compiler to be this rich and informative. I'm considering instrumenting the compiler to catch errors it has never seen before, and report those errors directly back to us (with the user's permission, of course). This way, as new syntax errors with poor messages are found, we can improve the compiler using real user data. This would provide valuable research data and a paper or two along the way. </p> <p>As we now have our source code viewable, we'll soon be making it publicly available. Please drop a note if you're interested in working with or contributing to any of the tools we're developing. </p> RoboDeb Away! /blog/2006/05/robodeb-away /blog/2006/05/robodeb-away Matthew Jadud Sat, 27 May 2006 18:08:46 +0000 <p>For those of you looking for an excellent way to explore programming in a concurrent programming language, and are ready to lead your own evil robotic warlord uprising, you should check out <a href="">RoboDeb</a>.</p> <p>Granted, I need to write more documentation, but there are plenty of books and tutorials available for the occam programming language, as it isn't new... it's been around for quite some time.</p> <p>And, in case you were worried, we submitted the most recent version of RoboDeb (an auspicious version <strong>007</strong>) to the <a href="">VMWare $100,000 uber-appliance challenge</a>. Our most recent release does little more than include the new Sun Java Distribution License; as far as we know, we're 100% legal on this release. </p> From v209 to v301.14 /blog/2006/05/from-v209-to-v30114 /blog/2006/05/from-v209-to-v30114 Matthew Jadud Fri, 19 May 2006 22:23:40 +0000 <p>If only everything was easy...</p> <p>To do a source release, we need the slinker. To do a source release that will work out-of-the-box on the MacIntel, we need the slinker running on the v300-series of <a href="">MzScheme</a>.</p> <p>Yesterday, I finished what looks like a successful upgrade of the slinker to the most recent (301.14) version of MzScheme. This is nice, as MzScheme/MzC now produces JIT-compiled code, and soon may drop its dependencies on external libraries. In other words, we may soon be able to distribute binaries on a number of platforms without requiring the user to install support libraries, which has always been an extra/annoying step.</p> <p>In other words, because of the continued excellence of the work done on <a href="">PLT Scheme</a>, we benefit. I return to the UK on what looks like a long flight spanning Monday and Tuesday; while we currently don't have 'make dist' working, we may be able to do a source release very soon. And I think that would make me happy.</p> <p>Or, at least, take something off the to-do list.</p> <p>Perhaps the opportunity will present itself during the weekend to do some additional updating of the TVM Blog; a bunch of things continue to fall into place, including improved floating point support, and some really nice performance improvements. More to come...</p> What is RoboDeb? /blog/2006/05/what-is-robodeb /blog/2006/05/what-is-robodeb Matthew Jadud Fri, 05 May 2006 00:20:55 +0000 <p>I visited <a href="">Oberlin College</a> today---I was in town, and I had some questions that I thought they might be able to answer. I saw presentations from two 4<sup>th</sup> year Honors students, and both <a href="">Assoc. Prof. Bob Geitz and Prof. Richard Salter</a> took time out of their busy days to chat with me. I appreciated it greatly. </p> <p>In talking with them about some of the projects I've worked on, I realized that it was very difficult to show them anything about the Transterpreter and <a href="">RoboDeb</a>. In short, we have very few images or videos on our website, despite the fact that we do some pretty darn cool things involving neat programming languages and robots. While there's a lot of information on these pages, there's nothing that you can browse through and say "Oh. That looks cool."</p> <p>So, that's something for me to work on. It's a different kind of documentation, but it is documentation never-the-less. I have videos from various talks and presentations we've given over the last two years; perhaps those can be cleaned up and integrated into the site here. </p> Porting to the Tmote Sky, round 1 /blog/2006/04/porting-to-the-tmote-sky-round-1 /blog/2006/04/porting-to-the-tmote-sky-round-1 Matthew Jadud Thu, 27 Apr 2006 23:37:40 +0000 <p>Porting to bare hardware is slightly more difficult than I thought it would be.</p> <p>The Transterpreter recompiles to new targets with little or no effort; it's just not a big deal. The VM is written in a subset of ANSI C that ports well from one machine to another. However, running on bare hardware means that you must first initialize the hardware.</p> <p>This has been my biggest battle with the <a href="">Tmote Sky</a>. I'm not particularly well versed at reading <a href="">datasheets</a> (PDF) for hardware boards, nor does everything in the <a href="">MSP430 processor specification</a> (PDF) make sense to me the first (or second, or sometimes the third) time through. I've spent a fair amount of time in the last few days just reading the lowest-level sources for <a href="">TinyOS</a>, which has taught me a great deal about how they initialize the MSP430 (and therefore, the Tmote Sky).</p> <p>For example, it took some time to discover the LED initialization sequence:</p> <pre> #define INIT_LEDS() P5DIR = 0x70 ; P5OUT = 0x70; </pre> <p>And, I'm not even sure about my setting of the <code>P5OUT</code> register. Certainly, those register settings ignore everything else on port 5.</p> <p>The serial communications took longer; I have to admit, there's still some things going on that I don't fully understand, as my initial explorations in this area were almost the same. Granted, one bit might make a difference in this case:</p> <pre> #define INIT_SERIAL() \ P3SEL |= 0xC0; \ U1CTL = SWRST; U1CTL |= CHAR; U1CTL = SWRST; \ U1CTL |= CHAR; U1TCTL &#38;= ~(SSEL_0 | SSEL_1 | SSEL_2 | SSEL_3); \ U1TCTL |= SSEL_ACLK; U1BR0 = 0x03; U1BR1 = 0x00; \ U1MCTL = 0x4A; ME2 &#38;= ~USPIE1; \ ME2 |= (UTXE1 | URXE1); U1CTL &amp;= ~SWRST; </pre> <p>That code is heavily based on some of the hardware initialization code in TinyOS. </p> <p>This is an important start, and not bad for three days of work. It's not as quick as I thought it would be... but it isn't terribly slow, either. Once I have the hardware initialization sequence done (the biggest pieces left are the flash and the radio), "porting" the Transterpreter is a piece of cake. That is, binding external channels to these interfaces isn't the hard part---it's just getting the mote to come to life with everything ready to go that is the trick!</p> <p>(Of course, learning how to turn these devices on and off is going to be another challenge...)</p> RoboDeb006 Released /blog/2006/04/robodeb006-released /blog/2006/04/robodeb006-released Matthew Jadud Sun, 09 Apr 2006 09:20:31 +0000 <p> Our virtual machine for exploring occam (running on the Transterpreter) as a massively concurrent control language for robotics platforms is now available for download. </p> <p> See the <a href="">RoboDeb Quickstart Guide</a> for more information. If you download it and start playing with it, please drop me a note (<strong>matt at transterpreter dot org</strong>) and ask to be put on the mailing list. We want to hear from users regarding their explorations and experiences! </p> RoboDeb... any moment now /blog/2006/03/robodeb-any-moment-now /blog/2006/03/robodeb-any-moment-now Matthew Jadud Wed, 29 Mar 2006 22:59:11 +0000 <p>In preparation for our talk and workshop next week at <a href="">DIKU</a>, we're prepping our release of RoboDeb. Before the end of the week, the link:</p> <p><a href=""></a></p> <p>will work, and you'll be able to download a VM full of occamy, robotic goodness. I can't wait!</p> <p>(Why can't I wait? Because it takes a lot of effort to release something like this. And I want it to be over. Oh... sleep! Sweet sleep!)</p> 0.7.3, RoboDeb /blog/2006/03/073-robodeb /blog/2006/03/073-robodeb Matthew Jadud Thu, 23 Mar 2006 22:49:40 +0000 <p>We have a new binary build of the Transterpreter up; version 0.7.3 rolls in some minor fixes, and includes support for <code>SHARED</code> channels. However, it does not support <code>SHARED</code> top-level channels; this is an issue with how the data is passed through the compiler (and therefore, ultimately, to the slinker), and those issues may (or may not) be resolved someday. For now, you may not (directly) share top-level channels. </p> <p>We also have a VMWare virtual machine available for download; if you are interested in using the Transterpreter and the <a href="">Player/Stage</a> library for controlling Pioneer class robots (a tastes-great combination, giving you world-class concurrency for robotic control), drop us a note. We've got around 60 pages of documentation to accompany it, and deployed it this semester to students enrolled in CO631, a second-year module in concurrency at the University of Kent. We'll be giving a talk and mini-workshop at <a href="">DIKU</a> in April regarding this same technology. </p> <p>Like all things, we'll do a complete release in due time; we're adding more things to the VM (like a JDK, JCSP, the FUSE libraries for userland filesystems, the Python Player/Stage bindings, so on and so forth) so we're not keen on doing a general release at this time. It's a 250MB download, and until it's finished, we'll be happy to release it to people upon request.</p> Coming reorganization /blog/2006/02/coming-reorganization /blog/2006/02/coming-reorganization Matthew Jadud Sun, 19 Feb 2006 22:57:47 +0000 <p><em>Software projects aren't</em>. The Transterpreter is a small and portable runtime for the occam programming language. However, there is more to the Transterpreter than just the TVM (the core interpreter). There are wrappers for each platform we target, the SWIG FFI generator, the toolchain (currently the slinker, and soon 42, our new compiler), as well as documentation, educational material (for use in the classroom), and applications. In our case, we have two primary areas of application---wireless sensor networks and robotics, both embedded contexts. </p> <p>However, there's a hidden aspect to the project that we have had our eyes on for some time, but really never kept up on---the WWW. The site was largely static when we first built it, and I believe that is inappropriate for reporting on the work we're doing as a team. The goal is to have an easy way for us to publish updates and information pertaining to our work in a flexible form that can easily be subscribed to in several different ways. </p> <img src="/images/blog/2006/structure.jpg" height="307" width="400" border="0" hspace="4" vspace="4" alt="Structure" /> <p>Currently, our site is a collection of static pages, and one weblog where we post updates. I recently promoted that to the top-level of the site, as it was the most interesting/dynamic part of the site. However, I think I can take this much further. In the coming site reorganization, I'll treat every aspect of the project as a flowing source of information and updates, easily aggregated and redistributed. Practically, this means that the website will become a collection of weblogs; it will be easy to add new weblogs, and they'll all be aggregated through the main part of the site. </p> <p>It is possible that we should just announce and discuss more about the Transterpreter on the mailing lists---in fact, I might encourage the group to begin doing so. However, people go to Google to find information when they're stuck, and therefore exposing as much information as possible to search engines in a structured format that can be repackaged and repurposed (a DB-backed weblog exporting an RSS feed) seems to make a great deal of sense. </p> 42 /blog/2006/02/42 /blog/2006/02/42 Matthew Jadud Tue, 14 Feb 2006 10:03:21 +0000 <p>We're pleased to announce the beginnings of <strong>42</strong>, a new compiler for occam, and ultimately, CSP-based languages targeting a wide variety of hardwares and platforms. </p> <p>This work will be guided by a number of principles, and will ultimately have its own space here on the <a href=""></a> website. Our ultimate goal is to support novel and heterogeneous compilation targets---things like the Cell and modern FPGAs that have a mix of traditional cores (PPC) and reconfigurable logic. Our initial goal is to provide an occam2.1 compiler that supports arbitrarily sized targets (8-bit through 2<sup>n</sup>-bit) machines, both big- and little-endian. Our runtime target will initially be the Transterpeter. </p> <p>The principles that will guide this work are simple. It is a <strong>learning exercise</strong> for those of us involved in the project; furthermore, it will provide an opportunity for students who follow after us to explore compilers and perhaps extend it themselves. Hence, simplicity of expression will be of primary concern for all code that we write. Second, the compiler must be <strong>reliable</strong>; it will not crash, and it will not generate bad code. Third, the project will be <strong>well-documented</strong>, both from an end-user's perspective and a developer's perspective. A project is no better than its documentation---something we're learning about the Transterpreter. Also, it is unlikely to be useful as a learning resource if it becomes a dead and undocumented pile of code. The compiler will be <strong>thoroughly tested</strong>, both from one pass to the next as well as front-to-back. Good tests contribute to reliability, and serve as a kind of documentation themselves. We are hoping that the compiler will be <strong>maintainable and extensible</strong>, but this is always a challenge---we think it is possible, however. </p> <p>And lastly, this will be an <strong>open and team-driven endeavor</strong>. As the language evolves from our starting point (or, as we look at the next language project we want to work on based on what we learn from this one), that process must be open and well-documented. The Transterpreter team currently communicates well, and must continue to do so if we're to make a success of our efforts. </p> <p>If you're interested in contributing to our ongoing efforts, feel free to drop me a note (matt at transterpreter plbbt org). It should be fun.</p> Join the team? /blog/2006/02/join-the-team /blog/2006/02/join-the-team Matthew Jadud Wed, 08 Feb 2006 11:57:28 +0000 <p>The Transterpreter is an small virtual machine for executing occam programs on a diverse set of computing platforms. Currently, the VM can run on Mac OS X, Windows, Linux, Solaris, FreeBSD... the list of desktop platforms goes on-and-on. We're known to run on the Nokia Series 60 platform, as well as the Sharp Zaurus (ARM), and the Texas Instruments MSP430 series, a power-savvy, 16-bit chip often found in a variety of embedded systems. </p> <p>However, our project is more than just the virtual machine. Our goal is to develop a viable platform for researching languages for concurrent and parallel control in embeded, desktop, and cluster applications; no other concurrent programming language currently spans that range. We see a number of tools as being important in this overall goal: </p> <ol> <li> <strong>The virtual machine</strong>. <p>Our current VM was designed to be small, portable, and simple to understand. It's a great place for students to start exploring virtual machines and hardware. For example, we're really hoping to end up in the <a href="">LEGO Mindstorms NXT</a> developer program, in which case we'll be porting the Transterpreter to the NXT to run native. That should be very cool. (There's a half-dozen other places we'd like to be as well... the Palm, Windows CE, Hawaii... to name just a few.)</p></li> <li> <strong>The compiler</strong>.<p> We're developing a new compiler, which we generally refer to as <strong>42</strong>. This is largely a learning exercise for the team; we're ultimately interested in novel, heterogeneous compilation targets, but are starting simple: occam2.1, targeting the Transterpreter. This compiler will be reliable, well documented, well tested, maintainable, open (in its design and for future linguistic exploration), and hopefully extensible. It is very important to us that this work takes place as a team, and that we learn about the language and process of writing compilers through our efforts.</p></li> <li> <strong>The editor</strong>.<p> Currently, we make the Transterpreter generally available as a plug-in to JEdit, an open-source editor written in Java. This is great, but occam is a highly visual language---process networks should, we think, be diagrams. While we have not yet started a visual process editor for occam, it would be an invaluable contribution to our overall project goals.</p></li> <li> <strong>Applications</strong>. <p>We are about to deploy our first major application, which is a complete Linux build for <a href="">VMWare</a> that includes both the <a href="">Player/Stage</a> simulator as well as libraries to connect occam programs to both virtual and real <a href="">Pioneer3</a> robots. </p> <p>However, the Transterpreter will be playing an increasing role in wireless sensor network research here at Kent, and the opportunity to explore this exciting and emerging field of research is really quite cool. Likewise, we have some good ideas for using occam for controlling parallel computing cluters, as well as more widely distributed frameworks like that used in <a href="">SETI@home</a>. Unfortunately, it takes time to develop and maintain the VM and compiler as well as develop applications (which, really, is the fun bit). So if you've got a mind for concurrency, or think you'd like to develop one, there's lots of things you can do to contribute without working on the core.</p></li> <li> <strong>Documentation</strong>. <p>Actually, everyone needs to do more of this. However, it is probably possible for the interested student (or anyone, really) to join us as a technical writer. As a third-year project, I suspect one could do solo or small-team work writing documentation and examples. This is not, however, an <em>easy</em> route. It's actually quite challenging---you have to understand what you're writing about, and be able to communicate those things simply and clearly, often with examples in code. While it is often considered unglamorous, I often consider a project no better than its documentation. In this regard, the Transterpreter is currently awful, and projects like the <a href="">DrScheme</a>/<a href="">MzScheme</a> project is amazing.</p> </li> </ol> <p>You don't have to be at Kent to join in the fun, but it certainly is easier at the moment. Drop us a note if you want to learn more.</p> Robotics Simulators /blog/2006/02/robotics-simulators /blog/2006/02/robotics-simulators Matthew Jadud Sun, 05 Feb 2006 11:25:11 +0000 <p>In working with the Player/Stage simulation environment, we're finding that the use of TCP for data transmission from robot (server) to user's code (client) was a problematic choice. In trying to develop occam programs that make use of the libplayerc library, it becomes very clear that a fundamentally concurrent programming language, capable of running tens of thousands of threads---casually---on modest hardware allows for much more natural control paradigms than a <em>polling loop</em>. Furthermore, the use of TCP means that the network itself becomes a buffer (and bottleneck) for data, whereas I'd rather be <em>guaranteed</em> the most recent data from the robot anytime I ask for it. If I could figure out how to ask the robot to give me data... </p> <p>As a result, I'm interested in other simulation environments---but only in a casual way. The problem is, we actually have a Pioneer3 robot, which means having a simulator for that particular robot is a Good Thing. </p> <p>However, I wanted to be able to find these links again, so here they are: </p> <ul> <li> <a href="">RobotBattle</a>, a robot programming game that apparently/probably uses a high-level "scripting language."</li> <li> <a href="">C++ Robots</a>, which lets you program a bot in C++ and pit it against others in an arena... somewhere.</li> <li> <a href="">Autonomous robotic soccer</a>, which has a Windows/DOS simulator, but no source.</li> <li> <a href="">Simbad</a>, a 3D robotics simulator written in Java. Open source.</li> <li> <a href="">Bugworks</a>, a little robot simulator that looks completely unlike what we want.</li> <li> eyeWyre makes <a href="">Simulation Studio</a>, which seems to be BASICStamp focused.</li> <li> <a href="">Webots</a> looks great, but costs <em>an arm and a leg</em>. I cannot afford this, without a substantial grant. And even then, our students wouldn't be able to go home and use it themselves.</li> <li> <a href="">MOBOTSIM</a> is Windows only, and simulates "differential drive" robots. In other words, no physics, no particular robot.</li> <li> <a href="">MOBS</a> is another distributed client/server architecture that is probably written to suit C hackers, not people programming in languages like occam and Erlang.</li> </ul> <p>Yes, Player/Stage is "best of breed" in many ways, but it doesn't seem to be trivial to support several thousand robots in the simulator. This is frustrating, especially when you have a language that makes it easy to control this many robots...</p> <p><strong>Update 20060208</strong>: I received some feedback from the P/S mailing list, and it looks like some of the problems we're having might be our own. That's reassuring. And, I found the <a href="">RoboCup Soccer Simulator</a>, which looks like it could make a great platform for a third-year project. There's a real opportunity for a student or team of students to do some interesting work researching architectures and approaches to robotic team control, and implementing a football team.</p> TVM 0.7.1 /blog/2006/02/tvm-071 /blog/2006/02/tvm-071 Matthew Jadud Sat, 04 Feb 2006 11:57:48 +0000 As seen on the <strong>tvm-announce</strong> mailing list: <blockquote> <p>We've been quiet, but we've been busy. </p> <p>First, I'd like to point out that the current binary release for Mac and Windows---version 0.7.1---can be considered our v1.0 "release candidate." Many people have poked and prodded this version, and we've deployed it to students for use in the concurrency 2nd-year module here at Kent. It looks good. Most notably to end-users, we've integrated a pure-Java VT220 emulator, so the JEdit IDE is now a one-stop solution for writing and executing occam programs. Christian has put a lot of time in over the last few months making things "just work." Excellent stuff. </p> <p>Second, I'll hint at two impending announcements (days... days away, my friends). Damian has dome some great work with the foreign function interface; this means, for example, that the same occam program can make use of OpenGL on both Windows and Linux (Mac coming soon). So, we've done something nifty involving OpenGL and the Player/Stage robotics API---on that count, I'll say no more. You'll have to wonder just how infinitely cool it is. </p> <p>Last, we are nigh upon our first source release. Most importantly, this means that you can start compiling the world's smallest and most portable occam runtime for all those devices you wished ran occam: your Nintendo DS (two processors, two Transterpreters!), your mobile phone, the toaster, the kids next door... </p> </blockquote> Of course, I shouldn't have sent, but instead talked to the team and bumped the version number to "-rc1" or something before sending. Oops. Sometimes, I'm just over-eager. Website tweaks, FFI, other fun /blog/2006/01/website-tweaks-ffi-other-fun /blog/2006/01/website-tweaks-ffi-other-fun Matthew Jadud Mon, 30 Jan 2006 00:00:11 +0000 <p>Having a static main page to the site seemed rather pointless; so, we've done away with it. From now on, the Transterpreter weblog will be the first point of entry into all things Transterpreterish. </p> <p>Its time to step up information dissemination. Christian and Damian have sorted many of the FFI issues we've been having on both Linux and Windows. For example, Damian has SWIG generating wrappers for SDL on both Linux and Windows, and the same occam program can be executed against the FFI on both platforms. Due to some endianness excitement, big-endian machines are not yet playing nicely. </p> <p>Meanwhile, I've been working on setting up a virtual machine usable in <a href="">VMWare Player</a>; this VM will include all of the development tools, as well as a complete Player/Stage environment. We'll be putting this into place this week for use with students in the Concurrency module at Kent, and can likely go about releasing it unto the world at the same time. </p> <p>Overall, we're moving forward on a number of fronts at once, but I think the most important thing is that we're bringing together the loose ends and generating what looks like a really useful and fun tool for developing concurrent systems. If you downloaded the Transterpreter some time ago, you might grab the latest builds available and have another go. Christian and Jon (I need a profile!) did some really nice work on the JEdit integration, and we now have a solid environment for writing and executing occam programs that runs equally well on the Mac, Windows, and Linux. </p> LEGO Mindstorms NXT link roundup /blog/2006/01/lego-mindstorms-nxt-link-roundup /blog/2006/01/lego-mindstorms-nxt-link-roundup Matthew Jadud Fri, 06 Jan 2006 04:33:50 +0000 <p>It's clear; I'll need one, and more importantly, I really hope we can get one at Kent to port the Transterpreter to it early. It looks like there are so many absolutely cool things we can do with it as a substrate.</p> <p>And, since our runtime should work perfectly well on both the original Mindstorms and the NXT, we're in a good position to have a nice bridge between the two. Yummy!</p> <ul> <li><a href="">Robot magazine</a></li> <li><a href=""> blog</a> (a weblog paid for by LEGO, and dedicated to the product)</li> <li><a href="">Gizmodo coverage</a></li> <li><a href="">Engadget coverage</a></li> <li><a href="">Pocket lint</a></li> <li><a href=",69946-0.html">Wired</a></li> <li>Mention on <a href="">Make</a></li> <li><a href="">ZDNet</a></li> <li><a href="">LinuxElectrons</a></li> <li><a href=""></a></li> </ul> (cross-posted from <a href="">HDYC</a>) The LEGO Mindstorms NXT /blog/2006/01/the-lego-mindstorms-nxt /blog/2006/01/the-lego-mindstorms-nxt Matthew Jadud Thu, 05 Jan 2006 21:55:06 +0000 <p> There's no two ways about it: we're excited about the <a href=";contentid=17278&amp;countrycode=2057&amp;yearcode=&amp;archive=false">LEGO Mindstorms NXT</a>. We've wrestled the Transterpreter onto the Mindstorms, and will be using it in the classroom this semester as part of the University of Kent's module on concurrent and parallel programming (how cool is that!). However, the NXT looks like it will offer some really excellent features, including Bluetooth communications, serial connections for sensors (or any other device), and what looks like some nice additions to the motors. </p><p> I would love to be part of their developer program over the next few months. I submit my dissertation shortly, and have dreamed about having time to develop course materials surrounding the LEGO Mindstorms and the Transterpreter. So, LEGO, if you're listening: we've got a team of six people out here working on the Transterpreter, plus many more on Kent's faculty involved in the First LEGO League in the SouthEast of the UK, and we'd love to take part in the developer program! </p> <p>Links:</p> <ul> <li><a href="">Call for Developers</a></li> <li><a href=";contentid=17278&amp;countrycode=2057&amp;yearcode=&amp;archive=false">Press Release</a></li> <li><a href="">CES Announcement</a></li> </ul> It's alive... IT'S ALIVE! /blog/2005/12/its-alive-its-alive /blog/2005/12/its-alive-its-alive Matthew Jadud Thu, 01 Dec 2005 21:45:01 +0000 <p>Christian has recently finished a massive reorganization of the Transterpreter code repository. This is something they never tell you during your undergraduate CS degree: sometimes, the repository needs refactoring, too. When it gets to the point that things are scattered all over, you should probably sit down and rethink how things should be done.</p> <p>I just did a clean checkout, and rebuilt from scratch. It was quick and painless. Now, we get a little more documentation in place, and do a Version 1.0 release. </p> Sensor Networks? /blog/2005/11/sensor-networks /blog/2005/11/sensor-networks Matthew Jadud Wed, 16 Nov 2005 09:33:09 +0000 <p>As we developed the Transterpreter, we slowly began to realize that we had a tiny CSP runtime that could run on all kinds of hardware. It wasn't until we did our first port to "bare" hardware that we realized how truly portable our design was. </p> <p>We may yet see the <a href="">DIAS project</a> employ the Transterpreter in the group's efforts. If nothing else, it will give us an additional focus ("customers" always do), which is good. As we continue reworking the repository, and getting our documentation lined up, having a better sense for uses will allow us to more profitably focus our time. </p> <p>Today, though, I'm going to try and get over the chest cold that has been knocking on the door for the last two weeks...</p> Novel software distribution mechanisms /blog/2005/10/novel-software-distribution-mechanisms /blog/2005/10/novel-software-distribution-mechanisms Matthew Jadud Thu, 13 Oct 2005 14:34:49 +0000 <p>Brainstorm of the moment: </p> <p>To increase the number of Transterpreter installations in the world, all we have to do is write a Windows virus. That virus does two things: </p> <ol> <li>It downloads and installs the Transterpreter.</li> <li>It propagates itself.</li> </ol> <p>I suppose, if we wanted to be swish, we could update any previous or existing installations... </p> <p>However, we're kinda busy at the moment, and I suspect that writing a virus to install software is generally rude. And probably illegal. And really doesn't seem terribly ethical, or any of a host of other things. It's a funny thought, though. :)</p> Two projects for a new runtime /blog/2005/10/two-projects-for-a-new-runtime /blog/2005/10/two-projects-for-a-new-runtime Matthew Jadud Wed, 12 Oct 2005 09:51:23 +0000 <p>One of the challenging things about developing the Transterpreter has been shifting our thinking from core issues regarding the slinker and VM to questions regarding applications for the software we've spent almost two years working on. </p> <p>Christian has achieved full occam-pi parity (with the exception of some floating point stuff). This means that we can write traditional, static occam2 code, as well as code that involves dynamic data, channel ends, and processes. To be honest, I assumed that the language extensions would take months to implement, not two or three weeks! </p> <p>So, we're going to tackle two projects, partially to become more familiar with the tools we've built, and partially to lay some groundwork for doing larger, more interesting projects later. </p> <ul> <li> We have access to a Pioneer3 robot; it's time to start thinking about architectures for robotic control. As long as we have a language that allows us to easily do concurrent and parallel computing, locally and over the network, we should dig into doing some cool things with the robots we have available.</li> <li> Likewise, while we suspect our performance might not be that of straight C, we think we can write much shorter, expressive, and maintainable code on clusters. So, it's time to break up Adam's "Life" demo, and run it on the cluster. This will stress not only the dynamic aspects of the language, but force us to think about some issues regarding the effective distribution and setup of many Transterpreter virtual machines in a cluster. </li> </ul> <p>While we're at it, it's also time to bring the whole group up-to-speed on the languages we're actively using. The Transterpreter is written in C, the slinker (or <strong>S</strong>cheme <strong>linker</strong>) is written in PLT Scheme, and we're a run-time that supports the occam-pi programming language. It's time to make sure the whole development team is competent enough with Scheme to help maintain the toolchain, and to spend some time working with the new, dynamic features of occam-pi! </p> Stale weblog! Many things happening! Woot! /blog/2005/09/stale-weblog-many-things-happening-woot /blog/2005/09/stale-weblog-many-things-happening-woot Matthew Jadud Fri, 23 Sep 2005 18:06:57 +0000 <p>We could, I think, update this weblog much more often. </p> <p>For example, we have yet to announce that we're up to version 0.6.2. Christian's hard work, along with some help from Damian, have yielded a Transterpreter that now passes thousands of test cases covering the occam2 programming language. Additionally, the JEdit IDE is stable and functional; we've had a number of people successfully download it and get things working all over the world at this point. (The IDE, it seems, was a larger undertaking than had been expected.) </p> <p><a href="">Damian's work with SWIG</a> has been paying off; along with wrapping OpenGL, he's recently wrappped the Player/Stage API. This gives us the ability to write occam programs that control a whole slew of robots, including the <a href="">Pioneer III we have in the lab</a>. This is excellent. I'm hoping to have time soon to learn more about using SWIG, and poke at some things myself. </p> <p>And, Damian and Christian's presentation at <a href="">CPA 2005</a> regarding the FFI (Damian's SWIG paper) went very well; a lot of people were interested in the technology, and we hope that some interesting and exciting collaborations might be born from those conversations. Also, while at CPA, it would appear that Christian was either out of his mind, or otherwise taunted and provoked beyond measure, as he decided to implement dynamic memory and mobiles! This marks the point where we move on from occam2 to occam-pi; these dynamic extensions give the language a great deal of power and flexibility. I suspect these features won't see release until they're better tested, but it continues to be a testament to the design choices we made in the VM and linker that Christian can sit down for a few hours and add new, and significant, language features. </p> <p>Perhaps, as Minister of Misinformation (I'm not allowed to do any significant development work until my own dissertation is done) I can keep on top of this weblog and turn it into a bit more of a consistent source of updates and information. So, if you have any questions, comments, or you have some odd hardware you'd like to see the Transterpreter running on, drop us a note! We're currently considering <a href="">Nelson's monument in Trafalgar Square</a><sup>*</sup> as our next runtime target, so don't be afraid---send us some mail! </p> <p>&nbsp;</p> <p>&nbsp;</p> <p><sup>*</sup> Yes, we are aware that Nelson's monument is not a computational device, <em>per se</em>.</p> Backend technologies /blog/2005/05/backend-technologies /blog/2005/05/backend-technologies Matthew Jadud Thu, 19 May 2005 07:43:52 +0000 Something we're always interested in are portable back-end technologies. So, this is as much a reminder to us as much as something of possible interest to you. Found via <a href="">Chris Double's</a> Weblog: <blockquote> <br /> <a href="">LLVM version 1.5 has been released</a>. <br />LLVM is a '<a href="">Low Level Virtual Machine</a>' that can be used for a programming language implementation. This release adds several new major features to the LLVM compiler, allows it to be used in domains where it couldn't before (e.g. for functional languages), supports new targets, generates faster code, and is mucheasier to port to a new target than before. <br /></blockquote><br /> <br /> Mailing lists! Whoo-hoo! /blog/2005/05/mailing-lists-whoo-hoo /blog/2005/05/mailing-lists-whoo-hoo Matthew Jadud Tue, 10 May 2005 19:00:18 +0000 <p>Ah! I was so busy getting the command-line version bundled up that I forgot that we have two new mailing lists!</p> <p> </p><p><strong>tvm-announce</strong> will be used to announce new releases and exciting advances in the world of the Transterpreter. It should, I think, be low volume.</p> <p><strong>tvm-discuss</strong> is for users of the Transterpreter. I'm not sure how you can get on the list... I have to look that up. For now, drop a note to <strong>matt</strong> or <strong>christian</strong> at <strong></strong>, and we'll hook you up. </p> <p>This ushers in a new era in communication between the authors of the Transterpreter and the eight people using it. <code>:)</code> Enjoy.</p> Command-line versions available /blog/2005/05/command-line-versions-available /blog/2005/05/command-line-versions-available Matthew Jadud Tue, 10 May 2005 18:57:04 +0000 <p>Despite the death of a hard drive (we use version control, and back up our repositories), we've finally had the time to sit down and get a command-line version of the compiler, linker, and virtual machine ready for download. </p> <p><a href=""></a></p> <p>Both Windows and the Mac versions can now be run from the command line, eliminating the ugliness of the JEdit IDE (which I suspect is responsible for many of the difficulties experienced by Transterpreter users). Both Vi and Emacs have occam modes that can be downloaded from here and there on the WWW. </p> <p>A quick walk-through for using the command-line version is online as well: </p> <p><a href=""></a> </p> <p>I will say this: we're assuming you've kicked occam around before, and are comfortable on the command line. If you get stuck, drop a note to <strong>tvm-discuss</strong> at <strong>transterpreter dot org</strong>, and we (or someone else on the list) will try and help get you started. </p> <p>I hope that helps get people started. Our apologies for the rough take-off. </p> The Zaurus and the Brainstem, The Transterpreter and the Zaurus /blog/2005/03/the-zaurus-and-the-brainstem-the-transterpreter-and-the-zaurus /blog/2005/03/the-zaurus-and-the-brainstem-the-transterpreter-and-the-zaurus Matthew Jadud Mon, 21 Mar 2005 23:03:23 +0000 <p>For so little to say, too.</p> <p>This evening, we verified that:</p> <ol> <li> We can slave the Brainstem to the Zaurus from C. I don't have my own 'hello, world' running yet, but I was able to cross-compile and run one of Acroname's test programs. That works for me. The next step is to study Acroname's code, understand how the libraries are used, and wrap them up for use from occam.</li> <li> The TVM has been ported to the Zaurus. Granted, we never thought this would be an issue, but it was good to know that it wasn't.</li> </ol> <p>That, and in unrelated news, pre-compiled libraries of occam code can now contain references to foreign functions. This is useful for the whole enterprise as well, although kinda less obvious and less glamorous. </p> Compiling occam to C /blog/2005/03/compiling-occam-to-c /blog/2005/03/compiling-occam-to-c Matthew Jadud Sat, 05 Mar 2005 17:46:20 +0000 <p>In presenting the Transterpreter at Indiana University Bloomington, we met Geoff Brown, a (new to me) faculty member in the Computer Science department. He gave us a number of ideas for improving the performance of the virtual machine. One which I thought was quite cool was the idea of compiling the bytecode directly to C. Since I don't think it is immediately obvious what I mean by this, I will explain. </p> <p>SPoC was an occam to C compiler. It would transform a piece of occam code into an equivalent C program (links <a href="">here</a>, <a href="">here</a>, and <a href="">here</a>). This was a very hard task, and the compiler is (I believe) no longer actively supported. So, a program like </p> <pre> SEQ x := 3 y := 5 + y), 0, scr) </pre> <p>might be turned into </p> <pre> x = 3; y = 5; printf("%d", (x + y)); </pre> <p>in C. Now, SPoC had to provide compile in a whole scheduler and other stuff to make this work, so it didn't produce code that was <em>quite</em> that simple. </p> <p>We wondered what kind of performance increase we could get by using our existing infrastructure to execute compiled C code instead of interpreting bytecode. You see, right now the Transterpreter looks like this: </p> <img src="/images/blog/2005/tvm-consume-numbers.png" height="174" width="429" border="0" align="middle" hspace="6" vspace="6" alt="Tvm-Consume-Numbers" /> <p>The Transterpreter reads in a big string of numbers (produced by KRoC, the occam compiler), and it "interprets" what those numbers really mean, and executes them. We might also look at it like this: </p> <img src="/images/blog/2005/tvm-consume-instructions.png" height="170" width="418" border="0" align="middle" hspace="6" vspace="6" alt="Tvm-Consume-Instructions" /> <p>The Transterpreter looks at the numbers, and figures out that a certain set of zeros and ones means "add two numbers". Or, "load a constant into memory", and so on. When working on the Transterpreter, we tend to think of things at this level; it makes things a lot simpler. So, it's a stream of assembly language instructions, and we just do what they say to do. </p> <p>Well, our entire machine is written in C. That means, we have a little piece of C code for every assembly language instruction. For example, the ADC (ADd Constant) instruction looks like </p> <pre> void ins_adc() { areg = areg + oreg; /* breg and creg stay the same */ CLEAR(oreg); } </pre> <p>So, whenever we see the assembly language instruction <code>ADC</code>, we simply call the function <code>ins_adc</code> in our interpreter. </p> <p>What we did last night was simply get rid of the instructions. You see, we were going through a process that looked like this: </p> <ol> <li> Read an instruction </li> <li> Look up the C function that represents that instruction </li> <li> Execute the function </li> </ol> <p>Over and over. So, instead of looking up every function, we instead converted our list of instructions into a list of C functions that look like </p> <pre> static int main() { ins_adc(); ins_move(); ins_ldc(); ... }</pre> <p>This way, our execution loop looks like </p> <ol> <li> Run C code.</li> </ol> <p>which is a good deal faster. </p> <p>We then take the C program that we generated (which behaves just like us looking up every instruction in sequence), and ran it. The performance improvement is impressive, and we haven't even begun to optimize. </p> <p>(There are a few small details missing here, but this is, generally speaking, what we did... I'm not going to go into the details of exactly what our C code looks like when we dump it, because it isn't important for this discussion right here, right now.) </p> <p>So, how do we fare? Well, when we interpret everything, we can run through a tight loop around 10,000,000 times (a loop that just does some simple arithmetic... not a very great benchmark) in, say, 300 time units. When we compile our code in C, we run in 60 time units; that is a 5x speedup. We're still a little way off KRoC on the Intel/Linux platform, as it runs the same loop in 1 time unit. Kinda a bummer. </p> <p>So, we get a <strong>5x</strong> speedup by a naive compilation to C. (Back in the day, people would publish papers about their compilers, and how they got a <strong>5%</strong> speedup!) This is good, though, as we can run that C code anywhere the Transterpreter can run, which is pretty much everywhere. And, there's a lot of room for improving the performance of both the compilation (from bytecode to C) and the Transterpreter itself. </p> SIGCSE a success! /blog/2005/02/sigcse-a-success /blog/2005/02/sigcse-a-success Matthew Jadud Mon, 28 Feb 2005 08:38:44 +0000 I think we'll call SIGCSE a success. Our presentation went well, and were glad to get a chance to meet the other speakers in the robotics track. While some were new faces (<a href="">Judy Challinger</a>), we were also able to catch up with old friends as well (like <a href="">Jerry Weinberg</a> at SIUE). While there is yet more to report about the last two weeks of travel around the US, as well as some exciting news that comes as a result of our presentation in the CS department at Indiana University Bloomington, the current "nifty" thing is the <a href="">XPort</a>. <img src="/images/blog/2005/gameboy-bot-1.jpg" height="370" width="340" border="0" align="middle" hspace="6" vspace="6" alt="Gameboy-Bot-1" /> This device rocks our nascent robotics world. It plugs into the Gameboy, and provides a fully programmable FPGA for handling all kinds of sensors, motors, and other stuff. Additionally, it provides you with an easy in for programming the Gameboy itself. There's too much to say about it, and we may have more to say, as I think we need to get one to experiment with. At 30 hours on an extended battery back, a Gameboy and one of these could make a really nice little robotics controller. This is why you go to conferences; you run into people who point out things like this to you (thanks, Jerry!), and you have a chance to rethink some of the ways you do things. Very cool. Portable? You bet. /blog/2005/02/portable-you-bet /blog/2005/02/portable-you-bet Matthew Jadud Wed, 23 Feb 2005 08:50:37 +0000 <p> We gave a talk in the CS department at Indiana University Bloomington on Monday to a good crowd; almost 40,000 in attendance... at the whole University that day. It was a good group, and we had a chance to run our talk we're giving Saturday. </p><p> As a side effect of stopping by, we met a <a href="">new faculty member</a> who was formerly at HP research; he's now developing an interesting research agenda in the department and teaching courses on concurrent programmming languages and also our low-level embedded/assembly language programming course. Therefore, he was interested/critical of our project (in a healthy, academic way, not a vindictive, destructive way). Tuesday afternoon, we got together with <a href="">Bryce</a>, the CS department's hardware magician, and in an hour we ported the Transterpreter to run on <a href="">Goofy Giggles</a>. </p><p> We were given a Goofy to borrow, and we're going to add a slide about this to the SIGCSE talk. So, the claims about the software being portable are true: given an hour, we were able to port the software to a completely new platform, and it didn't involve touching the "core," just the "wrapper." Given that he only has 10K of flash, and 2K of RAM, we only use around 1/4 of the flash and 1/6th of the RAM for a program that prints "TVM &#38; IUB" to the serial port while flashing the LEDs on his front. </p><p> I think that was the most impressive demo we gave at IUB, and will make a nice addition to the talk @ SIGCSE. :) </p> Test Suite, continued /blog/2005/02/test-suite-continued /blog/2005/02/test-suite-continued Matthew Jadud Sat, 05 Feb 2005 18:02:30 +0000 A little more work on Christian's part, and by the end of the day the Transterpreter was passing even more of the CG test suite. While the number of tests that have <b>Crashed</b> seems awfully high, it is important to note that we handle a huge proportion of the "typical" cases, and the crashing cases almost always involve instructions which have not been implemented yet. So, while the number of suites that crash seems high, passing <strong>3920</strong> tests in the CG suite is no small accomplishment. It means the machine is sound in the general case, and we simply need to implement or debug individual (and uncommon) instructions to improve our conformance to the test suite. Completely Passed: <ul> <li>cgtest01.occ</li><li>cgtest02.occ</li><li>cgtest03.occ</li><li>cgtest04.occ</li><li>cgtest05.occ</li><li>cgtest16.occ</li><li>cgtest27.occ</li></ul> Passed: <strong>3920</strong> Failed: 11 <ul><li>cgtest00.occ (TAB2 R10 E6)</li><li>cgtest07.occ (M200)</li> <li> cgtest08.occ (TA5 TA3)</li><li> cgtest57.occ (FNARR24 FNARR23 FNARR22 FNARR13)</li><li> cgtest67.occ (xxx)</li></ul> Crashed 27 (Oh, and we can run the suite now and walk away. That, too, is a good thing. It downloads the entire source of the project, compiles it, and then builds and runs each program in the test suite. I believe that's the way you want it to work.) Test Suite /blog/2005/02/test-suite /blog/2005/02/test-suite Matthew Jadud Thu, 03 Feb 2005 08:45:31 +0000 We did not have a test suite in place for the entire development cycle. However, as of last night, we now have the "cgtest" suite. This is the original compiler test suite for the INMOS occam compiler that has grown over the years along side the Kent Retargetable occam Compiler. With over 1,000 tests in the suite, it is intended to stress every language construct in occam, so as to provide a comprehensive test of the compiler. We're using it more to test our linker and virtual machine, but that's valid as well. At 2:30 AM, Christian dropped a note with the following results (HTML styling applied): cgtest00: <strong>all</strong> (305) - misc bits cgtest01: <strong>all</strong> (326) - more misc cgtest02: <strong>all</strong> (54) - SEQ + PAR cgtest03: Does not slink (widenshort) cgtest04: <strong>all</strong> (120) - Abbreviations cgtest05: <strong>all</strong> (405) - Loop + Replication - takes 21:38 to slink(!) on a G4 1.5Ghz cgtest06: Does not slink (widenshort) cgtest07: Does not slink (widenshort) cgtest08: Passes tests until TALT, when it bombs! cgtest09: Does not slink (widenshort) cgtest10: <em>Uses floating point numbers</em> cgtest11: <em>Uses floating point numbers</em> cgtest12: <em>Uses floating point numbers</em> cgtest13: Does not slink (starttable) cgtest14: <em>Uses floating point numbers</em> cgtest15: <em>Uses floating point numbers</em> cgtest16: <strong>all</strong> (26) - Multi assignment of arrays cgtest17: Does not compile (Special name .VSPTR is not permitted at this pos) cgtest18: Does not slink (realresult) cgtest19: <em>Uses floating point numbers</em> cgtest20: <em>Uses floating point numbers</em> This is very, very significant. Christian set the tests up yesterday, and with some minor additions and bugfixes, he got us to the point where we were passing several large tracts of the tests. That is, the <code>slinker</code> handles the code correctly, and the transterpreter virtual machine executes it <strong>correctly</strong>. These tests (with the exception of TALT) represent 99.999% of anything a student might do in working with a small robot. The floating point tests will, for now, all be emulated with a pure-occam library that can be included at compile time. We will eventually support these natively, but the library should get us past those tests without a problem. And the slinking problems should be solvable relatively easily. The next step is to finish the automation script so these are run automatically, every night, on a freshly checked out source tree. Overall, we're <em>very</em> pleased. Books online /blog/2005/01/books-online /blog/2005/01/books-online Matthew Jadud Fri, 28 Jan 2005 02:01:01 +0000 One of the aspects of bringing the Transterpreter site fully online is to provide a clearinghouse for resources on programming in occam. In particular, we want to bring some out-of-print texts to the digital world, so people can freely obtain high-quality resources for learning to program in this great little language. Our first "rescue" project is <a href="">Programming in occam2</a> by Alan Burns. Professor Burns is excited about our effort; our next step is to get through to the publisher. If you have good contacts within Addison Wesley, let us know; currently, email isn't working. New site, again /blog/2005/01/new-site-again /blog/2005/01/new-site-again Matthew Jadud Wed, 26 Jan 2005 21:37:16 +0000 We're in the final stages of pre-release. The Transterpreter has been used, live in the classroom, since September. We've been giving students in Cool Stuff in Computer Science a chance to build some little robots out of LEGO and program them in occam. While we wish we had infinite time to implement all the little, complex things that would make the Transterpreter better, we're pleased enough with the software to make a release next month. Now, we're going to present the fruits of the last year's worth of labor, and see what people think. Our hope is that we'll get a good reception for the revitalization of a little language for writing concurrent programs both on clusters, the desktop, and in embedded contexts like robots, PDAs, and other devices. That's not bad for one runtime. Granted, the details are what kill you. But we'll get there. We have 31 days until release. :) Concurrency for fun and ... /blog/2005/01/concurrency-for-fun-and /blog/2005/01/concurrency-for-fun-and Matthew Jadud Wed, 19 Jan 2005 18:36:21 +0000 <p> I'll settle for fun. </p><p> For the last year, <a href="">Christian</a> and I have been working on revitalizing an old, little language called occam. Where Scheme is tightly coupled to the lambda calculus, occam is a concise expression of the CSP calculus. Up until recently, occam was only "alive" on Linux/x86. With our work, we've brought occam to Mac OS X, Windows, Linux (all flavors), and the LEGO Mindstorms. I've mentioned it <a href="">a</a><a href="">few</a><a href="">times</a> on my own weblog; perhaps we should start making a concerted effort to go into some more detail. </p><p> What does occam look like? occam is a language of processes; everything, in fact, is a process. These processes are incredibly light-weight, and communicate with each-other via <em>channels</em> (a la Hoare's Communicating Sequential Processes algebra). How about a simple example: producer-consumer, perhaps? </p> <pre><font color="#999999">1</font> PROC producer(CHAN BYTE ch) <font color="#999999">2</font> WHILE TRUE <font color="#999999">3</font> ch ! 'x' <font color="#999999">4</font> : </pre> <p> This process does nothing but emit an 'x' on the channel <code>ch</code>. Because occam channels are not asynchronous by nature, the process blocks on the send down the channel <code>ch</code> (line 3) until someone performs a read. </p><p> A consumer of 'x's might look like </p> <pre><font color="#999999">1</font> PROC consumer(CHAN BYTE ch, CHAN BYTE scr) <font color="#999999">2</font> BYTE tmp: <font color="#999999">3</font> WHILE TRUE <font color="#999999">4</font> SEQ <font color="#999999">5</font> tmp ? ch <font color="#999999">6</font> scr ! tmp <font color="#999999">7</font> : </pre> <p> The consumer process reads a byte from <code>ch</code> into a locally declared variable (line 5) of type <code>BYTE</code>, and outputs the value received to the screen (line 6). </p><p> Of course, these processes are floating in the void. We need one more process that spawns both processes in parallel: </p> <pre><font color="#999999">1</font> PROC main(CHAN BYTE kyb, scr, err) <font color="#999999">2</font> CHAN BYTE c: <font color="#999999">3</font> PAR <font color="#999999">4</font> producer( c ) <font color="#999999">5</font> consumer( c , scr) <font color="#999999">6</font> : </pre> <p> This main process is special in that it binds three top-level, external channels that connect to stdin, stdout, and stderr (these are implementation dependent); commonly, these are the keyboard, screen, and error port (line 1). The two processes, <code>producer</code> and <code>consumer</code> are launched in parallel (lines 3,4, and 5), and then the process <code>main</code> exits. <code>PAR</code>, if it isn't obvious, is a process, too; everything indented under it is executed in parallel. </p><p> <strong>That's it</strong>. Having never seen the language before coming to Kent, I was impressed with how simple and clean it is in execution; it's a shame that it "died", so to speak. We've had good experiences working with occam on the LEGO Mindstorms, and look forward to presenting our work and releasing the software at SIGCSE next month. </p><p> (For the Schemers in the house, you may take a look at a <a href="">previous post</a> where I was playing with MzScheme's ConcurrentML library and a quickly hacked together s-expression-based syntax for occam-y constructs.) </p> <blockquote><h3>Interested in playing with occam?</h3> <p>We've built a few things you might enjoy: <ol><li>We've co-opted JEdit as an IDE. On Windows or the Mac, you can write and execute occam programs in the IDE.</li><li>From the JEdit IDE, you can also execute your programs on the LEGO Mindstorms. Currently, we only have this working on Windows machines.</li></ol> We're finalizing installers and getting ready for SIGCSE 2005; if you'd like to be informed when we release, and be kept up-to-date on updates, drop a note to matt at, and we'll add you to... well, we'll set up a mailing list. </p></blockquote> Blogs re: occam and Mindstorms /blog/2004/10/blogs-re-occam-and-mindstorms /blog/2004/10/blogs-re-occam-and-mindstorms Matthew Jadud Fri, 29 Oct 2004 02:13:38 +0000 We asked the students taking part in the <a href="">Cool Stuff in Computer Science</a> project if they would be willing to blog their experiences using occam on the LEGO Mindstorms. So far, I (Matt, slinker wrangler) have been writing about CSCS as it progresses this term on <a href="">my own weblog</a>. Another Matt (we have around 700 of them in CSCS this year) has started keeping track of things over at <a href=""></a>. His <a href="">first post</a> is a back-fill for the last few weeks. As we see more blogs on this topic, we'll mention them here, and try and get a coherent page of links together (or perhaps a blogroll somewhere on the margins of this page). Thanks, and welcome, <a href="">Matt</a>! Platforms, platforms, platforms... /blog/2004/10/platforms-platforms-platforms /blog/2004/10/platforms-platforms-platforms Matthew Jadud Thu, 28 Oct 2004 07:01:43 +0000 There's an endless number of platforms we could dream of targeting with the transterpreter, but time is limited. Just the same, I wanted to point out that the <a href="">Robosapien has been hacked</a> (via <a href=",guid,798802cc-1084-4d77-8a4c-60b6d0d20a15.aspx"></a>); we could plunk a PDA on top of it like these guys and make it autonomous. Wouldn't that be a fun target to play with? Welcome to the Transterpreter blog! /blog/2004/10/welcome-to-the-transterpreter-blog /blog/2004/10/welcome-to-the-transterpreter-blog Matthew Jadud Tue, 19 Oct 2004 06:23:44 +0000 <p> <strong>Welcome to the Transterpreter website!</strong> <br /> </p><p> While things have been appearing to be idle on the site for a while, lots has been happening behind the scenes. We have developed an integrated development environment (IDE), currently targeting development of Transterpreter applications running in a desktop environment and on the <a href="">Lego Mindstorms</a>. <!--The IDE is primarily aimed at an educational setting, where the syntax highlighting, error highlighting, easy compilation, running and downloading to Mindstorms, is meant to improve the learning experience of students, by taking away the distractions which have otherwise been present when faced with learning occam: learning how to program in a UNIX environment.--> </p><p> For those just showing up, the Transterpreter is a virtual machine for the <a href="">occam-pi</a> programming language (a language designed explicitly for parallel programming, it is based on the Communicating Sequential Processes (CSP) algebra). The Transterpreter is based on the <a href="">INMOS Transputer</a>; currently, it allows classical occam and occam-pi programs to be run on virtually any platform, ranging from Windows, Mac OS X and Linux desktops (on Intel X86, MIPS and PowerPC processors). We're especially interested in running occam programs on small devices like the Lego Mindstorms, PDAs, and mobile phones. The Transterpreter is essentially a portable runtime for the <a href="">KRoC compiler</a>. </p><p> <strong>Current efforts (hammer hammer, bang! bang!)</strong> The IDE is based around <a href="">jEdit</a>, and is provided in the form of a plugin. While the IDE is not currently ready for general (beta) release, it has been used successfully in the CSCS sessions which we run at the <a href="">University of Kent at Canterbury Computer Science</a> department. During these sessions we have students play with occam and the Lego Mindstorms, so far (we believe) with pretty good success. More on the Transterpreter, the Lego Mindstorms and how we use it in "Cool Stuff in Computer Science" should be available on the <a href="">CSCS blog</a>. </p><p> Finally we have begun a crusade make occam and Transputer programming resources available on the Transterpreter website. We are currently <a href="/books/">collecting books</a>, which we are hoping to be able to provide in online versions (with the agreement of the copyright-holders of the works of course). Ideally we would be able to get hold of original manuscripts, from which we can generate online versions of the books, though should the original manuscripts not be available, we are prepared to de-spine and scan any books that we own! </p><p> Anyone who have any books they would like to donate, know of books which we should add to the list of works to be acquired, or otherwise would like to help in this endeavor, we would be more than happy to hear from you at <strong>books @ transterpreter dot org</strong>. </p><p> <strong>Releases</strong> </p><p> The Transterpreter, both in its IDE and a command line version, should be available in the not so distant future. The Windows version of the IDE will be ready first, followed swiftly by the Mac OS X version and then a Unix/Linux version. Following that, pending some cleaning up, the command line version should appear. </p><p></p>