AppHack-e :: ‘Appcopter’ toy helicopter, hacked. Part II: Flying robot

In the previous post of this two-part series, I revealed the process by which the binary control signal for the ‘Appcopter’ toy could be deciphered. This was the first step towards the ultimate goal of commanding the helicopter to perform a sequence of predefined manoeuvres, like the well known educational toy robot ‘Turtle‘.

The remaining task is to write a program capable of generating an audio signal that looks like the image below. Chattering the signal out of a computer’s line out port and into the transmitter will send the toy off doing whatever I tell it to, so long as the parameters are within bounds – “fly, my pretty, fly!”.

Throttle at 10%

Generating the signal

I immediately set out to use Processing (a Java based programming tool) as my programming tool of choice; it’s a free, cross-platform tool that’s pretty easy to get into. Audio support ‘straight out of the box’ isn’t quite adequate for the purposes of easy audio manipulation, so I opted to plug in a third party library called Minim in order to manage my requirement for dynamic audio generation.

Brace yourselves for some code – there’s no way to make this section any easier than simply posting the full code with explanatory comments. Feel free to use this source as you wish (evil robot swarms excluded).

// IMPORTS

// requires Minim audio library: http://code.compartmental.net/tools/minim/
import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.effects.*;
// Java's AudioFormat is required by Minim's createSample() Function
import javax.sound.sampled.AudioFormat ;

// VARS
Minim minim ; // minim instance
int outputSampleRate = 44100 ; //audio sample rate (hz)

// arrays storing prefabricated signal elements - a full signal is built from
// assemblies of these elements in the correct order for a particular message
float[] sig_open ;
float[] sig_0 ;
float[] sig_1 ;

// affects how quickly one program value becomes another - rather than change e.g. throttle abruptly, smoothly ease it from one value to the next
float ease = 0.75 ;

// 'current' control values - these are the values that are actually sent to the transmitter. They are based on the desired value contained in the current instruction, modified by easing
int cThrottle ;
int cPitch ;
int cYaw ;
int cTrim ;
//

// signal is being generated and transmitted during main update
boolean signalOn = false ;

// time at which the program began (used to compare time to next instruction)
int programStartTime ;

// a modifible audio sample to host the signal data
AudioSample signal ;

// In the same way that a washing machine has a program - 'prewash, main wash, spin' - I want to be able to say 'take off, fly forwards a bit, turn left', etc. I use a list of Instruction objects, with a variable pointer to the current instruction. the suffix '_ch1' indicates that this could be made into an array of instruction arrays, thus supporting multiple airbourne agents simultaneously controlled by one computer
int programIndex_ch1 = 0 ; // current program index (1)
Instruction[] program_ch1 = new Instruction[0] ; // program instruction set (1)

// CONSTANTS
int MIN_THROTTLE = 0 ;
int MAX_THROTTLE = 127 ;
int DEFAULT_THROTTLE = 0 ;
int MAX_PITCH = 0 ;
int MIN_PITCH = 63 ;
int DEFAULT_PITCH = 32 ;
int MIN_YAW = 0 ;
int MAX_YAW = 31 ;
int DEFAULT_YAW = 16 ;
int MIN_TRIM = 0 ;
int MAX_TRIM = 63;
int DEFAULT_TRIM = 31 ;
int DEFAULT_CHANNEL = 1 ;
int DEFAULT_SPROG = 2 ; // in the last post I had no idea what this was - it apparently controls the on/off state of the the bright white nose light, but may also be something to do with resetting the helicopter control board between flights!
//
int throttleBits = 7 ;
int pitchBits = 6 ;
int yawBits = 5 ;
int trimBits = 6 ;
int channelBits = 2 ;
int sprogBits = 2 ;
//

// INITIALISATION
void setup()
{
 frameRate(6) ; // feels about right: repeats the full signal every 6th of a second
 size(256, 256); // any old window size - doesn't really matter since there's no UI

 minim = new Minim(this); // make a Minim audio instance

 int i = 0 ; // a counter

 // 0, 1 prefix: each signal value (0 or 1) is preceeded by a fixed length output close to zero. In this block of code I create this data 'building block', in such a way that it can be included in order with other building blocks, and a full signal thus made
 float[] sig_prefix ;
 float prefixSignalDuration_ms = 0.41 ; // the duration of the signal element
 int prefixSignalNumSamples = round( outputSampleRate * (prefixSignalDuration_ms/1000.0) ) ;
 sig_prefix = new float[prefixSignalNumSamples] ;
 for( i = 0 ; i < prefixSignalNumSamples ; i++) sig_prefix[i] = 0.01 ; // set a value very close to zero

 // open: construct the 'on' signal building block, which preceeds the data values in every full signal
 float openSignalDuration_ms = 1.6 ;
 int openSignalNumSamples = round( outputSampleRate * (openSignalDuration_ms/1000.0) ) ;
 sig_open = new float[openSignalNumSamples] ;
 for( i = 0 ; i < openSignalNumSamples ; i++) sig_open[i] = 1.0 ; // the 1 signal data (set to -1 for MacOS?)

 // the '0' value building block
 float zeroSignalDuration_ms = 0.4 ;
 int zeroSignalNumSamples = round( outputSampleRate * (zeroSignalDuration_ms/1000.0) ) ;
 float[] tSig_0 = new float[zeroSignalNumSamples] ;
 for( i = 0 ; i < zeroSignalNumSamples ; i++) tSig_0[i] = 1.0 ; // the 0 signal data (set to -1 for MacOS?)
 sig_0 = concat( sig_prefix, tSig_0 ) ; // prepend the 0, 1 prefix building block from above

 // the '1' value building block
 float oneSignalDuration_ms = 0.8 ;
 int oneSignalNumSamples = round( outputSampleRate * (oneSignalDuration_ms/1000.0) ) ;
 float[] tSig_1 = new float[oneSignalNumSamples] ;
 for( i = 0 ; i < oneSignalNumSamples ; i++) tSig_1[i] = 1.0 ; // the 1 signal data (set to -1 for MacOS?)
sig_1 = concat( sig_prefix, tSig_1 ) ; // prepend the 0, 1 prefix building block from above

// make a container for the full signal data - an assembly of the building blocks above in whatever order we choose. Length is some arbitrary value big enough to contain the entire signal
int nullSignalNumSamples = 2000 ;
float[] nullSig = new float[nullSignalNumSamples] ;
 for( i = 0 ; i < oneSignalNumSamples ; i++) nullSig[i] = 0 ; // set the full signal to zero

// build our audio sample, using our 'nullSignal' data source to drive it
 AudioFormat signalAudioFormat = new AudioFormat( outputSampleRate, 16, 1, true, false ) ;
 signal = minim.createSample( nullSig, signalAudioFormat ) ;

 // That's the building blocks ready: now use them to set up our custom helicopter control program!
 initProgram() ;
}

void initProgram()
{
 // as the battery drains, the appcopter loses power - it needs proportionally more throttle to climb to the same altitude, etc. tScale value allows uniform tweaking of the full program to keep the helicopter behaving consistently throughout multiple runs, across the entire range of the battery's charge
 float tScale = 0.5 ;

 float t = 0 ; // time offset

  int myTrim = DEFAULT_TRIM - int(11 * tScale) ;
// my heli needs this - one too many crashes I think!

  // spin up and stabilise - send '0' to the 'sprog' param briefly; seems to be the way that the proper Appcopter app does it (reset?)
  program_ch1 = appendInstruction( program_ch1, t, 0, DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, 0 ) ;
  program_ch1 = appendInstruction( program_ch1, t+=1, 0.3, DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  //

  // fly, my pretty, fly... Draw a square.

  // lift off and stabilise
  program_ch1 = appendInstruction( program_ch1, t+=1, 1 * tScale , DEFAULT_PITCH, DEFAULT_YAW, DEFAULT_TRIM, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.6, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  //

  // forwards - pitch forwards for a time
  program_ch1 = appendInstruction( program_ch1, t+=1, 1 * tScale , MAX_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  // strip momentum and stabilise - pitch briefly back to arrest forwards momentum and return to hover
  program_ch1 = appendInstruction( program_ch1, t+=1.5, 1 * tScale , MIN_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.3, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // turn 90 - yaw full for a brief time
  program_ch1 = appendInstruction( program_ch1, t+=1, 0.9 * tScale , DEFAULT_PITCH, MIN_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.5, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // forwards
  program_ch1 = appendInstruction( program_ch1, t+=1, 1 * tScale , MAX_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  // strip momentum and stabilise
  program_ch1 = appendInstruction( program_ch1, t+=1.5, 1 * tScale , MIN_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.3, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // turn 90
  program_ch1 = appendInstruction( program_ch1, t+=1, 0.9 * tScale , DEFAULT_PITCH, MIN_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.5, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // forwards
  program_ch1 = appendInstruction( program_ch1, t+=1, 1 * tScale , MAX_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  // strip momentum and stabilise
  program_ch1 = appendInstruction( program_ch1, t+=1.5, 1 * tScale , MIN_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.3, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // turn 90
  program_ch1 = appendInstruction( program_ch1, t+=1, 0.9 * tScale , DEFAULT_PITCH, MIN_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.5, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // forwards
  program_ch1 = appendInstruction( program_ch1, t+=1, 1 * tScale , MAX_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  // strip momentum and stabilise
  program_ch1 = appendInstruction( program_ch1, t+=1.5, 1 * tScale , MIN_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.3, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // turn 90
  program_ch1 = appendInstruction( program_ch1, t+=1, 0.9 * tScale , DEFAULT_PITCH, MIN_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;
  program_ch1 = appendInstruction( program_ch1, t+=0.5, 0.9 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // landing - throttle down to allow safe descent
  program_ch1 = appendInstruction( program_ch1, t+=1, 0.55 * tScale , DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, DEFAULT_SPROG ) ;

  // shutdown - set 'sprog' to 1, again seems to be the way appcopter app does it, but seems maybe unnecessary?
  program_ch1 = appendInstruction( program_ch1, t+=1.5, 0, DEFAULT_PITCH, DEFAULT_YAW, myTrim, DEFAULT_CHANNEL, 1 ) ;
}

// construct an instruction and put it into the control program array
Instruction[] appendInstruction( Instruction[] program, float time_s, float throttle, int pitch, int yaw, int trimOffset, int channel, int sprog )
{
  Instruction instruction = new Instruction( time_s, throttle * MAX_THROTTLE, pitch, yaw, trimOffset, channel, sprog ) ;
  program = (Instruction[]) append( program, instruction ) ;
  return program ;
}

// the main update loop
void draw()
{
 if( ! signalOn ) return ; // don't do anything if the signal is off

 // otherwise...
 readProgram() ; // fetch the latest instruction
 sendData() ; // transmit it as audio
}

void readProgram()
{
 float currentTime_s = float(millis() - programStartTime) / 1000.0 ;

 int nextInstructionIndex = programIndex_ch1 + 1 ;

 if( nextInstructionIndex < program_ch1.length )
 {
 Instruction nextInstruction = program_ch1[nextInstructionIndex] ;
 if( nextInstruction.time_s < currentTime_s )
 {
 programIndex_ch1 = nextInstructionIndex ; // next instruction is now due: step the program index along
 Instruction instr = program_ch1[ programIndex_ch1 ] ;
 println( "Set instruction: @time: " + instr.time_s + " throttle: " + instr.throttle + " pitch: " + instr.pitch + " yaw: " + instr.yaw + " trimOffset: " + instr.trimOffset + " sprog " + instr.sprog ) ;
 }
 }
}

void sendData()
{
 float[] buffer = signal.getChannel(1) ;

 int offset = 0 ;

 Instruction instr = program_ch1[ programIndex_ch1 ] ;

 // ease
 cThrottle += (instr.throttle - cThrottle) * ease ;
 cPitch += (instr.pitch - cPitch) * ease ;
 cYaw += (instr.yaw - cYaw) * ease ;
 cTrim += (instr.trimOffset - cTrim) * ease ;
 println( "Sent data: Throttle: " + cThrottle + " Pitch: " + cPitch + " Yaw: " + cYaw + " Trim: " + cTrim ) ;
 //

 // open
 offset = writeData( sig_open, buffer, offset ) ;

 // throttle
 offset = writeValue( throttleBits, cThrottle, buffer, offset ) ;

 // pitch
 offset = writeValue( pitchBits, cPitch, buffer, offset ) ;

 // yaw
 offset = writeValue( yawBits, cYaw, buffer, offset ) ;

 // trim
 offset = writeValue( trimBits, cTrim, buffer, offset ) ;

 // channel
 offset = writeValue( channelBits, instr.channel, buffer, offset ) ;

 // sprog
 offset = writeValue( sprogBits, instr.sprog, buffer, offset ) ;
 //offset = writeData( sig_1, buffer, offset ) ;
 //offset = writeData( sig_0, buffer, offset ) ;

 //
 signal.trigger() ;
}

void keyPressed()
{
 // any keypress: toggle signal on / off
 signalOn = ! signalOn ;

 if( signalOn )
 {
 // reset the values to default when the signal turns back on
 cThrottle = DEFAULT_THROTTLE ;
 cPitch = DEFAULT_PITCH ;
 cYaw = DEFAULT_YAW ;
 cTrim = DEFAULT_TRIM ;
 programStartTime = millis() ;
 programIndex_ch1 = 0 ;
 println( "program started" ) ;
 }
 else
 {
 println( "program stopped" ) ;
 }
}

int writeValue( int numBits, int value, float[] target, int offset )
{
 byte tByte = (byte)value ;

 // nudge extra bits off
 int mask = 0x80;
 int bitDiff = 8 - numBits ;
 for( int i = 0 ; i < bitDiff ; i++ ) mask >>= 1;

 while( mask > 0 )
 {
 if( (mask & tByte) != 0)
 {
 offset = writeData( sig_1, target, offset ) ;
 //println( "wrote 1" ) ;
 }
 else
 {
 offset = writeData( sig_0, target, offset ) ;
 //println( "wrote 0" ) ;
 }
 mask >>= 1;
 }
 //println( ".." ) ;
 return offset ;
}

int writeData( float[] src, float[] target, int offset )
{
 int i ;
 for( i = 0 ; i < src.length ; i++ ) target[offset + i] = src[i] ;
 return offset + i ;
}

void stop()
{
 minim.stop();
 signal.close() ;

 super.stop();
}

class Instruction
{
 float time_s ;
 float throttle ;
 int pitch ;
 int yaw ;
 int trimOffset ;
 int channel ;
 int sprog ;

 Instruction( float time_s, float throttle, int pitch, int yaw, int trimOffset, int channel, int sprog )
 {
 this.time_s = time_s ;
 this.throttle = throttle ;
 this.pitch = pitch ;
 this.yaw = yaw ;
 this.trimOffset = trimOffset ;
 this.channel = channel ;
 this.sprog = sprog ;
 }
}

Troubleshooting:

I’ve tried this code on both Mac and Windows machines – for some reason that I don’t understand, I needed to invert the sign of the signal data on MacOS (see comment in code, within setup function) for the thing to work.

I note that the Appcopter app sets the output volume to max when the transmitter is plugged into the iPhone/iPad. Similarly, you need to set an alarmingly high volume for the transmitter to push out any infrared! Needs an amplifier in there methinks.

Sprog:

In the last post I indicated that the last two bits were some unknown signal element. After further study, this appears to be a control for the ‘headlight’ LED on the nose of the Appcopter. I suspect there may be more to it however – perhaps a ‘reset’ signal? I base this on the discovery that if the signal doesn’t go from 0 to 2 to 1 during a flight, the helicopter can be landed in an inconsistent state within which it’ll not respond to the control signal at all.

Flying time:

Ultimately, my ambition of commanding the helicopter to fly a square kind of worked… Check out the really poor video. I need a proper video camera.

Immediately you’ll note: It’s not a very square square. Uhh no.

Conclusions:

I quickly realised the naivety of my ambitions after the first couple of automated flights. The control signal was adequately deciphered, and the program fully sufficient to tell the helicopter what to do in a robotic way. My ‘flying turtle’ kind of worked, being able to draw a (very rough) square! But I expected perhaps a little more… accuracy?

Unlike the Turtle robot I once enjoyed at school, no two of my helicopter’s flights were the same. ‘Tiny’ variables, like rotor positioning at the start of a run, became massive deviations in direction as the flight progressed – reality brings the full weight of the chaos effect to bear on my toy robot! In my control signal I’ve tried to avoid a predictable episode of turbulent disruption caused by ground effect (the helicopter buffeted by its own rotor down wash) by launching up to a respectable altitude and stabilising before setting off. My flying robot with it’s four degrees of freedom (up/down, forwards/backwards, yaw, pitch) is a lot more complicated than a terrestrial buggy already at peace with gravity (and a mere two degrees of freedom)!

So what next? A sensible answer would be to attach sensors, and allow the helicopter to take care of it’s own stability to some extent. But my little toy can’t possibly be expected to carry a load (gyroscopic sensors, distance sensors, altimeter, the list of possible enhancements goes on), and with each new system comes a greater requirement for power, which obliges a bigger battery, which requires more lift, so bigger rotors or more powerful motors… Basically, my explorations with the Appcopter have led me to consider the possibilities of constructing a multirotor helicopter in order to explore ‘real’ onboard flight AI. I’m currently working on a hexcopter built from three broken Appcopters, in order to get a feel for constructing my own flying machine: