From RL-Library

Introduction

Implements a simulator that models one of the Stanford autonomous helicopters (an XCell Tempest helicopter) in the flight regime close to hover. Additional pictures, as well as further information about the helicopter specifications are available.

The model was learned from data collected using "frequency sweeps" (which systematically vary each of the helicopter's four controls) around the hover flight regime, and should be reasonably accurate for this regime (up to effects described below). The learning procedure is described in more detail in (Abbeel et al., 2006). The simulator will enter RL_glue's terminal state whenever the helicopter leaves the hover regime. In particular, the simulator will enter the terminal state when the velocity along any of the main axes exceeds 5m/s, when the angular rate around any of the main axes exceeds 2*2pi rad/s, when the orientation is more than 30 degrees of from the target (level) orientation, or when the position is off by more than 20m.

The simulator is set up to run for 6000 timesteps, and each simulation step is 0.1 seconds, thus giving runs of 10 minutes. (The simulator runs faster than realtime.) If it enters the terminal state before 6000 timesteps have been completed, a large negative reward is given, corresponding to getting the most negative reward achievable for the remaining time.

The simulator implements a rigid body simulation, which is reasonably accurate around hover. However, it's important to keep in mind that two factors that complicate real helicopter control were left unmodeled:

(1) Uncertainty about the state. Real helicopters are often equipped with an IMU (inertial measurement unit) and magnetic compass (magnetometer), which measure acceleration, angular rates, and magnetic field. They also often have some position sensing, such as provided by GPS or camera systems. The sensor information is then fused using a (extended or unscented) Kalman filter. In practice, especially due to high vibration of the helicopter caused by the engine and blades, sensor information (especially for IMU sensors) is noisy, resulting in noisy state estimates. Noisy state estimates tend to strongly influence control performance for many control algorithms.

(2) Latency. All sensors and actuators have some latency. E.g., on the Stanford helicopter, the sensors are on-board, and their information is radioed down to a ground-based computer which runs the Kalman filter. Controls are computed based on the Kalman filter's output, and sent back up to the helicopter with a radio transmitter. The helicopter's receiver decodes the controls, and then applies them to the servomotors. Only after the servomotors have moved will the controls take effect on the helicopter. Typical latency is on the order of 0.1 (up to even 0.2) seconds.

Technical Details

The helicopter's full state is given by its velocity, position, angular rate and orientation.

Observation Space

In the simulator, rather than giving the raw state vector, we instead provide the observation/state vector in the helicopter's coordinate frame (because in our experience this simplifies the mapping from state to controls). In particular, the simulator provides the following observation/state vector:

 0: u [forward velocity] 
 1: v [sideways velocity (to the right)] 
 2: w [downward velocity] 
 3: xerr [helicopter x-coord position - desired x-coord position] -- helicopter's x-axis points forward 
 4: yerr [helicopter y-coord position - desired y-coord position] -- helicopter's y-axis points to the right 
 5: zerr [helicopter z-coord position - desired z-coord position] -- helicopter's z-axis points down 
 6: p [angular rate around helicopter's x axis] 
 7: q [angular rate around helicopter's y axis] 
 8: r [angular rate around helicopter's z axis] 
 9,10,11: quaternion x,y,z entries

Rewards

The reward function is computed as follows:

 reward = 0.0; 
 for (int i=0; i < 12; ++i) 
 reward -= heli.state[i]*heli.state[i];

Here heli.state contains the forementioned 12 variables. Because the tail rotor's thrust exerts a sideways force on the helicopter, even in the absence of wind, the helicopter needs to tilt sideways slightly to stay in place (such that main rotor thrust compensates for the tail rotor's sideways thrust). Thus, the helicopter cannot be held stationary in the zero cost state, even in the absence of noise in the dynamics.

Parameters

wind0

   Type: continuous
   Range: [-1.0d, 1.0d]
   Description: Used to sample wind: the wind is assumed to be stationary, and is sampled from a uniform distribution over the interval [-5m/s, 5m/s] for both the North and East axes.

wind1'

   Type: continuous
   Range: [-1.0d, 1.0d]
   Description: Used to sample the noise added on to the dynamics (to capture unmodeled effects) at every simulation time step. Note: This may not be correct.  Someone competent with the helicopter should investigate whether this is how it is being used.

Action Space

Termination

Other Details

RL-Viz Compatible
Has Visualizer
Language: Java
License: Apache 2.0

Download and Installation Instructions

Download Link: Helicopter-Java-R1306.tar.gz (Details)

This download contains the source code that can be used to change/rebuild the project as well as a pre-built JAR file that can be used immediately.

Using This Download

Before diving into this, you may want to check out the getting started guide.

This download can be used to augment your existing local RL-Library (if you have one), or as the basis to start a new one.

This Is Your First Project

#Create a directory for your rl-library. Call it whatever you like.
mkdir rl-library

That's all you have to do special for the first time you download a rl-library component. Continue on now to the next section.

Adding To An Existing RL-Library Download

#First, download the file.  Depending on your platform, you might have to do this manually with a web browser. 

#If you are on Linux, you can use wget which will download Helicopter-Java-R1306.tar.gz for you
wget http://rl-library.googlecode.com/files/Helicopter-Java-R1306.tar.gz

#Copy the download to your local rl-library folder (whatever it is called)
cp Helicopter-Java-R1306.tar.gz rl-library/
cd rl-library

#This will add any project-specific things necessary to system and products folders
#It will also create a folder for this particular project
tar -zxf Helicopter-Java-R1306.tar.gz

#Clean up
rm Helicopter-Java-R1306.tar.gz

After this step is completed, you will have several new files:

rl-library/
- products/
  - Helicopter.jar
- Helicopter-Java-R1306/
  - src/
  - build.xml
  - LICENSE.TXT
  - README.TXT
- system/
  - common/
    - libs/
    - ant/

Compiling This Project

You must have Apache Ant installed to build this project using these instructions.

You don't have to compile this project, because the JAR file has been compiled and placed into the products directory already. However, if you want to make changes and recompile, type:

cd Helicopter-Java-R1306
ant clean

#this will update ../products/Helicopter.jar
ant {{{Anttarget}}}

Running This Project

You can run this project by typing:

java -jar products/Helicopter.jar
#or from within the project's directory
cd Helicopter-Java-R1306
ant run

You can also use it in conjunction with RL-Viz by putting the JAR file products/Helicopter.jar in the appropriate directory, as long as the RL-Viz library jar file is in the appropriate relative location from where you put Helicopter.jar. The location is: ../system/common/libs/rl-viz/RLVizLib.jar

Getting Help

Please send all questions to either the current maintainer (below) or to the RL-Library mailing list.

Current Maintainer

Brian Tanner
btanner@rl-community.org

History

Pieter Abbeel, Adam Coates, Andrew Y. Ng, Stanford University. Ported by Mark Lee and Brian Tanner from C++ to Java for the 2008 RL-Competition and beyond.

References

Modeling Vehicular Dynamics, with Application to Modeling Helicopters, Pieter Abbeel, Varun Ganapathi and Andrew Y. Ng. In NIPS 18, 2006. (ps, .pdf) Stanford Autonomous Helicopter Project, Pieter Abbeel, Adam Coates, Morgan Quigley, Andrew Y. Ng, (www).

Partner Sites

Makers of ipad keyboard case and other gadgets.

Helicopter (Java)