Warning: Parameter 1 to Language::getMagic() expected to be a reference, value given in /home/btanner/library.rl-community.org/includes/StubObject.php on line 58
Mountain Car (Java) - RL-Library

Mountain Car (Java)

From RL-Library

Jump to: navigation, search
Mountain Car Illustration

Contents

Introduction

This environment is based on Sutton and Barto's (1998) Mountain Car Task as described in Reinforcement Learning: An Introduction. This implementation was ported to RL-Glue from Sutton's Mountain Car Software (2000). Consider a task where an agent must drive an underpowered car up a steep mountain road. Since gravity is stronger than the car’s engine, even at full throttle the car cannot simply accelerate up the steep slope. The car’s movement is described by two continuous output variables, position and velocity, and one discrete input representing the acceleration of the car.

This task is interesting because the car’s position on the hill and its velocity are real-valued. Therefore, a learning algorithm must use a function approximator to learn a good policy. Mountain car is also interesting because a successful control policy must drive the car backwards, up the other side of the valley, to gain enough momentum to drive forwards up the hill. This means the learning algorithm must move away from the goal, incurring additional negative reward, to discover the solution. Finally, actions do not have immediately measurable effects on the state of the system. Thus, learning algorithms must assign credit to actions taken several time steps in the past.

The Mountain Car task was originally presented by Andrew Moore in his PhD disertation (1990) and has been widely studied since. Singh and Sutton (1996) later used Mountain Car in their work on eligibility traces and formalized the state update equations for the position and velocity of the car based on Moore's original problem specification. Over the years there have been several variations on Singh and Sutton's version of the problem: different reward functions, starting states and termination conditions.

Technical Details

Observation Space

2 dimensional, continuous valued

 car position (-1.2 to 0.6)
 car velocity (-0.07 to 0.07)    

Action Space

1 dimensional, discrete valued

 reverse, neutral and forward

Rewards

negative reward per step

Termination

if car position is >= 0.6

Additional technical details

  • deterministic dynamics
  • optional random starting position and velocity

Parameters

Mountain Car currently has three parameters that can be set with RL-Viz:

  • Random Seed
  • Random Start States
  • Transition Noise

These parameters are not relevant to the conventional Mountain Car problem; they exist for experiments for generalized mountain car. This page will be updated at some near-future date to explain the implications of setting those parameters. For now, do not use special values for those settings when doing traditional Mountain Car experiments.

Other Details

  • RL-Viz Compatible
  • Has Visualizer
  • Language: Java
  • License: Apache 2.0

Download and Installation Instructions

Download Link: MountainCar-Java-R1308.tar.gz (Details)

This download contains the source code that can be used to change/rebuild the project as well as a pre-built JAR file that can be used immediately.

Using This Download

Before diving into this, you may want to check out the getting started guide.

This download can be used to augment your existing local RL-Library (if you have one), or as the basis to start a new one.

This Is Your First Project

#Create a directory for your rl-library. Call it whatever you like.
mkdir rl-library

That's all you have to do special for the first time you download a rl-library component. Continue on now to the next section.

Adding To An Existing RL-Library Download

#First, download the file.  Depending on your platform, you might have to do this manually with a web browser. 

#If you are on Linux, you can use wget which will download MountainCar-Java-R1308.tar.gz for you
wget http://rl-library.googlecode.com/files/MountainCar-Java-R1308.tar.gz

#Copy the download to your local rl-library folder (whatever it is called)
cp MountainCar-Java-R1308.tar.gz rl-library/
cd rl-library

#This will add any project-specific things necessary to system and products folders
#It will also create a folder for this particular project
tar -zxf MountainCar-Java-R1308.tar.gz

#Clean up
rm MountainCar-Java-R1308.tar.gz

After this step is completed, you will have several new files:

  • rl-library/
    • products/
      • MountainCar.jar
    • MountainCar-Java-R1308/
      • src/
      • build.xml
      • LICENSE.TXT
      • README.TXT
    • system/
      • common/
        • libs/
        • ant/

Compiling This Project

You must have Apache Ant installed to build this project using these instructions.

You don't have to compile this project, because the JAR file has been compiled and placed into the products directory already. However, if you want to make changes and recompile, type:

cd MountainCar-Java-R1308
ant clean

#this will update ../products/MountainCar.jar
ant {{{Anttarget}}}

Running This Project

You can run this project by typing:

java -jar products/MountainCar.jar
#or from within the project's directory
cd MountainCar-Java-R1308
ant run

You can also use it in conjunction with RL-Viz by putting the JAR file products/MountainCar.jar in the appropriate directory, as long as the RL-Viz library jar file is in the appropriate relative location from where you put MountainCar.jar. The location is: ../system/common/libs/rl-viz/RLVizLib.jar


Getting Help

Please send all questions to either the current maintainer (below) or to the RL-Library mailing list.


Authors

Various, over the years, including:

Current Maintainer

History

This version of Mountain Car is based on the C/C++ implementation from Richard Sutton. In 2005, the code was ported to RL-Glue by Adam White. Later, Brian Tanner ported it to Java and added RL-Viz capabilities.

References

[Moore, 1990] A. Moore, Efficient Memory-Based Learning for Robot Control, PhD thesis, University of Cambridge, November 1990.

[Singh and Sutton, 1996] Singh, S.P. and Sutton, R.S. (1996) Reinforcement learning with replacing eligibility traces. Machine Learning 22(1/2/3):123-158.

[Sutton, 2008] Mountain Car Software. Richard s. Sutton. http://www.cs.ualberta.ca/~sutton/MountainCar/MountainCar.html

[Sutton and Barto, 1998] Reinforcement Learning: An Introduction. Richard S. Sutton and Andrew G. Barto. A Bradford Book. The MIT Press Cambridge, Massachusetts London, England, 1998

Personal tools