From RL-Library

Contents 1 Introduction 2 Technical Details 2.1 Observation Space 2.2 Action Space 2.3 Additional technical details: 3 Other Details 4 Download and Installation Instructions 4.1 Using This Download 4.1.1 This Is Your First Project 4.1.2 Adding To An Existing RL-Library Download 4.2 Compiling This Project 4.3 Running This Project 4.4 Getting Help 4.5 Authors 4.5.1 Current Maintainer 5 History 6 References

Introduction

This environment is based on Sutton and Barto's (1998) Acrobot task as described in Reinforcement Learning: An Introduction.

The acrobot, a two-link, underactuated robot roughly analogous to a gymnast swinging on a high bar. The first joint (corresponding to the gymnast's hands on the bar) cannot exert torque, but the second joint (corresponding to the gymnast bending at the waist) can. The system has four continuous state variables: two joint positions and two joint velocities.

Technical Details

Observation Space

The system has four continuous observation variables: two joint positions and two joint velocities.

  θ1 ∈ [-π, π] angle between the horizontal base and the first link
  θ2 ∈ [-π, π] angle between the first and the scond link
  θ1_dot ∈ [-4π, 4π] angular velocity w.r.t. θ1
  θ2_dot ∈ [-9π, 9π] angular velocity w.r.t. θ2

Action Space

Acrobot has 3 discrete actions, {0,1,2}, correspond to torque to the joint between the first and second link of {-1, 0, 1} respectively. Reward:

    -1 per time step
    0 when goal state reached

Additional technical details:

• deterministic dynamics

Other Details

• RL-Viz Compatible
• Has Visualizer
• Language: Java
• License: Apache 2.0

Download and Installation Instructions

Download Link: Acrobot-Java-R1306.tar.gz (Details)

This download contains the source code that can be used to change/rebuild the project as well as a pre-built JAR file that can be used immediately.

Using This Download

Before diving into this, you may want to check out the getting started guide.

This download can be used to augment your existing local RL-Library (if you have one), or as the basis to start a new one.

This Is Your First Project

#Create a directory for your rl-library. Call it whatever you like.
mkdir rl-library

That's all you have to do special for the first time you download a rl-library component. Continue on now to the next section.

Adding To An Existing RL-Library Download

#First, download the file.  Depending on your platform, you might have to do this manually with a web browser. 

#If you are on Linux, you can use wget which will download Acrobot-Java-R1306.tar.gz for you
wget http://rl-library.googlecode.com/files/Acrobot-Java-R1306.tar.gz

#Copy the download to your local rl-library folder (whatever it is called)
cp Acrobot-Java-R1306.tar.gz rl-library/
cd rl-library

#This will add any project-specific things necessary to system and products folders
#It will also create a folder for this particular project
tar -zxf Acrobot-Java-R1306.tar.gz

#Clean up
rm Acrobot-Java-R1306.tar.gz

After this step is completed, you will have several new files:

• rl-library/
  • products/
    • Acrobot.jar
  • Acrobot-Java-R1306/
    • src/
    • build.xml
    • LICENSE.TXT
    • README.TXT
  • system/
    • common/
      • libs/
      • ant/

Compiling This Project

You must have Apache Ant installed to build this project using these instructions.

You don't have to compile this project, because the JAR file has been compiled and placed into the products directory already. However, if you want to make changes and recompile, type:

cd Acrobot-Java-R1306
ant clean

#this will update ../products/Acrobot.jar
ant {{{Anttarget}}}

Running This Project

You can run this project by typing:

java -jar products/Acrobot.jar
#or from within the project's directory
cd Acrobot-Java-R1306
ant run

You can also use it in conjunction with RL-Viz by putting the JAR file products/Acrobot.jar in the appropriate directory, as long as the RL-Viz library jar file is in the appropriate relative location from where you put Acrobot.jar. The location is: ../system/common/libs/rl-viz/RLVizLib.jar

Getting Help

Please send all questions to either the current maintainer (below) or to the RL-Library mailing list.

Authors

Various, over the years, including:

• Adam White
• Brian Tanner

Current Maintainer

• Brian Tanner
• btanner@rl-community.org

History

Adam White implemented the Acrobot task for RL-Glue (1.0) in C++. Since then, Brian Tanner ported it to Java.

References

[Sutton and Barto, 1998] Reinforcement Learning: An Introduction. Richard S. Sutton and Andrew G. Barto. A Bradford Book. The MIT Press Cambridge, Massachusetts London, England, 1998.