Sarsa Lambda Fourier Basis (Java)

From RL-Library

Jump to: navigation, search

Contents

Introduction

The Fourier Basis is a generic basis for linear function approximation in continuous-state reinforcement learning domains. It is simple, easy to use, and seems to work reliably.

In the simplest case, specifying a Fourier Basis for an n dimensional continuous domain requires only one parameter: k, which specifies the order terms to include (all terms of all combinations of variables of order between 0 and k inclusive are included). This parameter is necessary because the Fourier Basis has infinitely many terms.

Since the number of basis functions scales exponentially with the number of state variables n (as is true for all fixed bases), in high-dimensional problems the basis function coefficient ranges will probably need to be constrained. This code includes a FourierCoefficientGenerator class which can be used to write constrained coefficient generators. Two example derived classes are included: FullFourierCoefficientGenerator (which generates all coefficients) and IndependentFourierCoefficientGenerator (which only generates terms using individual variables). Both classes take k as a parameter. The Fourier Basis is also well suited as a fixed basis for use in feature selection, since it has infinitely many basis functions.

The Fourier Basis is described in the following paper:

Technical Details

Policy

  • e-greedy

Exploration Policy

  • e-greedy

State Representation/Function Approximation

  • State variables should be continuous. The function approximation code scales them to between 0 and 1 internally.

Actions

  • This implementation handles only discrete actions.

Other Details

  • RL-Viz Compatible
  • Value function visualizable
  • Language: Java
  • License: Apache 2.0

Download and Installation Instructions

Download Link: FourierSarsaAgent-Java-R1318.tar.gz (Details)

This download contains the source code that can be used to change/rebuild the project as well as a pre-built JAR file that can be used immediately.

Using This Download

Before diving into this, you may want to check out the getting started guide.

This download can be used to augment your existing local RL-Library (if you have one), or as the basis to start a new one.

This Is Your First Project 

#Create a directory for your rl-library. Call it whatever you like.
mkdir rl-library

That's all you have to do special for the first time you download a rl-library component. Continue on now to the next section.

Adding To An Existing RL-Library Download 

#First, download the file.  Depending on your platform, you might have to do this manually with a web browser. 

#If you are on Linux, you can use wget which will download FourierSarsaAgent-Java-R1318.tar.gz for you
wget http://rl-library.googlecode.com/files/FourierSarsaAgent-Java-R1318.tar.gz

#Copy the download to your local rl-library folder (whatever it is called)
cp FourierSarsaAgent-Java-R1318.tar.gz rl-library/
cd rl-library

#This will add any project-specific things necessary to system and products folders
#It will also create a folder for this particular project
tar -zxf FourierSarsaAgent-Java-R1318.tar.gz

#Clean up
rm FourierSarsaAgent-Java-R1318.tar.gz

After this step is completed, you will have several new files:

  • rl-library/
    • products/
      • fourierBasisAgentLib.jar
    • FourierSarsaAgent-Java-R1318/
      • src/
      • build.xml
      • LICENSE.TXT
      • README.TXT
    • system/
      • common/
        • libs/
        • ant/

Compiling This Project

You must have Apache Ant installed to build this project using these instructions.

You don't have to compile this project, because the JAR file has been compiled and placed into the products directory already. However, if you want to make changes and recompile, type: 

cd FourierSarsaAgent-Java-R1318
ant clean

#this will update ../products/fourierBasisAgentLib.jar
ant build

Running This Project

You can run this project by typing: 

java -jar products/fourierBasisAgentLib.jar
#or from within the project's directory
cd FourierSarsaAgent-Java-R1318
ant run

You can also use it in conjunction with RL-Viz by putting the JAR file products/fourierBasisAgentLib.jar in the appropriate directory, as long as the RL-Viz library jar file is in the appropriate relative location from where you put fourierBasisAgentLib.jar. The location is: ../system/common/libs/rl-viz/RLVizLib.jar


Getting Help

Please send all questions to either the current maintainer (below) or to the RL-Library mailing list.

Current Maintainer

Retrieved from "http://library.rl-community.org/wiki/Sarsa_Lambda_Fourier_Basis_(Java)"
Personal tools