Tutorial of Creating an RL Agent¶
Reinforcement learning (RL) is a method of black-box machine learning, meaning that the RL agent can produce actions that optimize performance without knowing the mechanics of the environment. Like a child learning new things, RL agents can try an action, observe the outcome, and extrapolate about which actions are good and bad and when without ever knowing the exact physics behind transitions from one state to the next. As the human behind the RL agent you will have to give the RL agent clues to sort the outcomes into “good” and “bad”. You will also be able to define other parameters of the RL agent such as which algorithms it uses, the amount of time you allow it to train, and which observations the RL should consider. We have provided a framework for you in the
rl_training.py file (found in the sister repo https://github.com/cugriffinglab/gnomes-rl) and a description of how to implement it to create a custom agent below:
We suggest that you copy the
training\ folder from the gnomes-rl repository.
Components of submission¶
The reward* function¶
While the GNOMES competition will provide holistic feedback on your performance, the reward(home) function can be useful to provide step-by-step feedback on performance.
For example, let’s say the competition reward algorithm was very complicated and nearly impossible to model but you notice it is strongly correlated with a low amount of energy consumed at any given timestep. You can “grade” your agent based off a very simple reward function such as:
reward = -1 * home.obs_dict["my_demand"]
but there is no techincal limit to how complicated your function can be (we don’t necessarily recommend this reward signal either, but you get the idea):
reward = -1 * home.obs_dict["my_demand"] ** (home.obs_dict["occupancy"]) * np.sin(home.obs_dict["h_out_6hr"])
We encourage you to try other transformations to get your home working well. You might look into different optimization objectives that you think could work to approximate the goals of the competition.
Reward should take the input argument home, and output the reward as a single floating point value.
The normalization* function¶
This function is intended to (1) filter the values that the reinforcement learning agent can consider and (2) normalize them to stabilize learning.
Since RL agents built on the OpenAI gym platform expect the observation values as a list of numbers (e.g. floats) you will need to parse the observation dictionary (home.obs_dict) to a list. You can build this list in any way you like, but consider that the list should pass the values in the same order every time (e.g. if you pass the values as [temperature, occupancy] you should continue to use that order; passing them as [occupancy, temperature] might confuse the agent).
Deep reinforcement learning is based on a series of neural networks that describe the relationships of inputs (actions and states) and outputs (rewards). Neural networks are “trained” to classify this relationship through gradient descent which depends on the error between the observed and predicted output. If the model uses multiple inputs (say temperature and occupancy status) the scale of the temperature value (e.g. 20 deg C) compared to the occupancy status (e.g. 0 or 1) can cause issues in updating weights of the network. Therefore it is recommended to normalize your values to be approximately have a mean of 0 and standard deviation of 1.
Normalization should take the input argument
home, and output the observation as a list (any length >= 1) of floating point values.
The train* function¶
The train function is what you will run locally to create an agent. The agent can be saved and evaluated later on by the official scoring mechanism of GNOMES.
It will be your responsibility to train your agent, save it, and provide a
predict() method in
submission.py that calls the agent from whatever method you used to store it. We have provided a framework using the popular RL library Stable-Baselines3 for you to get started.
You may add any arbitrary helper function to rl_training.py.
You may use any custom Python package in rl_training.py.
*Indicates that you should keep these function names to make use of the
train_player.py script. We will not be running
train_player.py or similar as part of the submission/scoring process, so feel free to modify these scripts if they don’t suit your application. We can offer limited one-on-one support (subject to the availability of the GNOMES support team) if you need help understanding how the environment is written and which variables are accessible for training.
Training using docker¶
Build the training using
docker-compose -f ./training/docker-compose.yml build
Run the training using
docker-compose -f ./training/docker-compose.yml up --abort-on-container-exit
The trained network should appear in submission as
my_agent.zip. You must include this file with your submission.
Training using non-docker¶
Follow steps 1-4 from the non-docker submission documentation.
Open two new terminal windows
In each window, change the directory to the sandbox simulation folder using
$ cd <your-username-gnomes>/training/simulation
In one terminal window, start the player submission using
$ python train_player.py
In the other terminal window, run the simualtion using
$ python run_aggregator.py
Submitting and receiving official feedback¶
The process for testing your agent is identical to the process you used for an RBC controller but you will have to modify
submission.py to load the agent. We will walk you through that process below:
The predict function¶
For each prediction we load the saved RL agent from Stable Baselines. The Stable Baselines agent contains a method
predict that takes an argument of the vectorized observation list and returns the action. To provide this to the agent we use the normalization function previously defined in
rl_training.py. Remember to ensure that the
predict() function is only returning a list of 3 floats that represent the desired action. For Stable Baselines this means calling the zero-indexed element of the
predict() method. (In Stable Baselines predict returns the action + some extraneous data.)
Loading external files¶
Loading external files (such as a pre-trained agent) can be done by modifying the following line in
You may import any pip installable package to your submission function. Please note that we will not support manual dependecy conflict resolution so your submission package should therefore be pip installable. To include these functions in the pip install process we use for the scoring process modify
<gnomes-your-username>/setup.py to include the custom package name under