What's the proper way to log reward per episode?

[GF-Huang]

I want to show each episode rewards on Tensorboard.

The trainer callbacks seems only based on epochs and steps.

[Trinkle23897]

The train/reward and test/reward have been recorded to tensorboard.

[GF-Huang]

It seems based on steps. I want to statistics on an episodes basis.

[Trinkle23897]

No, it's based on episode. You can actually have a test with a fixed length episode env (e.g., 100) with per step reward=1, set the n_step per collect to less than 100, and see what will be recorded into tensorboard.

[GF-Huang]

My env is fixed length with 390 steps. But seems the Tensorboard is plot data point per 3,900 steps. Is it because my train_envs is 10?

My code:

train_envs = tianshou.env.DummyVectorEnv([lambda: MarketEnv(train_df_list) for _ in range(10)])
test_envs = tianshou.env.DummyVectorEnv([lambda: MarketEnv(test_df_list) for _ in range(100)])

...
...

train_collector = tianshou.data.Collector(
    policy,
    train_envs,
    tianshou.data.VectorReplayBuffer(390 * 100, 10, stack_num=15),
    exploration_noise=True
)
test_collector = tianshou.data.Collector(policy, test_envs, exploration_noise=True)

trainer = tianshou.trainer.OnpolicyTrainer(
    policy=policy, 
    train_collector=train_collector, 
    test_collector=test_collector,
    max_epoch=10,
    repeat_per_collect=1,
    episode_per_test=len(train_df_list) // 10,
    batch_size=64,
    step_per_collect=390,
    step_per_epoch=sum([len(df) for df in train_df_list]),
    resume_from_log=True,
    logger=tianshou.utils.TensorboardLogger(SummaryWriter('log/PPO-LSTM-15-Timesteps'))
)
trainer.run()

[Trinkle23897]

I think so. At step 3900 it finishes 10 envs and then records the mean episodic reward.