如何提高我的DQN的性能？

Question

我创建了一个深度Q网络来玩蛇游戏。代码运行得很好，只是在训练周期中性能并没有真正提高。最后，它与采取随机动作的代理几乎没有区别。以下是训练代码：

def train(self):
        self.build_model()
        for episode in range(self.max_episodes):
            self.current_episode = episode
            env = SnakeEnv(self.screen)
            episode_reward = 0
            for timestep in range(self.max_steps):
                env.render(self.screen)
                state = env.get_state()
                action = None
                epsilon = self.current_eps
                if epsilon > random.random():
                    action = np.random.choice(env.action_space) #explore
                else:
                    values = self.policy_model.predict(env.get_state()) #exploit
                    action = np.argmax(values)
                experience = env.step(action)
                if(experience['done'] == True):
                    episode_reward += 5 * (len(env.snake.List) - 1)
                    episode_reward += experience['reward']
                    break
                episode_reward += experience['reward']
                if(len(self.memory) < self.memory_size):
                    self.memory.append(Experience(experience['state'], experience['action'], experience['reward'], experience['next_state']))
                else:
                    self.memory[self.push_count % self.memory_size] = Experience(experience['state'], experience['action'], experience['reward'], experience['next_state'])
                self.push_count += 1
                self.decay_epsilon(episode)
                if self.can_sample_memory():
                    memory_sample = self.sample_memory()
                    #q_pred = np.zeros((self.batch_size, 1))
                    #q_target = np.zeros((self.batch_size, 1))
                    #i = 0
                    for memory in memory_sample:
                        memstate = memory.state
                        action = memory.action
                        next_state = memory.next_state
                        reward = memory.reward
                        max_q = reward + self.discount_rate * self.replay_model.predict(next_state)
                        #q_pred[i] = q_value
                        #q_target[i] = max_q
                        #i += 1
                        self.policy_model.fit(memstate, max_q, epochs=1, verbose=0)
            print("Episode: ", episode, " Total Reward: ", episode_reward)
            if episode % self.target_update == 0:
                self.replay_model.set_weights(self.policy_model.get_weights())
        self.policy_model.save_weights('weights.hdf5')
        pygame.quit() 

以下是超参数：

learning_rate = 0.5
discount_rate = 0.99
eps_start = 1
eps_end = .01
eps_decay = .001
memory_size = 100000
batch_size = 256
max_episodes = 1000
max_steps = 5000
target_update = 10

以下是网络架构：

model = models.Sequential()
model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros', input_dim = 400))
model.add(Dense(500, activation = 'relu', kernel_initializer = 'random_uniform', bias_initializer = 'zeros'))
model.add(Dense(5, activation = 'tanh', kernel_initializer = 'random_uniform', bias_initializer = 'zeros')) #tanh for last layer because q value can be > 1
model.compile(loss='mean_squared_error', optimizer = 'adam')

作为参考，网络输出5个值，因为蛇可以移动的4个方向，以及一个额外的不采取行动的值。此外，我不像传统的DQN那样是游戏的屏幕截图，而是传入了一个400维向量作为游戏发生时所在的20x20网格的表示。智能体因靠近食物或食用食物而获得1的奖励，如果食物死亡，则获得-1的奖励。怎样才能提高性能呢？

Answer 1

我认为主要的问题是你的高学习率。请尝试使用小于0.001的值。Atari DQN使用了0.00025。

还要将traget_update设置为大于10。例如，设置为500或更多。

要查看某些内容，步骤数应至少为10000。

将Lower_batch大小设置为32或64。

你有没有考虑过实施其他一些改进？像PER，决斗DQN？看看这个：https://www.freecodecamp.org/news/improvements-in-deep-q-learning-dueling-double-dqn-prioritized-experience-replay-and-fixed-58b130cc5682/

也许您不想再次重新实现轮子，请考虑使用https://stable-baselines.readthedocs.io/en/master/

最后，您可以查看类似的项目：https://github.com/lukaskiss222/agarDQNbot