Policy gradients can be applied in the case of multiple continuous actions by using a policy network that outputs a probability density function (PDF) over the action space, and then sampling actions from this PDF.
Specifically, the policy network can output a mean vector and a diagonal covariance matrix that parameterize a multivariate Gaussian distribution over the action space. To sample an action from this distribution, we can first sample a noise vector from a standard Gaussian distribution, and then transform it using the mean and covariance parameters. This results in a continuous action that can be passed to the environment.
The loss function for training the policy network can be the negative log-likelihood of the chosen action under the PDF. This encourages the policy to assign high probabilities to actions that lead to high rewards, and low probabilities to actions that lead to low rewards.
The gradient of this loss function with respect to the policy parameters can be computed using the chain rule and the reparameterization trick, which allows for low-variance gradient estimates.
However, the use of continuous actions can pose additional challenges in the RL setting, such as the need for exploration and the possibility of numerical instability in the computation of the gradients. These challenges can be addressed using techniques such as entropy regularization and actor-critic methods.
