You can set this on your visual transformer config with the key patch_dropout. In the paper, they also finetuned without the patch dropout at the end. You can do this with the command-line argument ...