If you put * in the gitignore file you don't have to deal with ignoring the downloads directory. When you want to add a new file / directory just add the -f flag (git add -f) and everything else works as normal.