They may temporarily bump a site up based on new "authoritative" incoming links and then make judgements from a combination of user time on site/bounces back to search results and clicking on more results and additional pages. Longer term there is definitely manual human review looking for "spam."