• tal@lemmy.today
    link
    fedilink
    English
    arrow-up
    4
    ·
    edit-2
    4 months ago

    I guessed in a previous comment that given their new partnership, Reddit is probably feeding their comment database to Google directly, which reduces load for both of them and permits Google to have real-time updates of the whole kit-and-kaboodle rather than polling individual pages. Both Google and Reddit are better-off doing that, and for Google it’d make sense for any site that’s large-enough and valuable enough to warrant putting forth any effort special-case to that site.

    I know that Reddit built functionality for that before, used it for pushshift.io and I believe bots.

    I doubt that Google is actually using Googlebot on Reddit at all today.

    I would bet against either Google violating robots.txt or Reddit serving different robots.txt files to different clients (why? It’s just unnecessary complication).