Post Guidance progress report

https://www.reddit.com/r/modnews/comments/18fzfne/post_guidance_progress_report/

created by lift_ticket83 on 11/12/2023 at 17:09 UTC*

22 upvotes, 40 top-level comments (showing 25)

Howdy Mods,

In June, we announced a pilot program to test an exciting new mod tool - Post Guidance[1]. Since the unveiling of this program, over 200 subreddits have enlisted to participate, and we are sincerely grateful for the overwhelming support and enthusiasm expressed for this feature. A heartfelt thank you to every mod team who took the time to experiment with this new tool, provide us feedback, and collaborate with us on this product journey.

1: https://www.reddit.com/r/modnews/comments/149gyrl/announcing_mobile_mod_log_and_the_post_guidance/

Today we’re excited to share an update on the progress we’ve made and detail our upcoming plans.

*TL;DR** We’ve seen some incredible results with a sharp decrease in % of automod removals. While certain metrics, such as post starts and submissions, have shown mixed results, we acknowledge that there is still progress to be made. We still have a ways to go, but we’re feeling bullish about the progress thus far.

https://preview.redd.it/a81w2llmfq5c1.jpg?width=2190&format=pjpg&auto=webp&s=a539fe2a6d96efe69868acefaafae9d93b97ff28

*Not so TL;DR**

What you’re looking at above is an AB test analysis of 33 subreddits that we tracked for 4 weeks. Here we are comparing a “control” group (individuals who did not see the Post Guidance experience) against an “enabled experiment” group (individuals who did see the Post Guidance experience). Here’s a summary of some of the results.

We’ve seen a **35% drop** in Automod removals in the experiment group. * In other words, far fewer posters are getting their posts removed by automod. * We’re interpreting this to mean that Post Guidance is successfully educating users on how to post in a way that aligns with a community’s guidelines, and a community’s needs. Needless to say, we’re pretty stoked about this outcome.
Successful posts (as a percentage of submitted posts) have **increased by 6%**. * Seeing this stat elicited several high-fives and a few audible “hell yeahs” from our team. Overall we’ve seen a greater percentage of posts (out of all submitted posts) stay up in our experiment group with post guidance turned on.
Post starts (clicking “create post”) are down by 6% in our experiment group. * The good news: Post guidance reduces the number of individuals having to repeat posts after a failed submission. * What we still need to figure out: There are potentially good faith posters getting overwhelmed during the posting experience, that have abandoned their post.
Post submissions are down by 13% in our experiment group. * The good news: Posts that otherwise would have been removed, are no longer being submitted (e.g. off-topic submissions, low-effort posts, etc.). * What we still need to figure out: Similar to the above, some good faith users may be abandoning their posts after seeing a list of post requirements.
More Post Guidance configurations equals more successful posts! In communities that created more than 7 rules for Post Guidance, we found that the effect was roughly 10% stronger than in communities that created 7 or fewer rules. This indicates that extensive use of the feature can help communities thrive even further!

Note: FYI for those digging into the numbers, it’s worth mentioning that automod removals, mod removals, and admin removals aren’t mutually exclusive. These results are a bit more ad-hoc across 30 days of data. Our data scientist is working on looking at a longer time horizon to look at statistical significance.

*The future of Post Guidance**

At present, Post Guidance is exclusively accessible to subreddits that have enrolled in the pilot program. We have plans to release this feature more broadly in 2024. As part of our experimentation, we are selectively showcasing Post Guidance to 50% of desktop users within those communities. Given the positive results we’ve experienced, we are thrilled to announce our decision to expand the availability of Post Guidance to 100% of desktop users within subreddits participating in our pilot program.

While we do this, our team is also busy designing and building the following feature improvements:

Cross-platform parity within our mobile apps. We know the majority of users are on our apps, and we want them to benefit from the power of post guidance too.
Include URL detection and flair-based rule configurations.
Comment Guidance!

Thanks again to everyone who has helped us get this far. **If you’re interested in joining our pilot program, please feel free to sign up in the comments.** If you have any questions or feedback about the program, the results we’ve shared, or how this feature works don’t hesitate to ask us below!

Comments

Comment by SampleOfNone at 11/12/2023 at 17:36 UTC

10 upvotes, 1 direct replies

expand the availability of Post Guidance to 100% of desktop users within subreddits participating in our pilot program.

u/lift_ticket83 , what’s the timeframe for that?

Comment by shiruken at 11/12/2023 at 17:41 UTC

9 upvotes, 1 direct replies

Post Guidance has been exclusively tested on desktop users thus far. However, New Reddit traffic only accounts for ~10% of our monthly uniques. Have you tested on mobile app users yet? Do you anticipate similar results w.r.t. the decrease in AutoMod removals and increase in successful posts?

Given the positive results we’ve experienced, we are thrilled to announce our decision to expand the availability of Post Guidance to 100% of desktop users within subreddits participating in our pilot program.

For our internal tracking, what date will this go into effect?

Comment by TheRealWhoop at 11/12/2023 at 17:54 UTC

6 upvotes, 0 direct replies

Can /r/ukpersonalfinance join please?

Comment by ihopeurwholelifesux at 11/12/2023 at 19:42 UTC

6 upvotes, 0 direct replies

Could r/POTS and r/dysautonomia please be signed up? I think the redirect to a megathread feature will be so useful for chronic illness groups

Comment by Froggypwns at 11/12/2023 at 19:45 UTC*

7 upvotes, 0 direct replies

/r/Windows, /r/Windows10, and /r/Windows11 would like to join. We have a substantial amount of posts that get removed automatically due to people submitting tech support posts there instead of on /r/WindowsHelp or /r/TechSupport, this could help us reducing how many posts we have to remove or otherwise get filtered.

Edit - Add /r/fatsquirrelhate too, mostly so we can warn people for using death threats and similar messaging.

Comment by Sun_Beams at 11/12/2023 at 19:52 UTC

6 upvotes, 0 direct replies

Comment Guidance!!

https://i.imgur.com/8W7HTs5.gif

Comment by Iron_Fist351 at 11/12/2023 at 17:44 UTC

5 upvotes, 0 direct replies

r/lexus & r/meggyxmario would like to join the pilot program

Comment by ashamed-of-yourself at 11/12/2023 at 18:06 UTC

5 upvotes, 0 direct replies

r/Letterkenny is interested

Comment by 0spore13 at 11/12/2023 at 18:33 UTC

6 upvotes, 1 direct replies

I have been wanting post guidance as a mod tool so badly, is the pilot program still taking applicants?

Comment by SolariaHues at 11/12/2023 at 19:46 UTC

6 upvotes, 1 direct replies

Please sign up r/gardenwild, r/wildlifeponds, & r/learntoreddit

IIRC it wasn't working or ready for image based communities, is it now?

Thank you for the update and all the work you and your team have done on this. Is it too early to say Happy Holidays?

Comment by MajorParadox at 11/12/2023 at 18:08 UTC

10 upvotes, 1 direct replies

I'm so excited for this feature! Hopefully, the 100% of desktop users will make a bigger difference, but I suspect we won't see too big a difference as mods until it's enforced everywhere.

Does the 100% of desktop only include new Reddit, though? What about old and Shreddit?

Comment by StOoPiD_U at 11/12/2023 at 18:35 UTC

4 upvotes, 0 direct replies

/r/FreeGameFindings would be interested in trying this out as well.

Comment by Moggehh at 11/12/2023 at 20:08 UTC

3 upvotes, 0 direct replies

Could you add /r/AmItheCloaca to the pilot? We'd love to start using this to proactively catch posts about the rainbow bridge.

Comment by travjhawk at 11/12/2023 at 17:12 UTC

7 upvotes, 0 direct replies

r/britishcolumbia would join

Comment by lolbot-10000 at 12/12/2023 at 09:12 UTC

3 upvotes, 0 direct replies