Site Notes

New RTH Comment Voting System

The purpose of comment scoring is to encourage and reinforce civility, respect, and intellectual honesty in commenting.

By Ryan McGreal
Published May 11, 2009

Raise the Hammer is an online community of choice dedicated to urban revitalization and sustainable economic development in Hamilton, Ontario. One important form of participation on the site is in the comments on posted articles and blog entries.

Due to a recent escalation in the incidence and disruptive influence of "trolls" - comments posted for the express purpose of disrupting and undermining the discussion. One result has been that many RTH readers have become reluctant to: a) post comments for fear of getting trolled; or b) bother reading the comments as they increasingly constitute fruitless debates with a troll.

As a result, RTH proposed implementing a system of community-based comment moderation based on votes by registered users. The overwhelming response we received, both in the comments on that post and in direct correspondence, has been enthusiastic support for this system of user moderation.

However, we also received some comments that expressed concerns about a) suppressing free expression and b) RTH users 'gaming' the system with multiple sock-puppet accounts. The design of the comment moderating system attempts to address these valid concerns; and it will be a work in progress, adapting if and when problems emerge.

Comment Scoring System

Here is how the system will work:

  1. Registered users will be able to vote on comments - a simple vote up or down on each comment. You can always change your vote after the fact. Clicking the same arrow again cancels the vote, and clicking opposite arrow reverses the vote.

  2. Registered users will have an option to set a comment score threshold - a score below which comments will be hidden by default. The default is to show all comments, and even hidden comments can be revealed by clicking a button on the hidden comment.

  3. Registered users will also have an option to turn comment voting off entirely, so they only see the comments themselves, not their scores.

  4. No registered user will be able to change the settings for what another user can see. Each user will choose their own settings in their user profile.

  5. Anonymous users will still be able to leave anonymous comments and will see all posted comments (regardless of score) plus comment scores, but will not be able to vote. To do that, they will have to register a user account.

Commenting and Voting Etiquette

The purpose of comment scoring is not to downvote unpopular opinions but to encourage and reinforce civility, respect, and intellectual honesty in commenting. Toward that end, we propose the following guidelines:

  1. Please DO:

    1. Post comments that contribute meaningfully to the discussion. Good comments: add new information, correct misinformation, offer a new perspective on an issue, challenge a conclusion already drawn, or propose a resolution to an existing debate.

    2. Upvote comments that are well-researched and reasonably argued, regardless of whether you agree with the conclusion or how you feel about the person who wrote it. If someone known for trolling posts a reasonable comment, encourage more reasonable comments with an upvote instead of dismissing or punishing it because of who wrote it.

    3. Downvote inappropriate comments. Inappropriate comments: use rude or insulting language, are needlessly inflammatory, seek to provoke an emotional reaction from others, are attempts to disrupt and derail the discussion, are offtopic or irrelevant, or abuse evidence and reasoning to defend an unjustifiable conclusion.

    4. Write in defence of a comment you feel is being downvoted unfairly just because it draws an unpopular conclusion or was written by someone controversial.

  2. Please DO NOT:

    1. Do not post inappropriate comments that: use rude or insulting language, are needlessly inflammatory, seek to provoke an emotional reaction from others, are attempts to disrupt and derail the discussion, are offtopic or irrelevant, or abuse evidence and reasoning to defend an unjustifiable conclusion.

    2. Do not post comments just to agree or disagree with another comment. "Me too" posts don't add to the conversation and make threads long and awkward to navigate.

    3. Do not post comments to complain about being downvoted. It's off-topic and disrupts the conversation.

    4. Do not bait people to downvote you for posting an inflammatory comment.

    5. Do not post comments to complain that an article or comment is "biased" because it promotes an urban agenda. That's the whole point of this site.

    6. Do not post multiple consecutive comments. It's rude to monopolize the comment area and crowd out other potential commenters.

    7. Do not vote based on whether you agree with the author's opinion. Comments should be rated for the quality of their argument, not which side they've taken.

    8. Do not vote based on who wrote a comment. Try to judge each comment on its own merit.

    9. Do not create a phony "sock puppet" RTH account just to add upvotes for your comments or downvotes for someone else's comments.

It is up to all of us as members of an online community to reflect and reinforce our community standards of respect, civility and intellectual curiosity in our exchanges with each other.

We all have a responsibility to avoid using argumentative fallacies, to maintain a civil and respectful tone with each other, to make a sincere attempt to contribute meaningfully to the discussion, and to encourage others in the same responsible behaviour.

Ryan McGreal, the editor of Raise the Hammer, lives in Hamilton with his family and works as a programmer, writer and consultant. Ryan volunteers with Hamilton Light Rail, a citizen group dedicated to bringing light rail transit to Hamilton. Several of his articles have been published in the Hamilton Spectator. He also maintains a personal website and has been known to post passing thoughts on Twitter @RyanMcGreal.

29 Comments

View Comments: Nested | Flat

Read Comments

[ - ]

By Ryan (registered) - website | Posted May 11, 2009 at 13:19:24

Additional note: at the same time, we tweaked the algorithm that displays comments to speed up page loading on articles with lots of comments, and we also fixed the comments form so that once you log in, you don't have to keep entering your username and password to post each comment.

Reply | Permalink | Context

You must be logged in to vote on this comment.

By WRCU2 (registered) | Posted June 17, 2011 at 17:03:47 in reply to Comment 30976

This new system isn't so new anymore and I must admit IT kinda grew on me. And I really like the structure of the whole RTH site, in particular, the consistency of page formatting by Quandy has allowed me to assemble a collection of simple data processing scripts written in bash, to pinch interesting information from the RTH stash.

For example, near the end of this two year old thread, JonC and UrbanRenaissance mention the idea of rating a topic discussion via vote ratios:

JonC

What would be interesting is a ratio of total comment scores to number of votes (which would theoretically indicate articles with more interesting comments.

UrbanRenaissance

Not a bad idea, assign the rating to the discussion of the article, rather than the article itself.

I agree, that would be a neat idea so I did exactly that and more, for all 3500+ article and blog topics in the RTH store. On my Linux desktop in a Command Prompt X terminal emulator (Bourne Again SHell), a simple command reveals the information we seek:

wrcu2:$ cat article/874/DR
raisethehammer.org/article/874/
New RTH Comment Voting System
78 100 78%

This ain't that big of a deal however and compiling a discussion rating for all RTH submissions took only mere seconds. Nevertheless, there are many other calculations which may be performed with the plethora of OpenData contained in the RTH user-space strata.

As another example, here is a list of all the users who were part of the original discussion two years ago and myself, along with our current ratings determined by ourselves and other registered RTH users:

ARTICLE COMMENTS

Ryan 3280 3588 91%
reuben 69 77 89%
smz180 -4 14 -28%
Meredith 676 776 87%
Brioski8 37 39 94%
nobrainer 1573 1823 86%
jason 3924 4514 86%
z jones 1265 1467 86%
UrbanRenaissance 622 658 94%
Blossomful 1 1 100%
jason 3924 4514 86%
WRCU2 169 571 29%

I am somewhat embarrassed by my own personal score for comments I've made to articles, however, my score is much better on blogs so I guess there's still hope for me yet:

BLOG COMMENTS

Ryan 3179 3517 90%
reuben 95 109 87%
smz180 5 5 100%
Meredith 504 566 89%
Brioski8 35 39 89%
nobrainer 1442 1636 88%
jason 4513 5277 85%
z jones 1214 1364 89%
UrbanRenaissance 780 802 97%
jason 4513 5277 85%
WRCU2 58 86 67%

Great work Ryan and Staff, you and RTH are simply the best, Cheers!

Comment edited by WRCU2 on 2011-06-17 17:06:47

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By reuben (registered) - website | Posted May 11, 2009 at 14:48:25

im very interested in seeing how this works out. nice work getting this up and running so quickly.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By smz180 (registered) | Posted May 11, 2009 at 17:13:34

Comments with a score below -5 are hidden by default.

You can change or disable this comment score threshold by registering an RTH user account.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Meredith (registered) - website | Posted May 11, 2009 at 17:19:08

To date, there's 30 comments made (all at least 2 days old) on this RTH post asking users their opinion of the comment-voting idea: http://raisethehammer.org/blog.asp?id=13...

I don't know if that counts for you, but that's a fairly substantial effort (and response) in my eyes regarding user input.

RTH staff: Thanks for getting this in so quickly and seamlessly -- sometimes these discussions can drag on for weeks, or implementation can be messy. This looks quick and self-explanatory.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Brioski8 (registered) | Posted May 11, 2009 at 18:41:35

Thanks for this Ryan and RTH staff. You inspired me to register finally. As we can all see, the comment system is working already. SMZ's poorly researched and perhaps hasty comment has the negative score it deserves.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By nobrainer (registered) | Posted May 11, 2009 at 21:50:01

Here here! RTH is nothing if not responsive to it's readers/commenters. I'm not sure how complicated the voting is to put in place but kudos for doing it so fast.

Next up: voting on articles themselves...

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By jason (registered) | Posted May 11, 2009 at 21:59:57

good job Ryan. Should be interesting to see how this works.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By smz180 (registered) | Posted May 12, 2009 at 09:38:25

Yup, I read that post. If all opinions voiced we're so valued, I'd really like to know which additition functionality was added after that dialog and which user comments they were based on?

My opinion getting "bad ratings" based some assumptions of what I've read is half of why I'm posting this to begin with. Just because an opinion is contrary to most users doesn't mean it's less valid.

Lastly, for those who voted the dreaded "Negative". Exactly where does my comment fail in on the Comment Voting Etiquette above?

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By reuben (registered) - website | Posted May 12, 2009 at 10:08:22

i agree, smz180, the negative ratings on your comments are unwarranted. especially the negative votes on your followup comment questioning why.

this is not a good way to start with this new system, everyone! i dont think smz180 was trolling, just stating an opinion.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Ryan (registered) - website | Posted May 12, 2009 at 10:11:05

smz180 wrote:

If all opinions voiced we're so valued, I'd really like to know which addition functionality was added after that dialog

The additional functionality will all be rolled out in stage two, but specifically:

  • Added Comment Voting Etiquette to encourage responsible use;
  • Registered users will be able to turn off comment scoring completely, so they only see the comments themselves; and
  • Stated explicitly that this is a work in progress, open to future changes based on how it works in practice.

Just because an opinion is contrary to most users doesn't mean it's less valid.

I'm just guessing, but I suspect it has to do with what sounded like an accusatory tone. You wrote, "It was obviously decided before you asked any users their opinions" and "I'd rather you went ahead and updated the site however you see fit, than falsely allude to inclusiveness." [emphasis added]

In fact, I proposed the idea in specific response to a user comment complaining about a lack of user moderation and in general response to a significant uptick in similar sentiments of late.

When I wrote the blog that formally proposed the comment scoring system, I was fully prepared to back away from the idea if a large part of the community opposed it - registered and non-registered users alike.

With the overwhelmingly positive response in both comments and direct emails, I decided to go ahead and develop it over the weekend and implement it in two stages.

As for how quickly the function arrived, it was a simple implementation. I had to alter two database tables, add a table, and write maybe 50 lines of code - and that includes optimizing the string method that generates the list of comments and fixing the login process for commenting.

Again, I'm prepared to modify or even remove the function if it turns out not to work. In the meantime, let's try it out and see how it works.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By reuben (registered) - website | Posted May 12, 2009 at 10:15:22

i had one note on the etiquette list above:

DO NOT: Post comments just to say you agree with another a comment. Use the upvote arrow for that. DO NOT: Downvote a comment just because you disagree with it.

are these not conflicting points? if agreement earns a vote up, shouldnt that be balanced by disagreement earning a vote down? in my opinion, voting should never be used for agreement or disagreement of what is being said.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By z jones (registered) | Posted May 12, 2009 at 10:18:29

Well, I registered an account, so thanks for making it worth my while.

@smz180 -

I downvoted your first comment which basically accused the site admins of lying. And then I upvoted your second comment which asked a fair question and pointed out a problem with the voting.

This is an important issue that the users can deal with by encoraging responsible voting on each other. I bet you'll see people vote one way, then read a reply and reverse there vote, that's what I see on other sites that have comment voting.

Also, don't feed the trolls! I've been saying this for awhile now but now instead of feeding the troll you can just downvote and move on.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Ryan (registered) - website | Posted May 12, 2009 at 10:22:49

reuben wrote:

are these not conflicting points? if agreement earns a vote up, shouldnt that be balanced by disagreement earning a vote down? in my opinion, voting should never be used for agreement or disagreement of what is being said.

Heh, good point. :)

I guess what I was trying to suggest is that a good comment adds something to the discussion, rather than just a "me, too" sentiment. I'll try to think of better wording - and please feel free to make a suggestion if you have anything in mind.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By UrbanRenaissance (registered) | Posted May 12, 2009 at 10:41:26

Ryan wrote: "I'll try to think of better wording - and please feel free to make a suggestion if you have anything in mind."

How about...

DO NOT: Post comments just to say you agree/disagree with another comment. "Me too" posts don't add to the conversation and make threads long and awkward to navigate. DO NOT: Vote on comments based on the author's opinion. Comments should be rated for the quality of their argument, not which side they've taken.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Ryan (registered) - website | Posted May 12, 2009 at 10:49:22

Excellent, UrbanRenaissance. I've updated the article with your suggestion.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By UrbanRenaissance (registered) | Posted May 12, 2009 at 11:04:01

One more small thing Ryan, in the "DO NOTs" section the change in item c now makes item d redundant.

Though I suppose you could always replace it with

DO NOT: Nitpick the Editor

:)

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By reuben (registered) - website | Posted May 12, 2009 at 11:09:14

I bet you'll see people vote one way, then read a reply and reverse there vote, that's what I see on other sites that have comment voting.

good point zjones. i think that this could happen a lot, and an example of where voting a comment down will encourage the original poster to clarify or re-frame their argument.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Ryan (registered) - website | Posted May 12, 2009 at 11:11:00

UrbanRenaissance wrote:

in the "DO NOTs" section the change in item c now makes item d redundant.

Fixed. And by the way -

DO: Nitpick the Editor. :)

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Blossomful (anonymous) | Posted May 12, 2009 at 11:12:52

"Next up: voting on articles themselves"

Seconded. It's only fair...

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By Ryan (registered) - website | Posted May 12, 2009 at 11:30:59

Voting on articles and blog entries: what do people think of this? It wouldn't be that hard to implement.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By jason (registered) | Posted May 12, 2009 at 11:40:15

sure. go for it. Then we'll see which ones of us deserve a raise!

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By UrbanRenaissance (registered) | Posted May 12, 2009 at 11:43:43

I don't think voting on the articles themselves is necessary, or even appropriate.

If the voting is being used as its intended, (to downgrade low quality posts and upgrade high quality posts) then Ryan, as editor is already acting to prevent low quality articles from being added to the site.

If we agree or disagree with an article then we have the comments section to argue all sides of the issue.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By reuben (registered) - website | Posted May 12, 2009 at 11:54:19

i do not think voting on articles is needed. this site is an expression of the editors, the comments are the way readers can provide feedback, or by submitting articles themselves. this site is not a discussion forum where everything and everyone carries equal weight. it is a news source / blog. while there is a lot of two-way communication taking place here, it is primarily setup as a one-way communication tool.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By JonC (registered) | Posted May 12, 2009 at 12:38:47

I don't think voting on the articles is necessary either.
What would be interesting is a ratio of total comment scores to number of votes (which would theoretically indicate articles with more interesting comments.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By UrbanRenaissance (registered) | Posted May 12, 2009 at 12:53:04

JonC wrote: "What would be interesting is a ratio of total comment scores to number of votes (which would theoretically indicate articles with more interesting comments."

Not a bad idea, assign the rating to the discussion of the article, rather than the article itself. To that end why not use the ratio of comments that have positive vote counts to the number of comments with negative vote counts. That way you can see at a glance how derailed a thread has become, and at the same time it would (hopefully) prevent people from overusing the down vote since it would make that thread (and their own arguments) appear less relevant.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By jason (registered) | Posted May 12, 2009 at 18:54:18

I can tell how derailed a thread has become when I log in and see that there are 75 comments in 1 day about Burlingtons NHL arena. People simply need to stop feeding the troll.

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By hammertime (registered) | Posted March 01, 2011 at 16:18:32

I'm with smz 180, this is a stupid Idea. Why not let everyone think for themselves??

Comment edited by hammertime on 2011-03-01 16:19:49

Reply | Permalink | Context

You must be logged in to vote on this comment.
[ - ]

By hobbit (anonymous) | Posted May 26, 2012 at 15:18:03

comment voting and greying out makes this site utterly childish and monotone. I bet if you did a graph of users you'd a core group of 30-50 people providing 90% of the comment voting. I'd also be willing to bet that 90% of that 90 have their comment settings set so that they read every one. All the greying out does is allow one more way for the herd to tarnish the black sheep. It does not achieve its function of keeping 'conversations' on track. It should be replaced with a function where these 30-50 people who love to be in agreement have the option of sorting the comments so they see the top rated comments first. Then those of us who think those of you who vote on comments are ridiculous wouldn't have to be regularly confirmed of this idea.

Reply | Permalink | Context

You must be logged in to vote on this comment.
View Comments: Nested | Flat

Post a Comment

Comment Anonymously
Screen Name
What do you get if you divide 12 by 3?
Leave This Field Blank
Comment

Events Calendar

Recent Articles

Article Archives

Blog Archives

Site Tools

Feeds