As others (including Dave Slusher and Michael Geoghegan) have blogged, Chris McIntyre’s Podcast Alley site is on a path towards self destruction. It began as a fun and amusing site within the podcasting community, but the recent spate of coverage in the mainstream press has elevated Podcast Alley to a position of supposed authority.
A few podcasters asked their listeners, somewhat innocently, to go to Podcast Alley to vote for them. Seeing that IT Conversations was down in the #29 spot (and knowing our shows were far more popular than our rating would suggest) I asked our listeners to vote for us. Hey…it was nothing more than honest campaigning.
But somewhere along the line, two things happed. From what I’ve been told, in order to make the system “fairer” to new podcasts, Podcast Alley allowed one vote per person per day, and some sites encouraged their supporters to “vote early, vote often.” And then some podcasters whom I would refer to as less than scrupulous, encouraged their listeners to actually cast negative votes against other podcasts, even for podcasts they’d never listened to.
So why am I wasting so many bytes on what appears to be so trivial? It’s because that due to that major-media coverage, Podcast Alley has been granted a franchise. Lacking any alternative, journalists and others are turning to Podcast Alley as an authority of podcast popularity. It’s the lazy thing for a reporter to do, and they can cover their butts by writing loophole copy such as, “According to web site Podcast Alley…” rather than survey users for themselves.
There’s only one solution, and that’s for Chris McIntyre to solve the problem. Don’t allow more than one vote (total) per person, and log the verified email and IP addresses. That won’t stop hardcore hackers, but if you don’t do something, Chris, the franchise you’ve been handled will slip through your fingers.
Blogarithms 
To be honest, as a newcomer at Podcast Alley, I was surprised that you were allowed more than one vote at all. But I imagine that there will be quite a few up in arms if such a change is made. And, what if you change your mind? Perhaps use the same system, IP logging and all, but reset ever month/year/etc?
LikeLike
My suggestion: Clear the entire system of votes and limit everyone to one vote per show per person. The problem is that it’s hard to enforce the one vote limit since it’s easy to spoof IP and email addresses, so charge $1 per vote via Paypal with the proceeds either going to charity or to support Podcast Alley. Why not?!
LikeLike
Yes, I mean, the minute savvy listeners (who have heard a dozen or more shows) hear the current #1 ranked show, the credibility of the ranking and the site itself will go right out the window!
They may also want to consider a popular (i.e. “I enjoy it”) ranking AND a separate quality ranking. Honest listeners would not necessarily rank a show the same on both scales.
LikeLike
Why be obsessed with this type of ranking?
of course it is better to limit to one vote since it is fairer but in any case I hope bloggers are a little more intelligent than relying on this type of “voting” to select the blogs they visit.
If not they get what they deserve.
LikeLike
Apparently it has self destructed. I haven’t been able to access the domain at all this evening.
LikeLike
I’m new to podcasting and haven’t been able to get to Podcast Alley. 😦
traceroute podcastalley.com
traceroute: unknown host podcastalley.com
LikeLike
I’ve been a member of Podcast Alley for a month plus now and I’ve never once downloaded a show and listened to it based off of it’s ranking at the Alley. I look to see what others are saying about it first then listen to it. This goofy pissing contest over ranking is just that…goofy. While it’s nice to have some recognition, the fact is that the shows with larger listenership will always get better rankings than the ones that have lower listenership or even new casts that have a very, very small audience to start with.
I’m sorry but anyone with half a melon in their skull will realize that a shows ranking does not mean it is the best show out there. I’ve listened to the current #1 ranking show and while I don’t mind it, I don’t listen to it regularly since it is not my cup of tea.
Just my two cents – by the way, whatever happened to the cents sign on typewriters and keyboards? What a shame to have to fine a wingding for it instead of it being a key. Hell, ~ is on the keyboard and I’ve never used it. ~ should be replaced with the cents symbol 😉
LikeLike
Doug I have enjoyed your thoughts on voting and podcast directories. I would like to get an honest opion of our voting methods at http://www.podcastpickle.com
Thabk you
Gary
LikeLike
Gary, I went to your site but couldn’t find anything about voting.
LikeLike
I am sorry I was not more specific. You must go to a podcast on the site to vote and see the voting method. If you login and go to a podcast you will get a better idea of the system. I will try and expain it. I will use casts number 1,2,3 in the top 25 section as examples. Cast number 1 has 18 votes with an average rating in the 4’s. The cast in 2nd place has more votes with 20, but has a lower average in the 3’s. Now cast number 3 has a rating of 5 higher than the 1st or 2nd place cast. It only had 5 votes, to cast number 1s 18 and cast number 2s 20 so they rank higher because of the number of votes is so much higher.
LikeLike
Actually there are statistical methods you could adopt that would be much better based on the distribution of voting. You have three values: (a) the number of votes, (b) the average (mean) of those votes, and (c) the distribution of those votes. (a) and (c) determine the accuracy of (b). The most accurate rating is when you have a large number of votes all within a short range (the standard deviation) of the mean. As the number of votes drops *or* the distribution of the votes spreads, the accuracy is diminished. It’s possible to create a threshold at any level that takes into account all three variables, and below that threshold you have to say. “I don’t know because I don’t have enough data or the data varies too widely.” If you’re not a statistics junkie, you can still probably come up with some algorithms that works, so long as you’re willing to say “I don’t know” for perhaps the majority of sites you rank, because you probably *won’t* have a statistically significant for most podcasts. (The “long tail” is where comparisons fail, and that’s where most podcasts live.)
So why do Amazon ratings work so well? Because they’re not usually used to compare one book to another. A rating comes with a review, and it’s the review that’s truly valuable. The number of stars is merely an indication of the tone of the reviews, not a statistically significant number. Yes, you can search Amazon for highest-rated in a category, for example, but that’s far less accurate and merely a good way to get to the reviews of some of the books.
What I’d like to see isn’t a “Top ” site, but rather an Amazon-style site with good reviews and ratings. I don’t see the value, for example, in comparing Dawn and Drew with Reel Reviews or Coverville, but I’d like to know how each is rated by its listeners and why.
LikeLike