I’m still not happy with our profile scores and plenty of brewers tell me they aren’t happy either. Here are the issues/problems that I want to solve:
- Brewers only care about how they can expect a seller to perform today. They are not interested in a seller’s performance months or years ago.
- Brewers don’t want to be penalized for failing to rate sellers and/or lots.
Here are the changes that I’m considering to further simplify profile scores and address the issues above:
- Eliminate profile scores for buyers
- Eliminate the Community Engagement metric
- For the remaining 4 metrics, calculate averages based on the 5 most recent events only:
- Reviews = % of 5 most recent ratings I received that were positive
- Reliability = % of my last 5 sales that I canceled
- Shipping Speed = 100% if the carrier pickup scan is < 24hrs (excluding weekends) since the Hops Sold email is sent to seller. Subtract 11% for each additional day.
- Response Time = 100% - (1 / 7200 x (time of seller’s 1st response - time of buyer’s 1st message for a given order or listing))
I like using a number of most recent events because that incentivizes merchants & growers to always stay on top of things but it also works for the brewery or small farm that only sells occasionally. Using any given time period doesn’t work well for the latter.
How does everyone feel about this approach? Is it fair? Is it ideal? Is 5 the right number of events? Some sellers might have 5 orders in an hour, others might take a month or more to hit 5. I don’t want to make the number of events very high for several reasons:
- Scores need to accurately reflect current performance
- Most sellers will only get a review or message for a small % of their orders
- Occasionally, we notice sellers who suddenly become unreliable and/or unresponsive, so it’s important for sudden drops in performance to show up quickly in these scores.
What do you think? Am I missing anything?