Primary competition visual

Alvin Smart Money Management Classification Challenge

Helping Kenya
$3 000 USD
Completed (over 4 years ago)
Classification
511 joined
219 active
Starti
22 Jun 22
Closei
24 Jul 22
Reveali
24 Jul 22
Inappropriate Eval Metric being used?
Platform · 2 Jul 2022, 20:52 · edited ~14 hours later · 10

***Corection, I confused the metric with another competiton***

But there's still an issue on predictions made with no technique at all getting better scores than models with technique. What could be the cause?

----------------------------------------------------------------------------------------------

I suspect the reason we get "good" scores when submitting just ZEROs in the whole submission is that the incorrect Evaluation Metric is being used by Zindi. The threshold metric used assumes the data is balanced, and the data in this case is Imbalanced.

"Although widely used, classification accuracy is almost universally inappropriate for imbalanced classification. The reason is, a high accuracy (or low error) is achievable by a no skill model that only predicts the majority class." -

https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/

Discussion 10 answers
User avatar
Emmanuel360__RAIN

Thanks for the info.

Upvotes 0
User avatar
Raheem_Nasirudeen

Zindi is using Log_loss not Accuracy.

Upvotes 0
User avatar

Just checked, I confused the Metric with the Financial Inclusion Competition.

However, The Scoring Zindi is using here does seem problematic, Whether the submission has all 1s or all 0s it gives the same score, a better score than any model I'm running. That can't be appropriate?

Upvotes 0
User avatar
Emmanuel360__RAIN

I guess what you focus on is the probability that the classes ur model is predicting is actually correct with a minimal error (log loss).

Upvotes 0
User avatar
Emmanuel360__RAIN

In this regards, ur score should lie between 0 and 1, i.e, u should be getting something like 0.44, 0.33167, 0.7532. that's just aw it should look

Upvotes 0
User avatar

That's not my issue, I'm finding the scoring on Zindi problematic, others are as well if you read the Discussion Board. A model with no technique scores better than a model with technique

Upvotes 0
User avatar
Emmanuel360__RAIN

I wouldn't say an inappropriate eval metric was used, but i would say an inappropriate test values on Zindi portal mght have been used intentionally or unintentionally.

Upvotes 0
User avatar

I tell you even more guys. When I filled submission not with 0 or 1, not even with numbers, but string (i;ve used 'Zindi' :)) I've got the same result! Something wrong.

Upvotes 0
User avatar

Totally agree with you, we should do imbalanced classification to solve this problem

Upvotes 0