Home > Uncategorized > Dealing with #Overfitting in ML.NET (#AI / #ML)

Dealing with #Overfitting in ML.NET (#AI / #ML)

confidence

One of the issues with ML.NET, is that it will tend to be over-eager to provide a prediction, even when there is not enough information to go on, in order to make any sort of accurate prediction. This is where you should also look at the Score Property, and take the Max Value of this array in order to determine the level of confidence.

If you then plot confidence against correct predictions and incorrect predictions, and hopefully you should see that incorrect predictions should lie primarily on the lower end of the confidence scale, and correct predictions should lie on the higher end of the confidence scale.

Ideally, you should see a clearer delimination between correct and incorrect, to allow you say that if confidence is less than .3 (or whatever), then the prediction is unknown, not the wild guess that ML.NET has suggested.

To generate this graph, here is the SQL I used;

create table PercentileGraph
(
id int identity(0,1),
[Start] as cast(id as float)/100,
[End] as cast(id as float)/100 + 0.01,
Correct int,
Incorrect int,
)

while 1=1
begin
insert into PercentileGraph default values
end

— Run for a second, then stop

delete from PercentileGraph where id>100

select * from PercentileGraph

update PercentileGraph set correct =
(
select count(*) from Evaluation where
Model=Predicted
and ModelConfidence>=PercentileGraph.[start]
and ModelConfidence<=PercentileGraph.[end]
)

update PercentileGraph set incorrect =
(
select count(*) from Evaluation where
Model<>Predicted
and ModelConfidence>=PercentileGraph.[start]
and ModelConfidence<=PercentileGraph.[end]
)

select * from PercentileGraph

Categories: Uncategorized
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: