The 'PROPORTIONAL' option for FEATURE_WEIGHTS option (i.e. weight of each class) is not rendering predictions correctly for SVC models

[paulvpop]

Hello everyone,

I have recently noticed that the 'PROPORTIONAL' option for FEATURE_WEIGHTS option (i.e. weight of each class) is not rendering predictions correctly for SVC models. I have tried several SVC models, by changing the FEATURE_WEIGHTS to EQUALIZED, PROPORTIONAL, ANTIPROPORTIONAL, and it worked for EQUALIZED and ANTIPROPORTIONAL, but not PROPORTIONAL in all cases. It renders most times as complete black with no prediction values in the model prediction output. There must be an issue with this option. All three options work for RFC models. I have tried this both in Windows (10) and Linux/GNU (Ubuntu 24.04)
Please check. Attaching some photographs of the failed model predictions and train parameter

[
parameter_train_svc_six.txt
](url)

[davidfrantz]

Hi @paulvpop,

I am not sure if this is a bug. It could well be that setting the class priors for SVM might tweak the model to produce this result.

Do you see anything suspicious in the logfile?

Have a look at the Support Vector Machine parameters block. The prior information should be printed there

[paulvpop]

Thanks for the reply, David. Why could it be failing for only proportional, and not equivalized or antiproportional? Any idea? The overall accuracy (OA) predictions from the trained files doesn't match with the model predictions:

From the first class to the last (OA %):
86.86
93.50
94.21
91.95
93.08
90.11
93.50
94.63
92.66
91.95
92.66
93.36

Here are the SVM parameters for all of the 13 classes:

Support Vector Machine parameters
--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 99.999997
class weights: 0.879559 0.120441

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.889313 0.110687

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.930450 0.069550

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.939779 0.060221

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.931298 0.068702

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.939355 0.060645

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.908821 0.091179

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.932994 0.067006

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.938507 0.061493

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.927481 0.072519

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.917303 0.082697

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.932994 0.067006

--------------------------------------------------------------------
Termination type: 3
max iterations: 1000000
accuracy: 0.001000
k-fold CV: 10
P: 0.000000
C: 1.000000
Gamma: 0.000010
class weights: 0.932146 0.067854

[davidfrantz]

can you please reformat your post? it is hard to read... You can insert a code block by doing this:

also, you posted the parameter file content, please post the relevant content from the logfile.

[paulvpop]

Sorry. I have edited my comment again.

[davidfrantz]

OK, I do not quite understand what the 13 classes are. From what I can see, you have 13 times 2 classes.

And in each iteration, you have way more samples in the 1st class than in the 2nd. I suspect the first class is always the class with the black color in the images? Maybe, SVM is sensitive to that imbalance and hence produces a classification that mostly predicts the 1st class.

However, I do not see any technical issues from this...

[davidfrantz]

And this also matches the OA values you posted earlier. It seems like class 1 was predicted accurately, and all samples of class 2 were predicted incorrectly.

I guess, PROPORTIONAL is simply not the best choice here

[paulvpop]

These are my classes (13 classes):
Dense forest
Snow
Open forest
Bamboo
Cash crop
Active jhum
Wet rice cultivation
Shadow
Built-up
Alluvial grassland
Montane grassland
Water
Bare surface
Any idea why there are13 times two?

My features txt file is like this:

1288 1398 1665 1293 1424 1743 1254 1371 1657 1900 1989 2207 1951 2029 2060 1545 1595 1615 186 241 3112 886
1088 1111 1167 990 1019 1089 1078 1099 1153 1317 1340 1395 1877 1899 1908 1685 1703 1710 -191 21 1029 108
753 763 785 524 536 565 614 622 641 1399 1406 1425 2012 2024 2029 1527 1541 1546 262 13 3112 204
995 1017 1071 740 763 817 878 901 958 1357 1374 1415 2253 2290 2305 1885 1912 1923 -37 42 1737 285
995 1017 1071 740 763 817 878 901 958 1357 1374 1415 2253 2290 2305 1885 1912 1923 -37 42 1737 285

My response txt file like this (it only goes to 13 classes):

5.000000
5.000000
5.000000
7.000000
9.000000

My coordinates txt file is like this:

94.642646 29.335884
94.632111 29.327361
94.691035 29.306392
94.629078 29.292972
94.629081 29.292962

[davidfrantz]

From what I see here, it appears that you trained 13 separate models, rather than one with 13 classes.

[paulvpop]

Oh, that's interesting. So, in which stage in the workflow is this happening (I had followed the workflow in the documentation with some tweaking to adapt to my workflow, like using the synthetic mixtures for creating the sampling data, but not using the synthetic mixtures)? What options in which parameter file do I use to train it as 13 classes instead of 13 models? Sorry if these are too many questions.

[davidfrantz]

not sure. Did you split the response+features files at some point? You should just have one of each, with the response containing all labels from 1-13

[paulvpop]

No, I had only kept one copy of response and features in the samples folder like in the examples I shared above. I didn't split the files. I presume that my models are correct (barring the SVC for proportional samples for some reason). So, this workflow is working for me. I will investigate where in the workflow it is getting two sets of samples for each class.

[paulvpop]

Hello @davidfrantz I would like to understand which of these 13 models would the model prediction use to create a prediction. If there are 13 models which are trained, is it going to use the first one or use an average of all? I need this information so that I can extract the points which were not used for training so that it can be used for validation.

[paulvpop]

@davidfrantz apologies. I didn't realise that it was answered in the documentation. I missed it. "Within each modelset, the final prediction is generated by aggregating the results of each prediction. For regression, this is the average of the individual model’s predictions. For classification, this is the mode of the individual model’s predictions." So, it's using the mode of predictions. https://force-eo.readthedocs.io/en/latest/components/higher-level/ml/index.html

I have a query on how to avoid making it a model set, and instead just one model. Will ask it as a separate question later.

[paulvpop]

I am uploading the files from another model - Random Forest

Four of the 13 log files from the force-train module are these:
FILE_LOG_001.log
FILE_LOG_002.log
FILE_LOG_003.log
FILE_LOG_005.log
MODEL_CLASS_001.xml
MODEL_CLASS_002.xml
MODEL_CLASS_003.xml
MODEL_CLASS_005.xml

The 13 log files from the force-train module is coming because of the

%SET%: 001 002 003 005 006 007 008 009 010 011 012 013

in the train parameter file:

%SET%: 001 002 003 005 006 007 008 009 010 011 012 013

++PARAM_TRAIN_START++

# INPUT
# ------------------------------------------------------------------------
# File that is holding the features for training (and probably validation).
# The file needs to be a table with features in columns, and samples in rows.
# Column delimiter is whitespace. The same number of features must be given
# for each sample. Do not include a header. The samples need to match the
# response file.
# Type: full file path
FILE_FEATURES = /home/docker/force/force2/no_mixtures_rfc_22/NO_MIX_FEATURES_CLASS-{%SET%}_ITERATION-001.txt
# File that is holding the response for training (class labels or numeric
# values). The file needs to be a table with one column, and samples in rows.
# Do not include a header. The samples need to match the feature file.
# Type: full file path
FILE_RESPONSE = /home/docker/force/force2/no_mixtures_rfc_22/NO_MIX_RESPONSE_CLASS-{%SET%}_ITERATION-001.txt

The parameter train file is this:
parameter_train_rfc_35.txt

Two files from the "no_mixtures_rfc_22" folder:
NO_MIX_RESPONSE_CLASS-001_ITERATION-001.txt
NO_MIX_RESPONSE_CLASS-002_ITERATION-001.txt
NO_MIX_FEATURES_CLASS-001_ITERATION-001.txt
NO_MIX_FEATURES_CLASS-002_ITERATION-001.txt