Additional Threshold Algorithms #12

New issue

Closed

opened 2026-03-17 18:14:51 +00:00 by leon-adsk · 2 comments

leon-adsk commented

2026-03-17 18:14:51 +00:00

Owner

We will allow explicitly setting a threshold via flag, see issue #9. We will also implement the following methods to be selected via CLI args; these require a dataset:

Evaluate ROC (Youden's J): Find the threshold that maximizes (True Positive Rate - False Positive Rate).
Evaluate Confusion Matrix Metrics: F1, Recall, Precision, Accuracy, etc.
FBeta Score with Beta = 2
Evaluate Z-Score: Calculate μ+kσ on the reconstruction loss of strictly benign traffic to set the boundary.
Evaluate Median Absolute Deviation (MAD): Use MAD on benign traffic to set a threshold that is robust against occasional normal outliers.
Evaluate Percentile Cutoff: Pin the threshold to a fixed percentile (e.g., 99th or 99.9th) of benign validation losses based on acceptable false positive rates.

Originally posted by @leon-adsk in #5 (comment)

We will allow explicitly setting a threshold via flag, see issue #9. We will also implement the following methods to be selected via CLI args; these require a dataset: - [x] Evaluate ROC (Youden's J): Find the threshold that maximizes (True Positive Rate - False Positive Rate). - [x] Evaluate Confusion Matrix Metrics: F1, Recall, Precision, Accuracy, etc. - [x] FBeta Score with Beta = 2 - [x] Evaluate Z-Score: Calculate μ+kσ on the reconstruction loss of strictly benign traffic to set the boundary. - [x] Evaluate Median Absolute Deviation (MAD): Use MAD on benign traffic to set a threshold that is robust against occasional normal outliers. - [x] Evaluate Percentile Cutoff: Pin the threshold to a fixed percentile (e.g., 99th or 99.9th) of benign validation losses based on acceptable false positive rates. _Originally posted by @leon-adsk in https://git.leon-adsk.com/leon-adsk/http-anomaly-detection/issues/5#issuecomment-73_

leon-adsk added this to the Release v1.0 milestone

2026-03-17 18:14:59 +00:00

leon-adsk added the

enhancement

label

2026-03-17 18:15:09 +00:00

leon-adsk added this to the http-anomaly-detection project

2026-03-17 18:15:10 +00:00

leon-adsk self-assigned this

2026-03-17 18:15:11 +00:00

leon-adsk started working

2026-03-28 16:25:30 +00:00

leon-adsk commented

2026-03-28 17:05:46 +00:00

Author

Owner

Evaluate Alert Budgeting: Set the threshold dynamically to yield a maximum number of daily alerts matching SOC review capacity.
Load cached threshold (invalidate with different model)

- [ ] Evaluate Alert Budgeting: Set the threshold dynamically to yield a maximum number of daily alerts matching SOC review capacity. - [ ] Load cached threshold (invalidate with different model)

leon-adsk referenced this issue

2026-03-28 17:06:15 +00:00

Advanced Threshold Algorithms #13

leon-adsk referenced this issue from a commit

2026-03-28 17:08:04 +00:00

implement further threshold algorithms #12

leon-adsk commented

2026-03-28 17:09:20 +00:00

Author

Owner

Done, but moved the additional two algos to a new issue because they depend on multi-model support (#3) or aren't clear in their design yet.

leon-adsk closed this issue