Multimodal Dark-Pattern Detection with YOLO, Text Encoders, and LLM-Ready Consumer-Risk Explanations
- Ryan Shao
- DOI: 10.5281/zenodo.21625016
- ISA Journal of Business, Economics and Management (ISAJBEM)
Dark patterns steer consumers toward decisions they may
not intend and often combine visual hierarchy with manipulative microcopy. This
study presents an auditable multimodal pipeline comprising a YOLO-style branch
for deceptive interface components, sparse word and character encoders for
e-commerce text, and a structured consumer-risk-card layer. The empirical
evaluation uses 2,356 unique ec-darkpattern strings (1,178 positive and 1,178
negative) grouped by 1,248 page identifiers to limit page-level leakage. The
best binary model, a word-plus-character linear support vector machine,
achieved 0.958 accuracy, 0.966 precision, 0.950 recall, 0.958 F1, and 0.988
area under the receiver operating characteristic curve. A probability ensemble
produced the highest area under the curve (0.990) and a Brier score of 0.035.
For positive-category classification, character n-grams reached 0.955 accuracy
and 0.955 weighted F1, although macro performance remained lower for the rare
forced-action and sneaking classes. The DarkLens package was also audited: its
documentation declares 1,960 YOLO-format samples and five visual classes, but
the available archive contains a Git-LFS pointer rather than materialized
images and labels; consequently, no visual detection metrics are claimed.
Deterministic templates generated schema-valid risk cards for every record,
while risk-tier accuracy (0.593) and macro F1 (0.364) show that score-to-harm
mapping requires human validation. The results support compact leakage-aware
text detection and transparent data-readiness reporting, but not yet an
end-to-end visual benchmark or validated consumer intervention.
Keywords: dark patterns, multimodal detection, interface microcopy, probability calibration, consumer-risk cards.