Rare Not Random — Using Token Efficiency for Secrets Scanning
Summary
Rare Not Random discusses using Byte-Pair Encoding-based token efficiency as a post-regex filter for secrets scanning, comparing it to entropy. It presents experimental results on CredData, showing token efficiency can improve precision and recall for secret detection, with noted caveats for passwords and short strings, and provides code references and benchmarks.