๐๐ผ๐ ๐ฑ๐ผ ๐๐ผ๐ ๐ด๐ฒ๐ ๐๐๐ -๐พ๐๐ฎ๐น๐ถ๐๐ ๐๐ธ๐ถ๐น๐น ๐บ๐ฎ๐ฝ๐ฝ๐ถ๐ป๐ด ๐๐ถ๐๐ต๐ผ๐๐ ๐๐๐ -๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐ฐ๐ผ๐๐๐? At CIKM2025, Megagon Labs presented ๐ ๐๐๐ฝ๐: ๐๐ฃ๐๐ค๐ง๐ฅ๐ค๐ง๐๐ฉ๐๐ฃ๐ ๐๐๐๐๐ก๐๐ ๐๐๐ฃ๐ฉ๐๐ฃ๐๐๐จ ๐๐ฃ ๐ฝ๐โ๐๐ฃ๐๐ค๐๐๐ง ๐๐ฃ๐๐๐ง๐๐ฃ๐๐ ๐๐ค๐ง ๐๐๐จ๐ฉ ๐๐ฃ๐ ๐ผ๐๐๐ช๐ง๐๐ฉ๐ ๐๐ ๐๐ก๐ก ๐๐๐ฅ๐ฅ๐๐ฃ๐ โ a hybrid approach to large-scale skill mapping in HR.
In skill mapping, we assign ๐ผ๐ป๐๐ผ๐น๐ผ๐ด๐-๐ฑ๐ฒ๐ณ๐ถ๐ป๐ฒ๐ฑ ๐๐ธ๐ถ๐น๐น๐ (e.g., ESCO) to job texts. Bi-encoders are fast enough for millions of postings, but struggle with ๐ณ๐ถ๐ป๐ฒ-๐ด๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ, ๐ผ๐๐ฒ๐ฟ๐น๐ฎ๐ฝ๐ฝ๐ถ๐ป๐ด ๐๐ธ๐ถ๐น๐น๐. Cross-encoders and LLM-based rerankers are more accurate, yet too slow and expensive for production-scale pipelines.
๐ธ๐ก๐ก๐๐ ๐ฏ๐ฟ๐ถ๐ฑ๐ด๐ฒ๐ ๐๐ต๐ถ๐ ๐ด๐ฎ๐ฝ.
It keeps the bi-encoder backbone, but augments its score with a ๐ธ-๐ก๐ก ๐น๐ผ๐ผ๐ธ๐๐ฝ ๐ผ๐๐ฒ๐ฟ ๐น๐ฎ๐ฏ๐ฒ๐น๐ฒ๐ฑ ๐๐๐ป๐๐ต๐ฒ๐๐ถ๐ฐ ๐๐ฒ๐ป๐๐ฒ๐ป๐ฐ๐ฒ๐ stored in a memory bank. At inference time, the model:
- Encodes the input sentence and candidate skills with a bi-encoder
- Retrieves the k nearest synthetic sentences (each labeled with a skill)
- ๐๐ป๐๐ฒ๐ฟ๐ฝ๐ผ๐น๐ฎ๐๐ฒ๐ bi-encoder similarity with similarity to these nearest labeled examples
This lets the system:
- Better separate ๐๐ฒ๐ฟ๐ ๐๐ถ๐บ๐ถ๐น๐ฎ๐ฟ ๐๐ธ๐ถ๐น๐น๐
- ๐๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐น๐ถ๐๐ฒ ๐๐ผ ๐๐ป๐๐ฒ๐ฒ๐ป ๐๐ธ๐ถ๐น๐น๐ by updating the memory, not retraining the encoder
- Stay ๐ต๐ถ๐ด๐ต-๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐ ๐ฎ๐ป๐ฑ ๐ฝ๐ฟ๐ผ๐ฑ๐๐ฐ๐๐ถ๐ผ๐ป-๐ณ๐ฟ๐ถ๐ฒ๐ป๐ฑ๐น๐
On three benchmark datasets built from real job postings, kNNBE rivals ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ฟ๐ฒ๐ฟ๐ฎ๐ป๐ธ๐ฒ๐ฟ๐ ๐ถ๐ป ๐ฎ๐ฐ๐ฐ๐๐ฟ๐ฎ๐ฐ๐ while remaining ๐ผ๐ฟ๐ฑ๐ฒ๐ฟ๐ ๐ผ๐ณ ๐บ๐ฎ๐ด๐ป๐ถ๐๐๐ฑ๐ฒ ๐ณ๐ฎ๐๐๐ฒ๐ฟ, making it practical for labor-market-scale deployments.
Weโve also ๐ผ๐ฝ๐ฒ๐ป-๐๐ผ๐๐ฟ๐ฐ๐ฒ๐ฑ ๐ธ๐ก๐ก๐๐ on the ๐ ๐ฒ๐ด๐ฎ๐ด๐ผ๐ป ๐๐ฎ๐ฏ๐ ๐๐ถ๐๐๐๐ฏ, so research scientists and engineers can plug this hybrid memory + bi-encoder idea into their own retrieval and skill-mapping stacks.
If youโre exploring retrieval, representation learning, or HR tech, this is an example of innovation through research with a clear path to real-world, human-centered impact.
Read the paper!
