Hosted with nbsanity. See source notebook on GitHub.

Open In Colab

Advent of Haystack - Day 6

Task 1 - Santa’s Workshop: Optimal Elf Assignment

There’s always so much to do to prepare for Christmas! All the toys need to be ready, the visitors to Christmas village have to be welcomed, and the assembly line machines need to be maintained.

Fortunately, Santa has learnt a very important skill throughout the years - delegating. This year, he tasked elf Rita to make sure everything runs smoothly!

Elf Rita is managing a growing team of elves, each with unique skills, and an ever-expanding list of holiday tasks. She needs to assign the best elf to each task.

Option1.png

Elf Rita is a big fan of AI technology, and thought the best idea would be to use a ranking model to rank the likelihood of a task being a good fit for an elf. Matching elves to tasks isn’t easy, since some of them require specific skills or personality traits - still, it’s fine if more than one elf is assigned to the same task.

Can you help elf Rita implement this system?

Components to use:

Pre-Requisites

Create a free personal account in https://build.nvidia.com/ by clicking on the top right button. This will give you access to free tokens to access NVIDIA models and complete this exercise. Once you have an account, generate an API key by clicking on any model and choosing Build this with NIM on the top right side.

Environment Downloads

!pip install -q nvidia-haystack gdown
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/391.4 kB ? eta -:--:--   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 389.1/391.4 kB 12.6 MB/s eta 0:00:01   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 391.4/391.4 kB 8.6 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/109.8 kB ? eta -:--:--   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.8/109.8 kB 6.9 MB/s eta 0:00:00
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/54.8 kB ? eta -:--:--   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.8/54.8 kB 3.4 MB/s eta 0:00:00

Datasets

  1. task1_elf_profiles.csv: a short description of each elf’s expertise, traits, and preferences
  2. task1_descriptions.csv: a description of what each task involves
  3. task1_solution_key.csv: the optimal elf-task assignment (don’t peek!🫣)
  4. task2_parent_notes.xlsx: the messages that Santa got from the parents all over the world requesting to leave the gifts in specific places
  5. task2_solution_key.xlsx: the right assignment of messages-places (don’t open!🔒)
import gdown

url = "https://drive.google.com/drive/folders/1Oxn9M7cpIeuDlQ4tn4wUvqAKVRW0bNTC?usp=sharing"
output_dir = "challenge_files"

gdown.download_folder(url, quiet=True, output=output_dir)
['challenge_files/task1_descriptions.csv',
 'challenge_files/task1_elf_profiles.csv',
 'challenge_files/task1_solution_key.csv',
 'challenge_files/task2_parent_notes.xlsx',
 'challenge_files/task2_solution_key.xlsx']

Choose how to add your API key.
Option 1: add it directly to the cell below
Option 2: add the API key to your Colab environment by hitting the “Secrets” tab on the left hand side of this notebook. Choose Add new secret, add NVIDIA_API_KEY as name, with your key as value.

import os
from google.colab import userdata


os.environ["NVIDIA_API_KEY"] = userdata.get("NVIDIA_API_KEY")

Task 1

Complete the code below to use Haystack and a ranking model from NVIDIA to match the best task to each elf.
Your tasks are marked with ### COMPLETE ### comments.

Data Loading

import pandas as pd

elf_profiles_df = pd.read_csv("challenge_files/task1_elf_profiles.csv")
elf_profiles_df.head(3)
Elf Name Profile Description
0 Jack Loves theatrics and excels at bringing magic t...
1 Sally Fantastic at sewing and enjoys creating colorf...
2 Zero Knows all about Christmas history and is very ...
task_descriptions_df = pd.read_csv("challenge_files/task1_descriptions.csv")
task_descriptions_df.head(3)
Task ID Task Description
0 1 Paint intricate details on 100 wooden toys.
1 2 Act as a tour guide for visitors to Christmas ...
2 3 Move heavy boxes of toys to the sleigh loading...
elf_profiles = elf_profiles_df.to_dict()

elf_names = list(elf_profiles["Elf Name"].values())
elf_descriptions = list(elf_profiles["Profile Description"].values())

task_descriptions = task_descriptions_df.to_dict()
task_descriptions = list(task_descriptions["Task Description"].values())

Ranker Implementation

from haystack_integrations.components.rankers.nvidia import NvidiaRanker
from haystack import Document
from haystack.utils import Secret

tasks = []

for idx, t in enumerate(task_descriptions):
    tasks.append(Document(content=t, id=idx + 1))

ranker = NvidiaRanker(
    model="nvidia/nv-rerankqa-mistral-4b-v3",
    api_key=Secret.from_env_var("NVIDIA_API_KEY"),
)
ranker.warm_up()

all_elf_compatibility = []
for i, elf in enumerate(elf_descriptions):
    # For each elf, run the ranker model on all tasks to find the best match
    elf_compatibility = ranker.run(elf, tasks, top_k=1)
    all_elf_compatibility.append(elf_compatibility)
tasks[0]
Document(id=1, content: 'Paint intricate details on 100 wooden toys.', score: -9.515625)
elf_descriptions[0]
'Loves theatrics and excels at bringing magic to life.'
from IPython.display import display, Markdown
# Printing the solution! - Best task for each elf
for i in range(len(all_elf_compatibility)):
    display(
        Markdown(
            f"The best task for elf **{elf_names[i]}** is: *{all_elf_compatibility[i]['documents'][0].content}*"
        )
    )

The best task for elf Jack is: Design and sew 100 elf uniforms for the holiday season.

The best task for elf Sally is: Design and sew 100 elf uniforms for the holiday season.

The best task for elf Zero is: Act as a tour guide for visitors to Christmas Village.

The best task for elf Mayor is: Carve and assemble 100 custom wooden trains.

The best task for elf Oogie is: Host storytime sessions for children visiting Christmas Village.

The best task for elf Lock is: Prepare and decorate 200 holiday cookies for Santa’s snack station.

The best task for elf Shock is: Fix the glitchy conveyor belt in the toy assembly line.

The best task for elf Barrel is: Move heavy boxes of toys to the sleigh loading area.

The best task for elf Finkelstein is: Paint intricate details on 100 wooden toys.

The best task for elf Sandy is: Groom and care for Santa’s reindeer, ensuring they’re ready for the big night.

# Repeat the exercise with an alternative ranker model in build.nvidia.com
# TIP💡: If you used nv-rerankqa-mistral-4b-v3, use llama-3.2-nv-rerankqa-1b-v1 now or vice-versa)
alternative_ranker = NvidiaRanker(
    model="nvidia/llama-3.2-nv-rerankqa-1b-v1",
    api_key=Secret.from_env_var("NVIDIA_API_KEY"),
)
alternative_ranker.warm_up()

alternative_all_elf_compatibility = []
for i, elf in enumerate(elf_descriptions):
    # For each elf, run the alternative ranker model on all tasks and append them to the list alternative_all_elf_compatibility
    elf_compatibility = alternative_ranker.run(elf, tasks, top_k=1)
    alternative_all_elf_compatibility.append(elf_compatibility)
# Printing the solution! - Best task for each elf
for i in range(len(alternative_all_elf_compatibility)):
    display(
        Markdown(
            f"The best task for elf **{elf_names[i]}** is: *{alternative_all_elf_compatibility[i]['documents'][0].content}*"
        )
    )

The best task for elf Jack is: Rehearse and direct a dramatic holiday performance for Christmas Village.

The best task for elf Sally is: Design and sew 100 elf uniforms for the holiday season.

The best task for elf Zero is: Act as a tour guide for visitors to Christmas Village.

The best task for elf Mayor is: Carve and assemble 100 custom wooden trains.

The best task for elf Oogie is: Host storytime sessions for children visiting Christmas Village.

The best task for elf Lock is: Prepare and decorate 200 holiday cookies for Santa’s snack station.

The best task for elf Shock is: Fix the glitchy conveyor belt in the toy assembly line.

The best task for elf Barrel is: Move heavy boxes of toys to the sleigh loading area.

The best task for elf Finkelstein is: Carve and assemble 100 custom wooden trains.

The best task for elf Sandy is: Groom and care for Santa’s reindeer, ensuring they’re ready for the big night.

Ranker Performance Comparison

from haystack.components.evaluators import DocumentMRREvaluator

# Getting the ground truth: optimal task-elf placement
ground_truth = pd.read_csv("challenge_files/task1_solution_key.csv").to_dict()
ground_truth = list(ground_truth["Task ID"].values())

# Getting the ranking and alternative_ranking actual results
ranking = [
    all_elf_compatibility[index]["documents"][0].id
    for index in range(len(all_elf_compatibility))
]
alternative_ranking = [
    alternative_all_elf_compatibility[index]["documents"][0].id
    for index in range(len(alternative_all_elf_compatibility))
]
# Compare the results of the two ranker models using the Mean Reciprocal Rank (MRR) score. This is a ranking quality metric that only considers the position of the first relevant item in the ranked list.
mrr_evaluator = DocumentMRREvaluator()

ground_truth_documents = []
ranking_documents = []
alternative_ranking_documents = []

for idx, gt in enumerate(ground_truth):
    ground_truth_documents.append([Document(content=gt)])
    ranking_documents.append([Document(content=ranking[idx])])
    alternative_ranking_documents.append([Document(content=alternative_ranking[idx])])

### COMPLETE ###
# Run the mrr_evaluator over the original and the alternative rankers' results
result_ranker = mrr_evaluator.run(
    ground_truth_documents=ground_truth_documents,
    retrieved_documents=ranking_documents,
)
result_alternative_ranker = mrr_evaluator.run(
    ground_truth_documents=ground_truth_documents,
    retrieved_documents=alternative_ranking_documents,
)

print(f"MRR for the first ranker is {result_ranker['score']}")
print(f"MRR for the second ranker is {result_alternative_ranker['score']}")
MRR for the first ranker is 0.9
MRR for the second ranker is 0.9
result_ranker
{'score': 0.9,
 'individual_scores': [0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]}
result_alternative_ranker
{'score': 0.9,
 'individual_scores': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0]}

Task 2 - Santa’s Workshop: Delivery Organiser

Congratulations! With your help, elf Rita managed to assign the best task for each elf, making everyone happy and preserving the spirit of Christmas.

Santa is very happy with this outcome. To reward elf Rita for doing a great job, he makes her responsible for more work (with the vague promise of a promotion in the near future).

The new task is this: every year, Santa receives Christmas letters from children. On the backs of the letters, he gets notes from parents with instructions on where to leave the gifts. These notes, written in languages from all over the word, give indications of where best to leave the gifts. The options are:

  1. Under the Christmas Tree
  2. Under the Children’s Beds
  3. Near the Fireplace
  4. In the Stockings

Elf Rita does not know all the languages in the world - however, she knows AI! Help her by using a multilingual embedder and a cosine similarity algorithm to match the diverse descriptions to their categories.

Task 2

Complete the code snippets below to use Haystack and a multilingual embedding model from NVIDIA to group these notes based on their semantic meaning, regardless of the language they are written in.
Again, your tasks are marked with ### COMPLETE ### comments.

Read the Task Data

parent_notes_df = pd.read_excel("challenge_files/task2_parent_notes.xlsx")
parent_notes_df.head(3)
Notes
0 Just leave everything under the tree; it’s tra...
1 Mijn dochter zal het geweldig vinden om 's och...
2 Deja los regalos junto a la chimenea, por favor.
parent_notes = parent_notes_df.to_dict()
parent_notes = list(parent_notes["Notes"].values())

# Based on the task description above, input the possible locations for the presents
# TIP💡: They are 4
location_options = [
    "Under the Christmas Tree",
    "Under the Children's Beds",
    "Near the Fireplace",
    "In the Stockings",
]

Embedding the text

from haystack_integrations.components.embedders.nvidia import NvidiaTextEmbedder

### COMPLETE ###
# Look into the NvidiaTextEmbedder component and choose an embedding model from https://build.nvidia.com/explore/retrieval
embedder = NvidiaTextEmbedder(
    model="nvidia/llama-3.2-nv-embedqa-1b-v2",
    api_url="https://integrate.api.nvidia.com/v1",
    api_key=Secret.from_env_var("NVIDIA_API_KEY"),
)
embedder.warm_up()
embedded_notes = []
for note in parent_notes:
    embedded_notes.append(embedder.run(note))

embedded_locations = []
for l in location_options:
    embedded_locations.append(embedder.run(l))

Computing the similarity between notes and location options

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

max_cos_similarity_per_note = []
for i in range(len(embedded_notes)):
    cos_similarity_per_note = []
    # Extract the embedding vector for the current note
    note_embedding = np.array(embedded_notes[i]["embedding"]).reshape(1, -1)

    for j in range(len(embedded_locations)):
        # Extract the embedding vector for the current location
        location_embedding = np.array(embedded_locations[j]["embedding"]).reshape(1, -1)

        # Compute cosine similarity between the note and location embeddings
        similarity = cosine_similarity(note_embedding, location_embedding)[0][0]
        cos_similarity_per_note.append(similarity)

    # Get the index of the best matching location
    max_cos_similarity_per_note.append(np.argmax(cos_similarity_per_note))

Embedder Performance Evaluation

# Read the ground truth data
ground_truth_data = pd.read_excel("challenge_files/task2_solution_key.xlsx").to_dict()
ground_truth_data = list(ground_truth_data["Key"].values())

print(ground_truth_data)
print(max_cos_similarity_per_note)

### COMPLETE ###
# Calculate the accuracy of the solution by comparing max_cos_similarity_per_note to ground_truth_data
# TIP💡: The accuracy is the number of True Positives divided by all solutions!

# Count the number of correct predictions
correct_predictions = sum(
    1
    for pred, true in zip(max_cos_similarity_per_note, ground_truth_data)
    if pred == true
)

# Calculate accuracy by dividing correct predictions by total number of predictions
accuracy = correct_predictions / len(ground_truth_data)
[0, 0, 2, 3, 2, 2, 1, 1, 1, 2, 0, 2, 3, 3, 0, 1, 0, 1, 2, 2, 3, 1, 2, 1, 3, 0, 3, 3, 0, 0]
[0, 0, 2, 3, 2, 2, 0, 1, 0, 2, 0, 2, 3, 3, 0, 1, 0, 1, 2, 2, 3, 1, 2, 1, 2, 0, 3, 3, 0, 0]
print(f"The accuracy of this embedder is {accuracy}")
The accuracy of this embedder is 0.9

Conclusion:

With your incredible AI skills, Elf Rita was able to successfully manage both Santa’s Workshop and gift deliveries. Everything should now be ready for Christmas!🎄

PS - Due to her customer obsession and ability to innovate and disrupt using AI, Elf Rita got her promotion after all!🌟

Congrats Elf Rita!