fix: enforce unique id column values in input dataclasses

[frayle-ons]

resolves #167

✨ Summary

These changes modify the Pandera schemas of the input dataclasses to ensure that the user doesn't supply dataclasses with non-unique id column values. Non unique values in this case adversely affect data processing in the package because the codebase relies on df.groupby('id') statements which assume that the 'id' column has unique values.

I've added unique=True to each of the input dataclass' pandera schemas id colmumn specifications. I've also included a servers module update, to catch and raise Pandera validation errors when JSON body information is converted to dataclass input objects, ensuring users know when a specific Pandera issue arises behind the base Pydantic validation.

(Below) An example screenshot passing non-unique IDs to a VectorStore reverse_search() endpoint

📜 Changes Introduced

• (fix:) Updates to the indexers.dataclasses.py file to specify id columns must be unique
• (fix:) try catch statements added to servers.main.py file

✅ Checklist

Code passes all pre-commit checks, including optional Docker secret checker step.

🔍 How to Test

Below is a simple startup script, using fake data from the ClassifAI repo, that will run a server which the user can pass JSON requests to each endpoint to see the new changes in actions:

from classifai.vectorisers import HuggingFaceVectoriser
from classifai.indexers import VectorStore
from classifai.servers import run_server
from classifai.indexers.dataclasses import VectorStoreReverseSearchInput, VectorStoreSearchInput, VectorStoreEmbedInput


#start a vectoriser
my_vectoriser = HuggingFaceVectoriser(model_name="sentence-transformers/all-MiniLM-L6-v2")


#build a vector store
my_vector_store = VectorStore(
    file_name="./DEMO/data/fake_soc_dataset.csv",
    data_type="csv",
    vectoriser=my_vectoriser,
    skip_save=True,
)

#run a live server for the vectorstore
run_server([my_vector_store], endpoint_names=["test_vectorstore"], log_level="info", demo_mode=True)

The final line of code starts a RESTful API service. It would also be good to call the VectorStore methods programatically in Python instead of over a network. See below some example Python code that could be included in the above script replacing the line which runs the server.

For the search method:

# a good input that always worked
test_input_data_1  = VectorStoreSearchInput(
    {"id": ["1", "2"], "query": ["golden farmer", "golden software engineer"]}
)
print(my_vector_store.search(test_input_data_1, n_results=3))


############################

# duplicate ids would have caused problems before changes
test_input_data_2  = VectorStoreSearchInput(
    {"id": ["1", "1"], "query": ["golden farmer", "golden software engineer"]}
)
print(my_vector_store.search(test_input_data_2, n_results=3))

For the reverse_search method:

# good test data that's always worked
test_input_data_3  = VectorStoreReverseSearchInput(
    {"id": ["1", "2"], "doc_label": ["101", "130"]}
)
print(my_vector_store.reverse_search(test_input_data_3, max_n_results=3))


############################

# duplicate ids can cause issues
test_input_data_4  = VectorStoreReverseSearchInput(
    {"id": ["1", "1"], "doc_label": ["101", "130"]}
)
print(my_vector_store.reverse_search(test_input_data_4, max_n_results=3))

For the embed method:

# another good example that works normally
test_input_data_5 = VectorStoreEmbedInput(
    {"id": ["1", "2"], "text": ["golden farmer", "golden software engineer"]}
)
print(my_vector_store.embed(test_input_data_5))

############################

# duplicate ids can cause issues
test_input_data_6  = VectorStoreEmbedInput(
    {"id": ["1", "1"], "text": ["golden farmer", "golden software engineer"]}
)
print(my_vector_store.embed(test_input_data_6))

The above examples can also be applied to the REST API testing.