GenAI Text Mining

Problem

A large volume of open-text customer feedback that could not be categorised by hand at the cadence or granularity the business needed.

What was built

• Collected business needs during workshops and designed the solution’s overall architecture.
• Successfully implemented a proof-of-concept solution and presented a High-Level Design document to the client’s architectural council.
• Scaled classification from an 11-category proof of concept to 61 production categories on Google Gemini and VertexAI (Google Cloud Platform).
• Deployed the pipeline on production, orchestrated with Cloud Composer (Airflow) and connected to the client’s Big Data platform.
• Designed and shipped a profanity-censoring model to production (VertexAI, Gemini, BigQuery, Airflow).
• Iterative prompt tuning with domain experts, plus automated sample re-scoring and AI-driven correction passes.
• Business-facing self-service tooling for verifying results and updating evolving category definitions directly in the prompts, without needing the cloud environment or data scientists.
• Built a validation set with business users, used for model-version migrations and testing.
• Designed and implemented a production system for sending email notifications to business stakeholders based on classification data, using a pre-existing email service.

Outcome

• The first model deployed to production on the client’s new MLOps platform, and the client’s first GenAI model overall.
• 1,200–3,200 surveys/day in production.
• 85%+ agreement with expert labels (Jaccard coefficient, multi-label classification).
• Marketing built dashboards and Controlling built its own reporting procedures on top of the output.

Challenges and Lessons

• Simpler models (such as deprecated Gemini 2.0 Flash) tend to hallucinate a lot.
• Prompt engineering can achieve better and cheaper results than fine-tuning.
• Each model can behave differently even though they were created by the same vendor.
• Enabling Gemini 2.0/2.5 Flash’s “thinking” mode does not result in better classification — requesting weighted classification scores in the response yields more accurate assignment.
• The order of response parts in LLMs matters — sometimes it reasons first and then categorizes, other times it returns categories first and explains its reasoning afterward.
• Human classifications can contain errors.