LLM Ops Engineer

As an LLMOps Engineer at Quantios, you will play a foundational role in building and operating the company's first generation of Large Language Model–powered agentic products.

Requirements

• Design, implement, and maintain ingestion pipelines for LLM training and retrieval-augmented generation (RAG) datasets.
• Develop and optimise chunking, embedding, enrichment, and indexing processes using LangChain or equivalent frameworks.
• Manage the lifecycle of prompt templates, embedding models, LLM chains, evaluators, and model configurations.
• Support experimentation, evaluation, and benchmarking of foundation models, prompts, and retrieval strategies.
• Build and operate infrastructure for AI components using Azure AI Foundry, Azure OpenAI, Azure App Services, and related cloud services.
• Implement secure hosting for RAG applications, vector databases, and agent runtimes.
• Define and maintain CI/CD pipelines for LLM artefacts (datasets, prompts, model configs, evaluation suites) using Azure DevOps.
• Establish foundational observability for LLM-based systems, including telemetry, latency tracking, cost visibility, and model diagnostics.
• Monitor and surface signals such as hallucination rates, evaluation scores, retrieval quality, and content safety triggers.
• Implement automated evaluation pipelines for prompts, responses, and RAG relevance metrics.
• Ensure LLM quality gates are integrated into CI/CD workflows.
• Apply responsible AI principles in line with Quantios' AI and ISMS policies.
• Ensure privacy, access control, and logging for all model interactions and vector index operations.
• Support red-team style penetration testing for prompt injection, leakage, and model-based social engineering risks.
• Contribute to documenting LLM pipelines, governance patterns, and internal standards.
• Work with AI developers to integrate LLM and RAG components into product features.
• Partner with Portfolio Architects to evaluate new AI technologies, patterns, and architectural approaches.
• Collaborate with Product Owners to shape technical feasibility, performance considerations, and release planning for AI-enabled features.
• Participate in Agile ceremonies, contribute to estimation, and help the team deliver high-quality AI capabilities.
• Stay up to date with emerging tools in LLMOps, RAG optimisation, evaluation methodologies, and vector search technologies.
• Propose improvements to scalability, model performance, prompt engineering practices, and developer workflows.
• Contribute to establishing early LLMOps best practices that will scale as the organisation's AI capability grows.

Benefits

• Competitive salary
• Stock options
• Health insurance
• Retirement plan
• Generous parental leave
• Flexible work hours
• Professional development opportunities
• Casual work environment