Working at Allegis Global Solutions (AGS) is more than just a job. It’s a career. It’s a community of people who invest in your development and empower you to blaze your own trail. Each of us is here to create real, measurable impact that moves needles. We operate beyond "roles" or "jobs" to realize the opportunity to make meaningful contributions to a bigger idea.
Requirements
- Define testing strategies for AI agents, conversational interfaces, and agentic workflows
- Design behavioral test suites for non-deterministic outputs—where "correct" isn't binary
- Build evaluation frameworks that measure groundedness, factuality, relevance, and task completion
- Identify failure modes specific to AI systems: hallucinations, prompt injection, context window limitations, drift
- Develop testing approaches for each architecture pattern: RAG, function calling, human-in-the-loop, autonomous workflows
- Build automated evaluation pipelines that run as part of CI/CD
- Create test harnesses for LLM-based systems—mocking, fixtures, and reproducible test scenarios
- Develop regression suites that detect quality degradation when prompts, models, or data change
- Build monitoring and alerting for production agent quality (accuracy, latency, error rates)
- Maintain test infrastructure: test data management, environment setup, reporting dashboards
- Define quality metrics for each solution—what to measure and what thresholds matter
- Build and maintain evaluation datasets (ground truth, reference outputs, edge case collections)
- Conduct systematic prompt evaluation when prompts or models change
- Track quality trends over time and identify when re-evaluation is needed
- Report quality metrics to the team and stakeholders in clear, actionable terms
- Partner with AI Solutions Engineers to define testability requirements during design
- Work with AI Solutions Analysts to translate acceptance criteria into test scenarios
- Review solution designs from a quality and testability perspective
- Advocate for quality practices across the team—testing isn't an afterthought, it's part of delivery
- Contribute to incident response by diagnosing quality failures and building regression tests
Benefits
- Generous Paid Time Off
- 401k Matching
- Retirement Plan
- Relocation Assistance
- Tuition Reimbursement
