LLMOps Engineer - Operations & Quality Assurance QA, Test 🏆

LLMOps Engineer - Operations & Quality Assurance 💰 Salary: $70,000 - 80,000 per year

At Acti we are looking for a QA, Test engineer!

🛠️ Our tech stack

AI, API, Architect, CI/CD, CloudWatch, Datadog, DevOps, GitHub, Grafana, Support, Jenkins, LLM, Looker, Metabase, Prometheus, Python, Tester

📝 Rquirements

• 3+ years of experience in DevOps, SRE, MLOps, or a similar operations-oriented engineering role
• Strong proficiency in Python with experience in developing monitoring and observability systems
• Familiar with logging and monitoring tools like Datadog, Grafana, Prometheus, CloudWatch, or similar
• Understanding of LLM APIs, token-based pricing structures, and architectures of AI systems
• Experience in establishing evaluation frameworks, testing pipelines, or quality assurance systems for software products
• Knowledgeable about CI/CD tools and deployment automation methods, such as GitHub Actions, Jenkins, or equivalent
• Strong analytical abilities to identify data patterns and convert them into actionable recommendations
• Direct exposure to LLMOps tools (LangSmith, Weights & Biases, Humanloop, or similar) is a plus
• Experience in managing costs and optimizing resource consumption for API-intensive systems
• Background in creating dashboards and data visualization (Metabase, Looker, custom solutions) is advantageous
• Familiarity with prompt engineering and understanding the impact of prompt variations on model behavior
• Experience with multi-agent systems or orchestration platform monitoring is beneficial
• Knowledge of AI safety, content moderation, and ethical AI deployment practices is preferred

👩‍💻👨‍💻 Your responsibilities are

• Design and implement detailed monitoring and observability systems for all operational AI agents, tracking response quality, latency, error rates, and conversation results
• Build and uphold evaluation frameworks to assess agent performance against established benchmarks, including automated quality scoring and regression identification
• Oversee token utilization, API expenses, and resource distribution across all agents and LLM suppliers, providing regular cost analysis and optimization suggestions
• Develop and maintain robust conversation logging infrastructure for analysis, debugging, and compliance purposes
• Implement hallucination detection, content safety filters, and guardrail systems to safeguard end users and uphold brand integrity
• Create and manage alerting mechanisms for agent failures, performance declines, and unusual behavior patterns
• Establish A/B testing and prompt version control systems to assist the Prompt Architect in ongoing agent enhancements
• Create and maintain CI/CD pipelines for prompt releases, ensuring testing, staging, and safe rollouts of changes
• Develop dashboards and reporting tools to provide leadership with insights into agent performance, return on investment (ROI), and operational health
• Collaborate with the AI/ML Engineer on infrastructure enhancements and with the Solutions Engineer to ensure production reliability

View this job and over 500 other transparent jobs with salaries (💰💰💰) & tech stacks (🛠️) on DevITJobs

Category: QA, Test Developer / Engineer

Location address: From Road 650, Hackensack, United States

Salary: $70,000 - 80,000 per year

Benefits & perks that we offer

Acti - More about us and the role

At Actualized Intelligence (ACTi), we manage an expanding array of AI agents that engage in crucial conversations—from legal outreach to personal coaching. The efficacy, dependability, and cost-effectiveness of these agents play a vital role in our success and our clients results. We are seeking an LLMOps Engineer to take charge of the operational integrity and ongoing enhancement of our entire agent ecosystem. We offer a hybrid remote work environment based in Paramus, NJ, with competitive salaries ranging from $70,000 to $80,000 annually and attractively comprehensive benefits including health, dental, and vision insurance, as well as a 401(k) plan and paid time off.

Are you looking for QA, Test jobs in Hackensack?