LLM Ops Engineer

Join the Legal Tech Revolution at Litera
Are you ready to shape the future of how law is practiced? At Litera, we’re on a mission to Raise The Bar™️ for the legal profession by delivering transformational and globally-trusted solutions to law firms and corporate legal teams worldwide. We’ve been a leader in legal tech innovation for 30 years and are leading the legal AI revolution to this day with most of the world’s largest law firms as our clients. If you’re passionate about building AI-forward solutions that scale globally and want your work to impact millions of legal professionals worldwide, this is your opportunity to be part of an extraordinary team that’s elevating the craft of law.

This position is based in Denver, Colorado or Raleigh, North Carolina and candidates should reside within reasonable commuting distance, as this role requires on-site presence at least three days per week.

About the Role
We are seeking an experienced LLM Engineer to join our newly established LLM Ops Team. In this role, you will be responsible for managing the complex lifecycle of Large Language Models from development to deployment, monitoring, and continuous improvement.

Key Responsibilities:
Model Development & Training
• Fine-tune pre-trained models for specific use cases
• Curate and prepare datasets for training
• Manage training infrastructure, resources, and computational environments
• Implement optimization techniques to improve model performance
Deployment & Serving
• Develop and manage APIs for model serving
• Scale infrastructure to handle varying demand loads
• Build and maintain the GenAI middleware/sidecar layer
• Integrate LLMs with existing systems and data sources
Monitoring & Evaluation
• Track performance metrics including latency and throughput
• Monitor quality metrics such as hallucination rates and accuracy
• Optimize costs associated with model inference and training
• Create and maintain dashboards for real-time performance insights
Testing & Quality Assurance
• Create and maintain golden datasets for benchmark testing
• Implement statistical validation methods for model outputs
• Set up similarity matching criteria for response evaluation
• Develop confidence score thresholds for production systems
Feedback Loops & Iteration
• Design and implement user feedback collection systems
• Establish continuous improvement processes
• Create A/B testing frameworks for model and feature evaluation
• Conduct trace analysis to identify areas for performance optimization
Safety & Compliance
• Implement content moderation systems
• Detect and mitigate bias in model outputs
• Ensure regulatory compliance in AI systems
• Develop output validation frameworks
Prompt Management
• Version and store prompts systematically
• Create and maintain prompt templates
• Set up playground environments for prompt testing
• Abstract prompts from application code for better maintainability

Qualifications
• Experience with LLM development, fine-tuning, and deployment
• Strong programming skills, particularly in Python
• Experience with Kubeflow, Apache Airflow, MLFlow, or other LLM Pipeline technology
• Experience with Azure OpenAI, AWS Sagemaker, and/or Vertex AI
• Understanding of machine learning operations and MLOps principles
• Knowledge of infrastructure scaling and optimization
• Experience with AI monitoring tools and dashboard creation
• Familiarity with AI safety, bias detection, and compliance requirements
• Strong problem-solving abilities and analytical thinking
• Familiarity with ISO 27001 and SOC2 Certification

Why Join Litera?

· The company culture: We emphasize helping each other grow, doing the right thing always, and being part of a journey to amplify impact, creating an exciting and fulfilling work environment

· Commitment to Employees: Our people commitment is based on what employees love most about being part of the team, focusing on tools that matter to the difference-makers in the legal world and amplifying their impact

· Global, Dynamic, and Diverse Team: Our is a global company with ambitious goals and unlimited opportunities, offering a dynamic and diverse work environment where employees can grow, listen, empathize, and problem-solve together

· Comprehensive Benefits Package: Experience peace of mind with our health insurance, retirement savings plans, generous paid time off, and a supportive work-life balance. We invest in your well-being and future, ensuring a rewarding career journey.

· Career Growth and Development: We provide career paths and opportunities for professional development, allowing employees to progress through various technical and leadership roles.

Pay Transparency Notice for Colorado Applicants:

The annual salary range for this position is $140,000 to $180,000. Actual compensation is determined by factors including education, work experience, certifications, and other relevant qualifications. Litera offers a comprehensive benefits package including health, dental, and vision insurance, 401(k) with company contribution, and incentive and recognition programs. All benefits are subject to eligibility requirements.