. Instrument services with logging, tracing, and metrics (latency percentiles, token usage, error rates) so production behavior...
to continuously improve model accuracy Build model evaluation frameworks with rigorous metrics (F1, precision, recall, false... degradation, and emerging failure modes Implement centralized token usage monitoring for leadership visibility...
-based features: Offline metrics (accuracy, relevance, latency, cost) Human-in-the-loop evaluations where needed Use..., latency, and cost through: Model selection and routing (e.g., small vs. large models, Google vs. OpenAI) Prompt/token...
& Operations Instrument services with logging, metrics, and traces;support monitoring dashboards and alerting. Contribute... (OAuth/OIDC, token handling). Familiarity with agentic concepts: tool/function calling, structured outputs, multi-step...
challenges—such as optimizing LLM token usage, automating model routing, and fine-tuning LLM models—to ensure our platform scales... results of fine-tuning open source LLMs with sharply defined goal metrics. Compare these with similar out of box industry...
, and troubleshoot complex issues Proven track record of exceeding quota or outperforming key metrics Ability to anticipate customer...) Flexible PTO Unlimited AI token usage Centralized home-office equipment ordering Health and wellness stipend...
to standardize Gemini usage across F5 teams. Optimize prompt engineering, token efficiency, grounding strategies, and structured... evaluation pipelines and AI quality metrics. Familiarity with structured output enforcement (JSON schemas, function calling...
, and troubleshoot complex issues Proven track record of exceeding quota or outperforming key metrics Ability to anticipate customer...) Flexible PTO Unlimited AI token usage Centralized home-office equipment ordering Health and wellness stipend...
detection, online/offline metrics, and retraining strategies for AI/ML systems;applies statistical methods and evaluation..., classification, clustering, and time series;ability to design and validate experiments, define online/offline metrics...
and applications Experience in developing metrics, alerts, integrating monitoring solutions, and dashboards Proficiency in Azure... in PingFederate Knowledge of Access Token Management, Access Token Mapping, and OIDC policies in PingFederate Experience in creating...