Deliver Results (Project Sign-off)
It not only evaluates the end result but also gauges the behavioral traits the candidate demonstrated during the process of delivering those results.
Points to consider while evaluating this competency:
- Did the candidate operate independently to deliver quality software?
- Did the candidate demonstrate effectiveness despite ambiguous or incomplete specification by driving refinement of those specifications?
- Does the candidate take testability into account during business/design reviews?
- Did the candidate identify missing customer scenarios or impact during the feature review?
- Did the candidate came up with traceability matrix to ensure thorough coverage of test scenarios?
- Did the candidate circulate a quality test plan and got it reviewed with relevant stakeholders?
- Was the test plan comprehensive enough to call out the scope, effort estimates, QA deliverables along with timelines, entry/exit criteria, automation plan, risks and mitigation, external dependencies etc.?
- Did the candidate document the use cases with proper steps in a test repository?
- Did the candidate send regular status updates on test execution status explicitly calling out the overall progress (executed vs. pending), blocker issues and path to green?
- Did the candidate prevent any defect from happening through code reviews, identifying edge case/missing scenarios, understanding the impact of bug fixes on regression scenarios etc.?
- Did the candidate find obvious defects early in the development lifecycle thereby reducing code churn due to bug fixes?
- Did the candidate define the Dev/QA engagement model, enforce unit test coverage, contribute towards integration testing etc.?
- Did the candidate perform any non-functional testing like performance, latency, security, accessibility, service level testing etc. as deemed by project requirements?
- Did the candidate collaborate with external/dependent partner teams based on project requirements?
- Did the candidate deliver on automation as part of project sign-off?
- Were there any obvious scenarios missed out during the testing process that resulted in high severity or Sev-2 tickets post launch?
- Did the candidate provide process guidance to developers/TPMs using deep understanding of the entire software development life cycle?
- Did the candidate proactively followed up and documented what went well, areas for improvement and corrective actions as part of the project retrospective meeting?
Automation (Operational Excellence)
It evaluates technical competency demonstrated by the candidate towards developing and maintaining automation scripts and other tooling initiatives for improving QA efficiency.
Points to consider while evaluating this skill set:
- Does the candidate show tendency to prioritize automation work as part of project deliverables?
- Did the candidate follow basic coding standards while developing automation scripts?
- Did the candidate test the automation code changes and provide specific details in testing done section while sending it for review?
- Is the candidate responsive to code review feedback?
- Did the candidate proactively follow-up in maintaining the pass percentage of automated scripts (excluding valid failures or real issues)?
- Were the automated scripts developed scalable enough to be used across locales with minimal changes?
- Did the candidate provide valid review comments on automation CRs submitted by other engineers?
- Did the candidate leverage automation to identify regression issues and improve upon the overall QA cycle time?
- Did the candidate provide any recommendation or feedback on enhancing the automation framework?