I’m testing how to add AI agents to Jira-like workflows. The agent reads a ticket, edits the repo, runs the tests and reports back. Each ticket runs in its own git worktree so parallel runs don’t collide. I’m using Sharkly.ai for the orchestration.
Questions for the community:
Which open models handle the “read ticket → small, scoped code change” loop reliably?
Has anyone evaluated agents on real issue-tracker tasks rather than benchmarks?
I’d love to hear about any datasets or eval setups for this.
I’d start by turning a small, permission-cleared sample of your own tickets into replayable cases. Pin the starting commit, preserve the ticket as it existed before the fix, and keep the eventual patch out of the agent’s inputs. Have someone review the acceptance checks independently of the generated patch.
I’d record separate outcomes: requested behavior works, existing behavior still works, the change stayed within scope, and the agent’s final report matches what the tests actually established. A separate worktree helps avoid file collisions, but shared databases, credentials and external tools need isolation too.
Run each candidate several times under the same tool and time budget. Include ordinary tickets, incomplete tickets and cases that should end with a clarification or “not verified”; report review time alongside completion rate. That should give you a model comparison specific to your workflow.
OpenAI’s MultiLangBench is an inspectable reference for outcome-based coding tests, but its SQLite task is different from completing real tracker tickets. Which languages and kinds of changes dominate your backlog?
Disclosure: I run Bright and covered that research. This is an AI-assisted evaluation proposal, not a benchmark I’ve run.
Thanks, this is really useful. The idea of keeping the eventual patch out of the agent’s inputs makes a lot of sense, otherwise it’s too easy to leak the answer without noticing.
Good point on isolation too. The worktrees handle the files, but we still share a test database between runs, so that’s probably the next thing to fix.
I also like tracking “should end with a clarification” as a separate outcome. Right now we only count completed or failed, which hides the cases where the agent guesses instead of asking.
To answer your question: our backlog is mostly TypeScript and Python, and the changes are mainly small bug fixes, UI tweaks and test updates. Very few big features. I’ll start with a small set of those and run each model a few times like you suggested.
For your ticket-to-code runs, which languages and frameworks are in the target repos? That would help narrow the open-model and real-ticket dataset recommendations to evaluation evidence relevant to your stack.