Been building agent workflows for a bit now, and every time I need to pick a framework + model combo, it feels like I’m going on vibes — a blog post, whatever worked last time, whatever’s trending on Twitter that week.
So I started building an open-source tool that runs the same task (document summarization, tool-use benchmarks so far) through LangGraph and AutoGen, across different models (GPT-4o, Claude, Gemini), and scores them side by side on quality, latency, and cost. Basically trying to replace “I think X is better” with an actual number.
Before I go further with it, I’m curious how this community actually handles that decision today — do you benchmark at all, or is it mostly experience/gut feel? Genuinely want to know if this is a real gap or if people have already solved it some other way.
If you’re up for it, I turned this into a quick 2-minute survey (8 multiple choice, no login required): SmartWrapperOSS — PM Workflow Survey
And if anyone wants to try running it on your own task and tell me what breaks, drop a comment or DM — happy to walk you through it.
Repo’s here if you want to poke at the code: GitHub - SmartWrapperOSS/SmartWrapperOSS: Benchmark the same task across agent frameworks (AutoGen, LangGraph) and LLMs — quality, latency, and cost in one table. · GitHub