AI Engineer NYC kicked off yesterday with a day devoted to workshops: 18 hands-on sessions built for learning by doing. You can watch someone demo an eval framework and nod along. Running it against your own agent is different, especially when it fails on a case you never thought to test and there’s an expert a few feet away who can tell you why. That kind of learning sticks. The Expo floor opened at 4:00pm, and a welcome reception sponsored by Sonar closed out the evening.
Opening days at conferences tend to be about listening. This one put people at their keyboards from the first session, and the lineup ended up doubling as a snapshot of the problems engineering teams are wrestling with right now. Three came up again and again.
Evals take center stage. Seven of the 18 workshops focused on proving an agent actually works. Plenty of teams have learned the hard way that getting an agent to run is easy and knowing whether to trust it is not. Laurie Voss taught Evaluations with Phoenix 101, SallyAnn DeLucia led AI Evaluation 101, and Xander Song’s Harbor intro covered shipping agents “on evidence, not vibes.” Sessions from Sonar and Langfuse, plus Nick Nisi and Zack Proser’s workshop on verification loops, pushed the same message: “done” needs a definition. Evals used to be something teams added after a demo impressed someone. In these sessions, they were treated as part of the build itself, written early and run constantly, the way software engineers treat unit tests.
Finance is a proving ground. Six workshops targeted financial services and other regulated industries, from a wealth management concierge built in ChatGPT to an anti-money-laundering agent for financial crime detection to insurance programs running on unstructured data. Regulated industries make a demanding test case for agents. Every decision needs an audit trail, mistakes carry legal and financial consequences, and the data rarely arrives in a clean format. An agent that holds up under compliance review and messy policy documents has cleared a much higher bar than one answering questions in a sandbox. It was a fitting focus for a New York crowd, in a city where so many banks, insurers, and fintechs are headquartered.
The harness matters as much as the model. A third group of workshops looked at everything surrounding the model: orchestration, permissions, reliability, and cost. Attendees could orchestrate Claude Managed Agents, build agents that self-improve overnight, secure long-running agents with Keycard and Temporal, and dig into Lance Martin’s Unhobbling the Frontier. Eugene Cheah’s Tokenomics 101 tackled the bill that comes after. Model capability keeps climbing, but what users actually experience depends just as heavily on the system wrapped around it: how an agent is scheduled, what it’s allowed to access, how it recovers when something fails halfway through a task, and what each run costs. An agent that works on its own for hours needs guardrails a chatbot never did, and its spend can grow faster than anyone budgeted.
For builders coming off Sunday’s AI Engineer New York Hackathon with MLH, Monday was a natural next step. The hackathon rewarded speed: pick an idea, ship something that works, demo it before time runs out. The workshops asked people to slow down and learn the parts a weekend sprint skips, like testing, security, and cost control. Less sprinting, more depth, and real hands-on time with the tools shaping how agents get built and trusted.
Top comments (0)