Blogs — Alex Polozov

Reflection

polozov@reflection.ai (Alex Polozov) — Thu, 06 Nov 2025 00:00:00 +0000

Next week, I am excited to join Reflection AI as a Member of Technical Staff to help build the frontier open intelligence ecosystem of the Western world.

Spending the last few years at Google (X → Labs → DeepMind) during the founding years of the AI revolution was an opportunity of a lifetime. Working on Gemini and Jules with some of the smartest and most ambitious (and fun!) people on Earth – even more so. I will miss them fondly. But as 2025 unfolded, I found myself reflecting on the state of the AI ecosystem in the U.S. and on my work’s impact on empowering software builders… something critical was missing. This post is my attempt to articulate what.

NOTE: This is also my first blog post in… 7 years, apparently? Most of the older ones were from grad school 😅 Hope that changes – I remember enjoying blogging and tweeting about my work, which got somewhat disrupted by the whole “highly-confidential-frontier-lab” thing.

AI as a platform technology

I started my career in programming languages and developer tools. The devtools culture should be an inspiration to all technologists – they focus less on building a product and more on building a platform that enables millions of builders to create new products. What constitutes a “platform” varies – Microsoft popularized the desktop OS and SDKs; Amazon popularized cloud computing; Apple popularized mobile apps. We are now in the next, and possibly last, cycle of platform technology – AI enables all kinds of individuals, tiny teams, or companies to create applications previously thought impossible. It seems to be growing faster than any technology in modern history, with Gemini alone processing 1.3 quadrillion monthly tokens.

But this platform is different – as a builder, you don’t control or fully understand it. Sure, you might not control the binaries of Windows OS or machines in the AWS data center either – but you understand the behavior of the components you use. OS APIs are documented and usually deterministic; the VM specs are well-known and can be re-provisioned at will. Not so for an LLM API. You make a black-box call into a model with limited customization (context engineering, sampling parameters, tool definitions, multiturn agent logic) and iterate for a consistently satisfactory result. Often it is. Sometimes not.

If I were building the next Bloomberg Terminal with AI insights, or AI-driven customer support at a major airline, or a government-contracted public benefits assistant for an EU country… that level of control and reliability would be unacceptable. Yes, models will improve, but human psychology won’t. For any of the above, I would emphasize finetuning my models and owning as much of the stack as I can – just to derisk. A good platform empowers competent builders to own, tinker with, and build upon its components.

This realization helps us deduce a few things about the medium-term future:

We shall see a growth of AI tuning expertise – engineers post-training their own domain- or skill-specialized LLMs. While today “AI Engineer” denotes LLM-aware application building expertise, the next tech cycle will expand it onto LLM post-training too, once again blurring the line with “ML Engineer”. We can already see that in the blossoming ecosystem of RL environments.
Serious enterprises (e.g., Fortune 500s, hedge funds, sensitive-industry startups, governments) will integrate AI in their applications with more homegrown models, environments, and evals. The art and science of LLM development, currently contained in ~5 frontier labs, will commoditize as a subdiscipline. Specific trade secrets may stay hidden for months or even years – but the methodology will spread as soon as the application builders are ready.
Infrastructure layers of the ecosystem will happily provide for this newfound demand. Jensen Huang is already building 100 of his so-called AI factories – data centers specialized for AI, possibly serving individual industries or companies. Serious enterprises will want their homegrown, user-data-tuned AI to run on-prem or in a leased AI factory. A private cloud with an alphabet soup of compliance acronyms won’t cut it.

NOTE: I am not making any predictions about a hypothetical AGI-assisted world here. The frontier-lab race to develop and own AGI makes a lot of sense, will continue, and, if successful, will change most of the technological tenets of this essay unrecognizably. That won’t be easy to “control”. I reserve any judgements about that world.

But I think the medium-term road there, using “AI as a normal platform technology”, looks clearer.

The coding bellwether

For evidence of this trend, look at coding 🙂 Code AI has always been a bellwether for our industry – largely because researchers and engineers love building tools to accelerate ourselves first, and thus it becomes a frontier indicator of AI innovations. Github Copilot was the first successful LLM product, more than a year before ChatGPT. Cursor and Devin were the first useful agents – reliably completing 10-step tasks months prior to the current wave of travel booking demos.

So what are Cognition (Devin, Windsurf) and Anysphere (Cursor) doing recently? Training models. Just last week, Cognition released SWE-1.5 and Cursor released Composer – domain-specialized agentic models finetuned on in-house traces of real-world SWE tasks. Perhaps more interestingly, Cognition also presented SWE-grep – a specialized LLM finetuned for one component of the agentic system, in this case context retrieval. These are our first signs of model innovation in the agentic app layer, not just software/context/tooling innovation. That will multiply.

And the second most interesting thing about these models? They are both built on open Chinese LLMs.

We need good Western open models

Finetuning does not require open weights – all frontier closed-LLM providers have offered tuning APIs for the last ~2 years. However, anecdotally, they have not taken off much:

Iterating on a closed-model finetune is slow, expensive, and can’t unlock the full tinkering abilities of an AI engineer. It does not compare to the experimental velocity of your own H100 or DGX SPARK (measured as ablated insights per week rather than raw hardware throughput).
For the aforementioned control & ownership reasons, closed models are not even in consideration for serious enterprises. They hope to close the frontier gap with customer data instead. As we are starting to see in coding, that’s viable for task- or domain-specialized LLMs.

Now, setting aside the open-vs-closed gap, where are our best open models coming from?

This chart is from The ATOM Project - American Truly Open Models manifesto. As of early November 2025, the past six months saw four American open model releases and 14 Chinese ones.¹ The Chinese open models are solid – while not reaching the state-of-the-art closed-model frontier, they are good enough to build upon and ship to real products (as Windsurf and Cursor successfully demonstrated). The Western alternatives… less so.

Market adoption is a more important argument that any benchmarks, so HuggingFace Trends and the Windsurf/Cursor deployments should speak for themselves. But if they don’t, look at our favorite marketing datapoint for agents – SWE-bench. Keeping the harness constant (“bash-only” minimal agent) and filtering to open-weight models, we have:

The 29-point gap between Qwen3-Coder and gpt-oss-120b is wild. Especially so as the closed Western models (Claude 4.5 Sonnet and GPT-5) range in 65-70% territory (and surpass 80% with bespoke agent harnesses). We don’t have comparable numbers from Llama 4, Magistral, or Command-A, but nobody I know expects them to match Qwen3 or GLM 4.6.² Nobody I know is building upon them either.

Does it matter where your open model originates? Even setting aside regulatory and PR risks (which alone are sufficient to discourage most Fortune 500 CTOs, I imagine), a Chinese model is a perilous and inconvenient technical choice for an American application stack:

I prefer reading my model’s reasoning traces in a language I understand;
I prefer to avoid lawsuits about data provenance and copyright choices in my AI features;³
I don’t want to wonder about cultural alignment of my customer-support agent;
And if I do discover a mismatch, realigning it with company-specific post-training is really hard. Even frontier labs notoriously struggle training away behavioral quirks all the time, OpenAI’s sycophancy incident being the most famous recent example. For enterprise AI engineers, I don’t envision alignment as a core competency any time soon, unlike task-oriented post-training.

Put another way, if a Western and an Eastern open model match in performance, any American company should surely pick the former. But we are not even close to such a match.

This disappointing state of affairs is obvious to many, has existential political connotations to some, and thus obviously going to change. By the end of 2026, I expect at least three strong Western open models on the market, Reflection chiefly among them. Will Brown predicts 7+ but that might be overly optimistic. After all, there are reasons for the underperformance of Western open models to date – not for the lack of trying – and it’s hard to predict who will avoid the past mistakes.

Ingredients

Suppose you decided to capture the open-weight frontier. In 2025-2026, for this recipe you shall need:

✔️ Funding for enough compute to supply your staff with experimental velocity. Most of the compute goes to ablations rather than hero runs, so the compute-per-researcher ratio is a more reliable predictor of success than raw FLOPS.
✔️ A continuous partnership with a hyperscaler. It’s not enough to procure your GPUs once, you must be able to expand compute when you hit milestones, cross-license your models for enterprise serving, benefit from SaaS support and GTM, and rely on the partner for subsequent funding rounds. Reflection has the backing of NVIDIA who was the largest investor in their $2B Series B.
✔️ Independent company structure from closed frontier. This is ironic, but there are deep organizational reasons why it’s difficult for the same lab to lead both closed-weight and open-weight frontiers. I will not elaborate on them. To be fair, Gemma was the best model in its weight class with every generation, but those classes now lag substantially behind stronger Chinese models. It remains to be seen how the weight classes evolve.⁴
✔️ Talent. For a foundation-model company, I would overemphasize pretraining, infrastructure, evals, data, and tool-assisted reasoning – in that order of importance. SFT is reasonably well understood; RL is by now probably the hottest research topic in the open-source AI community. But the topics above require expertise with relatively less attention outside frontier labs. Reflection understand this, already has the right talent, and is hiring more 😉
✔️ Experimental culture. Really, it flows from the right talent and almost religious fervor for the importance of protocol, methodology, ablations, and deep understanding of your evals and vibe checks.
✔️ Customer obsession. LLMs often fall into a trap of “benchmaxxing” – excellent at, say, scientific reasoning but useless for everyday SWE tasks, data wrangling, or automating enterprise workflows. gpt-oss may be the most famous example. A good foundation-model company is not just a research lab – it should follow the Startups 101 of “know your user needs” and build customer relationships early. And if your customers are Fortune 500 enterprises, it helps to live close 🗽

At this time, I only see Thinking Machines, SSI, Reflection, and maybe Poolside with enough capitalization to fund all the compute, talent, and data acquisitions. Out of these, Thinky and SSI are uncertain and stealthy in their product goals, so difficult to analyze. Poolside is now pivoting toward training foundation models – but I’m unsure about some of the ingredients, plus they haven’t mentioned any commitment to openness to date. Reflection has the partnership with NVIDIA and hits every ingredient on that list – with of course lots of work remaining. It will not be easy, and we’ll have plenty of chances to fail. But this is a journey I’m actually excited to begin.

Which brings me to my most important point.

Join us!

You don’t have to already know pretraining, or post-training, or GPU training infrastructure, or data curation, or evals – but of course better if you do 😉 What matters is whether this vision of the beautiful world of independent AI tinkerers, builder empowerment, and open frontier-model science resonates with you. If you’d like to contribute your expertise to help make it happen, please by all means drop me a note, we’d love to chat. Reflection is hiring in NYC, SF, and London.

Thanks to Misha Laskin, Ioannis Antonoglou, and Aakanksha Chowdhery for feedback on the early drafts of this post.

ATOM’s timeline omits Cohere’s Command-A from Canada and Mistral’s Codestral/Magistral from France. They don’t change the narrative substantially. ↩
From a brief scan of the SWE-bench Verified leaderboard that does not control for scaffolding, the closest entrants appear to be DevStral Small 2505 evaluated in the OpenHands harness (46.80%) and Nemotron-CORTEXA, which is a complex parallel-compute harness combining different models for different subtasks (68%). GLM-4.6 achieves the same score with a ~~box of scraps~~ minimal harness. ↩
For example – do we know if GLM 4.6, which Cognition allegedly used to post-train their SWE-1.5, properly removed restrictively-licensed code such as GPL from their base model? Their tech report does not discuss it. I wonder how the software companies using Windsurf feel about this. ↩
This is also why I’m skeptical of MAI and Grok capturing the open-weight SOTA. Meta, too, has infamously abandoned openness after its MSL/TBD restructuring. ↩

Program Synthesis in 2017-18

polozov@reflection.ai (Alex Polozov) — Tue, 10 Jul 2018 00:00:00 -0700

Introduction

In early 2017, Sumit Gulwani, Rishabh Singh, and I wrote a survey on Program Synthesis. Its goal was to provide a comprehensive overview of the state of the field at the time and to serve as an introduction for newcomers. While complete at the time, the survey got surprisingly outdated in the mere two years. Nowadays, every time someone asks me for an introduction to program synthesis, I point them to this survey but also add 10-20 other links that have recently pushed state of the art. On one hand, this is great: program synthesis is expanding rapidly (ICLR’18 alone had 14 papers on the subject!). On the other hand, I’m getting tired of compiling and annotating these lists of recent links every time 🙂 Ergo, this post — a high-level overview of the recent ideas and representative papers in program synthesis as of mid-2018.

Before we dive in, a few disclaimers:

I assume you are familiar with program synthesis. If not, you’re welcome to check out the survey in question or this excellent blogpost by James Bornholt.
As expected, it’s impossible to cover everything, although I tried my best. We never set this goal even for the original survey – there’s an enormous body of great work out there. That said, if you notice some glaring omission, please let me know in the comments, and I’ll be happy to include it!
The list is skewed toward ML/AI rather than PL. This isn’t because of any personal biases of mine, but simply because the volume of synthesis-related papers in NIPS/ICLR/ACL/ICML over the last couple of years started exceeding PLDI/POPL/OOPSLA. Many prominent researchers now publish in both communities.
As with the original survey, I tried to organize the post to highlight high-level ideas rather than individual papers. An idea appears in more than one research paper. I also listed notable individual papers in their own section.
Unlike the original survey, this post focuses only on the ideas and techniques, and not on, say, new applications. There’s also a small section on datasets in the end.
There’s a fair number of Microsoft Research papers in this list, although I tried my best to maintain diversity.

Neural-guided search

Many program synthesis applications are built on a top-down enumerative search as a means to construct the desired program. The search can be purely enumerative (with some optimizations or heuristics), deductively guided, or constraint-guided. A common theme of many research papers of 2018 has been to augment this search with a learned guiding function. Its role is to predict, for each branch at each step of the search process, how likely it is to yield a desired program, after which the search can focus only on the likely branches. The guiding function is usually a neural network or some other probabilistic model. The overall setup can be framed slightly differently depending on the interaction of components in the search process, but the key idea of combining a top-down search procedure with a trained model guiding it remains the same.

DeepCoder pioneered such combination of symbolic and statistical techniques in early 2017. That system learns a model that, given a task specification (input-output examples in the original work), predicts a distribution over the operators that may appear in the desired program. This distribution can be then used as a guiding function in any enumerative search.¹

A fragment of the deductive search process looking for the most generalizable program that satisfies the given input-output example. At each branching point in the search tree, the current state is fed into a neural model, which estimates the quality of the best program that is likely to be produced from each branch (shown as a green distribution curve; higher shapes correspond to more promising branches). Source: NGDS.

The subsequent research in 2017-18 pushed this idea further, allowing the guidance to happen at every step of the search process instead of just its root. The guidance can now take into account a partial search state instead of just a global task specification. As expected, this greatly improves the synthesis speed (thanks to trimming down the combinatorial explosion of branches) and accuracy (thanks to the exploration of more promising branches). The key take-away: if you can obtain enough data to train a reliable guiding function, there’s no reason not to do it, it’ll definitely help the search. The main caveat here is that invoking a neural network has a non-zero cost, and sometimes (especially nearing the leaves of the search tree) this can be more expensive than ignoring its prediction and exploring the entire subtree anyway. Also, if the prediction is wrong, there must be a way to detect this, backtrack, and pick a better branch. Thus, careful engineering and something like A*, branch-and-bound, or beam search must be a part of the solution.

Accelerating Search-Based Program Synthesis using Learned Probabilistic Models PLDI'18

Woosuk Lee, Kihong Heo, Rajeev Alur, Mayur Naik
Leveraging Constraint Logic Programming for Neural Guided Program Synthesis ICLR'18 Workshops

Lisa Zhang, Gregory Rosenblatt, Ethan Fetaya, Renjie Liao, William Byrd, Raquel Urtasun, Richard Zemel
Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples ICLR'18

Ashwin Kalyan, Abhishek Mohta, Alex Polozov, Dhruv Batra, Prateek Jain, Sumit Gulwani
Neural Program Search: Solving Programming Tasks from Description and Examples ICLR'18 Workshops

Illia Polosukhin, Alexander Skidanov
Program Synthesis using Conflict-Driven Learning PLDI'18

Yu Feng, Ruben Martins, Osbert Bastani, Isil Dillig

Sketch generation

Human programmers often write a program top-down, first producing a high-level description of the desired program structure, and then concretizing the missing details. In other words, we often first produce a sketch of a program instead of writing the entire program end-to-end. In program synthesis, manually sketching the desired program has long been a technique of choice for many challenging domains. However, what if we allow a model to learn the right sketch?

The idea of sketch generation is to split the program synthesis process (Spec $\to$ Program) into two phases (Spec $\to$ Sketch $\to$ Program). The first phase (sketch generation) is usually automatically learned; the second one (program generation) can either be a different learned model or some kind of search. In either case, in order to train the corresponding model, one defines (a) a language of sketches, (b) an abstraction function that produces a sketch of a given code snippet, and (c) a dataset of sketches (usually automatically rewritten from a dataset of concrete programs) associated with the corresponding task specs.

A two-step program generation process, with separate decoder models for (1) a sketch conditioned on the spec and (2) a program conditioned on the sketch and the spec. Source: Coarse2Fine.

Sketch generation outperforms end-to-end program generation. Intuitively, it is easier for a model to learn high-level patterns of the desired program structure rather than its fine implementation details. Moreover, filling in implementation details in a given sketch is much easier than coming up with them on the fly as part of whole program generation. Thus, non-surprisingly, the technique works pretty well on two very different application domains and specification kinds.

Sketch generation was developed independently in Bayou and Coarse2Fine. Bayou takes as input syntactic specs about elements of the desired program (e.g. “must include a call to readLines and an Iterator type”), and produces the most typical Java program that includes these elements. Coarse2Fine takes as input a natural language spec, and produces a snippet that implements it (evaluated on Python and SQL snippets from the Geo, ATIS, Django, and WikiSQL semantic parsing datasets).

Coarse-to-Fine Decoding for Neural Semantic Parsing ACL'18

Li Dong, Mirella Lapata
Neural Sketch Learning for Conditional Program Generation ICLR'18

Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, Chris Jermaine

Graph neural networks

Graph Neural Networks, or GNNs, arose recently as a particularly useful architecture for tasks that involve reasoning over source code, including program analysis and program synthesis. The motivation here is the need to treat a program as an executable object. Most of the ML applications to code before 2017 treated programs either as a sequence of tokens or, at best, a syntax tree. This captures natural language-like properties of code, i.e. the patterns that arise in programs thanks to their intent to communicate meaning to other programmers. However, code also has a second meaning – it is a formal executable structure, which communicates a particular behavior to the computer. In other words, NLP-inspired methods, when applied to code, have to infer the semantics of for, if, API chaining patterns, and so on. One would hope that this is learnable by a deep neural network from the ground up, but in practice, it’s been exceedingly difficult.

A portion of the data flow graph for a given code snippet, represented with three kinds of auxiliary data flow edges that describe how the variables in the snippet are read and written when the program gets executed. Syntactic AST edges are omitted for clarity. Source: IntelliCode.

A simple way to communicate such information about program semantics to a network is to encode the program’s data flow graph. It enriches the AST with additional edges that describe how the information might flow in a typical program execution. For instance, if a variable x is read somewhere in the program, we can connect it with all possible locations where x could have been last written. The beauty of the method is that it doesn’t care about the semantics of these additional edges, as long as they can be deterministically computed and added to the graph. For practical tasks (like detecting when a variable may be used incorrectly) the program is augmented with about 10 kinds of such semantic edges. This augmented graph is then fed into a GNN encoder, which learns the representations for program variables and locations that might be useful for a downstream task, such as variable misuse detection. This is the approach that worked best for a number of software engineering tasks.

What does it have to do with synthesis? Well, once we’ve established GNNs as a good architecture for processing programs,² then it’s only natural to apply them for program synthesis as well. Recently, we’ve tried to do just that. The idea is to frame program synthesis as a top-down generative model where at each step you (a) decide a grammar production to expand the next “hole” in your partial program, (b) use GNN propagation to update the representations for the relevant nodes of the partial program, (c) refocus and repeat once the entire program is complete. The whole process can be conditioned on any specification, but in our initial experiments we’ve focused on the hardest possible setup – infer a missing program subexpression just from its context, i.e. from surrounding source code alone. That’s hard. Especially on “real-life” source code (we applied it to various C# subexpressions from popular GitHub projects). We’ve reached 50% top-1 accuracy, which for this kind of setup, I think, is already pretty remarkable.

Learning to Represent Programs with Graphs ICLR'18

Miltiadis Allamanis, Marc Brockschmidt, Mahmoud Khademi
Generative Code Modeling with Graphs ICLR'19

Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, Alex Polozov

Datasets

The recent transition of program synthesis into an AI discipline had another nice side-effect — a flurry of new datasets. Neural networks are data-hungry, so researchers are forced to find problems with tens of thousands of data samples, or invent new problems where such a dataset can be creatively generated/collected, and pose a challenge for the field. Sometimes even collecting this dataset can be a genuine ML problem in itself.

A typical problem from the Karel dataset. The system is given two input-output examples $I_1 \to O_1$, $I_2 \to O_2$, and it has to synthesize a program that also passes a held-out test example $\check{I} \to \check{O}$. Every example is a grid environment for the robot, with walls and markers (potentially of different sizes). Source: Neural Program Meta-Induction.

Here are some datasets that the community has lately focused on:

Two new semantic parsing (English $\to$ Python) datasets, collected from StackOverflow: StaQC and CoNaLa.
Karel, a toy robot programming language that used be to used for introductory programming classes at Stanford. The dataset contains synthetically generated programs, each with 5 input-output examples.
NAPS, a really young and quite challenging dataset containing preprocessed problems from algorithmic competitions along with imperative descriptions and examples.
WikiSQL, a large semantic parsing dataset (English $\to$ limited SQL). It’s a bit controversial due to its lack of program variety and semi-artificial language generation, but its size and variety of tables/questions has generated a lot of attention after Salesforce released it a year ago.
A standardized collection of 10 semantic parsing datasets from NLP and DB communities, preprocessed into SQL instead of various logical forms.

Notable mentions

RobustFill: Neural Program Learning under Noisy I/O ICML'17

Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, Pushmeet Kohli

An end-to-end differentiable version of FlashFill that’s trained on a large volume of synthetically generated tasks and can adapt to small noise in the input-output examples. The core architecture is surprisingly simple, but works well for its DSL. Demonstrates an unreasonable effectiveness of attention 🙂

Program Synthesis using Conflict-Driven Learning PLDI'18

Yu Feng, Ruben Martins, Osbert Bastani, Isil Dillig

In addition to a variant of statistical branch prioritization, this paper introduces the notion of conflict-driven search from SAT/SMT solving to the program synthesis world. Loosely speaking, in modern SAT/SMT search algorithms, when a search branch results in a failure, we can learn a reason for this failure – a new conflict clause (i.e., a particular contradiction) that can be added to the working set of constraints to avoid a similar mistake in the future. In contrast, search algorithms in program synthesis couldn’t do that — until now. Now, for instance, an input-output example $[1, 2, 3] \to [2, 4]$ not only immediately eliminates any program of form x => map(x, ...) (because map can’t change the length of its input list), but also generates a constraint that eliminates x => reverse(x, ...) and x => sort(x, ...) for the same reason.

Learning to Infer Graphics Programs from Hand-Drawn Images NeurIPS'18

Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, Joshua B. Tenenbaum

A neat paper that introduces a novel program synthesis problem (recovering $\mathrm{\LaTeX}$ programs from hand-drawn mock-ups) that nicely combines perceptual and symbolic techniques. There are many technical challenges: dealing with noise in the input, recovering the right primitives, optimizing the resulting constraint problem with a learned policy, and so on.

Programmatically Interpretable Reinforcement Learning ICML'18

Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri

Have you ever wanted to understand the behavior of your learned RL policy? Or maybe not 100% understand, but at least get some insight into its logic, or maybe replace its logic with something more interpretable and generalizable. Turns out, program synthesis can help. The method is straightforward and beautiful in its simplicity: train a policy to, say, drive a race car, and then search for a program in an appropriate DSL to approximate the behavior of this policy. (The devil, of course, is in the details: DSL design, good sketches, and a method to intelligently sample examples from the policy are all key to making this work.) The program is not as perfectly optimized for the fastest track completion as a neural policy, but it drives smoother, provably generalizes to other tracks, and, most importantly, you can read it. All these benefits become apparent in a video.

With sincere apologies to Yoshua Bengio, who hasn’t been asked for consent to using his name in the illustrations. We all know how hard it is to make new figures, so the choices you make in a preprint often live on in posters, talks, and blog posts. 🙂 ↩
I’ve heard people say that GNNs are to programs are what CNNs are to images and LSTMs are to text — if not necessarily a silver bullet, then at least a baseline architecture of choice. I personally think it’s not as clear-cut yet, but explicit reasoning over graphs and relations should definitely be involved in most AI tasks out there. ↩

On performance in .NET

polozov@reflection.ai (Alex Polozov) — Sun, 28 Sep 2014 21:41:39 -0700

Formerly, when someone asked me “How to make my C# program faster?”, my go-to advice was:

Improve your computational complexity.
Switch to a native language.

Of these suggestions, step #1 should take care of performance in most situations, if done properly. However, sometimes it is not enough. For example, you are writing a user-facing real-time performance-critical application, such as the C# compiler, and you need to look not only at the average performance, but at the 99^th percentile. This is where step #2 used to take over… but not anymore. Currently, my go-to advice looks like this:

Improve your computational complexity.
1. Read “Writing High-Performance .NET Code”, Roslyn team blog, and the rest of the links provided below.
2. Switch to a native language.

Measurement tools

“Writing High-Performance .NET Code” should be your desk book for any performance-critical .NET application. I found the following knowledge it provides most important:

What profiling tools exist in the .NET world (PerfView, ETW, CLR Profiler, dotTrace, dotMemory, Windbg…), how to use each one of them, how to answer specific questions with a combination of these tools, which metrics should you look at and why.
How exactly .NET GC works and why you should care. How to help .NET GC by managing your allocations and object lifecycle carefully.
JIT mechanics, NGEN, when to use each of them, startup rates.

In other words, when you have a specific problem with your .NET application, this book will show how to use the tools to locate who exactly is responsible for the bottleneck, how often it occurs, and so on. Often you will be surprised, because (assuming you’ve already gone through step #1) your greatest bottleneck is going to be either GC, allocations, JIT, boxing, or incorrect usage of .NET API. All of these subtle engineering details are quite unobvious if you don’t train yourself to keep them in mind.

Performance-minded coding

When you know the consequence of the problem (e.g. “my app spends 20% of the time in GC”, which arises because of “there’s this pinned handle that points to a long retention path that is being walked on every gen-2 collection”), the next step is to eliminate it. The series of links below (along with the accompanying chapters in “Writing High-Performance .NET Code”) explains the techniques for doing it. If you know that this particular path in your code is going to be performance-critical up to the 99^th percentile (never optimize prematurely!), you should really think of these techniques as guiding principles to keep in mind every second during your coding process.

Here are some must-read articles for any .NET programmer that works with performance-critical code:

“Essential Truths Everyone Should Know about Performance in a Large Managed Codebase” — this BUILD 2013 talk was done by Dustin Campbell, a program manager on the Visual Studio/Roslyn team. In it, Dustin discusses the techniques they used when they rewrote the VB.NET/C# compiler from C++ to C# to make the new version perform almost as fast as the native one.
“Roslyn code base – performance lessons” (part 1, part 2) — written by Matt Warren, an enthusiast who looked through the open-sourced Roslyn code, and outlined even more tecnhiques from the codebase in a series of blog articles.
Really, you should read the entire Matt’s blog, it is full of performance wisdom.
“In managed code we trust, our recent battles with the .NET Garbage Collector” — this article is from the StackOverflow team, and it tells a similar performance investigation story. It’s interesting to see how they tracked down the issue and what did they do to eliminate it.

I’m going to list some of the mentioned techniques below, briefly. Again, the book and the sources above have already done it much better than me.

Avoid allocations

For every allocation you make, the .NET GC will spend a little more time on every future collection. If this instance is relatively long-lived, it’s going to eventually move to gen-1 and gen-2, and greatly affect the future GC times. But even if it is short-lived, the performance of gen-0 collection is going to matter in your higher percentiles anyway.

Some of the common sources of hidden allocations are:

LINQ queries: they allocate enumerators and lambda-representing objects that capture references in a closure. Also, eventually you need to convert a query to a list/array, and it’s going to expand itself several times in the process. Instead you should allocate a list/array of a known capacity beforehand, and do all the required work directly, without allocating hidden LINQ enumerators.
Lambdas: if they capture any references from the outer scope, they will be rewritten to compiler-generated objects. If your lambda operates in a generic context this object will not even be cached in place, and will be re-created every time. Consider refactoring your logic in such a way that lambda becomes non-capturing (such lambdas compile down to static methods), or get rid of the lambda altogether.

UPD: This behavior will change in the upcomping C# 6. In Roslyn, lambda functions now always compile down to instance calls, which causes up to 35% performance gain in common scenarios.
Strings: because they are immutable, every string modification allocates a new string object. This is most subtle in calls like string.Trim(' '), which actually returns a new string. Instead use index-based arithmetic on hot paths. For case-insensitive comparison, use the corresponding string.Compare overload instead of converting all your strings to lowercase with string.ToLower. Also, string concatenation for simple cases is much faster than string formatting.
Invoking a method with params always allocates an array, possibly of zero length. Consider creating specialized overloads for most common numbers of parameters.

Use object pooling

If you use some objects often but temporarily, consider allocating a pool of objects of this type once, and reuse them where necessary. For example, the interface that Roslyn uses for StringBuilders look somewhat like this:

internal static class StringBuilderPool {
    public static StringBuilder Allocate() {
        return SharedPools.Default<StringBuilder>().AllocateAndClear();
    }
 
    public static void Free(StringBuilder builder) {
        SharedPools.Default<StringBuilder>().ClearAndFree(builder);
    }
 
    public static string ReturnAndFree(StringBuilder builder) {
        SharedPools.Default<StringBuilder>().ForgetTrackedObject(builder);
        return builder.ToString();
    }
}

Which is then later used in the following manner:

public override string GetTextBetween(SyntaxToken token1, SyntaxToken token2) {
    var builder = StringBuilderPool.Allocate();
    CommonFormattingHelpers.AppendTextBetween(token1, token2, builder);
    return StringBuilderPool.ReturnAndFree(builder);
}

Compile your reflection calls into delegates

Method calling through Reflection is horrendously slow. If you know the signature of your method beforehand (and for some reason cannot agree on a common interface with the third party who implements the external method), you can use LINQ Expression Trees to compile your MethodInfo to a strongly-typed delegate. This needs to be done only once, after that you can keep a reference to the delegate, and call it whenever necessary with standard C# syntax.

The technique is described at Jon Skeet’s blog. Here’s how I used it in one of my recent projects:

public static TDelegate ToDelegate<TDelegate>(this MethodInfo method) {
     var parameterTypes = method.GetParameters().Select(p => p.ParameterType).ToArray();
     MethodInfo invokeMethod = typeof (TDelegate).GetMethod("Invoke");
     var @params = invokeMethod.GetParameters()
                               .Select((p, i) => Expression.Parameter(p.ParameterType, "arg" + i))
                               .ToArray();
     MethodCallExpression methodCall = Expression.Call(method,
         @params.ZipWith(parameterTypes).Select2(Expression.Convert));
     return Expression.Lambda<TDelegate>(Expression.Convert(methodCall, invokeMethod.ReturnType), @params).Compile();
 }

Avoid boxing

Apart from obvious cases where people unnecessarily convert value types into objects (such as using now deprecated System.Collections namespace), there are some subtler ones:

String.Format boxes its arguments
Structs are boxed when used through their implemented interface. In particular, this happens when you use LINQ: List<T>.Enumerator is a struct, but LINQ methods treat it as IEnumerator<T>
Object.Equals(object other) is the source of all evil
Do not ever concatenate a string with anything other than a string
Calling GetHashCode on an enum causes boxing. Convert the enum instance to int first.

Use specialized collections for smaller sizes

An array outperforms a dictionary up to a certain number of elements, even though its lookup is $\mathcal{O}(n)$. The reason is all the boilerplate overhead that accompanies a dictionary (hashtable, bucket exploration, etc.). The exact tripping point depends on your particular code and has to be measured experimentally.

Know when to use structs and when not to

Structs give better performance than classes when they are small, consist of primitive types, and when they are not copied around often. The overhead of a class vtable is noticeable during garbage collection. However, classes are copied faster, because they are represented with just a pointer. On another hand, an array of classes will cause you a nightmare of cache misses (see reference locality). An array of structs neatly lies in memory as a single block, but would be a nightmare to copy as a function result. Bottom line, know your usecase and measure everything.

Note: you should always re-implement Equals, GetHashCode, and IEquatable<T>.Equals for your structs, because the default implementations use reflection to enumerate over the fields.

Compile your regular expressions

If you are going to reuse the same regular expression often, build it with a RegexOptions.Compiled flag. Do not ever use static methods of the Regex class, they are horrendously slow.

If your regular expressions do not require non-regular features such as lookaround or backreferences, consider using re2 instead of a default regex engine. It might yield you up to a 10x speedup. There is a .NET wrapper available, called Re2.Net.

I will stop here because I cannot possibly outline all of the techniques. Again, for the complete picture read the book, watch the talk, read the articles referenced above, know your tools, and measure everything carefully. Also, feel free to comment here if you want to contribute more advice!