Module Utils

What a configuration key does to the identity a compilation cache keys on. A cache replays a schedule crowned by an earlier process onto freshly lowered code, and the only thing between "replay" and "replay from another regime" is that identity: gh-ocannl-568 measured a default-flags run replaying a tf32-tuned winner at 5.9x slower than not tuning at all, because the numerics policy was absent from the key. So every key is classified here, and test/operations/digest_completeness fails on one that is not — adding a knob forces the question "does this change what a cached schedule means?" while the answer is still fresh.

The classification of every key in known_config_keys, grouped by class and by reason. The reason is the reviewable part: the class alone does not say why, and why is what a reviewer has to check when a key is added to a group or moved between two.

The class and the reason recorded for key, or None when it is unclassified (which test/operations/digest_completeness rejects for every known key).

Whether to print where each configuration setting comes from (commandline, environment, config file, or the hard-coded default). It gates the logging of the config-reading functions themselves, so it is bootstrapped directly rather than via get_global_arg: the initial value only reflects the commandline and the environment, and the config file setting is applied once the config file is read. Can also be set programmatically, e.g. to trace configs a test reads.

Opt-in (gh-ocannl-595): the trace is a debugging tool -- some eighty lines on a run that reads a config file -- and it shares stderr with the unknown-config-key warning, the one startup message that means the user made a mistake. Enabling it traces every key, without a second dependence on log_level: it says "tell me where the configuration came from", and answering that for one key is not a useful reading of it.

The environment spelling of a config key: OCANNL_ followed by the key in uppercase, and nothing else. The prefix is mandatory here (unlike on the commandline and in a config file) so that OCANNL does not read an unrelated tool's variable.

ONE spelling (gh-ocannl-652). A lowercase ocannl_<key> used to be read as well, and won over the uppercase one besides -- which test/operations/profiles/dune had to document as a real precedence trap. It bought nothing anyone could name: no caller in this repository spelled a variable that way and no documentation recommended it, while every dune rule that has to declare the ambient variables it is invalidated by paid two lines per key, 228 of them, forever (gh-ocannl-628). The reserved namespaces this file introduced alongside them (OCANNL_TOOL_..., OCANNL_LOG_LEVEL_<MODULE>) were uppercase-only from the start, for the same reason: uppercase is what the shell convention is unambiguous about.

Dropping a spelling someone may have exported is a silent demotion, so the spellings this file no longer reads are not merely ignored: classify_env_var reports one as Env_unread_spelling, and the check at the foot of this file makes it FATAL when it names a known key. See unread_env_vars.

Dashes are not a spelling either: ocannl-log_level and its uppercase form were dropped in gh-ocannl-605, documented nowhere and used by nobody, while costing every such dune rule four spellings per key -- of which the natural-looking all-dashed ocannl-log-level was never one. Dashes remain idiomatic on the commandline, where cmdline_var_names accepts them.

The ocannl-prefixed environment namespaces that are deliberately NOT configuration, so that a name in one of them is never reported as a misspelt key (gh-ocannl-629).

Two of them, and each is a namespace someone else reads:

• ocannl_tool_… belongs to this repository's own tooling and test harnesses -- tools/sweep.sh's state directory, tools/test-run.sh's time cap, the hardware hook in test/operations/test_cpu_topology.ml. None of them is a library setting, and every one of them is exported into the environment of processes that link OCANNL.
• ocannl_log_level_… is read by ppx_minidebug at PREPROCESSING time: the per-module tracing gates, one %%global_debug_log_level_from_env_var at the top of tensor/row.ml and of its eighteen siblings. The name after the prefix is a module, not a setting, and it is consumed before this file's initialization exists to have an opinion.

A reserved prefix rather than a list of exempt names: a list is a second place to update when a tool grows a variable, nothing forces the update, and the failure mode is a warning the reader cannot act on -- which trains people to ignore the warning that matters, the exact outcome gh-ocannl-595 was fixed to prevent. A prefix costs the tooling a rename once and nothing thereafter.

Whether the platform resolves environment variable names case-insensitively. On Windows ocannl_Log_level and OCANNL_LOG_LEVEL are ONE variable, so a spelling this file would call unread there is in fact read, and reporting it would be wrong on that platform only.

Native Windows only, NOT Cygwin (Codex P1 on PR #389): a Cygwin runtime sets Sys.cygwin rather than Sys.win32, and its POSIX environment is case-SENSITIVE -- getenv "OCANNL_BACKEND" does not find ocannl_backend there. Folding case on that runtime would call the lowercase spelling read while read_env_var found nothing, which is the silent demotion this whole check exists to prevent, delivered by the check itself. What matters is what the runtime's own getenv does, not whether the host kernel is Windows.

The DEFAULT of a parameter, rather than the only answer (gh-ocannl-661). Baked in from Sys.win32 at definition, it made the Windows reading of classify_env_var unreachable on every other host: Linux and macOS CI could not execute that branch, so a change collapsing it stayed green on the ordinary PR path and surfaced days later on a scheduled Windows run -- which is how the dashed spellings came to be classified as read there in the first place. Every caller in this file still takes the platform's answer; what the parameter buys is that test/operations/config_var_spellings pins BOTH readings on EVERY host.

What an environment variable name is, to OCANNL. The classification is shared by the startup warning at the foot of this file and by test/operations/env_var_deps, which asks it of every (env_var …) a dune file declares -- so a name a rule tracks and a name a run warns about are decided by one function.

Whether two environment variable names denote the same variable on this platform: case-folded where the environment is, and otherwise exactly. case_insensitive defaults to env_names_case_insensitive and is passed explicitly only by the test that pins both readings on every host -- see there for why the platform's answer is not the only one reachable.

Every environment variable that addresses OCANNL's configuration and that nothing reads: the name, whether it is fatal, and the reason, in the order the names sort.

Fatal for a known key spelled in a way nothing reads, and a warning for everything else (gh-ocannl-652). The distinction is whether a VALUE that was meant to decide something is being dropped: ocannl_backend=cuda names a real key, so somebody wrote it to choose a backend, and silently running on the default instead is the failure mode this whole check exists to prevent -- it is also what dropping the lowercase spelling would otherwise have inflicted on anyone who had exported one. A name that matches no key (OCANNL_BACKEDN) never decided anything to begin with, and a lowercase name in a reserved namespace belongs to a tool rather than to the configuration; both stay warnings, as they were.

An EMPTY value is not reported at all, in either class: "" counts as unset at every source (see read_env_var), a dune rule clears a variable by setting it empty, and a launcher expanding $OCANNL_BACKEND with nothing set must not thereby abort the run.

Separate from the loop that consumes it (at the foot of this file) so that the walk is a value rather than an effect: what is reported is then a list something else -- a test, a tool refusing to run under a misconfigured environment -- can also ask for. Sorted so that a stream capturing the messages does not depend on the order the C library hands the environment over.

The commandline spellings of a config key, up to the value separator: the ocannl_-qualified ones -- then, unless qualified_only, the prefix-free ones.

Every spelling carries a leading dash, one or two. A bare ocannl_log_level=1 used to be read as well, and gh-ocannl-605 dropped it (Codex P2 on PR #363): a bare argument is a host application's positional, and an OCANNL-linked tool taking a path -- ocannl_config is the obvious one -- was one key name away from having it eaten. It also left the unknown-argument warning with a spelling it could not diagnose, since a bare argument is exactly what it must NOT claim to know about.

The dashing is two independent choices, not one per separator: the prefix separator dashes on its own, and the key's own separators dash TOGETHER. For log_level that is ocannl_log_level, ocannl_log-level, ocannl-log_level and ocannl-log-level (each in lowercase and in uppercase, each with one leading dash or two). A key dashed halfway (ocannl-print_decimals-precision) is not a spelling -- and, since cmdline_var_prefixes is also what the unknown-argument warning matches, it is reported as unknown rather than silently ignored, which is what makes the narrower contract safe to have (Codex P2 on PR #363). Enumerating every separator independently is the alternative, at 2^separators spellings per key; nothing asked for it.

qualified_only exists because OCANNL is a library: it scans the host executable's Sys.argv, so a prefix-free key claims an application's own option of that name. That is tolerable for keys nobody else would spell (--virtualize_max_visits) and not for --profile, which is a common application flag and which OCANNL treats as fatal when it does not name a known bundle -- a host passing --profile=prod would die during module initialization (Codex P2 on PR #291).

What an argument setting n begins with: a spelling from cmdline_var_names followed by the value separator, which is _, -, = or nothing at all. Whatever remains of the argument is the value.

This is the single source of truth for "an argument OCANNL reads", and the unknown-argument warning at the bottom of this file matches against it rather than parsing arguments a second way. It used to parse: split on =, normalize every dash to an underscore, look the result up -- which accepted spellings the reader ignored (`--ocannl-log-level=1`, before gh-ocannl-605 made it real) and rejected ones the reader honoured (`--ocannl_log_level_1`, whose separator is not an =, so the key came out as `log_level_1`). Both directions are silent contradictions: one applies nothing while saying nothing, the other applies the setting while warning that it is unknown. One table cannot disagree with itself.

The commandline sublevel of get_global_arg: returns the setting's value and the Sys.argv element it came from. Pure -- the sourcing log lives at the resolution seam, which is the only place that knows which sublevel actually won.

qualified_only defaults per key from qualified_only_config_keys, so a caller need not remember which keys renounce their prefix-free spellings.

Whether a raw command-line argument addresses a known configuration key under any spelling read_cmdline_var accepts — prefixed or prefix-free, dashed or underscored, any separator. For executables that parse their own flags (tools/): such an argument belongs to the config machinery and should be passed over rather than rejected as unknown, while an argument matching no known key under any spelling can still be flagged as a probable typo.

The environment sublevel of get_global_arg: returns the setting's value and the variable it came from. An empty value counts as unset.

The bootstrap reader: the few keys that are consulted before the config file exists (and hence before profiles are resolved) come from the commandline or the environment only.

Silent, deliberately. These keys are read before log_config_sourcing is resolved -- one of the reads IS that resolution -- so nothing here can know whether anyone asked for a trace, and each is read more than once besides (three call sites consult suppress_welcome_message). Their provenance is reported once, in full, by bootstrap_config_report below.

Parses the ocannl_config syntax: one key=value per line, # and ~~ lines are comments, empty values mean "unset", the ocannl_ key prefix is optional and keys are case-insensitive. Shared by the config file and by the embedded profile payloads (which are literally partial config files); source names the origin in error messages.

The source levels a setting can come from, in decreasing priority. Each level splits into two sublevels: the keys stated explicitly at that level, then the payload of a profile picked at that level -- so a specific setting always beats an aggregate one of equal immediacy, and a profile named on the commandline still overrides an exhaustive config file.

Where get_global_arg found a value; reported by the log_config_sourcing trace.

A short provenance tag, e.g. "profile 'reproducible' via the commandline".

The precedence walk, factored out of get_global_arg so it can be exercised on synthetic sources (see test/operations/config_profiles.ml). The lookups are pure; logging happens at the call site, which is the only place that knows which sublevel won.

The keys a profile payload may not set: they are read before profiles are resolved (or would make the resolution recursive).

The embedded profile payloads, by name. Each is literally a partial ocannl_config file: same syntax, same parser, setting only the keys where the profiles' goals disagree.

The profile picked for this run, if any: its level (which decides the priority of its payload), its name, and the parsed payload.

Retrieves the arg_name setting from the commandline, the environment, the config file, or the payload of the profile picked at one of those levels; returns default if none has it, together with where the value came from.

The directory generated-code debug files are written to (created if missing): build_files/<prefix>/, where the prefix defaults to the executable's base name.

The directory diagnostic and routine-debug logs are written to (created if missing): log_files/<prefix>/, sharing the prefix resolution of build_files_dir.

A fresh non-negative integer per call, wrapping to 0 on overflow. Backends use it to tell one run of a kernel from the next: cc names each dynamically loaded library by it (there can be only one library of a given name in a process), and the GPU backends prefix a launch's routine logs with it. Process-wide rather than per backend, so that ids never collide across the backends a process links.

Filters the list keeping the first occurrence of each element.

Returns the multiset difference of l1 and l2, where l1 and l2 must be sorted in increasing order.

Removes the first occurrence of an element from the list that is equal to the given element.

parallel_merge merge num_devices progressively invokes the pairwise merge callback, converging on the 0th position, with from ranging from 1 to num_devices - 1, and to_ < from.

s with any exponent's leading zeros removed, so that the rendering does not depend on which C runtime formatted it.

OCaml's %g goes through the platform's snprintf, and the Windows runtimes pad the exponent to three digits (1e+020 where glibc writes 1e+20) -- which would make generated kernels, IR dumps, and any golden quoting one, differ by platform. Both spellings denote the same number, so this is about the artifact being reproducible rather than about the value. Digits that are not padding are kept: 1e-300 stays itself.

The decimal spelling of the double c: a floating literal -- a radix point or an exponent is always present -- that parses back to exactly c.

Two properties %.16g alone gets wrong, and both of them matter wherever a constant is written down: in emitted kernel text (C_syntax.c_float_literal, gh-ocannl-623) and in the IR dumps (Low_level.to_doc and Low_level.to_doc_cstyle, gh-ocannl-713).

• It is a floating literal, not an integer one. %.16g of a value with no fractional part has no radix point, so 2. comes out as "2" and -0. as "-0" -- the integer zero, hence +0.0 once read as C, and indistinguishable from +0. to a reader of a dump. That is the one distinction the gh-ocannl-615 chase was about: asking whether a -0.0 in host data survives to the kernel, and being answered := 0 either way. Forcing the radix point closes the whole class at once.
• It round-trips. 16 significant digits do not recover every double: 0.1 +. 0.2 prints as "0.3", which is a different double, and so a constant whose 17th digit matters displays identically to one whose does not. The rendering therefore retries at %.17g -- the width IEEE-754 guarantees -- whenever the 16-digit spelling does not parse back to c. Values that already round-trip keep their exact previous spelling, which is why goldens move only by the appended .0.

Non-finite values get %.16g's words (inf, -inf, nan): they have no decimal spelling at all, and appending a radix point to one would produce a token that is neither. A caller whose dialect needs its own spelling handles them before calling -- C_syntax.c_float_literal spells C's INFINITY and NAN, while an IR dump, which is not C, keeps the words.

A mutable linked list structure.

A dynamic array of weak references.

A lazy value that can be safely forced and compared by unique ID.

A persistent map implemented as a balanced binary tree. The sexp_of function preserves and displays the tree structure.

CPU topology facts for the worker-pool uniformity policy (gh-ocannl-530, docs/proposals/gh-ocannl-530-pool-uniformity.md). Facts only — the policy that consumes them lives in Cc_backend. Every probe degrades to an "unknown" answer ([], `Unknown, a fallback count) rather than raising; the consumers treat unknown conservatively.

Publishing a file so that no reader ever observes a half-written one (gh-ocannl-780).