Module Utils

type settings = {
  1. mutable log_level : Base.int;
  2. mutable debug_log_from_routines : Base.bool;
    (*

    If the debug_log_from_routines flag is true _and_ the flag log_level > 1, backends should generate code (e.g. fprintf statements) to log the execution, and arrange for the logs to be emitted via ppx_minidebug.

    *)
  3. mutable output_debug_files_in_build_directory : Base.bool;
    (*

    Writes compilation related files in the build_files subdirectory of the run directory (additional files, or files that would otherwise be in temp directory). When both output_debug_files_in_build_directory = true and log_level > 1, compilation should also preserve debug and line information for runtime debugging.

    *)
  4. mutable fixed_state_for_init : Base.int Base.option;
  5. mutable print_decimals_precision : Base.int;
    (*

    When rendering arrays etc., outputs this many decimal digits.

    *)
  6. mutable check_half_prec_constants_cutoff : Base.float Base.option;
    (*

    If given, generic code optimization should fail if a half precision FP16 constant exceeds the cutoff.

    *)
  7. mutable default_prng_variant : Base.string;
    (*

    The default variant of threefry4x32 PRNG to use. Options: "crypto" (20 rounds) or "light" (2 rounds). Defaults to "light" for better performance.

    *)
  8. mutable large_models : Base.bool;
    (*

    If true, use uint64 for indexing arithmetic. If false, use uint32 for indexing arithmetic. This affects all backends' kernel index parameters and local index variables, and gates the per-pool offset width (uint32 caps a pool at 4 GB; see the pool allocator). Decoupled in intent from element indexing within a single tensor, though both currently follow it.

    *)
}
val settings_of_sexp : Sexplib0.Sexp.t -> settings
val sexp_of_settings : settings -> Sexplib0.Sexp.t
val settings : settings
val accessed_global_args : Base.String.t Base.Hash_set.t
val str_nonempty : f:(Base.String.t -> 'a) -> Base.String.t -> 'a option
val pair : 'a -> 'b -> 'a * 'b
val known_config_keys : (Base.String.t, Base.String.comparator_witness) Base.Set.t
val bool_of_config_string : arg_name:Base.String.t -> Base.String.t -> bool
val log_config_sourcing : bool Base.ref

Whether to print where each configuration setting comes from (commandline, environment, config file, or the hard-coded default). It gates the logging of the config-reading functions themselves, so it is bootstrapped directly rather than via get_global_arg: the initial value only reflects the commandline and the environment, and the config file setting is applied once the config file is read. Can also be set programmatically, e.g. to trace configs a test reads.

val env_var_names : Base.String.t -> Base.String.t Base.List.t
val read_cmdline_var : ?qualified_only:bool -> Base.String.t -> (Base.String.t * Base.String.t) option

The commandline sublevel of get_global_arg: returns the setting's value and the Sys.argv element it came from. Pure -- the sourcing log lives at the resolution seam, which is the only place that knows which sublevel actually won.

qualified_only drops the prefix-free spellings, leaving the ocannl_-qualified ones. OCANNL is a library: it scans the host executable's Sys.argv, so a prefix-free key claims an application's own option of that name. That is tolerable for keys nobody else would spell (--virtualize_max_visits) and not for --profile, which is a common application flag and which OCANNL treats as fatal when it does not name a known bundle -- a host passing --profile=prod would die during module initialization (Codex P2 on PR #291).

val read_env_var : Base.String.t -> (Base.String.t * Base.String.t) option

The environment sublevel of get_global_arg: returns the setting's value and the variable it came from. An empty value counts as unset.

val read_cmdline_or_env_var : Base.string -> Base.String.t Base.Option.t

The bootstrap reader: the few keys that are consulted before the config file exists (and hence before profiles are resolved) come from the commandline or the environment only.

val filename_parts : string -> string list
val filename_of_parts : string list -> string
val log_config_sourcing_arg : Base.String.t Base.Option.t
val parse_config_lines : source:Base.String.t -> Base.String.t Base.List.t -> (Base.String.t * Base.String.t) Base.List.t

Parses the ocannl_config syntax: one key=value per line, # and ~~ lines are comments, empty values mean "unset", the ocannl_ key prefix is optional and keys are case-insensitive. Shared by the config file and by the embedded profile payloads (which are literally partial config files); source names the origin in error messages.

val config_table_of_lines : source:Base.String.t -> Base.String.t Base.List.t -> (Base.String.t, Base.String.t) Base.Hashtbl.t
val config_file_args : (Base.String.t, Base.String.t) Base.Hashtbl.t

Configuration profiles (gh-ocannl-559)

type config_level =
  1. | Cmdline_level
  2. | Env_level
  3. | Config_file_level

The source levels a setting can come from, in decreasing priority. Each level splits into two sublevels: the keys stated explicitly at that level, then the payload of a profile picked at that level -- so a specific setting always beats an aggregate one of equal immediacy, and a profile named on the commandline still overrides an exhaustive config file.

val equal_config_level : config_level -> config_level -> bool
val describe_config_level : config_level -> string
type config_source =
  1. | From_cmdline of Base.string
    (*

    the matching Sys.argv element

    *)
  2. | From_env of Base.string
    (*

    the matching environment variable

    *)
  3. | From_config_file
  4. | From_profile of config_level * Base.string
    (*

    the level that picked the profile, and its name

    *)
  5. | From_default

Where get_global_arg found a value; reported by the log_config_sourcing trace.

val config_source_label : config_source -> string

A short provenance tag, e.g. "profile 'reproducible' via the commandline".

val describe_config_source : value:string -> default:string -> config_source -> string
val resolve_config_value : cmdline:('a -> ('b * Base.string) Base.Option.t) -> env:('a -> ('b * Base.string) Base.Option.t) -> file:('a -> 'b Base.Option.t) -> profile:(config_level * Base.string * ('a -> 'b Base.Option.t)) option -> default:'b -> arg_name:'a -> 'b * config_source

The precedence walk, factored out of get_global_arg so it can be exercised on synthetic sources (see test/operations/config_profiles.ml). The lookups are pure; logging happens at the call site, which is the only place that knows which sublevel won.

val profile_ineligible_keys : (Base.String.t, Base.String.comparator_witness) Base.Set.t

The keys a profile payload may not set: they are read before profiles are resolved (or would make the resolution recursive).

val reproducible_profile_payload : string
val performance_profile_payload : string
val profile_payloads : (string * string) list

The embedded profile payloads, by name. Each is literally a partial ocannl_config file: same syntax, same parser, setting only the keys where the profiles' goals disagree.

val parse_profile_payload : name:string -> Base.String.t -> (Base.String.t, Base.String.t) Base.Hashtbl.t
val active_profile : (config_level * Base.String.t * (Base.String.t, Base.String.t) Base.Hashtbl.t) Base.Option.t

The profile picked for this run, if any: its level (which decides the priority of its payload), its name, and the parsed payload.

val profile_lookup : (config_level * Base.String.t * (Base.String.t Base.Hashtbl.key -> Base.String.t option)) Base.Option.t
val get_global_arg_with_source : default:Base.String.t -> arg_name:Base.string -> Base.String.t * config_source

Retrieves the arg_name setting from the commandline, the environment, the config file, or the payload of the profile picked at one of those levels; returns default if none has it, together with where the value came from.

val get_global_arg : default:Base.String.t -> arg_name:Base.string -> Base.String.t
val get_global_flag : default:bool -> arg_name:Base.String.t -> bool
val original_log_level : Base.Int.t
val filename_concat : Base.String.t -> Base.String.t -> Base.String.t
val clean_filename : Base.String.t -> Base.String.t
val artifacts_subdir : unit -> Base.String.t option
val ensure_artifacts_dir : Base.String.t -> Base.String.t Base.Option.t -> Base.String.t
val build_files_dir : unit -> Base.String.t

The directory generated-code debug files are written to (created if missing): build_files/<prefix>/, where the prefix defaults to the executable's base name.

val build_file : Base.String.t -> Base.String.t
val log_files_dir : unit -> Base.String.t

The directory diagnostic and routine-debug logs are written to (created if missing): log_files/<prefix>/, sharing the prefix resolution of build_files_dir.

val diagn_log_file : Base.String.t -> Base.String.t
val get_local_debug_runtime : unit -> (module Minidebug_runtime.Debug_runtime)
val _get_local_debug_runtime : unit -> (module Minidebug_runtime.Debug_runtime)
val set_log_level : Base.int -> unit
val restore_settings : unit -> unit
val with_runtime_debug : unit -> Base.bool
val debug_log_from_routines : unit -> Base.bool
val never_capture_stdout : unit -> bool
val enable_runtime_debug : unit -> unit
val union_find : equal:('a -> 'a -> bool) -> ('a, 'a, 'b) Base.Map.t -> key:'a -> rank:Base__Int.t -> 'a * Base__Int.t
val union_add : equal:('a -> 'a -> bool) -> ('a, 'a, 'b) Base.Map.t -> 'a -> 'a -> ('a, 'a, 'b) Base.Map.t
val unique_keep_first : equal:('a -> 'a -> bool) -> 'a list -> 'a Base.List.t

Filters the list keeping the first occurrence of each element.

val sorted_diff : compare:('a -> 'b -> int) -> 'a Base.List.t -> 'b list -> 'a Base.List.t

Returns the multiset difference of l1 and l2, where l1 and l2 must be sorted in increasing order.

val remove_elem : equal:('a -> 'b -> bool) -> 'a -> 'b list -> 'b Base.List.t

Removes the first occurrence of an element from the list that is equal to the given element.

val parallel_merge : (from:Base__Int.t -> to_:Base__Int.t -> unit) -> Base.int -> Base.unit

parallel_merge merge num_devices progressively invokes the pairwise merge callback, converging on the 0th position, with from ranging from 1 to num_devices - 1, and to_ < from.

val (!@) : 'a Stdlib.Atomic.t -> 'a
type atomic_bool = Base.bool Stdlib.Atomic.t
val sexp_of_atomic_bool : Base.bool Stdlib.Atomic.t -> Sexplib0.Sexp.t
type atomic_int = Base.int Stdlib.Atomic.t
val sexp_of_atomic_int : Base.int Stdlib.Atomic.t -> Sexplib0.Sexp.t
val sexp_append : elem:Base.Sexp.t -> Base.Sexp.t -> Base.Sexp.t
val sexp_mem : elem:Base.Sexp.t -> Base.Sexp.t -> bool
val sexp_deep_mem : elem:Base.Sexp.t -> Base.Sexp.t -> Base.bool
val split_with_seps : Re.re -> string -> string Base.List.t
module Lazy : sig ... end
type requirement =
  1. | Skip
  2. | Required
  3. | Optional of {
    1. callback_if_missing : Base.unit -> Base.unit;
    }
val compare_requirement : requirement -> requirement -> Base.int
val requirement_of_sexp : Sexplib0.Sexp.t -> requirement
val sexp_of_requirement : requirement -> Sexplib0.Sexp.t
val default_indent : int Base.ref
val doc_of_sexp : Base.Sexp.t -> PPrint.document
val output_to_build_file : fname:Base.String.t -> (PPrint.ToChannel.document -> Base.unit) option
val get_debug_output_channel : fname:Base.String.t -> Stdio.Out_channel.t option
exception User_error of Base.string
val header_sep : Re.re
val log_trace_tree : 'a -> unit
module Set_O : sig ... end
val no_ints : (Base.Int.t, Base.Int.comparator_witness) Base.Set.t
val one_int : Base.Int.t -> (Base.Int.t, Base.Int.comparator_witness) Base.Set.t
val map_merge : ('a, 'b, 'c) Base.Map.t -> ('a, 'b, 'c) Base.Map.t -> f:('b -> 'b -> 'b) -> ('a, 'b, 'c) Base.Map.t
val mref_add : ('a, 'b, 'c) Base.Map.t Base.ref -> key:'a -> data:'b -> or_:('b -> Base.unit) -> Base.unit
val mref_add_missing : ('a, 'b, 'c) Base.Map.t Base.ref -> 'a -> f:(unit -> 'b) -> Base.unit
type 'a mutable_list =
  1. | Empty
  2. | Cons of {
    1. hd : 'a;
    2. mutable tl : 'a mutable_list;
    }

A mutable linked list structure.

val equal_mutable_list : 'a. ('a -> 'a -> Base.bool) -> 'a mutable_list -> 'a mutable_list -> Base.bool
val mutable_list_of_sexp : 'a. (Sexplib0.Sexp.t -> 'a) -> Sexplib0.Sexp.t -> 'a mutable_list
val sexp_of_mutable_list : 'a. ('a -> Sexplib0.Sexp.t) -> 'a mutable_list -> Sexplib0.Sexp.t
val empty : 'a mutable_list
val cons : hd:'a -> tl:'a mutable_list -> 'a mutable_list
val is_empty : 'a mutable_list -> bool
val is_cons : 'a mutable_list -> bool
val empty_val : 'a mutable_list -> unit Stdlib.Option.t
val cons_val : 'a mutable_list -> ([> `hd of 'a ] * [> `tl of 'a mutable_list ]) Stdlib.Option.t
module Variants_of_mutable_list : sig ... end
val insert : next:'a -> 'a mutable_list -> 'a mutable_list
val tl_exn : 'a mutable_list -> 'a mutable_list
type 'a weak_dynarray = 'a Stdlib.Weak.t Base.ref

A dynamic array of weak references.

val weak_create : unit -> 'a weak_dynarray
val sexp_of_weak_dynarray : ('a -> Sexplib0.Sexp.t) -> 'a Stdlib.Weak.t Base.ref -> Sexplib0.Sexp.t
val register_new : 'a weak_dynarray -> ?grow_by:Base__Int.t -> (int -> 'a) -> 'a
val weak_iter : 'a weak_dynarray -> f:('a -> unit) -> unit
type 'a safe_lazy = {
  1. mutable value : [ `Callback of Base.unit -> 'a | `Value of 'a ];
  2. unique_id : Base.string;
}

A lazy value that can be safely forced and compared by unique ID.

val safe_lazy : Base.string -> (Base.unit -> 'a) -> 'a safe_lazy
val safe_force : 'a safe_lazy -> 'a
val is_safe_val : 'a safe_lazy -> bool
val safe_map : upd:Base.String.t -> f:('a -> 'b) -> 'a safe_lazy -> 'b safe_lazy
val equal_safe_lazy : ('a -> 'b -> bool) -> 'a safe_lazy -> 'b safe_lazy -> bool
val compare_safe_lazy : ('a -> 'b -> int) -> 'a safe_lazy -> 'b safe_lazy -> int
val hash_fold_safe_lazy : 'a -> Base__.Ppx_hash_lib.Std.Hash.state -> 'b safe_lazy -> Base__.Ppx_hash_lib.Std.Hash.state
val sexp_of_safe_lazy : ('a -> Base.Sexp.t) -> 'a safe_lazy -> Base.Sexp.t
module Tree_map : sig ... end

A persistent map implemented as a balanced binary tree. The sexp_of function preserves and displays the tree structure.

type build_file_channel = {
  1. f_path : Base.string;
  2. oc : Stdlib.out_channel;
  3. finalize : Base.unit -> Base.unit;
}
val open_build_file : base_name:Base.String.t -> extension:Base.String.t -> build_file_channel
val captured_log_prefix : string Base.ref
type captured_log_processor = {
  1. log_processor_prefix : Base.string;
  2. process_logs : Base.string Base.list -> Base.unit;
}
val captured_log_processors : captured_log_processor Base.list Base.ref
val add_log_processor : prefix:Base.string -> (Base.string Base.list -> Base.unit) -> Base.unit
val input_scan_line : Stdlib.in_channel -> Base.int
val flush_c_streams : Base.unit -> Base.unit
val input_line : Stdlib.in_channel -> bool * Base.String.t
val capture_stdout_logs : (unit -> 'a) -> 'a
val log_debug_routine_logs : log_contents:Base.string Base.List.t -> stream_name:Base.String.t -> unit
val log_debug_routine_file : log_file_name:Base.string -> stream_name:Base.String.t -> unit
val gcd : Base__Int.t -> Base__Int.t -> Base__Int.t