push sheeet

pkgs/development/misc/resholve/README.md

@@ -0,0 +1,336 @@
+# Using resholve's Nix API
+resholve replaces bare references (subject to a PATH search at runtime) to external commands and scripts with absolute paths.
+
+This small super-power helps ensure script dependencies are declared, present, and don't unexpectedly shift when the PATH changes.
+
+resholve is developed to enable the Nix package manager to package and integrate Shell projects, but its features are not Nix-specific and inevitably have other applications.
+
+<!-- generated from resholve's repo; best to suggest edits there (or at least notify me) -->
+
+This will hopefully make its way into the Nixpkgs manual soon, but
+until then I'll outline how to use the functions:
+- `resholve.mkDerivation` (formerly `resholvePackage`)
+- `resholve.writeScript` (formerly `resholveScript`)
+- `resholve.writeScriptBin` (formerly `resholveScriptBin`)
+- `resholve.phraseSolution` (new in resholve 0.8.0)
+
+> Fair warning: resholve does *not* aspire to resolving all valid Shell
+> scripts. It depends on the OSH/Oil parser, which aims to support most (but
+> not all) Bash. resholve aims to be a ~90% sort of solution.
+
+## API Concepts
+
+The main difference between `resholve.mkDerivation` and other builder functions
+is the `solutions` attrset, which describes which scripts to resolve and how.
+Each "solution" (k=v pair) in this attrset describes one resholve invocation.
+
+> NOTE: For most shell packages, one invocation will probably be enough:
+> - Packages with a single script will only need one solution.
+> - Packages with multiple scripts can still use one solution if the scripts
+>   don't require conflicting directives.
+> - Packages with scripts that require conflicting directives can use multiple
+>   solutions to resolve the scripts separately, but produce a single package.
+
+`resholve.writeScript` and `resholve.writeScriptBin` support a _single_
+`solution` attrset. This is basically the same as any single solution in `resholve.mkDerivation`, except that it doesn't need a `scripts` attr (it is automatically added). `resholve.phraseSolution` also only accepts a single solution--but it _does_ still require the `scripts` attr.
+
+## Basic `resholve.mkDerivation` Example
+
+Here's a simple example of how `resholve.mkDerivation` is already used in nixpkgs:
+
+<!-- TODO: figure out how to pull this externally? -->
+
+```nix
+{
+  bash,
+  coreutils,
+  gnused,
+  goss,
+  lib,
+  resholve,
+  which,
+}:
+
+resholve.mkDerivation rec {
+  pname = "dgoss";
+  version = goss.version;
+  src = goss.src;
+
+  dontConfigure = true;
+  dontBuild = true;
+
+  installPhase = ''
+    sed -i '2i GOSS_PATH=${goss}/bin/goss' extras/dgoss/dgoss
+    install -D extras/dgoss/dgoss $out/bin/dgoss
+  '';
+
+  solutions = {
+    default = {
+      scripts = [ "bin/dgoss" ];
+      interpreter = "${bash}/bin/bash";
+      inputs = [
+        coreutils
+        gnused
+        which
+      ];
+      keep = {
+        "$CONTAINER_RUNTIME" = true;
+      };
+    };
+  };
+
+  meta = {
+    homepage = "https://github.com/goss-org/goss/blob/v${version}/extras/dgoss/README.md";
+    changelog = "https://github.com/goss-org/goss/releases/tag/v${version}";
+    description = "Convenience wrapper around goss that aims to bring the simplicity of goss to docker containers";
+    license = lib.licenses.asl20;
+    platforms = lib.platforms.linux;
+    maintainers = with lib.maintainers; [
+      hyzual
+      anthonyroussel
+    ];
+    mainProgram = "dgoss";
+  };
+}
+```
+
+
+## Basic `resholve.writeScript` and `resholve.writeScriptBin` examples
+
+Both of these functions have the same basic API. The examples are a little
+trivial, so I'll also link to some real-world examples:
+- [shell.nix from abathur/tdverpy](https://github.com/abathur/tdverpy/blob/e1f956df3ed1c7097a5164e0c85b178772e277f5/shell.nix#L6-L13)
+
+```nix
+{
+  resholvedScript =
+    resholve.writeScript "name"
+      {
+        inputs = [ file ];
+        interpreter = "${bash}/bin/bash";
+      }
+      ''
+        echo "Hello"
+        file .
+      '';
+  resholvedScriptBin =
+    resholve.writeScriptBin "name"
+      {
+        inputs = [ file ];
+        interpreter = "${bash}/bin/bash";
+      }
+      ''
+        echo "Hello"
+        file .
+      '';
+}
+```
+
+
+## Basic `resholve.phraseSolution` example
+
+This function has a similar API to `writeScript` and `writeScriptBin`, except it does require a `scripts` attr. It is intended to make resholve a little easier to mix into more types of build. This example is a little
+trivial for now. If you have a real usage that you find helpful, please PR it.
+
+```nix
+{
+  stdenv,
+  resholve,
+  module1,
+}:
+
+stdenv.mkDerivation {
+  # pname = "testmod3";
+  # version = "unreleased";
+  # src = ...;
+
+  installPhase = ''
+    mkdir -p $out/bin
+    install conjure.sh $out/bin/conjure.sh
+    ${resholve.phraseSolution "conjure" {
+      scripts = [ "bin/conjure.sh" ];
+      interpreter = "${bash}/bin/bash";
+      inputs = [ module1 ];
+      fake = {
+        external = [
+          "jq"
+          "openssl"
+        ];
+      };
+    }}
+  '';
+}
+```
+
+
+## Options
+
+`resholve.mkDerivation` maps Nix types/idioms into the flags and environment variables
+that the `resholve` CLI expects. Here's an overview:
+
+| Option | Type | Containing |
+|--------|------|------------|
+| scripts | `<list>` | scripts to resolve (`$out`-relative paths) |
+| interpreter | `"none"` `<path>` | The absolute interpreter `<path>` for the script's shebang. The special value `none` ensures there is no shebang. |
+| inputs | `<packages>` `<paths>` | A list of packages and string paths to directories/files to resolve external dependencies from. |
+| fake | `<directives>` | pretend some commands exist |
+| fix | `<directives>` | fix things we can't auto-fix/ignore |
+| keep | `<directives>` | keep things we can't auto-fix/ignore |
+| lore | `<directory>` | control nested resolution |
+| execer | `<statements>` | modify nested resolution |
+| wrapper | `<statements>` | modify nested resolution |
+| prologue | `<file>` | insert file before resolved script |
+| epilogue | `<file>` | insert file after resolved script |
+
+<!-- TODO: section below is largely custom for nixpkgs, but I would LIKE to wurst it. -->
+
+## Controlling resolution with directives
+
+In order to resolve a script, resholve will make you disambiguate how it should
+handle any potential problems it encounters with directives. There are currently
+3 types:
+1. `fake` directives tell resholve to pretend it knows about an identifier
+   such as a function, builtin, external command, etc. if there's a good reason
+   it doesn't already know about it. Common examples:
+   - builtins for a non-bash shell
+   - loadable builtins
+   - platform-specific external commands in cross-platform conditionals
+2. `fix` directives give resholve permission to fix something that it can't
+   safely fix automatically. Common examples:
+   - resolving commands in aliases (this is appropriate for standalone scripts
+     that use aliases non-interactively--but it would prevent profile/rc
+     scripts from using the latest current-system symlinks.)
+   - resolve commands in a variable definition
+   - resolve an absolute command path from inputs as if it were a bare reference
+   - force resholve to resolve known security wrappers
+3. `keep` directives tell resholve not to raise an error (i.e., ignore)
+   something it would usually object to. Common examples:
+   - variables used as/within the first word of a command
+   - pre-existing absolute or user-relative (~) command paths
+   - dynamic (variable) arguments to commands known to accept/run other commands
+
+> NOTE: resholve has a (growing) number of directives detailed in `man resholve`
+> via `nixpkgs.resholve` (though protections against run-time use of python2 in nixpkgs mean you'll have to set `NIXPKGS_ALLOW_INSECURE=1` to pull resholve into nix-shell).
+
+Each of these 3 types is represented by its own attrset, where you can think
+of the key as a scope. The value should be:
+- `true` for any directives that the resholve CLI accepts as a single word
+- a list of strings for all other options
+<!--
+TODO: these should be fully-documented here, but I'm already maintaining
+more copies of their specification/behavior than I like, and continuing to
+add more at this early date will only ensure that I spend more time updating
+docs and less time filling in feature gaps.
+
+Full documentation may be greatly accelerated if someone can help me sort out
+single-sourcing. See: https://github.com/abathur/resholve/issues/19
+-->
+
+This will hopefully make more sense when you see it. Here are CLI examples
+from the manpage, and the Nix equivalents:
+
+```nix
+{
+  # --fake 'f:setUp;tearDown builtin:setopt source:/etc/bashrc'
+  fake = {
+    # fake accepts the initial of valid identifier types as a CLI convenience.
+    # Use full names in the Nix API.
+    function = [
+      "setUp"
+      "tearDown"
+    ];
+    builtin = [ "setopt" ];
+    source = [ "/etc/bashrc" ];
+  };
+
+  # --fix 'aliases $GIT:gix /bin/bash'
+  fix = {
+    # all single-word directives use `true` as value
+    aliases = true;
+    "$GIT" = [ "gix" ];
+    "/bin/bash" = true;
+  };
+
+  # --keep 'source:$HOME /etc/bashrc ~/.bashrc'
+  keep = {
+    source = [ "$HOME" ];
+    "/etc/bashrc" = true;
+    "~/.bashrc" = true;
+  };
+}
+```
+
+
+> **Note:** For now, at least, you'll need to reference the manpage to completely understand these examples.
+
+## Controlling nested resolution with lore
+
+Initially, resolution of commands in the arguments to command-executing
+commands was limited to one level for a hard-coded list of builtins and
+external commands. resholve can now resolve these recursively.
+
+This feature combines information (_lore_) that the resholve Nix API
+obtains via binlore ([nixpkgs](../../tools/analysis/binlore), [repo](https://github.com/abathur/resholve)),
+with some rules (internal to resholve) for locating sub-executions in
+some of the more common commands.
+
+- "execer" lore identifies whether an executable can, cannot,
+  or might execute its arguments. Every "can" or "might" verdict requires:
+  - an update to the matching rules in [binlore](https://github.com/abathur/binlore)
+    if there's absolutely no exec in the executable and binlore just lacks
+    rules for understanding this
+  - an override in [binlore](https://github.com/abathur/binlore) if there is
+    exec but it isn't actually under user control
+  - a parser in [resholve](https://github.com/abathur/resholve) capable of
+    isolating the exec'd words if the command does have exec under user
+    control
+  - overriding the execer lore for the executable if manual triage indicates
+    that all of the invocations in the current package don't include any
+    commands that the executable would exec
+  - if manual triage turns up any commands that would be exec'd, use some
+    non-resholve tool to patch/substitute/replace them before or after you
+    run resholve on them (if before, you may need to also add keep directives
+    for these absolute paths)
+
+- "wrapper" lore maps shell exec wrappers to the programs they exec so
+  that resholve can substitute an executable's verdict for its wrapper's.
+
+> **Caution:** At least when it comes to common utilities, it's best to treat
+> overrides as a stopgap until they can be properly handled in resholve and/or
+> binlore. Please report things you have to override and, if possible, help
+> get them sorted.
+
+There will be more mechanisms for controlling this process in the future
+(and your reports/experiences will play a role in shaping them...) For now,
+the main lever is the ability to substitute your own lore. This is how you'd
+do it piecemeal:
+
+```nix
+{
+  # --execer 'cannot:${openssl.bin}/bin/openssl can:${openssl.bin}/bin/c_rehash'
+  execer = [
+    /*
+      This is the same verdict binlore will
+      come up with. It's a no-op just to demo
+      how to fiddle lore via the Nix API.
+    */
+    "cannot:${openssl.bin}/bin/openssl"
+    # different verdict, but not used
+    "can:${openssl.bin}/bin/c_rehash"
+  ];
+
+  # --wrapper '${gnugrep}/bin/egrep:${gnugrep}/bin/grep'
+  wrapper = [
+    /*
+      This is the same verdict binlore will
+      come up with. It's a no-op just to demo
+      how to fiddle lore via the Nix API.
+    */
+    "${gnugrep}/bin/egrep:${gnugrep}/bin/grep"
+  ];
+}
+```
+
+
+The format is fairly simple to generate--you can script your own generator if
+you need to modify the lore.

pkgs/development/misc/resholve/default.nix

@@ -0,0 +1,62 @@
+{
+  lib,
+  pkgsBuildHost,
+  resholve,
+}:
+
+let
+  removeKnownVulnerabilities =
+    pkg:
+    pkg.overrideAttrs (old: {
+      meta = (old.meta or { }) // {
+        knownVulnerabilities = [ ];
+      };
+    });
+  # We are removing `meta.knownVulnerabilities` from `python27`,
+  # and setting it in `resholve` itself.
+  python27' = (removeKnownVulnerabilities pkgsBuildHost.python27).override {
+    self = python27';
+    pkgsBuildHost = pkgsBuildHost // {
+      python27 = python27';
+    };
+    # strip down that python version as much as possible
+    openssl = null;
+    bzip2 = null;
+    readline = null;
+    ncurses = null;
+    gdbm = null;
+    sqlite = null;
+    rebuildBytecode = false;
+    stripBytecode = true;
+    strip2to3 = true;
+    stripConfig = true;
+    stripIdlelib = true;
+    stripTests = true;
+    enableOptimizations = false;
+  };
+  callPackage = lib.callPackageWith (pkgsBuildHost // { python27 = python27'; });
+  source = callPackage ./source.nix { };
+  deps = callPackage ./deps.nix { };
+  # not exposed in all-packages
+  resholveBuildTimeOnly = removeKnownVulnerabilities resholve;
+in
+rec {
+  # resholve itself
+  resholve = (
+    callPackage ./resholve.nix {
+      inherit (source) rSrc version;
+      inherit (deps.oil) oildev;
+      inherit (deps) configargparse;
+      inherit resholve-utils;
+      # used only in tests
+      resholve = resholveBuildTimeOnly;
+    }
+  );
+  # funcs to validate and phrase invocations of resholve
+  # and use those invocations to build packages
+  resholve-utils = callPackage ./resholve-utils.nix {
+    # we can still use resholve-utils without triggering a security warn
+    # this is safe since we will only use `resholve` at build time
+    resholve = resholveBuildTimeOnly;
+  };
+}

pkgs/development/misc/resholve/deps.nix

@@ -0,0 +1,78 @@
+{
+  lib,
+  callPackage,
+  fetchFromGitHub,
+  python27,
+  fetchPypi,
+  ...
+}:
+
+/*
+  Notes on specific dependencies:
+  - if/when python2.7 is removed from nixpkgs, this may need to figure
+  out how to build oil's vendored python2
+*/
+
+rec {
+  oil = callPackage ./oildev.nix {
+    inherit python27;
+    inherit six;
+    inherit typing;
+  };
+  configargparse = python27.pkgs.buildPythonPackage rec {
+    pname = "configargparse";
+    version = "1.5.3";
+    format = "setuptools";
+
+    src = fetchFromGitHub {
+      owner = "bw2";
+      repo = "ConfigArgParse";
+      rev = "v${version}";
+      sha256 = "1dsai4bilkp2biy9swfdx2z0k4akw4lpvx12flmk00r80hzgbglz";
+    };
+
+    doCheck = false;
+
+    pythonImportsCheck = [ "configargparse" ];
+
+    meta = with lib; {
+      description = "Drop-in replacement for argparse";
+      homepage = "https://github.com/bw2/ConfigArgParse";
+      license = licenses.mit;
+    };
+  };
+  six = python27.pkgs.buildPythonPackage rec {
+    pname = "six";
+    version = "1.16.0";
+
+    src = fetchPypi {
+      inherit pname version;
+      sha256 = "1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926";
+    };
+
+    doCheck = false;
+
+    meta = {
+      description = "Python 2 and 3 compatibility library";
+      homepage = "https://pypi.python.org/pypi/six/";
+      license = lib.licenses.mit;
+    };
+  };
+  typing = python27.pkgs.buildPythonPackage rec {
+    pname = "typing";
+    version = "3.10.0.0";
+
+    src = fetchPypi {
+      inherit pname version;
+      sha256 = "13b4ad211f54ddbf93e5901a9967b1e07720c1d1b78d596ac6a439641aa1b130";
+    };
+
+    doCheck = false;
+
+    meta = with lib; {
+      description = "Backport of typing module to Python versions older than 3.5";
+      homepage = "https://docs.python.org/3/library/typing.html";
+      license = licenses.psfl;
+    };
+  };
+}

pkgs/development/misc/resholve/oildev.nix

@@ -0,0 +1,121 @@
+{
+  lib,
+  stdenv,
+  python27,
+  callPackage,
+  fetchFromGitHub,
+  makeWrapper,
+  re2c,
+  # oil deps
+  glibcLocales,
+  file,
+  six,
+  typing,
+}:
+
+{
+  /*
+    Upstream isn't interested in packaging this as a library
+    (or accepting all of the patches we need to do so).
+    This creates one without disturbing upstream too much.
+  */
+  oildev = python27.pkgs.buildPythonPackage rec {
+    pname = "oildev-unstable";
+    version = "2024-02-26";
+    format = "setuptools";
+
+    src = fetchFromGitHub {
+      owner = "oilshell";
+      repo = "oil";
+      # rev == present HEAD of release/0.20.0
+      rev = "f730c79e2dcde4bc08e85a718951cfa42102bd01";
+      hash = "sha256-HBj3Izh1gD63EzbgZ/9If5vihR5L2HhnyCyMah6rMg4=";
+
+      /*
+        It's not critical to drop most of these; the primary target is
+        the vendored fork of Python-2.7.13, which is ~ 55M and over 3200
+        files, dozens of which get interpreter script patches in fixup.
+
+        Note: -f is necessary to keep it from being a pain to update
+        hash on rev updates. Command will fail w/o and not print hash.
+      */
+      postFetch = ''
+        rm -rf $out/{Python-2.7.13,metrics,py-yajl,rfc,gold,web,testdata,services,demo}
+      '';
+    };
+
+    # patch to support a python package, pass tests on macOS, drop deps, etc.
+    patchSrc = fetchFromGitHub {
+      owner = "abathur";
+      repo = "nix-py-dev-oil";
+      rev = "v0.20.0.0";
+      hash = "sha256-qoA54rnzAdnFZ3k4kRzQWEdgtEjraCT5+NFw8AWnRDk=";
+    };
+
+    patches = [
+      "${patchSrc}/0001-add_setup_py.patch"
+      "${patchSrc}/0002-add_MANIFEST_in.patch"
+      "${patchSrc}/0006-disable_failing_libc_tests.patch"
+      "${patchSrc}/0007-namespace_via_init.patch"
+      "${patchSrc}/0009-avoid_nix_arch64_darwin_toolchain_bug.patch"
+      "${patchSrc}/0010-disable-line-input.patch"
+      "${patchSrc}/0011-disable-fanos.patch"
+      "${patchSrc}/0012-disable-doc-cmark.patch"
+      "${patchSrc}/0013-fix-pyverify.patch"
+      "${patchSrc}/0015-fix-compiled-extension-import-paths.patch"
+    ];
+
+    configureFlags = [
+      "--without-readline"
+    ];
+
+    nativeBuildInputs = [
+      re2c
+      file
+      makeWrapper
+    ];
+
+    propagatedBuildInputs = [
+      six
+      typing
+    ];
+
+    doCheck = true;
+
+    preBuild = ''
+      build/py.sh all
+    '';
+
+    postPatch = ''
+      patchShebangs asdl build core doctools frontend pyext oil_lang ysh
+      rm cpp/stdlib.h # keep modules from finding the wrong stdlib?
+      # work around hard parse failure documented in oilshell/oil#1468
+      substituteInPlace osh/cmd_parse.py --replace 'elif self.c_id == Id.Op_LParen' 'elif False'
+    '';
+
+    # See earlier note on glibcLocales TODO: verify needed?
+    LOCALE_ARCHIVE = lib.optionalString (
+      stdenv.buildPlatform.libc == "glibc"
+    ) "${glibcLocales}/lib/locale/locale-archive";
+
+    # not exhaustive; sample what resholve uses as a sanity check
+    pythonImportsCheck = [
+      "oil"
+      "oil.asdl"
+      "oil.core"
+      "oil.frontend"
+      "oil._devbuild"
+      "oil._devbuild.gen.id_kind_asdl"
+      "oil._devbuild.gen.syntax_asdl"
+      "oil.osh"
+      "oil.tools.ysh_ify"
+    ];
+
+    meta = {
+      license = with lib.licenses; [
+        psfl # Includes a portion of the python interpreter and standard library
+        asl20 # Licence for Oil itself
+      ];
+    };
+  };
+}

pkgs/development/misc/resholve/resholve-utils.nix

@@ -0,0 +1,293 @@
+{
+  lib,
+  stdenv,
+  resholve,
+  binlore,
+  writeTextFile,
+}:
+
+rec {
+  /*
+    These functions break up the work of partially validating the
+    'solutions' attrset and massaging it into env/cli args.
+
+    Note: some of the left-most args do not *have* to be passed as
+    deep as they are, but I've done so to provide more error context
+  */
+
+  # for brevity / line length
+  spaces = l: builtins.concatStringsSep " " l;
+  colons = l: builtins.concatStringsSep ":" l;
+  semicolons = l: builtins.concatStringsSep ";" l;
+
+  # Throw a fit with dotted attr path context
+  nope = path: msg: throw "${builtins.concatStringsSep "." path}: ${msg}";
+
+  # Special-case directive value representations by type
+  phraseDirective =
+    solution: env: name: val:
+    if builtins.isInt val then
+      toString val
+    else if builtins.isString val then
+      name
+    else if true == val then
+      name
+    else if false == val then
+      "" # omit!
+    else if null == val then
+      "" # omit!
+    else if builtins.isList val then
+      "${name}:${semicolons (map lib.escapeShellArg val)}"
+    else
+      nope [
+        solution
+        env
+        name
+      ] "unexpected type: ${builtins.typeOf val}";
+
+  # Build fake/fix/keep directives from Nix types
+  phraseDirectives =
+    solution: env: val:
+    lib.mapAttrsToList (phraseDirective solution env) val;
+
+  # Custom ~search-path routine to handle relative path strings
+  relSafeBinPath =
+    input:
+    if lib.isDerivation input then
+      ((lib.getOutput "bin" input) + "/bin")
+    else if builtins.isString input then
+      input
+    else
+      throw "unexpected type for input: ${builtins.typeOf input}";
+
+  # Special-case value representation by type/name
+  phraseEnvVal =
+    solution: env: val:
+    if env == "inputs" then
+      (colons (map relSafeBinPath val))
+    else if builtins.isString val then
+      val
+    else if builtins.isList val then
+      spaces val
+    else if builtins.isAttrs val then
+      spaces (phraseDirectives solution env val)
+    else
+      nope [
+        solution
+        env
+      ] "unexpected type: ${builtins.typeOf val}";
+
+  # Shell-format each env value
+  shellEnv =
+    solution: env: value:
+    lib.escapeShellArg (phraseEnvVal solution env value);
+
+  # Build a single ENV=val pair
+  phraseEnv =
+    solution: env: value:
+    "RESHOLVE_${lib.toUpper env}=${shellEnv solution env value}";
+
+  /*
+    Discard attrs:
+    - claimed by phraseArgs
+    - only needed for binlore.collect
+  */
+  removeUnneededArgs =
+    value:
+    removeAttrs value [
+      "scripts"
+      "flags"
+      "unresholved"
+    ];
+
+  # Verify required arguments are present
+  validateSolution =
+    {
+      scripts,
+      inputs,
+      interpreter,
+      ...
+    }:
+    true;
+
+  # Pull out specific solution keys to build ENV=val pairs
+  phraseEnvs =
+    solution: value: spaces (lib.mapAttrsToList (phraseEnv solution) (removeUnneededArgs value));
+
+  # Pull out specific solution keys to build CLI argstring
+  phraseArgs =
+    {
+      flags ? [ ],
+      scripts,
+      ...
+    }:
+    spaces (flags ++ scripts);
+
+  phraseBinloreArgs =
+    value:
+    let
+      hasUnresholved = builtins.hasAttr "unresholved" value;
+    in
+    {
+      drvs = value.inputs ++ lib.optionals hasUnresholved [ value.unresholved ];
+      strip = if hasUnresholved then [ value.unresholved ] else [ ];
+    };
+
+  # Build a single resholve invocation
+  phraseInvocation =
+    solution: value:
+    if validateSolution value then
+      # we pass resholve a directory
+      "RESHOLVE_LORE=${binlore.collect (phraseBinloreArgs value)} ${phraseEnvs solution value} ${resholve}/bin/resholve --overwrite ${phraseArgs value}"
+    else
+      throw "invalid solution"; # shouldn't trigger for now
+
+  injectUnresholved =
+    solutions: unresholved:
+    (builtins.mapAttrs (name: value: value // { inherit unresholved; }) solutions);
+
+  # Build resholve invocation for each solution.
+  phraseCommands =
+    solutions: unresholved:
+    builtins.concatStringsSep "\n" (
+      lib.mapAttrsToList phraseInvocation (injectUnresholved solutions unresholved)
+    );
+
+  /*
+    subshell/PS4/set -x and : command to output resholve envs
+    and invocation. Extra context makes it clearer what the
+    Nix API is doing, makes nix-shell debugging easier, etc.
+  */
+  phraseContext =
+    {
+      invokable,
+      prep ? ''cd "$out"'',
+    }:
+    ''
+      (
+        ${prep}
+        PS4=$'\x1f'"\033[33m[resholve context]\033[0m "
+        set -x
+        : invoking resholve with PWD=$PWD
+        ${invokable}
+      )
+    '';
+  phraseContextForPWD =
+    invokable:
+    phraseContext {
+      inherit invokable;
+      prep = "";
+    };
+  phraseContextForOut = invokable: phraseContext { inherit invokable; };
+
+  phraseSolution = name: solution: (phraseContextForOut (phraseInvocation name solution));
+  phraseSolutions =
+    solutions: unresholved: phraseContextForOut (phraseCommands solutions unresholved);
+
+  writeScript =
+    name: partialSolution: text:
+    writeTextFile {
+      inherit name text;
+      executable = true;
+      checkPhase = ''
+        ${
+          (phraseContextForPWD (
+            phraseInvocation name (
+              partialSolution
+              // {
+                scripts = [ "${placeholder "out"}" ];
+              }
+            )
+          ))
+        }
+      ''
+      + lib.optionalString (partialSolution.interpreter != "none") ''
+        ${partialSolution.interpreter} -n $out
+      '';
+    };
+  writeScriptBin =
+    name: partialSolution: text:
+    writeTextFile rec {
+      inherit name text;
+      executable = true;
+      destination = "/bin/${name}";
+      checkPhase = ''
+        ${phraseContextForOut (
+          phraseInvocation name (
+            partialSolution
+            // {
+              scripts = [ "bin/${name}" ];
+            }
+          )
+        )}
+      ''
+      + lib.optionalString (partialSolution.interpreter != "none") ''
+        ${partialSolution.interpreter} -n $out/bin/${name}
+      '';
+    };
+  mkDerivation =
+    {
+      pname,
+      src,
+      version,
+      passthru ? { },
+      solutions,
+      ...
+    }@attrs:
+    let
+      inherit stdenv;
+
+      /*
+        Knock out our special solutions arg, but otherwise
+        just build what the caller is giving us. We'll
+        actually resholve it separately below (after we
+        generate binlore for it).
+      */
+      unresholved = (
+        stdenv.mkDerivation (
+          (removeAttrs attrs [ "solutions" ])
+          // {
+            inherit version src;
+            pname = "${pname}-unresholved";
+          }
+        )
+      );
+    in
+    /*
+      resholve in a separate derivation; some concerns:
+      - we aren't keeping many of the user's args, so they
+        can't readily set LOGLEVEL and such...
+      - not sure how this affects multiple outputs
+    */
+    lib.extendDerivation true passthru (
+      stdenv.mkDerivation {
+        src = unresholved;
+        inherit version pname;
+        buildInputs = [ resholve ];
+        disallowedReferences = [ resholve ];
+
+        # retain a reference to the base
+        passthru = unresholved.passthru // {
+          unresholved = unresholved;
+          # fallback attr for update bot to query our src
+          originalSrc = unresholved.src;
+        };
+
+        # do these imply that we should use NoCC or something?
+        dontConfigure = true;
+        dontBuild = true;
+
+        installPhase = ''
+          cp -R $src $out
+        '';
+
+        # enable below for verbose debug info if needed
+        # supports default python.logging levels
+        # LOGLEVEL="INFO";
+        preFixup = phraseSolutions solutions unresholved;
+
+        # don't break the metadata...
+        meta = unresholved.meta;
+      }
+    );
+}

pkgs/development/misc/resholve/resholve.nix

@@ -0,0 +1,96 @@
+{
+  lib,
+  callPackage,
+  python27,
+  fetchFromGitHub,
+  installShellFiles,
+  rSrc,
+  version,
+  oildev,
+  configargparse,
+  gawk,
+  binlore,
+  resholve,
+  resholve-utils,
+}:
+
+let
+  sedparse = python27.pkgs.buildPythonPackage {
+    pname = "sedparse";
+    version = "0.1.2";
+    format = "setuptools";
+    src = fetchFromGitHub {
+      owner = "aureliojargas";
+      repo = "sedparse";
+      rev = "0.1.2";
+      hash = "sha256-Q17A/oJ3GZbdSK55hPaMdw85g43WhTW9tuAuJtDfHHU=";
+    };
+  };
+
+in
+python27.pkgs.buildPythonApplication {
+  pname = "resholve";
+  inherit version;
+  src = rSrc;
+
+  nativeBuildInputs = [ installShellFiles ];
+
+  propagatedBuildInputs = [
+    oildev
+    configargparse
+    sedparse
+  ];
+
+  makeWrapperArgs = [
+    "--prefix PATH : ${lib.makeBinPath [ gawk ]}"
+  ];
+
+  postPatch = ''
+    for file in setup.cfg _resholve/version.py; do
+      substituteInPlace $file --subst-var-by version ${version}
+    done
+  '';
+
+  postInstall = ''
+    installManPage resholve.1
+  '';
+
+  # Do not propagate Python; may be obsoleted by nixos/nixpkgs#102613
+  # for context on why, see abathur/resholve#20
+  postFixup = ''
+    rm $out/nix-support/propagated-build-inputs
+  '';
+
+  passthru = {
+    inherit (resholve-utils)
+      mkDerivation
+      phraseSolution
+      writeScript
+      writeScriptBin
+      ;
+    tests = callPackage ./test.nix {
+      inherit
+        rSrc
+        binlore
+        python27
+        resholve
+        ;
+    };
+  };
+
+  meta = with lib; {
+    description = "Resolve external shell-script dependencies";
+    homepage = "https://github.com/abathur/resholve";
+    changelog = "https://github.com/abathur/resholve/blob/v${version}/CHANGELOG.md";
+    license = with licenses; [ mit ];
+    maintainers = with maintainers; [ abathur ];
+    platforms = platforms.all;
+    knownVulnerabilities = [
+      ''
+        resholve depends on python27 (EOL). While it's safe to
+        run on trusted input in the build sandbox, you should
+        avoid running it on untrusted input.
+      ''
+    ];
+  };
+}

pkgs/development/misc/resholve/source.nix

@@ -0,0 +1,14 @@
+{
+  fetchFromGitHub,
+  ...
+}:
+
+rec {
+  version = "0.10.6";
+  rSrc = fetchFromGitHub {
+    owner = "abathur";
+    repo = "resholve";
+    rev = "v${version}";
+    hash = "sha256-iJEkfW60QO4nFp+ib2+DeDRsZviYFhWRQoBw1VAhzJY=";
+  };
+}

pkgs/development/misc/resholve/test.nix

@@ -0,0 +1,352 @@
+{
+  lib,
+  stdenv,
+  callPackage,
+  resholve,
+  shunit2,
+  coreutils,
+  gnused,
+  gnugrep,
+  findutils,
+  jq,
+  bash,
+  bats,
+  libressl,
+  openssl,
+  python27,
+  file,
+  gettext,
+  rSrc,
+  runDemo ? false,
+  binlore,
+  sqlite,
+  unixtools,
+  gawk,
+  rlwrap,
+  gnutar,
+  bc,
+  # override testing
+  esh,
+  getconf,
+  libarchive,
+  locale,
+  mount,
+  ncurses,
+  nixos-install-tools,
+  nixos-rebuild,
+  procps,
+  ps,
+  # known consumers
+  aaxtomp3,
+  arch-install-scripts,
+  bashup-events32,
+  dgoss,
+  git-ftp,
+  lesspipe,
+  locate-dominating-file,
+  mons,
+  msmtp,
+  nix-direnv,
+  pdf2odt,
+  pdfmm,
+  rancid,
+  s0ix-selftest-tool,
+  unix-privesc-check,
+  wgnord,
+  wsl-vpnkit,
+  xdg-utils,
+  yadm,
+  zxfer,
+}:
+
+let
+  default_packages = [
+    bash
+    file
+    findutils
+    gettext
+  ];
+  parsed_packages = [
+    coreutils
+    sqlite
+    unixtools.script
+    gnused
+    gawk
+    findutils
+    rlwrap
+    gnutar
+    bc
+    msmtp
+  ];
+in
+rec {
+  module1 = resholve.mkDerivation {
+    pname = "testmod1";
+    version = "unreleased";
+
+    src = rSrc;
+    setSourceRoot = "sourceRoot=$(echo */tests/nix/libressl)";
+
+    installPhase = ''
+      mkdir -p $out/{bin,submodule}
+      install libressl.sh $out/bin/libressl.sh
+      install submodule/helper.sh $out/submodule/helper.sh
+    '';
+
+    solutions = {
+      libressl = {
+        # submodule to demonstrate
+        scripts = [
+          "bin/libressl.sh"
+          "submodule/helper.sh"
+        ];
+        interpreter = "none";
+        inputs = [
+          jq
+          module2
+          libressl.bin
+        ];
+      };
+    };
+
+    is_it_okay_with_arbitrary_envs = "shonuff";
+  };
+  module2 = resholve.mkDerivation {
+    pname = "testmod2";
+    version = "unreleased";
+
+    src = rSrc;
+    setSourceRoot = "sourceRoot=$(echo */tests/nix/openssl)";
+
+    installPhase = ''
+      mkdir -p $out/bin $out/libexec
+      install openssl.sh $out/bin/openssl.sh
+      install libexec.sh $out/libexec/invokeme
+      install profile $out/profile
+    '';
+    # LOGLEVEL="DEBUG";
+    solutions = {
+      openssl = {
+        fix = {
+          aliases = true;
+        };
+        scripts = [
+          "bin/openssl.sh"
+          "libexec/invokeme"
+        ];
+        interpreter = "none";
+        inputs = [
+          shunit2
+          openssl.bin
+          "libexec"
+          "libexec/invokeme"
+        ];
+        execer = [
+          /*
+            This is the same verdict binlore will
+            come up with. It's a no-op just to demo
+            how to fiddle lore via the Nix API.
+          */
+          "cannot:${openssl.bin}/bin/openssl"
+          # different verdict, but not used
+          "can:${openssl.bin}/bin/c_rehash"
+        ];
+      };
+      profile = {
+        scripts = [ "profile" ];
+        interpreter = "none";
+        inputs = [ ];
+      };
+    };
+  };
+  # demonstrate that we could use resholve in larger build
+  module3 = stdenv.mkDerivation {
+    pname = "testmod3";
+    version = "unreleased";
+
+    src = rSrc;
+    setSourceRoot = "sourceRoot=$(echo */tests/nix/future_perfect_tense)";
+
+    installPhase = ''
+      mkdir -p $out/bin
+      install conjure.sh $out/bin/conjure.sh
+      ${resholve.phraseSolution "conjure" {
+        scripts = [ "bin/conjure.sh" ];
+        interpreter = "${bash}/bin/bash";
+        inputs = [ module1 ];
+        fake = {
+          external = [
+            "jq"
+            "openssl"
+          ];
+        };
+      }}
+    '';
+  };
+
+  cli = stdenv.mkDerivation {
+    name = "resholve-test";
+    src = rSrc;
+
+    dontBuild = true;
+
+    installPhase = ''
+      mkdir $out
+      cp *.ansi $out/
+    '';
+
+    doCheck = true;
+    buildInputs = [ resholve ];
+    nativeCheckInputs = [
+      coreutils
+      bats
+    ];
+    # LOGLEVEL="DEBUG";
+
+    # default path
+    RESHOLVE_PATH = "${lib.makeBinPath default_packages}";
+    # but separate packages for combining as needed
+    PKG_FILE = "${lib.makeBinPath [ file ]}";
+    PKG_FINDUTILS = "${lib.makeBinPath [ findutils ]}";
+    PKG_GETTEXT = "${lib.makeBinPath [ gettext ]}";
+    PKG_COREUTILS = "${lib.makeBinPath [ coreutils ]}";
+    RESHOLVE_LORE = "${binlore.collect {
+      drvs = default_packages ++ [ coreutils ] ++ parsed_packages;
+    }}";
+    PKG_PARSED = "${lib.makeBinPath parsed_packages}";
+
+    # explicit interpreter for demo suite; maybe some better way...
+    INTERP = "${bash}/bin/bash";
+
+    checkPhase = ''
+      patchShebangs .
+      mkdir empty_lore
+      touch empty_lore/{execers,wrappers}
+      export EMPTY_LORE=$PWD/empty_lore
+      printf "\033[33m============================= resholve test suite ===================================\033[0m\n" > test.ansi
+      if ./test.sh &>> test.ansi; then
+        cat test.ansi
+      else
+        cat test.ansi && exit 1
+      fi
+    ''
+    + lib.optionalString runDemo ''
+      printf "\033[33m============================= resholve demo ===================================\033[0m\n" > demo.ansi
+      if ./demo &>> demo.ansi; then
+        cat demo.ansi
+      else
+        cat demo.ansi && exit 1
+      fi
+    '';
+  };
+
+  # Caution: ci.nix asserts the equality of both of these w/ diff
+  resholvedScript =
+    resholve.writeScript "resholved-script"
+      {
+        inputs = [ file ];
+        interpreter = "${bash}/bin/bash";
+      }
+      ''
+        echo "Hello"
+        file .
+      '';
+  resholvedScriptBin =
+    resholve.writeScriptBin "resholved-script-bin"
+      {
+        inputs = [ file ];
+        interpreter = "${bash}/bin/bash";
+      }
+      ''
+        echo "Hello"
+        file .
+      '';
+  resholvedScriptBinNone =
+    resholve.writeScriptBin "resholved-script-bin"
+      {
+        inputs = [ file ];
+        interpreter = "none";
+      }
+      ''
+        echo "Hello"
+        file .
+      '';
+  # spot-check lore overrides
+  loreOverrides =
+    resholve.writeScriptBin "verify-overrides"
+      {
+        inputs = [
+          coreutils
+          esh
+          getconf
+          libarchive
+          locale
+          mount
+          ncurses
+          procps
+          ps
+        ]
+        ++ lib.optionals stdenv.hostPlatform.isLinux [
+          nixos-install-tools
+          nixos-rebuild
+        ];
+        interpreter = "none";
+        execer = [
+          "cannot:${esh}/bin/esh"
+        ];
+        fix = {
+          mount = true;
+        };
+      }
+      (
+        ''
+          env b2sum fake args
+          b2sum fake args
+          esh fake args
+          getconf fake args
+          bsdtar fake args
+          locale fake args
+          mount fake args
+          reset fake args
+          tput fake args
+          tset fake args
+          ps fake args
+          top fake args
+        ''
+        + lib.optionalString stdenv.hostPlatform.isLinux ''
+          nixos-generate-config fake args
+          nixos-rebuild fake args
+        ''
+      );
+
+  # ensure known consumers in nixpkgs keep working
+  inherit aaxtomp3;
+  inherit bashup-events32;
+  inherit bats;
+  inherit git-ftp;
+  inherit lesspipe;
+  inherit locate-dominating-file;
+  inherit mons;
+  inherit msmtp;
+  inherit nix-direnv;
+  inherit pdf2odt;
+  inherit pdfmm;
+  inherit shunit2;
+  inherit xdg-utils;
+  inherit yadm;
+}
+// lib.optionalAttrs stdenv.hostPlatform.isLinux {
+  inherit arch-install-scripts;
+  inherit dgoss;
+  inherit rancid;
+  inherit unix-privesc-check;
+  inherit wgnord;
+  inherit wsl-vpnkit;
+  inherit zxfer;
+}
+//
+  lib.optionalAttrs
+    (stdenv.hostPlatform.isLinux && (stdenv.hostPlatform.isi686 || stdenv.hostPlatform.isx86_64))
+    {
+      inherit s0ix-selftest-tool;
+    }