fix(registry): repair two regressions from the domain migration; port the manifest validator to python
Fixes 4 test failures introduced by8e814ca0, which I pushed after running only the container-crate tests while the full suite was still compiling. Both failures were real defects, not stale assertions. 1. Catalog-driven installs would have failed fleet-wide.8e814ca0dropped the old registry address from TRUSTED_REGISTRIES, but the signed catalog still advertises image refs on it — deliberately, since rewriting a signed artifact invalidates its signature. Nodes resolve apps through the catalog, so every install would have been refused with "not from a trusted registry". Reinstated as LEGACY_REGISTRY_HOST, documented as transitional and removable only once the catalog is re-signed. 2. The update fallback lost the property it exists for. update.rs keeps two mirrors on purpose: the domain as primary, and the old IP over plain HTTP as a fallback, because a node whose DNS or clock is wrong (both break TLS) must still be able to update itself — the signature, not the transport, is what makes either source safe. The bulk rewrite pointed both constants at the domain, leaving the escape hatch dependent on exactly what it exists to survive. Restored to its original value. Separately, validate-app-manifest.sh is ported from ruby to python3+PyYAML. It shelled out to ruby with stderr discarded, so on any machine without ruby a missing interpreter was reported as "Valid YAML with top-level app block: FAIL" and every manifest came back REJECTED. This is the first tool an app developer runs, and it sent them to fix YAML that was never broken. Ruby was also the odd dependency out — the repo already ships three python scripts. It now checks for python3 and PyYAML up front and names what is missing, then parses with PyYAML. Missing keys resolve to an absent-value object that indexes to itself and prints empty, so call sites lost their per-hop guards: (((app["container"] || {})["build"] || {})["context"]) becomes app["container"]["build"]["context"]. Booleans still print as true/false rather than Python's True/False — call sites compare == "true", so Python's capitalisation would have silently inverted the readonly_root and no_new_privileges security checks. Verified: full rust suite 1148/1148, 0 failed. All 56 app manifests validate (0 rejected, 0 errored) where previously every one was rejected. No signed artifact modified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
8e814ca06a
commit
44b50a39d4
@@ -4,11 +4,27 @@
|
||||
//! manifest-supplied ref can't reach `pull_image` unchecked (§A of the
|
||||
//! 1.8.0 hardening plan).
|
||||
|
||||
/// The registry's previous address, before it moved behind a domain.
|
||||
///
|
||||
/// TRANSITIONAL — remove once the app catalog has been regenerated and
|
||||
/// re-signed against `source.archipelago-foundation.org`. The catalog is a
|
||||
/// signed artifact, so its image refs cannot be rewritten in place without
|
||||
/// invalidating the signature; until the signing ceremony runs, deployed
|
||||
/// nodes still resolve every app through a catalog that names this host.
|
||||
/// Dropping it from the trusted list before then makes each catalog-driven
|
||||
/// install fail with "not from a trusted registry".
|
||||
pub const LEGACY_REGISTRY_HOST: &str = "146.59.87.168:3000";
|
||||
|
||||
/// Registries images may be pulled from with an explicit host part.
|
||||
/// (git.tx1138.com was removed 2026-07-10: the host is retired and must
|
||||
/// never be pulled through again.)
|
||||
pub const TRUSTED_REGISTRIES: &[&str] =
|
||||
&["docker.io", "ghcr.io", "localhost", "source.archipelago-foundation.org"];
|
||||
pub const TRUSTED_REGISTRIES: &[&str] = &[
|
||||
"docker.io",
|
||||
"ghcr.io",
|
||||
"localhost",
|
||||
"source.archipelago-foundation.org",
|
||||
LEGACY_REGISTRY_HOST,
|
||||
];
|
||||
|
||||
/// Validate a container image reference.
|
||||
///
|
||||
|
||||
@@ -88,7 +88,7 @@ const DEFAULT_UPDATE_MANIFEST_URL: &str =
|
||||
/// whose DNS or TLS is broken still updates. Dropped from the mirror list
|
||||
/// once the fleet has moved.
|
||||
const LEGACY_UPDATE_MANIFEST_URL: &str =
|
||||
"https://source.archipelago-foundation.org/lfg2025/archy/raw/branch/main/releases/manifest.json";
|
||||
"http://146.59.87.168:3000/lfg2025/archy/raw/branch/main/releases/manifest.json";
|
||||
const UPDATE_STATE_FILE: &str = "update_state.json";
|
||||
const UPDATE_MIRRORS_FILE: &str = "update-mirrors.json";
|
||||
/// Marker written by apply_update() just before the service restart and
|
||||
|
||||
@@ -47,24 +47,68 @@ check() {
|
||||
esac
|
||||
}
|
||||
|
||||
# Preflight the YAML parser BEFORE any check runs. This used to shell out to
|
||||
# ruby with stderr discarded, so a machine without ruby reported "invalid YAML"
|
||||
# and rejected every manifest that was in fact perfectly valid — the first tool
|
||||
# an app developer runs, failing with a message that sent them to fix the wrong
|
||||
# thing. Fail loudly about the real cause instead.
|
||||
if ! command -v python3 >/dev/null 2>&1; then
|
||||
echo "ERROR: python3 is required to validate manifests, but was not found." >&2
|
||||
exit 3
|
||||
fi
|
||||
if ! python3 -c 'import yaml' >/dev/null 2>&1; then
|
||||
echo "ERROR: the PyYAML module is required to validate manifests." >&2
|
||||
echo " Install it with: python3 -m pip install pyyaml" >&2
|
||||
echo " (Debian/Ubuntu: apt-get install python3-yaml)" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Evaluate a path expression against the manifest's top-level `app` block.
|
||||
# Missing keys yield an empty string rather than an error, so callers can write
|
||||
# a plain chain like app["container"]["build"]["tag"] without guarding each hop.
|
||||
yaml_eval() {
|
||||
ruby -ryaml -e '
|
||||
path, expr = ARGV
|
||||
data = YAML.load_file(path)
|
||||
app = data.is_a?(Hash) ? data["app"] : nil
|
||||
abort "missing top-level app block" unless app.is_a?(Hash)
|
||||
value = eval(expr)
|
||||
case value
|
||||
when Array
|
||||
puts value.join("\n")
|
||||
when Hash
|
||||
puts value.to_a.map { |k, v| "#{k}=#{v}" }.join("\n")
|
||||
when NilClass
|
||||
puts ""
|
||||
else
|
||||
puts value
|
||||
end
|
||||
' "$MANIFEST" "$1"
|
||||
python3 -c '
|
||||
import sys, yaml
|
||||
|
||||
class Nil:
|
||||
"""Absent value: indexes to itself, is falsy, prints as empty."""
|
||||
def __getitem__(self, key): return self
|
||||
def __bool__(self): return False
|
||||
def __str__(self): return ""
|
||||
def __iter__(self): return iter(())
|
||||
|
||||
NIL = Nil()
|
||||
|
||||
class SafeDict(dict):
|
||||
def __missing__(self, key): return NIL
|
||||
|
||||
def wrap(value):
|
||||
if isinstance(value, dict):
|
||||
return SafeDict({k: wrap(v) for k, v in value.items()})
|
||||
if isinstance(value, list):
|
||||
return [wrap(v) for v in value]
|
||||
return value
|
||||
|
||||
path, expr = sys.argv[1], sys.argv[2]
|
||||
with open(path) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
app = data.get("app") if isinstance(data, dict) else None
|
||||
if not isinstance(app, dict):
|
||||
sys.exit("missing top-level app block")
|
||||
app = wrap(app)
|
||||
|
||||
value = eval(expr, {"__builtins__": {}}, {"app": app})
|
||||
if isinstance(value, list):
|
||||
print("\n".join(str(v) for v in value))
|
||||
elif isinstance(value, dict):
|
||||
print("\n".join(f"{k}={v}" for k, v in value.items()))
|
||||
elif value is None or isinstance(value, Nil):
|
||||
print("")
|
||||
elif isinstance(value, bool):
|
||||
print("true" if value else "false")
|
||||
else:
|
||||
print(value)
|
||||
' "$MANIFEST" "$1"
|
||||
}
|
||||
|
||||
echo "Validating: $MANIFEST"
|
||||
@@ -76,7 +120,12 @@ if [[ ! -f "$MANIFEST" ]]; then
|
||||
fi
|
||||
check "File exists" "pass"
|
||||
|
||||
if ! ruby -ryaml -e 'data = YAML.load_file(ARGV[0]); exit(data.is_a?(Hash) && data["app"].is_a?(Hash) ? 0 : 1)' "$MANIFEST" 2>/dev/null; then
|
||||
if ! python3 -c '
|
||||
import sys, yaml
|
||||
with open(sys.argv[1]) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
sys.exit(0 if isinstance(data, dict) and isinstance(data.get("app"), dict) else 1)
|
||||
' "$MANIFEST" 2>/dev/null; then
|
||||
check "Valid YAML with top-level app block" "fail"
|
||||
echo ""
|
||||
echo "Results: $PASS passed, $FAIL failed, $WARN warnings"
|
||||
@@ -90,9 +139,9 @@ APP_NAME="$(yaml_eval 'app["name"]')"
|
||||
APP_VERSION="$(yaml_eval 'app["version"]')"
|
||||
APP_DESCRIPTION="$(yaml_eval 'app["description"]')"
|
||||
APP_INTERNAL="$(yaml_eval 'app["internal"]')"
|
||||
IMAGE="$(yaml_eval '(app["container"] || {})["image"]')"
|
||||
BUILD_CONTEXT="$(yaml_eval '(((app["container"] || {})["build"] || {})["context"])')"
|
||||
BUILD_TAG="$(yaml_eval '(((app["container"] || {})["build"] || {})["tag"])')"
|
||||
IMAGE="$(yaml_eval 'app["container"]["image"]')"
|
||||
BUILD_CONTEXT="$(yaml_eval 'app["container"]["build"]["context"]')"
|
||||
BUILD_TAG="$(yaml_eval 'app["container"]["build"]["tag"]')"
|
||||
|
||||
if [[ "$APP_ID" =~ ^[a-z0-9]+(-[a-z0-9]+)*$ ]]; then
|
||||
check "app.id is lowercase kebab-case ($APP_ID)" "pass"
|
||||
@@ -164,15 +213,15 @@ if [[ -n "$IMAGE" ]]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
MEMORY_LIMIT="$(yaml_eval '((app["resources"] || {})["memory_limit"] || (app["resources"] || {})["memory"])')"
|
||||
CPU_LIMIT="$(yaml_eval '((app["resources"] || {})["cpu_limit"] || (app["resources"] || {})["cpu"])')"
|
||||
MEMORY_LIMIT="$(yaml_eval 'app["resources"]["memory_limit"] or app["resources"]["memory"]')"
|
||||
CPU_LIMIT="$(yaml_eval 'app["resources"]["cpu_limit"] or app["resources"]["cpu"]')"
|
||||
[[ -n "$MEMORY_LIMIT" ]] && check "resources.memory_limit specified ($MEMORY_LIMIT)" "pass" || check "resources.memory_limit specified" "warn"
|
||||
[[ -n "$CPU_LIMIT" ]] && check "resources.cpu_limit specified ($CPU_LIMIT)" "pass" || check "resources.cpu_limit specified" "warn"
|
||||
|
||||
READONLY_ROOT="$(yaml_eval '((app["security"] || {})["readonly_root"])')"
|
||||
NO_NEW_PRIVS="$(yaml_eval '((app["security"] || {})["no_new_privileges"])')"
|
||||
NETWORK_POLICY="$(yaml_eval '((app["security"] || {})["network_policy"])')"
|
||||
CONTAINER_NETWORK="$(yaml_eval '((app["container"] || {})["network"])')"
|
||||
READONLY_ROOT="$(yaml_eval 'app["security"]["readonly_root"]')"
|
||||
NO_NEW_PRIVS="$(yaml_eval 'app["security"]["no_new_privileges"]')"
|
||||
NETWORK_POLICY="$(yaml_eval 'app["security"]["network_policy"]')"
|
||||
CONTAINER_NETWORK="$(yaml_eval 'app["container"]["network"]')"
|
||||
|
||||
if [[ "$READONLY_ROOT" == "true" || -z "$READONLY_ROOT" ]]; then
|
||||
check "security.readonly_root true (explicit or Rust default)" "pass"
|
||||
@@ -200,7 +249,7 @@ else
|
||||
check "container.network does not share another namespace" "pass"
|
||||
fi
|
||||
|
||||
SECRET_ENV="$(yaml_eval '(app["environment"] || [])')"
|
||||
SECRET_ENV="$(yaml_eval 'app["environment"]')"
|
||||
if echo "$SECRET_ENV" | grep -iqE '^[A-Z0-9_]*(PASSWORD|PASS|SECRET|TOKEN|API_KEY|PRIVATE_KEY)[A-Z0-9_]*=.+$'; then
|
||||
check "no hardcoded secret-like values in app.environment" "warn"
|
||||
else
|
||||
@@ -218,33 +267,47 @@ if [[ -n "$APP_ID" && -n "$MANIFEST" ]]; then
|
||||
fi
|
||||
fi
|
||||
|
||||
PORT_CHECK="$(ruby -ryaml -e '
|
||||
current = ARGV[0]
|
||||
current_id = File.basename(File.dirname(current))
|
||||
ports = {}
|
||||
Dir.glob("apps/*/manifest.yml").sort.each do |path|
|
||||
data = YAML.load_file(path)
|
||||
app = data.is_a?(Hash) ? data["app"] : nil
|
||||
next unless app.is_a?(Hash)
|
||||
id = app["id"] || File.basename(File.dirname(path))
|
||||
next if id == current_id
|
||||
Array(app["ports"]).each do |p|
|
||||
next unless p.is_a?(Hash)
|
||||
proto = p["protocol"] || "tcp"
|
||||
bind = p["bind"] || ""
|
||||
host = p["host"]
|
||||
ports[[host, proto, bind]] = id if host
|
||||
end
|
||||
end
|
||||
data = YAML.load_file(current)
|
||||
app = data["app"]
|
||||
conflicts = []
|
||||
Array(app["ports"]).each do |p|
|
||||
next unless p.is_a?(Hash)
|
||||
key = [p["host"], p["protocol"] || "tcp", p["bind"] || ""]
|
||||
conflicts << "#{key[2].empty? ? "*" : key[2]}:#{key[0]}/#{key[1]} already used by #{ports[key]}" if ports.key?(key)
|
||||
end
|
||||
puts conflicts.join("\n")
|
||||
PORT_CHECK="$(python3 -c '
|
||||
import glob, os, sys, yaml
|
||||
|
||||
def load_app(path):
|
||||
try:
|
||||
with open(path) as fh:
|
||||
data = yaml.safe_load(fh)
|
||||
except Exception:
|
||||
return None
|
||||
return data.get("app") if isinstance(data, dict) else None
|
||||
|
||||
current = sys.argv[1]
|
||||
current_id = os.path.basename(os.path.dirname(current))
|
||||
|
||||
def port_keys(app):
|
||||
for entry in (app.get("ports") or []):
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
host = entry.get("host")
|
||||
if not host:
|
||||
continue
|
||||
yield (host, entry.get("protocol") or "tcp", entry.get("bind") or "")
|
||||
|
||||
claimed = {}
|
||||
for path in sorted(glob.glob("apps/*/manifest.yml")):
|
||||
app = load_app(path)
|
||||
if not isinstance(app, dict):
|
||||
continue
|
||||
app_id = app.get("id") or os.path.basename(os.path.dirname(path))
|
||||
if app_id == current_id:
|
||||
continue
|
||||
for key in port_keys(app):
|
||||
claimed[key] = app_id
|
||||
|
||||
app = load_app(current)
|
||||
if isinstance(app, dict):
|
||||
for key in port_keys(app):
|
||||
if key in claimed:
|
||||
host, proto, bind = key
|
||||
shown = bind if bind else "*"
|
||||
print(f"{shown}:{host}/{proto} already used by {claimed[key]}")
|
||||
' "$MANIFEST")"
|
||||
if [[ -n "$PORT_CHECK" ]]; then
|
||||
while IFS= read -r conflict; do
|
||||
|
||||
Reference in New Issue
Block a user