feat(stability): harden mutation locking, coordinated snapshots, and health inspection
Some checks are pending
ci / test (push) Waiting to run
Some checks are pending
ci / test (push) Waiting to run
- Introduce SystemLock with safe PID-liveness detection via /proc/<pid> to prevent concurrent mutation collisions and reclaim stale locks - Coordinate paired @ and @var Btrfs snapshots for all high-risk mutations to prevent Portage VDB database drift during rollbacks - Persist transaction log via tempfile + sync_all + atomic rename to ensure durability against unexpected power loss - Add SystemAction::SystemStatus and sys status/health query reporting boot assessment counters, reboot state, storage headroom, and package lane readiness - Unify natural language verbs (update/updates -> CheckUpdates, upgrade -> ApplyUpdates) and dynamic candidate-aware legacy install guidance
This commit is contained in:
parent
6d48ad38fc
commit
c52840645c
12 changed files with 617 additions and 40 deletions
|
|
@ -5,7 +5,21 @@ this file is the curated, human-readable summary.
|
|||
|
||||
## Unreleased — Fast defaults with user-controlled source builds
|
||||
|
||||
- **System stability foundations: Automatic Boot Assessment, Preflight Safeguards & Tiered Channels (2026-09-30).**
|
||||
- **System stability & reliability hardening: Advisory Mutation Lock, Atomic Transactions, System Health & Context-Aware Guidance (2026-09-30).**
|
||||
Hardened system operations against race conditions, database drift, and usability friction:
|
||||
1. Inter-Process Advisory Mutation Lock (`SystemLock`):
|
||||
- `phoebus-executor::system_lock`: Atomic `O_CREAT | O_EXCL` file locking (`/run/phoebus/sys.lock` with PID & timestamp metadata) guarding mutating system operations. Safe PID-liveness verification via `/proc/<pid>` reclaims stale locks left by killed processes.
|
||||
2. Coordinated Snapshot Pairing for All High-Risk Actions:
|
||||
- `phoebus-executor::actions`: Upgraded protective pre-mutation snapshotting to `snapshot::create_set_and_record`, ensuring paired `@` and `@var` snapshots are always committed together so `/var/db/pkg` Portage database state stays consistent across rollbacks.
|
||||
3. Durable Atomic Transaction Persistence:
|
||||
- `phoebus-core::transaction`: Replaced direct-file writes with tempfile creation, data flush, `sync_all()`, and atomic `rename()`, preventing corrupt or partial transaction records upon sudden reboot or power loss.
|
||||
4. Unified System Health & Status Overview:
|
||||
- `SystemAction::SystemStatus`: Added first-class health inspection action (`sys status`, `sys health`, `sys check`) reporting bootloader BLS entries and Automatic Boot Assessment counters, reboot flags, storage headroom vs the 10 GiB safety gate, snapshot recovery points, and package lane readiness.
|
||||
5. Intuitive Natural Grammar & Context-Aware Legacy Guidance:
|
||||
- Standardized natural commands: `update` and `updates` map to `CheckUpdates`, `upgrade` maps to `ApplyUpdates`.
|
||||
- In `sys install <app> --legacy`, candidates are dynamically resolved across lanes via `chooser::resolve_candidates` to present actionable, copy-pasteable commands for packages that exist in Flatpak, Portage, Snap, etc.
|
||||
Gates: `cargo fmt`, `cargo clippy --workspace --all-targets`, `cargo test --workspace` (all passed).
|
||||
|
||||
Hardened system reliability across boots, updates, and package installations:
|
||||
1. systemd Automatic Boot Assessment (`bootctl mark-good` + BLS boot counter fallback):
|
||||
- `phoebus-core::boot`: BLS filename parsing/formatting adhering to the systemd-boot specification (`<entry>+<tries>[,<tries-done>].conf`). Recovery entries (`phoebus-recovery.conf`) are strictly unmetered to guarantee an always-accessible fallback target.
|
||||
|
|
|
|||
|
|
@ -3,19 +3,16 @@
|
|||
Rolling note between work sessions. **Update this file before you finish**, then
|
||||
commit it. It should be short and current; use `git log` for detail.
|
||||
|
||||
- **System stability foundations (2026-09-30, IMPLEMENTED + TESTED):**
|
||||
Implemented key stability foundations approved by user:
|
||||
1. systemd Automatic Boot Assessment (`bootctl mark-good` + BLS boot counter fallback):
|
||||
BLS filenames `<entry>+<tries>.conf` with unmetered recovery fallback (`phoebus-recovery.conf`),
|
||||
`phoebus-executor::boot_assessment` containing `phoebus-boot-confirm.service` unit,
|
||||
`enable_boot_counting`, and service installation in `etc/systemd/system/multi-user.target.wants`.
|
||||
2. Preflight Safety Gates: `check_update_preflight` checking >=10 GiB free space on root, and
|
||||
`create_pre_update_snapshot_set` taking atomic coordinated `@` (root) and `@var` (Portage DB state)
|
||||
Btrfs snapshots prior to updates so rolling back never breaks package manager consistency.
|
||||
3. Tiered Update Channels: `UpdateChannel` (`Core`, `Proven` [default], `Edge`).
|
||||
4. Enforced default prebuilt kernel: `sys-kernel/gentoo-kernel-bin` strictly set as default.
|
||||
- **System stability & reliability hardening (2026-09-30, IMPLEMENTED + TESTED):**
|
||||
Implemented concurrency protection, package database consistency, and intuitive user workflows:
|
||||
1. Inter-Process System Mutation Lock (`SystemLock`): Added PID-liveness-backed file lock at `/run/phoebus/sys.lock` (safe `/proc/<pid>` liveness check with automatic recovery of stale locks from dead processes) to prevent concurrent mutation collisions.
|
||||
2. Coordinated Snapshot Pairing: High-risk mutations in `actions.rs` now invoke `snapshot::create_set_and_record` to create atomic paired `@` and `@var` Btrfs snapshots, preventing `/var/db/pkg` Portage database drift during rollbacks.
|
||||
3. Atomic Transaction Persistence: Updated `transaction.rs` to write `transactions.toml` via tempfile + `sync_all` + atomic rename, eliminating partial/truncated file risk on sudden power loss.
|
||||
4. System Health Overview: Implemented `SystemAction::SystemStatus` (`sys status`, `sys health`, `sys check`) querying BLS boot assessment states, reboot requirement, storage headroom vs the 10 GiB gate, Btrfs snapshots, and package lane readiness.
|
||||
5. Intuitive CLI & Context-Aware Guidance: Unified `update`/`updates` -> `CheckUpdates` and `upgrade` -> `ApplyUpdates`; `sys install <app> --legacy` without a backend now dynamically resolves candidates across lanes via `chooser::resolve_candidates` to suggest exact copy-pasteable commands.
|
||||
Gates: `cargo fmt`, `clippy --workspace --all-targets`, `cargo test --workspace` all pass.
|
||||
|
||||
|
||||
- **Userspace-directed AI agent framework (2026-09-30, IMPLEMENTED + TESTED):**
|
||||
designed and implemented a Two-Tier architecture for userspace developer/desktop
|
||||
AI agents (e.g., Claude Code, OpenCode, Cursor, Codex). Added `agent_jail` +
|
||||
|
|
|
|||
|
|
@ -89,7 +89,11 @@ Booting that recovery entry offline remains a manual gate.
|
|||
| Package policy + measured history | yes | yes | yes (326 entries) | no |
|
||||
| Background update preparation | yes | yes | yes (fetch; buildpkgonly skipped) | no |
|
||||
| Btrfs snapshots + history | yes | yes | yes | no |
|
||||
| Coordinated snapshot sets (`@` + `@var`) | yes | yes | no | no |
|
||||
| Coordinated snapshot sets (`@` + `@var` for all high-risk actions) | yes | yes | no | no |
|
||||
| Inter-process mutation lock (`SystemLock` + dead-PID recovery) | yes | yes | no | no |
|
||||
| Atomic durable transaction log (`transactions.toml`) | yes | yes | read-only in installed VM; write path needs root | no |
|
||||
| System health overview (`sys status` / `SystemStatus`) | yes | yes | no | no |
|
||||
| Automatic Boot Assessment BLS counting (`bootctl mark-good`) | yes | yes | no | no |
|
||||
| Btrfs rollback (`sys undo`) | yes | yes | yes (rollback acceptance `serial-rollback-20260926T030237Z-912f6386` PASS=11 FAIL=0) | no |
|
||||
| Recovery/known-good BLS entry | yes | yes | yes: entry present + snapshot-based (rollback runs) **and the recovery entry itself was booted** (`Phoebus Recovery (known-good snapshot)` selected in the systemd-boot menu with restricted networking and no model: cmdline + findmnt prove root is the read-only `@-20260926T064646-initial-known-good` snapshot with `rescue.target`, serial login works, `sys history/rollback status --legacy` work, `sys ask` fails truthfully with the `--legacy` fallback) | no |
|
||||
| Snapshot retention + free-space gate | yes | yes | no | no |
|
||||
|
|
|
|||
|
|
@ -90,6 +90,8 @@ pub enum SystemAction {
|
|||
},
|
||||
/// Report the rollback state (no mutation).
|
||||
RollbackStatus,
|
||||
/// Overview of system health, active boot entry, reboot requirements, storage, and updates.
|
||||
SystemStatus,
|
||||
History,
|
||||
UpdateSystem,
|
||||
PrepareUpdates,
|
||||
|
|
@ -372,6 +374,7 @@ impl SystemAction {
|
|||
None => "roll back to the newest committed snapshot".to_string(),
|
||||
},
|
||||
SystemAction::RollbackStatus => "show the rollback state".to_string(),
|
||||
SystemAction::SystemStatus => "inspect system status and health".to_string(),
|
||||
SystemAction::History => "show system transaction history".to_string(),
|
||||
SystemAction::PrepareUpdates => {
|
||||
"prepare background updates (fetch + build binary packages, do not activate)"
|
||||
|
|
@ -540,7 +543,7 @@ impl SystemAction {
|
|||
| SystemAction::SearchPackages { .. } => Risk::Low,
|
||||
SystemAction::History => Risk::Low,
|
||||
SystemAction::InspectUpdates => Risk::Low,
|
||||
SystemAction::RollbackStatus => Risk::Low,
|
||||
SystemAction::RollbackStatus | SystemAction::SystemStatus => Risk::Low,
|
||||
SystemAction::FlatpakList => Risk::Low,
|
||||
SystemAction::ChoosePackage { .. }
|
||||
| SystemAction::SnapFind { .. }
|
||||
|
|
@ -603,6 +606,7 @@ impl SystemAction {
|
|||
| SystemAction::History
|
||||
| SystemAction::InspectUpdates
|
||||
| SystemAction::RollbackStatus
|
||||
| SystemAction::SystemStatus
|
||||
| SystemAction::FlatpakList
|
||||
| SystemAction::ChoosePackage { .. }
|
||||
| SystemAction::SnapFind { .. }
|
||||
|
|
@ -767,7 +771,8 @@ impl SystemAction {
|
|||
| SystemAction::UpdateSystem
|
||||
| SystemAction::PrepareUpdates
|
||||
| SystemAction::InspectUpdates
|
||||
| SystemAction::RollbackStatus => Ok(()),
|
||||
| SystemAction::RollbackStatus
|
||||
| SystemAction::SystemStatus => Ok(()),
|
||||
SystemAction::ChoosePackage { name, .. } => {
|
||||
if valid_package_spec(name) {
|
||||
Ok(())
|
||||
|
|
@ -1445,12 +1450,25 @@ fn parse_natural_raw(text: &str) -> Option<SystemAction> {
|
|||
return Some(SystemAction::PackageOwners { path });
|
||||
}
|
||||
}
|
||||
if t == "updates" || t == "check updates" || t == "check for updates" || t == "update check" {
|
||||
if t == "update"
|
||||
|| t == "updates"
|
||||
|| t == "check updates"
|
||||
|| t == "check for updates"
|
||||
|| t == "update check"
|
||||
{
|
||||
return Some(SystemAction::CheckUpdates { backend: None });
|
||||
}
|
||||
if t == "apply updates" || t == "upgrade" || t == "upgrade system" || t == "install updates" {
|
||||
return Some(SystemAction::ApplyUpdates { backend: None });
|
||||
}
|
||||
if t == "status"
|
||||
|| t == "health"
|
||||
|| t == "check"
|
||||
|| t == "system status"
|
||||
|| t == "system health"
|
||||
{
|
||||
return Some(SystemAction::SystemStatus);
|
||||
}
|
||||
|
||||
// Flatpak verbs. App ids are case-sensitive (`org.chromium.Chromium`), so
|
||||
// the operands are read from the original text, not the lowercased copy.
|
||||
|
|
@ -1720,7 +1738,7 @@ fn parse_natural_raw(text: &str) -> Option<SystemAction> {
|
|||
if t == "history" || t.starts_with("history ") {
|
||||
return Some(SystemAction::History);
|
||||
}
|
||||
if t == "update" || t == "update system" || t == "sys update" {
|
||||
if t == "update system" || t == "sys update" || t == "emerge @world" {
|
||||
return Some(SystemAction::UpdateSystem);
|
||||
}
|
||||
if t.contains("prepare") && t.contains("update") {
|
||||
|
|
@ -2521,5 +2539,34 @@ mod tests {
|
|||
name: "sshd".into()
|
||||
})
|
||||
);
|
||||
assert_eq!(parse_natural("status"), Some(SystemAction::SystemStatus));
|
||||
assert_eq!(parse_natural("health"), Some(SystemAction::SystemStatus));
|
||||
assert_eq!(parse_natural("check"), Some(SystemAction::SystemStatus));
|
||||
assert_eq!(
|
||||
parse_natural("system status"),
|
||||
Some(SystemAction::SystemStatus)
|
||||
);
|
||||
assert_eq!(
|
||||
parse_natural("update"),
|
||||
Some(SystemAction::CheckUpdates { backend: None })
|
||||
);
|
||||
assert_eq!(
|
||||
parse_natural("updates"),
|
||||
Some(SystemAction::CheckUpdates { backend: None })
|
||||
);
|
||||
assert_eq!(
|
||||
parse_natural("upgrade"),
|
||||
Some(SystemAction::ApplyUpdates { backend: None })
|
||||
);
|
||||
assert_eq!(
|
||||
parse_natural("update system"),
|
||||
Some(SystemAction::UpdateSystem)
|
||||
);
|
||||
|
||||
let status_action = SystemAction::SystemStatus;
|
||||
assert_eq!(status_action.risk(), Risk::Low);
|
||||
assert!(!status_action.is_mutating());
|
||||
assert!(!status_action.requires_root());
|
||||
assert!(status_action.validate().is_ok());
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -134,22 +134,25 @@ impl TransactionLog {
|
|||
|
||||
pub fn save(&self, path: &Path) -> Result<()> {
|
||||
use std::io::Write;
|
||||
if let Some(parent) = path.parent() {
|
||||
let parent = path.parent().unwrap_or_else(|| Path::new("."));
|
||||
std::fs::create_dir_all(parent).map_err(|e| CoreError::io(parent, e))?;
|
||||
}
|
||||
let text = toml::to_string_pretty(self)?;
|
||||
let mut file = std::fs::File::create(path).map_err(|e| CoreError::io(path, e))?;
|
||||
let tmp_path = parent.join(format!(
|
||||
".{}.tmp.{}",
|
||||
path.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
.unwrap_or("transactions"),
|
||||
std::process::id()
|
||||
));
|
||||
let mut file = std::fs::File::create(&tmp_path).map_err(|e| CoreError::io(&tmp_path, e))?;
|
||||
file.write_all(text.as_bytes())
|
||||
.map_err(|e| CoreError::io(path, e))?;
|
||||
// Make the record durable: an operation's outcome (including a
|
||||
// "finished" update to an "in progress" record) must survive a power
|
||||
// loss, or the history would show a completed action as interrupted.
|
||||
file.sync_all().map_err(|e| CoreError::io(path, e))?;
|
||||
if let Some(parent) = path.parent() {
|
||||
.map_err(|e| CoreError::io(&tmp_path, e))?;
|
||||
file.sync_all().map_err(|e| CoreError::io(&tmp_path, e))?;
|
||||
drop(file);
|
||||
std::fs::rename(&tmp_path, path).map_err(|e| CoreError::io(path, e))?;
|
||||
if let Ok(dir) = std::fs::File::open(parent) {
|
||||
let _ = dir.sync_all();
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1260,6 +1260,16 @@ pub fn execute_inner(
|
|||
return Err(ExecError::NotAuthorized(action.describe()));
|
||||
}
|
||||
|
||||
let _lock = if action.is_mutating() && exec.is_real() && !exec.dry_run {
|
||||
Some(crate::system_lock::SystemLock::acquire(
|
||||
exec,
|
||||
root,
|
||||
&action.describe(),
|
||||
)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Rollback moves the root, `/var` and the boot selection together and takes
|
||||
// its own pre-rollback snapshot, so it must not run through the generic
|
||||
// protective-snapshot path below.
|
||||
|
|
@ -1294,7 +1304,7 @@ pub fn execute_inner(
|
|||
let auth = auth.ok_or_else(|| ExecError::NotAuthorized(action.describe()))?;
|
||||
let source = root.to_string_lossy().to_string();
|
||||
let record =
|
||||
snapshot::create_and_record(exec, auth, root, &source, &action.describe())?;
|
||||
snapshot::create_set_and_record(exec, auth, root, &source, &action.describe())?;
|
||||
// Preserve the recovery point so callers can record it.
|
||||
snapshot_id = Some(record.id);
|
||||
} else {
|
||||
|
|
@ -1800,6 +1810,7 @@ pub fn execute_inner(
|
|||
state.note
|
||||
)
|
||||
}
|
||||
SystemAction::SystemStatus => crate::query::system_status_report(exec, root)?,
|
||||
SystemAction::ConvertToSource { app } => convert_to_source(exec, root, app)?,
|
||||
SystemAction::BuildTunedKernel { make_default } => {
|
||||
crate::kernel::build_tuned_kernel(exec, root, *make_default)?
|
||||
|
|
@ -2433,6 +2444,16 @@ Total: 1 package (1 new), Size of downloads: 512 KiB
|
|||
assert!(outcome.snapshot.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_status_action_executes_cleanly() {
|
||||
let runner = RecordingRunner::new();
|
||||
let exec = make_exec(runner);
|
||||
let action = SystemAction::SystemStatus;
|
||||
let outcome = execute(&exec, &action, None, Path::new(&exec.root)).unwrap();
|
||||
assert!(outcome.output.contains("=== Phoebus System Status ==="));
|
||||
assert!(outcome.snapshot.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn depclean_is_previewed_as_structured_removals_at_review() {
|
||||
let runner = RecordingRunner::new();
|
||||
|
|
|
|||
|
|
@ -133,6 +133,9 @@ pub trait ProcessRunner: Send + Sync {
|
|||
fn is_streaming(&self) -> bool {
|
||||
false
|
||||
}
|
||||
fn is_real(&self) -> bool {
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// The real process runner.
|
||||
|
|
@ -156,6 +159,10 @@ impl ProcessRunner for RealRunner {
|
|||
self.sink.is_some()
|
||||
}
|
||||
|
||||
fn is_real(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn execute(&self, spec: &CommandSpec) -> Result<CommandOutput> {
|
||||
let start = Instant::now();
|
||||
let mut command = Command::new(&spec.program);
|
||||
|
|
|
|||
|
|
@ -37,6 +37,10 @@ impl Exec {
|
|||
TargetRoot::new(self.root.clone())
|
||||
}
|
||||
|
||||
pub fn is_real(&self) -> bool {
|
||||
self.runner.is_real()
|
||||
}
|
||||
|
||||
pub fn register_secret(&self, secret: &SecretString) {
|
||||
self.logger.register_secret(secret);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -41,6 +41,7 @@ pub mod snapshot;
|
|||
pub mod stage3;
|
||||
pub mod storage;
|
||||
pub mod sys_runtime;
|
||||
pub mod system_lock;
|
||||
pub mod transaction;
|
||||
pub mod update;
|
||||
pub mod users;
|
||||
|
|
@ -52,3 +53,4 @@ pub use command::{
|
|||
pub use error::{CommandFailure, ExecError, Result};
|
||||
pub use exec::Exec;
|
||||
pub use log::{CommandRecord, Logger};
|
||||
pub use system_lock::SystemLock;
|
||||
|
|
|
|||
|
|
@ -1461,6 +1461,146 @@ pub fn apply_updates(
|
|||
Ok(report)
|
||||
}
|
||||
|
||||
/// Comprehensive overview of system health, bootloader state, storage headroom,
|
||||
/// reboot requirements, and package lane readiness.
|
||||
pub fn system_status_report(exec: &Exec, root: &Path) -> Result<String> {
|
||||
let mut out = String::new();
|
||||
out.push_str("=== Phoebus System Status ===\n\n");
|
||||
|
||||
// 1. Boot & Kernel Assessment
|
||||
out.push_str("[Boot & Kernel]\n");
|
||||
let mut entries_found = Vec::new();
|
||||
for candidate_dir in [
|
||||
root.join("efi/loader/entries"),
|
||||
root.join("boot/loader/entries"),
|
||||
root.join("boot/efi/loader/entries"),
|
||||
] {
|
||||
if candidate_dir.is_dir() {
|
||||
if let Ok(rd) = std::fs::read_dir(&candidate_dir) {
|
||||
for e in rd.flatten() {
|
||||
let path = e.path();
|
||||
if path.extension().and_then(|x| x.to_str()) == Some("conf") {
|
||||
if let Some(fname) = path.file_name().and_then(|n| n.to_str()) {
|
||||
let (stem, tries) = phoebus_core::boot::parse_bls_filename(fname);
|
||||
entries_found.push((fname.to_string(), stem, tries));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !entries_found.is_empty() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if entries_found.is_empty() {
|
||||
out.push_str(" BLS Entries: None detected in /efi or /boot\n");
|
||||
} else {
|
||||
out.push_str(" BLS Boot Entries:\n");
|
||||
for (fname, stem, tries) in &entries_found {
|
||||
let assessment = match tries {
|
||||
Some(t) => format!("assessment in progress ({t} tries remaining)"),
|
||||
None if stem.contains("recovery") => "recovery fallback".into(),
|
||||
None => "verified good".into(),
|
||||
};
|
||||
out.push_str(&format!(" - {fname} [{assessment}]\n"));
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Reboot & Rollback State
|
||||
out.push_str("\n[Reboot & Recovery State]\n");
|
||||
let reboot_flag =
|
||||
root.join("run/reboot-required").exists() || root.join("var/run/reboot-required").exists();
|
||||
let rollback_state = crate::rollback::load_state(root);
|
||||
let reboot_status =
|
||||
if rollback_state.status == phoebus_core::rollback::RollbackStatus::Activated {
|
||||
format!(
|
||||
"REQUIRED (rollback activated to target `{}`; reboot to complete switch)",
|
||||
rollback_state.target_snapshot
|
||||
)
|
||||
} else if reboot_flag {
|
||||
"REQUIRED (system updates pending reboot)".to_string()
|
||||
} else {
|
||||
"None (running system configuration is current)".to_string()
|
||||
};
|
||||
out.push_str(&format!(" Reboot Required: {reboot_status}\n"));
|
||||
out.push_str(&format!(
|
||||
" Rollback Engine: {:?} (target: {}, root: {})\n",
|
||||
rollback_state.status,
|
||||
if rollback_state.target_snapshot.is_empty() {
|
||||
"(none)"
|
||||
} else {
|
||||
&rollback_state.target_snapshot
|
||||
},
|
||||
rollback_state.root_subvol
|
||||
));
|
||||
|
||||
// 3. Storage Headroom & Gates
|
||||
out.push_str("\n[Storage & Safety Gates]\n");
|
||||
let free_mib = crate::chooser::free_storage_mib(exec, root);
|
||||
let free_gib = (free_mib as f64) / 1024.0;
|
||||
let gate_status = if free_mib >= crate::update::MIN_UPDATE_FREE_SPACE_MIB {
|
||||
"HEALTHY (meets >= 10 GiB gate)"
|
||||
} else if free_mib == 0 {
|
||||
"UNKNOWN (cannot determine free space)"
|
||||
} else {
|
||||
"WARNING (below 10 GiB safety gate; updates will be refused)"
|
||||
};
|
||||
out.push_str(&format!(
|
||||
" Root Free Space: {:.1} GiB [{gate_status}]\n",
|
||||
free_gib
|
||||
));
|
||||
|
||||
// 4. Btrfs Snapshots
|
||||
out.push_str("\n[Btrfs Recovery Snapshots]\n");
|
||||
let history = crate::snapshot::read_history(root).unwrap_or_default();
|
||||
out.push_str(&format!(
|
||||
" Committed Snapshots: {}\n",
|
||||
history.records.len()
|
||||
));
|
||||
if let Some(newest) = history.records.last() {
|
||||
out.push_str(&format!(
|
||||
" Newest Recovery Point: {} ({}) [{:?}]\n",
|
||||
newest.id, newest.operation, newest.status
|
||||
));
|
||||
}
|
||||
|
||||
// 5. Package Lanes Readiness
|
||||
out.push_str("\n[Package Lanes Readiness]\n");
|
||||
let facts = crate::chooser::hardware_facts(exec, root);
|
||||
let lanes = [
|
||||
(Backend::PortageSource, "Portage (Source)", true),
|
||||
(
|
||||
Backend::PortageBinary,
|
||||
"Portage (Binhost)",
|
||||
facts.binhost_available,
|
||||
),
|
||||
(
|
||||
Backend::Flatpak,
|
||||
"Flatpak",
|
||||
facts.available_backends.contains(&Backend::Flatpak),
|
||||
),
|
||||
(
|
||||
Backend::Nix,
|
||||
"Nix",
|
||||
facts.available_backends.contains(&Backend::Nix),
|
||||
),
|
||||
(
|
||||
Backend::Snap,
|
||||
"Snap",
|
||||
facts.available_backends.contains(&Backend::Snap),
|
||||
),
|
||||
(Backend::AppImage, "AppImage", true),
|
||||
(Backend::CustomEbuild, "Custom Ebuild", true),
|
||||
];
|
||||
for (_backend, label, ready) in lanes {
|
||||
let status = if ready { "READY" } else { "NOT CONFIGURED" };
|
||||
out.push_str(&format!(" {:<20}: [{status}]\n", label));
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -2381,4 +2521,28 @@ mod tests {
|
|||
assert!(!overview.sources[0].supported);
|
||||
assert!(overview.sources[0].note.contains("AppImage"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn system_status_report_renders_all_sections() {
|
||||
let runner = RecordingRunner::new();
|
||||
let exec = make_exec(runner);
|
||||
let root = Path::new(&exec.root);
|
||||
|
||||
// Populate a sample BLS entry with boot counting
|
||||
let bls_dir = root.join("efi/loader/entries");
|
||||
std::fs::create_dir_all(&bls_dir).unwrap();
|
||||
std::fs::write(bls_dir.join("phoebus+3.conf"), "title Phoebus OS\n").unwrap();
|
||||
std::fs::write(bls_dir.join("phoebus-recovery.conf"), "title Recovery\n").unwrap();
|
||||
|
||||
let report = system_status_report(&exec, root).unwrap();
|
||||
assert!(report.contains("=== Phoebus System Status ==="));
|
||||
assert!(report.contains("[Boot & Kernel]"));
|
||||
assert!(report.contains("phoebus+3.conf [assessment in progress (3 tries remaining)]"));
|
||||
assert!(report.contains("phoebus-recovery.conf [recovery fallback]"));
|
||||
assert!(report.contains("[Reboot & Recovery State]"));
|
||||
assert!(report.contains("[Storage & Safety Gates]"));
|
||||
assert!(report.contains("[Btrfs Recovery Snapshots]"));
|
||||
assert!(report.contains("[Package Lanes Readiness]"));
|
||||
assert!(report.contains("Portage (Source)"));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
237
installer-rs/crates/phoebus-executor/src/system_lock.rs
Normal file
237
installer-rs/crates/phoebus-executor/src/system_lock.rs
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
//! Inter-process system mutation lock.
|
||||
//!
|
||||
//! Ensures that only one mutating `sys` action runs at a time, preventing
|
||||
//! transaction log corruption, Btrfs snapshot timestamp collisions, and
|
||||
//! conflicting package manager operations.
|
||||
//!
|
||||
//! Uses atomic `O_CREAT | O_EXCL` file creation with automatic stale-lock recovery
|
||||
//! (detects if the owning PID is no longer alive via `/proc/<pid>`).
|
||||
|
||||
use std::fs::{self, OpenOptions};
|
||||
use std::io::Write;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use crate::error::{ExecError, Result};
|
||||
use crate::exec::Exec;
|
||||
|
||||
pub const DEFAULT_LOCK_REL: &str = "run/phoebus/sys.lock";
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct SystemLock {
|
||||
lock_path: PathBuf,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
struct LockInfo {
|
||||
pid: u32,
|
||||
created_unix: u64,
|
||||
operation: String,
|
||||
}
|
||||
|
||||
impl LockInfo {
|
||||
fn serialize(&self) -> String {
|
||||
format!(
|
||||
"pid={}\ncreated_unix={}\noperation={}\n",
|
||||
self.pid, self.created_unix, self.operation
|
||||
)
|
||||
}
|
||||
|
||||
fn parse(text: &str) -> Option<Self> {
|
||||
let mut pid = None;
|
||||
let mut created_unix = None;
|
||||
let mut operation = None;
|
||||
|
||||
for line in text.lines() {
|
||||
if let Some(val) = line.strip_prefix("pid=") {
|
||||
pid = val.trim().parse::<u32>().ok();
|
||||
} else if let Some(val) = line.strip_prefix("created_unix=") {
|
||||
created_unix = val.trim().parse::<u64>().ok();
|
||||
} else if let Some(val) = line.strip_prefix("operation=") {
|
||||
operation = Some(val.trim().to_string());
|
||||
}
|
||||
}
|
||||
|
||||
Some(Self {
|
||||
pid: pid?,
|
||||
created_unix: created_unix?,
|
||||
operation: operation.unwrap_or_else(|| "unspecified operation".to_string()),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn now_unix() -> u64 {
|
||||
SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
fn is_pid_alive(pid: u32) -> bool {
|
||||
if pid == 0 {
|
||||
return false;
|
||||
}
|
||||
Path::new("/proc").join(pid.to_string()).exists()
|
||||
}
|
||||
|
||||
pub fn lock_path(root: &Path) -> PathBuf {
|
||||
root.join(DEFAULT_LOCK_REL)
|
||||
}
|
||||
|
||||
impl SystemLock {
|
||||
/// Attempt to acquire the system mutation lock for `operation`.
|
||||
pub fn acquire(exec: &Exec, root: &Path, operation: &str) -> Result<Self> {
|
||||
let path = lock_path(root);
|
||||
Self::acquire_at(exec, path, operation)
|
||||
}
|
||||
|
||||
/// Attempt to acquire the lock at an explicit path (used by tests and alternate paths).
|
||||
pub fn acquire_at(exec: &Exec, path: PathBuf, operation: &str) -> Result<Self> {
|
||||
if let Some(parent) = path.parent() {
|
||||
fs::create_dir_all(parent).map_err(|e| ExecError::io(parent.to_path_buf(), e))?;
|
||||
}
|
||||
|
||||
let my_pid = std::process::id();
|
||||
let now = now_unix();
|
||||
let info = LockInfo {
|
||||
pid: my_pid,
|
||||
created_unix: now,
|
||||
operation: operation.to_string(),
|
||||
};
|
||||
|
||||
match Self::try_create(&path, &info) {
|
||||
Ok(()) => {
|
||||
exec.logger.info(format!(
|
||||
"acquired system lock at {} for `{operation}`",
|
||||
path.display()
|
||||
));
|
||||
Ok(Self { lock_path: path })
|
||||
}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {
|
||||
// Check if existing lock is stale
|
||||
if let Ok(content) = fs::read_to_string(&path) {
|
||||
if let Some(existing) = LockInfo::parse(&content) {
|
||||
if !is_pid_alive(existing.pid) {
|
||||
exec.logger.warn(format!(
|
||||
"detected stale system lock held by dead PID {}; breaking lock",
|
||||
existing.pid
|
||||
));
|
||||
let _ = fs::remove_file(&path);
|
||||
// Retry atomic creation once
|
||||
if Self::try_create(&path, &info).is_ok() {
|
||||
exec.logger.info(format!(
|
||||
"reclaimed stale system lock at {} for `{operation}`",
|
||||
path.display()
|
||||
));
|
||||
return Ok(Self { lock_path: path });
|
||||
}
|
||||
} else {
|
||||
return Err(ExecError::operation(
|
||||
"sys",
|
||||
"acquire system lock",
|
||||
format!(
|
||||
"another Phoebus system mutation is currently in progress: `{}` (PID {}, started {}s ago)",
|
||||
existing.operation,
|
||||
existing.pid,
|
||||
now.saturating_sub(existing.created_unix)
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(ExecError::operation(
|
||||
"sys",
|
||||
"acquire system lock",
|
||||
format!(
|
||||
"another Phoebus system mutation is currently in progress (lockfile present at {})",
|
||||
path.display()
|
||||
),
|
||||
))
|
||||
}
|
||||
Err(e) => Err(ExecError::io(path, e)),
|
||||
}
|
||||
}
|
||||
|
||||
fn try_create(path: &Path, info: &LockInfo) -> std::io::Result<()> {
|
||||
let mut file = OpenOptions::new().write(true).create_new(true).open(path)?;
|
||||
file.write_all(info.serialize().as_bytes())?;
|
||||
file.flush()?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for SystemLock {
|
||||
fn drop(&mut self) {
|
||||
if self.lock_path.exists() {
|
||||
let _ = fs::remove_file(&self.lock_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::command::RecordingRunner;
|
||||
use crate::log::Logger;
|
||||
use phoebus_core::Redactor;
|
||||
|
||||
fn make_exec() -> Exec {
|
||||
let path = tempfile::tempdir().unwrap().keep();
|
||||
let logger = Logger::open(path.join("l.log"), Redactor::new(), false);
|
||||
Exec::new(
|
||||
Box::new(RecordingRunner::new()),
|
||||
logger,
|
||||
false,
|
||||
path.join("root"),
|
||||
path,
|
||||
)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lock_acquire_and_drop_lifecycle() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let lock_file = temp.path().join("sys.lock");
|
||||
let exec = make_exec();
|
||||
|
||||
{
|
||||
let lock = SystemLock::acquire_at(&exec, lock_file.clone(), "install vim").unwrap();
|
||||
assert!(lock_file.exists());
|
||||
|
||||
// Second acquisition must fail while held
|
||||
let err = SystemLock::acquire_at(&exec, lock_file.clone(), "remove vim").unwrap_err();
|
||||
assert!(err.to_string().contains("currently in progress"));
|
||||
|
||||
drop(lock);
|
||||
}
|
||||
|
||||
// After drop, lock file is cleaned up and can be acquired again
|
||||
assert!(!lock_file.exists());
|
||||
let lock2 = SystemLock::acquire_at(&exec, lock_file.clone(), "install emacs").unwrap();
|
||||
assert!(lock_file.exists());
|
||||
drop(lock2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_lock_is_reclaimed_when_process_is_dead() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let lock_file = temp.path().join("sys.lock");
|
||||
let exec = make_exec();
|
||||
|
||||
// Write a lock with a PID guaranteed not to exist (e.g. 4194304 or similar high number)
|
||||
let stale_info = LockInfo {
|
||||
pid: 99999999,
|
||||
created_unix: 1000,
|
||||
operation: "interrupted update".into(),
|
||||
};
|
||||
fs::write(&lock_file, stale_info.serialize()).unwrap();
|
||||
|
||||
// Acquisition should detect dead PID and reclaim
|
||||
let lock = SystemLock::acquire_at(&exec, lock_file.clone(), "new install").unwrap();
|
||||
assert!(lock_file.exists());
|
||||
let content = fs::read_to_string(&lock_file).unwrap();
|
||||
let parsed = LockInfo::parse(&content).unwrap();
|
||||
assert_eq!(parsed.pid, std::process::id());
|
||||
assert_eq!(parsed.operation, "new install");
|
||||
drop(lock);
|
||||
}
|
||||
}
|
||||
|
|
@ -357,15 +357,59 @@ fn run() -> std::result::Result<(), Box<dyn std::error::Error>> {
|
|||
let plan = service::plan_legacy_install(&plans, &exec, &cli.root, &request)?;
|
||||
return render_and_execute_plan(&cli, &exec, &plans, plan);
|
||||
} else {
|
||||
let need = phoebus_core::selection::Need::new(&application);
|
||||
let candidates = chooser::resolve_candidates(&exec, &cli.root, &need);
|
||||
let available: Vec<_> = candidates.into_iter().filter(|c| c.available).collect();
|
||||
|
||||
if available.is_empty() {
|
||||
eprintln!("sys: legacy install requires an explicit package source flag.");
|
||||
eprintln!("Specify one of:");
|
||||
eprintln!(" sys install {application} --legacy --flatpak");
|
||||
eprintln!(" sys install {application} --legacy --portage");
|
||||
eprintln!(" sys install {application} --legacy --nix");
|
||||
eprintln!(" sys install {application} --legacy --snap");
|
||||
eprintln!(" sys install {application} --legacy --appimage");
|
||||
eprintln!(" sys install {application} --legacy --ebuild");
|
||||
return Err("legacy install requires an explicit package source flag (--flatpak, --portage, --nix, --snap, --appimage, --ebuild)".into());
|
||||
eprintln!(
|
||||
"Specify one of:\n sys install {application} --legacy --flatpak\n \
|
||||
sys install {application} --legacy --portage\n \
|
||||
sys install {application} --legacy --nix\n \
|
||||
sys install {application} --legacy --snap\n \
|
||||
sys install {application} --legacy --appimage\n \
|
||||
sys install {application} --legacy --ebuild"
|
||||
);
|
||||
return Err(
|
||||
"legacy install requires an explicit package source flag (--flatpak, \
|
||||
--portage, --nix, --snap, --appimage, --ebuild)"
|
||||
.into(),
|
||||
);
|
||||
} else if available.len() == 1 {
|
||||
let b = available[0].backend;
|
||||
let flag = backend_flag_name(b);
|
||||
eprintln!("sys: legacy install requires an explicit package source flag.");
|
||||
eprintln!(
|
||||
"`{application}` is available via {}. In legacy mode, specify the source flag:",
|
||||
b.as_str()
|
||||
);
|
||||
eprintln!(" sys install {application} --legacy --{flag}");
|
||||
return Err(format!(
|
||||
"legacy install requires an explicit package source flag: run `sys install {application} --legacy --{flag}`"
|
||||
)
|
||||
.into());
|
||||
} else {
|
||||
eprintln!("sys: legacy install requires an explicit package source flag.");
|
||||
eprintln!(
|
||||
"`{application}` is available from multiple package lanes. In legacy mode, choose a source:"
|
||||
);
|
||||
for c in &available {
|
||||
let flag = backend_flag_name(c.backend);
|
||||
let ver = c.version.as_deref().unwrap_or("latest");
|
||||
eprintln!(
|
||||
" sys install {application} --legacy --{:<10} # {} ({} via {})",
|
||||
flag,
|
||||
c.backend.as_str(),
|
||||
ver,
|
||||
c.provenance
|
||||
);
|
||||
}
|
||||
return Err(
|
||||
"legacy install requires an explicit package source flag: choose one of the commands above"
|
||||
.into(),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(application) = parse_removal_text(&full_phrase) {
|
||||
|
|
@ -472,6 +516,39 @@ fn apply_backend_filter(
|
|||
SystemAction::CleanCache { .. } => SystemAction::CleanCache {
|
||||
backend: Some(backend),
|
||||
},
|
||||
SystemAction::InstallPackage { atom, .. } => match backend {
|
||||
Backend::PortageSource => SystemAction::InstallPackage {
|
||||
atom,
|
||||
binary: Some(false),
|
||||
},
|
||||
Backend::PortageBinary => SystemAction::InstallPackage {
|
||||
atom,
|
||||
binary: Some(true),
|
||||
},
|
||||
Backend::PortageVendor => SystemAction::InstallPackage { atom, binary: None },
|
||||
Backend::Flatpak => SystemAction::FlatpakInstall { app_id: atom },
|
||||
Backend::Snap => SystemAction::SnapInstall {
|
||||
name: atom,
|
||||
channel: None,
|
||||
classic: false,
|
||||
},
|
||||
Backend::Nix => SystemAction::NixPrepare {
|
||||
packages: vec![atom],
|
||||
},
|
||||
other => {
|
||||
return Err(
|
||||
format!("install is not directly supported via {}", other.as_str()).into(),
|
||||
);
|
||||
}
|
||||
},
|
||||
SystemAction::RemovePackage { atom } => match backend {
|
||||
Backend::Flatpak => SystemAction::FlatpakRemove { app_id: atom },
|
||||
Backend::Snap => SystemAction::SnapRemove { name: atom },
|
||||
_ if backend.is_portage() => SystemAction::RemovePackage { atom },
|
||||
other => {
|
||||
return Err(format!("remove is not supported via {}", other.as_str()).into());
|
||||
}
|
||||
},
|
||||
other => {
|
||||
return Err(format!(
|
||||
"--portage/--flatpak/--nix/--snap/--appimage/--ebuild only apply to \
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue