3.2-clock-and-time.md

3.2 Clock and Time

Architecture documentation > 3. Virtual Devices

The NVX time ABI is the single guest-visible time, timer, and clock contract on every backend and launch path. It is versioned independently of the machine ABI. Every rule is a hard check: a host or snapshot that cannot meet the contract is rejected with a stable error code, a guest that observes a violation fails fast, and no rule falls back to another behavior.

The time ABI applies the architecture goal of keeping the guest-visible machine independent of the selected hypervisor to clocks and timers. It provides:

• The same CPUID identity, synthetic MSRs, CPU time bits, clocksource, tick, timers, and restore semantics on every backend and launch path.
• No guest kernel patches and no clock parameters on the kernel command line.
• Restore within one backend across hosts of the same CPU generation and CPU profile whose TSC rates match within a small tolerance, including after a host reboot.
• Guest monotonic time that advances by the snapshot downtime, and a wall clock that is set on every restore and disciplined afterwards.
• Kernel stall, lockup, clocksource, and TSC-warp detectors that never trip across capture and restore.

The guest sees:

• A TSC that ticks at a constant rate in every processor power and frequency state (an invariant TSC) as its only clocksource.
• The declared TSC and LAPIC timer rates through a minimal synthetic hypervisor identity.
• A one-shot LAPIC timer at the backend's fixed rate.
• A PIT that the guest disables at boot.
• A programmable RTC anchored to UTC.

An invariant TSC keeps one rate on a given host; it does not give every host the same rate. The guest TSC runs at its host's native rate, which can differ between hosts of one CPU generation and is not part of the CPU profile, so the guest reads the declared rate from a frequency MSR instead of deriving it from CPUID. A cold boot declares its host's rate, and a snapshot restores only on a host whose rate lies within 250 ppm of that declaration; see Rates.

Table 3.4 lists the notation that the time ABI uses.

Table 3.4. Time ABI notation.

Symbol	Meaning
F	Declared guest TSC rate in Hz
F_s	Declared rate recorded in a snapshot
F_d	Native TSC rate of the destination partition in Hz
L	LAPIC timer rate in Hz
C	Processor capacity of the machine
D	Snapshot downtime in nanoseconds
T_c	Guest TSC of the first processor at the capture anchor
Q	Host monotonic time from the VM-time cut at quiesce to the capture anchor
g	Restore generation counter of a VM process

3.2.1 Hypervisor Identity

As Table 3.5 shows, every processor sees a minimal Hyper-V frequency and invariant-TSC identity, configured identically on every backend. Each row lists the values that one leaf returns in the four general-purpose registers through which CPUID reports its results (EAX, EBX, ECX, and EDX).

Table 3.5. Hypervisor identity leaves.

Leaf	EAX	EBX	ECX	EDX
0x40000000	0x40000005 (maximum leaf)	0x7263694d ("Micr")	0x666f736f ("osof")	0x76482074 ("t Hv")
0x40000001	0x31237648 ("Hv#1")	0	0	0
0x40000002	0x0058564e ("NVX")	0x00010000 (time ABI 1.0)	0	0
0x40000003	0x00008860	0	0	0x00000100
0x40000004	0	0xffffffff	0	0
0x40000005	C	C	0	0

Leaf 0x40000003 grants exactly the privileges that Linux needs to detect the platform, hypercall-MSR and virtual-processor-number (VP-index) access, plus frequency-register and invariant-TSC-control access, and advertises only the frequency registers. Leaf 0x40000004 recommends nothing and never requests spinlock notification. Leaves 0x40000006..0x4000000f and 0x40000080..0x40000082 read as explicit zeros. Every other leaf in 0x40000006..0x400000ff returns either zeros or the vendor's architectural out-of-range result, and no base from 0x40000100 through 0x4000ff00 carries a hypervisor signature, so Linux selects only the Hyper-V platform. Beyond the leaves of Table 3.5, no leaf returns Hyper-V feature data, and leaf 0x40000081 carries no VS#1 virtualization-stack interface signature.

The identity deliberately deviates from the Hyper-V specification:

• The hypercall MSRs raise a general-protection fault (#GP), there is no hypercall page, and hypercall instructions are outside the contract.
• The TSC frequency MSR returns the declared rate F, which may differ from the physical rate by up to the rate tolerance after a restore.
• The invariant-TSC control can be cleared after it is set.
• The invariant-TSC CPUID bit is set from boot rather than after the control is written.
• Leaf 0x40000002 carries an NVX signature and the time ABI version instead of a build number.
• Leaf 0x40000005 reports the processor capacity.

The guest's only side effect is selecting the Hyper-V platform. No paravirtual clock, interrupt, TLB, spinlock, steal-time, or page-fault feature exists on any backend, and the scheduler clock and vDSO use the raw TSC.

3.2.2 MSRs

As Table 3.6 shows, the VMM serves the identity MSR range 0x40000000..0x400001ff identically on every backend.

Table 3.6. Identity MSRs.

MSR	Name	Read	Write
0x40000002	VP index	Processor index	#GP
0x40000022	TSC frequency	F	#GP
0x40000023	APIC frequency	L	#GP
0x40000118	TSC invariant control	Last value written; 0 after reset	0 or 1 accepted; any other value #GP
Any other MSR in the range		#GP	#GP

The TSC invariant control is one partition-wide value. It is guest state, saved and restored with the VM. Linux writes 1 once at boot without fault handling, so that write must never fault on any backend.

The MSRs that hold an offset added to the processor's TSC (IA32_TSC_ADJUST, 0x3b) and the TSC value at which the LAPIC timer fires in TSC-deadline mode (IA32_TSC_DEADLINE, 0x6e0) raise #GP on every backend, as do the cache-control registers of Intel's P6 processor generation (the legacy P6 L2-cache MSRs).

As Table 3.7 shows, under an AMD CPU profile, the VMM also serves two configuration MSRs with fixed values.

Table 3.7. AMD configuration MSRs.

MSR	Name	Read	Write
0xc0010015	Hardware configuration (HWCR)	0x1000000: only the bit that makes the TSC count at a fixed frequency (TscFreqSel)	The value read changes nothing; any other value #GP
0xc0011029	Decode configuration (DE_CFG)	0x2: only the bit that makes the load-fence instruction hold later instructions until earlier ones complete (LFENCE serialization)	The value read changes nothing; any other value #GP

3.2.3 CPU Time Bits

Table 3.8 lists the bits that every CPU profile fixes. A CPU profile that violates them is rejected.

Table 3.8. CPU time bits.

Leaf	Register and bits	Value
0x1	ECX[31] hypervisor present	1
0x1	ECX[24] TSC-deadline timer	0
0x1	ECX[15] performance and debug capabilities MSR (PDCM)	0
0x1	EDX[4] TSC	1
0x6	EAX, EBX, ECX, EDX	0x00000004 (only the always-running APIC timer, ARAT), 0, 0, 0
0x7.0	EBX[1] IA32_TSC_ADJUST	0
0xa	EAX, EBX, ECX, EDX	0 (no PMU)
0x15, 0x16	EAX, EBX, ECX, EDX	0, when within the maximum basic leaf
0x80000001	EDX[27] RDTSC variant that also returns a processor identifier (RDTSCP)	1
0x80000007	EAX, EBX, ECX, EDX	0, 0, 0, 0x00000100 (invariant TSC only)

Leaves 0x15 and 0x16 are zeroed rather than synthesized. Linux takes both rates from the frequency MSRs, which keeps the CPU profile's CPUID independent of any host's rate. The invariant-TSC bit is policy rather than a fingerprinted feature: every CPU profile sets it, and every backend presents it.

3.2.4 Rates

The guest TSC runs at the destination's native rate F_d and is never scaled. The declared rate F is fixed for the life of a snapshot lineage: a cold boot declares its native rate, and every restored process keeps F = F_s. The kernel computes its TSC frequency once and never recalibrates.

At restore, the native rate must lie within 250 ppm of the declared rate, computed with integer arithmetic of at least 128 bits:

accept  iff  |F_d - F_s| * 1_000_000  <=  250 * F_s

The boundary is accepted. The manifest records the tolerance, and any other recorded value is rejected. Within the tolerance, the TSC is neither scaled nor trapped, the frequency MSR keeps returning F_s, and guest clocks run fast or slow by at most the deviation. The restore packet reports that deviation in the kernel's frequency-adjustment unit, ppm scaled by 2^16, so that the guest can compensate:

rate_deviation = round((F_d - F_s) * 1_000_000 * 65_536 / F_s)

Every rate must also be plausible, 500 MHz <= F <= 10 GHz. A backend that cannot report its native rate is rejected at cold boot, capture, and restore alike; there is no calibration fallback.

The LAPIC timer rate L is a backend constant: 1 GHz on KVM and 200 MHz on MSHV and WHP. The kernel presets its timer period from the APIC frequency MSR and never calibrates the LAPIC. On restore, the destination's rate must equal the recorded rate exactly.

3.2.5 Clocksource, Tick, and Timers

With this identity, the guest kernel takes the TSC rate from the frequency MSR, trusts the TSC as reliable, which disables the clocksource watchdog and the CPU-online warp test, presets the LAPIC timer period, and disables the PIT at boot. The booted guest must then be in this state:

• Every CPU reports a constant, nonstop, known-frequency, reliable TSC with RDTSCP and ARAT, and none reports a TSC-deadline timer or TSC_ADJUST.
• The TSC is the current clocksource and, after each CPU's first tick, the only available one.
• Ticks are tickless-idle with high-resolution timers, and every online CPU's tick device is the LAPIC timer in one-shot mode.
• No PIT, high-precision event timer (HPET), or broadcast clock-event device exists.
• The LAPIC timer is never periodic and never in TSC-deadline mode after boot.
• PIT channel 0 never counts after boot, and IRQ 0 has no timer handler.

Capture and restore reject an armed periodic or TSC-deadline LAPIC timer and a periodically counting PIT channel 0. Across a downtime D, a counting one-shot LAPIC timer advances by exactly:

ticks = floor(D * L / 1_000_000_000 / divide)

where divide is the timer's divide configuration. If ticks reaches the current count, the count becomes zero and the timer interrupt is queued unless the LAPIC register that configures it (its LVT entry) is masked; otherwise, the count decreases by ticks. A queued or pending vector must reach its processor when the processor is released, even if it is halted.

3.2.6 Backend Obligations

Each backend implements the same obligations behind one VMM interface. The guest-visible result is identical; only the mechanism differs. Every backend must:

• Present exactly the identity leaves and their explicit zero leaves, with no other hypervisor signature.
• Present the CPU profile's effective CPUID, including the CPU time bits.
• Route the identity MSRs, and under an AMD CPU profile the two configuration MSRs, to the VMM.
• Report the native TSC rate and never scale the TSC.
• Run the LAPIC timer at the backend's fixed rate.
• Deliver a pending LAPIC vector after a LAPIC state write, even to a halted processor.
• Hide the TSC-deadline timer and TSC_ADJUST, and fault both MSRs.
• Expose the invariant TSC.
• Offer no paravirtual or synthetic features, and never enable a hypervisor-specific emulator or its saved state.
• Pair the first processor's TSC with a host time sample within 100 µs for the capture anchor, keeping the tightest of at most 64 samples.
• Set every instantiated processor's TSC at one host instant and verify it by read-back before any processor runs.
• Keep live cross-processor skew within 1 µs after release.

3.2.7 Cross-Processor Consistency

At every guest-observable instant, the TSCs of any two online processors differ by at most 1 µs, that is F_s / 1,000,000 cycles. The guest kernel no longer checks TSC consistency, so the bound is enforced outside the guest. Cold boot starts every processor's TSC together. Restore writes one target to every instantiated processor at one host instant and verifies it by read-back, so the VMM introduces no skew of its own. After release, each backend keeps live skew within the bound, and a host that cannot hold it does not qualify for the contract.

3.2.8 Downtime and Continuity

Guest monotonic time advances by the downtime D, the host time elapsed between the capture anchor and the restore anchor, plus at most the restore latency R from the restore anchor to the first restored instruction. The guest observes no other discontinuity between the snapshot request and the first restored instruction.

The capture anchor records:

• T_c, the first processor's guest TSC.
• Host UTC in nanoseconds since the Unix epoch.
• Suspend-inclusive host monotonic time.
• The host identity.
• The host boot identity.

A host that cannot provide all of them rejects capture and restore. As Table 3.9 shows, restore takes the same sample at its own anchor and selects exactly one downtime source.

Table 3.9. Downtime source selection.

Condition	Source	D
Same host identity, boot identity, and host clock kind	Host monotonic time	monotonic_now - monotonic_capture
Anything else	Host UTC	utc_now - utc_capture

D must satisfy 0 <= D <= 30 days. On the monotonic path, a UTC delta that differs from D by more than one second is reported as a host wall-clock step and does not change D. The restore packet reports the selected source.

VM time, the clock of the VMM's emulated devices, and the RTC stop at quiesce, before the capture anchor. Capture therefore records Q, the host monotonic interval from that cut to the anchor, and restore advances VM time and the RTC by D + Q so that they resume in step with the guest TSC, which kept running through Q. The TSC, the LAPIC timers, and the restore packet use D. D + Q must not exceed 30 days. Q is best effort: a snapshot without it restores with Q = 0 and a warning.

The restore anchor is the instant of the synchronized TSC set, and guest time runs from that instant on every backend; no backend defers it to the first processor run. No backend stops guest time for a capture either, so a capture that rolls back leaves the guest's clocks continuous, and the guest sees the quiesce as elapsed time, as it would a long preemption.

3.2.9 Time Samples

Figure 3.1 shows the layout of the 16-byte sample that writing 0xa7 to the byte-channel selector latches into the time window.

offset size field
     0    1 version = 1
     1    1 flags                bit 3 mirrors the restore packet's override flag; other bits zero
     2    2 reserved = 0
     4    4 generation           u32, little-endian, g of this VM process
     8    8 utc_ns               u64, little-endian, host UTC at the latch

Figure 3.1. Time-sample layout.

The VMM cannot observe a running processor's TSC on every backend, so the guest pairs host UTC with its own clock. It reads its realtime clock as t0 immediately before the selector write and as t1 immediately after it. The host instant lies in [t0, t1], so:

theta = utc_ns - (t0 + t1) / 2      uncertainty epsilon = (t1 - t0) / 2

The same bracket applies to the UTC sample in the restore packet. As Table 3.10 shows, the guest accepts a pairing only if epsilon is within the bound for its use.

Table 3.10. Time-pairing uncertainty bounds.

Use	Bound	Attempts	When every attempt exceeds the bound
Restore repair	1 ms	3	Repair fails, and the guest powers off
Initial synchronization at boot	1 ms	3	Conformance fails, and the guest powers off
Wall-clock discipline	50 µs	3	The poll is skipped; never fatal

3.2.10 Guest Obligations

The guest agents perform conformance checks, watch for violations, discipline the wall clock, and execute the snapshot time steps in every guest mode. Production paths keep the console quiet; status goes to a state file that the workload can read through a read-only mount. The state file reports the outcome of each phase's checks, the declared rates, the generation, the last downtime and its source, and the wall clock's discontinuities, synchronization, offset, uncertainty, and frequency correction.

Conformance. Before readiness, the guest only takes a time sample within the bound and steps its clock to host UTC, so the workload starts on host time. It runs every other boot check asynchronously after readiness, starting at idle priority, so the checks never delay readiness or take a processor from the workload. The checks cover the identity leaves and explicit zero leaves, the CPU time bits on every CPU, the identity MSRs, the kernel's platform, clocksource, and LAPIC-timer selection, CPU flags, tick devices, the absence of forbidden clock parameters, stall-detector settings, a running violation watcher, and kernel W+X integrity. Capture reruns the clocksource, tick-device, and watcher checks. Restore reruns them too, checks the declared rate in the frequency MSR again, and checks the identity leaves, the CPU time bits, and the CPU flags of every processor that it brings online. Every check is fail-fast: a failure powers the guest off even if the workload is running.

Violation watcher. The watcher follows the kernel log from before the boot checks, so no record is missed. TSC instability; clocksource instability, switches, or skew; TSC warps; TSC_ADJUST faults; RCU stalls or starvation; soft or hard lockups; hung tasks; unchecked MSR accesses; APIC identity mismatches; and lost log records are violations. The watcher also polls the kernel's RCU stall count, so a stall is caught even if its record is lost. A violation is reported as one bounded event line on the kernel log, the byte channel, and the console, and the guest then powers off through the shutdown port.

Stall suppression. Restoring after downtime makes kernel time jump, so the guest suppresses RCU stall detection, and the debug stall detectors, before a capture request. After restore, it releases the suppression only when the grace period that was in flight at capture has ended, forcing a full normal grace period first. If that grace period stays stuck beyond the stall timeout, the guest releases the suppression anyway so that the kernel reports the real stall.

Wall-clock discipline. The guest keeps its realtime clock on host UTC through the kernel's phase-locked loop. It polls every 16 seconds for the first four polls after boot or a restore, then every 64 seconds, and takes up to three samples per poll. An offset of at least 128 ms is stepped and counted as a discontinuity; a smaller one is slewed with error bounds derived from the offset and its uncertainty, which marks the clock synchronized for applications. The kernel limits the frequency correction to ±500 ppm, of which the rate tolerance uses at most 250 ppm. A sample whose generation differs from the recorded one is discarded, because a restore happened and restore repair resets the discipline. The discipline follows host wall-clock steps and never powers off the guest. Restore repair and the discipline step the clock through the kernel, so a workload can observe every step through the kernel's clock-set notification, such as a timerfd armed with TFD_TIMER_CANCEL_ON_SET.

The snapshot protocol and guest repair define the snapshot time steps.