Keep retries inside a total deadline
Inject a monotonic clock and limit retries without turning failures into endless work.
Attempts and elapsed time are different budgets
A maximum attempt count prevents an endless retry loop, but it does not bound how long each attempt can wait. A total deadline covers all attempts and the pauses between them. Before another probe starts, calculate the remaining milliseconds and cap its requested timeout to that amount.
Use a monotonic clock for elapsed time. Wall clocks can move when synchronization or manual changes occur. The lesson injects a clock and sleep callback so tests can advance virtual time instantly. ManualClock in lab_clock.py is deliberately not a claim about real browser scheduling or network latency.
Retry a defined observation
Only timed_out results retry in this policy. Open and refused observations are useful terminal evidence; an error needs diagnosis rather than an automatic retry storm. A returned result is assumed to have already passed the probe-record validation from earlier lessons. Cancellation and exceptions propagate to the caller.
The reason field describes why the attempt loop ended. observed means a terminal observation arrived before the deadline; it is not a claim that the port was open. attempt_limit means timeouts used every permitted attempt. deadline means the budget was exhausted, a result arrived too late, or the next backoff cannot fit.
Check the clock after each probe as well as before it. An adapter that ignores its timeout can still return late; preserve that last result for diagnosis but label the deadline outcome honestly. A synchronous caller cannot forcibly preempt an arbitrary injected function. The runtime's Stop control remains the outer interruption mechanism.
from lab_clock import ManualClock
clock = ManualClock()
clock.advance_ms(75)
clock.sleep(0.025)
print(round(clock() * 1000))
Put it into practice.
Define retry_probe(probe, clock, sleep, address, port, timeout_ms=100, budget_ms=250, max_attempts=3, backoff_ms=10). Validate integer non-bool policy values: timeout 1..5000, budget 1..30000, attempts 1..3, backoff 0..1000. clock supplies monotonic seconds; sleep accepts seconds; probe returns a validated state record. Set a total deadline. Each timeout is min(timeout_ms, floor(remaining milliseconds)); fewer than 1 ms ends with reason deadline. After a probe, reaching/passing the deadline also ends with deadline. Only timed_out retries; other states end with observed. Do not sleep after the final allowed attempt (reason attempt_limit), or if backoff reaches/exceeds remaining time (reason deadline). Return result (last row or None), attempts, reason. Propagate exceptions/cancellation. Use injected time, no direct real sleeping.
See an explained solution
def retry_probe(probe, clock, sleep, address, port, timeout_ms=100, budget_ms=250, max_attempts=3, backoff_ms=10):
for value,low,high in [(timeout_ms,1,5000),(budget_ms,1,30000),(max_attempts,1,3),(backoff_ms,0,1000)]:
if type(value) is not int or not low <= value <= high:
raise ValueError("Invalid retry policy")
deadline=clock()+budget_ms/1000
result=None;attempts=0
while attempts < max_attempts:
remaining=round((deadline-clock())*1000,9)
if remaining < 1:
return {"result":result,"attempts":attempts,"reason":"deadline"}
result=probe(address,port,min(timeout_ms,int(remaining)))
attempts+=1
if clock() >= deadline:
return {"result":result,"attempts":attempts,"reason":"deadline"}
if result["state"] != "timed_out":
return {"result":result,"attempts":attempts,"reason":"observed"}
if attempts == max_attempts:
return {"result":result,"attempts":attempts,"reason":"attempt_limit"}
delay=backoff_ms/1000
if delay >= deadline-clock():
return {"result":result,"attempts":attempts,"reason":"deadline"}
if delay:sleep(delay)
return {"result":result,"attempts":attempts,"reason":"attempt_limit"}
if __name__ == "__main__":
from lab_clock import ManualClock
clock=ManualClock()
print(retry_probe(lambda *args:{"state":"open"}, clock, clock.sleep, "192.0.2.10", 443))
The total deadline and attempt ceiling constrain different failure modes. Injected time makes boundaries testable without slow sleeps or changes to the browser runtime.