compiler/docs/en/python/implementation-plan.md

# TypePHP Python Interop Phased Implementation Plan

> This plan takes `python/design.md` as its specification. Each phase strictly follows: first add PHPUnit/PHPT/pytest tests and confirm they fail, then implement, then run the relevant tests and the full regression.

## Phase 1: Python module names, use, and lazy binding

The goal is to complete a minimal runnable closed loop, without implementing operators and general conversion:

1. Recognize `python\module\member()` / `python\module\member` in the global namespace, `\python\module\member()` / `\python\module\member` in other namespaces, the optional `use python\module` shorthand, case-insensitive root names, and case-sensitive Python subsequent names.
2. Do not build a Python-specific alias table, and do not specially handle `use`; use PHP's ordinary namespace, `use function`, `use const`, `as` alias, conflict checking, and fully qualified name resolution. Inside a namespace it must be written as `\python\module`; the relative `python\module` still resolves to a name under the current PHP namespace.
3. Allocate a module ID only when `python\module\attr`, `python\module\func()`, or their alias forms appear.
4. Generate a `pythonModuleMap` of the same kind as `funcMap`, a lazy getter, and request-clean code.
5. Dynamically call `PyCore::import()` using the Zend class/function map; do not include, link, or detect phpy.
6. Use the Zend object API to read module attributes and call module callables.
7. When phpy is not loaded, throw a PHP `Error` at the first actual use; an unused Python use does not trigger an error.

Test order: PHPUnit code generation and diagnostic tests → PHPT runtime tests → existing compiler regression.

Implementation status: completed. Fully qualified names and any alias share the same runtime slot per Python dotted module name; fully qualified names do not require `use`, and both syntaxes lazy-import on first actual execution. Before recognition, PHP namespace resolution is strictly applied: for example, in `namespace App`, the relative name `python\math\sqrt()` is an ordinary `App\python\math\sqrt()`, and only `\python\math\sqrt()` points to the Python root namespace.

## Phase 2: builtins, construction syntactic sugar, and static types

1. `python\name()` is dynamically called through the phpy Zend Facade: explicit `PyCore` methods are reused directly, and other names go through a Python `builtins` module lookup.
2. `python\list/dict/tuple/set/str/object()` maps to existing phpy Zend classes or methods.
3. `new PyList()` and `python\list()` and the like obtain the same logical static type.
4. Python call results remain `PyObject` or known phpy subclasses, turning off implicit scalar conversion on the TypePHP path.
5. Tests for missing phpy, nonexistent builtins, argument errors, and exception mapping.

Implementation status: completed. TypePHP lazily enables phpy's `return_as_object` the first time a Python expression is actually executed; merely declaring unused Python symbols still does not trigger a runtime dependency.

## Phase 3: argument conversion and explicit result conversion

1. TypePHP arguments are automatically converted to Python values after left-to-right evaluation.
2. Conversion of scalars, arrays, empty arrays, nested containers, and TypePHP callables.
3. Leave the Python object rules via `$py->toValue()` or `python\scalar($py)`; when a native type is needed, continue to use ordinary TypePHP conversion, e.g. `$py->toValue()->toInt()`. Containers and iterators can directly use `$py->toArray()`.
4. Tests for deep copying, recursive containers, overflow, Unicode/bytes, and exception paths.
5. Review and refactor phpy's conversion strategy, removing global temporary conversion state that affects synchronous reentrancy.

Implementation status: the core boundary is complete. TypePHP arguments are strictly evaluated left to right, supporting scalars, empty arrays, nested list/dict, and callables; `PyObject::toValue()` and `python\scalar()` reuse phpy's explicit conversion entry point, and `PyObject::toArray()` converts supported containers and iterators. phpy has removed the process-level conversion function pointer, replacing it with a local stateful converter, RAII recursion protection, and a 128-level depth limit, and covers invalid UTF-8, PHP self-referential arrays, and Python cyclic container error paths. The final language mapping for Python big integers and bytes remains in this phase's follow-up work.

## Phase 4: operators

1. Rewrite operators into dynamic calls to the Python standard library `operator` module.
2. Mixed operands are first converted to `PyObject`.
3. Strictly guarantee left-to-right evaluation, each evaluated once.
4. Use `operator.is_/is_not/truth` to implement identity and truthiness, and `iadd/isub/...` to implement compound assignment.
5. Verify that `operator` automatically handles `NotImplemented`, reflected dunders, and subclass priority.
6. Against phpy's opcode-handler behavior, fix existing issues such as `/` incorrectly mapping to floor division.

Implementation status: completed. Binary arithmetic, bitwise operations, comparison, `===`/`!==`, unary operations, conditional truthiness, short-circuit logic, and compound assignment are all executed through the implicit `operator` module binding; `/` uses `truediv`. Mixed TypePHP operands are converted by phpy at the call boundary, results continue to remain `PyObject`, and comparison and truthiness results explicitly converge to a TypePHP `bool`. Attribute and subscript lvalues are written back through the dynamic write protocol of Phase 5.

## Phase 5: the complete object protocol

1. Reading, writing, and deleting Python object attributes.
2. Subscript read/write, delete, and `isset()`.
3. iterator/foreach.
4. Synchronous reentrancy of Python callables and TypePHP callable proxies.
5. keyword arguments, argument unpacking, and error semantics.

Implementation status: completed. Dynamic attributes, unknown methods, subscripts, deletion, `isset()`, `foreach`, and callables of Python proxies all reuse phpy's Zend object protocol, without generating phpy C++ symbols. Method, attribute, subscript, and callable results continue to propagate as `PyObject`, supporting chained access and subsequent Python operations. Named arguments and unpacking reuse the unified call argument pipeline and keep left-to-right evaluation; attribute and subscript compound assignments use the returned object of `operator.i*()` to write back the original lvalue.

phpy simultaneously completed object protocol hardening: `__set()` conversion reference release, `__unset()`, list/tuple negative indexing, list deletion, `isset()` semantics for missing keys and Python `None`, deletion and contains state checks, and iterator/count exception propagation. The related bugs are independently covered by phpy PHPUnit and TypePHP PHPT.

## Phase 6: phpy stability and performance finishing

1. Full audit of the CPython/ZendVM lifecycle, GIL, and owned/borrowed/stolen references.
2. Audit of Python/Zend exception state and traceback.
3. Cross-VM reference cycles, destruction, and exception injection tests.
4. ASan/UBSan, PHP leak report, Python debug build, and stress tests.
5. Benchmark dynamic Zend calls, the module map, argument conversion, and `operator` module calls; only optimize hotspots proven by data.
6. Full PHPUnit, pytest, PHPT, and existing TypePHP compiler regression.

Implementation status: completed. The first round of CPython failure-path audits covered the construction and subscript writes of general objects, list, dict, tuple, and set, as well as sequence/set `contains()`. PHP-to-Python key/value conversion failures are now immediately mapped to `PyError`, all newly acquired references are released by scope guards; construction failure no longer leaves an unhandled CPython error indicator, and the `-1` error result of `contains()` is no longer misjudged as `true`. Paths such as invalid UTF-8, unhashable set members, and containers remaining usable after failure have phpy PHPUnit regression tests.

The second round covered module import, exception conversion, callable checks, and the explicit iterator API. New references transferred to the caller by `PyImport_ImportModule()`, `PyErr_Fetch()`, and `PyIter_Next()` are now uniformly released after the Zend wrapper acquires an independent reference; repeated imports, Python exceptions, or explicit iterator next no longer continuously increase the reference count. Calling a non-callable Python attribute or `PyObject` reliably throws `PyError(TypeError)`, no longer silently returning `null` because `PyCallable_Check()` did not set an error indicator. `PyCore::next()` also distinguishes normal end-of-iteration from iterator exceptions. All the above paths first establish failing phpy PHPUnit regression tests, with object call behavior additionally covered by TypePHP PHPT integration.

The third round covered conversion failures and function caching of the `PyCore` Facade. `PyCore::eval()` throws a `PyError` immediately after globals conversion fails, and `PyCore::bytes()` uses the converted `zend_string` for non-string scalars; neither dereferences a null pointer or the wrong zval union field, which would crash the process. `PyCore::next()` simultaneously releases the iterator reference produced by argument conversion. The builtin/operator function cache uses a `std::string` content key, no longer using request-level `char*` addresses as long-term keys, and also avoids the same-named dynamic call repeatedly caching and increasing Python function reference counts; calling a builtin that exists but is not callable releases the temporary reference and throws `PyError(TypeError)`. All issues are covered by independent PHPUnit tests that first fail, of which two crashes were confirmed in isolated processes with core dumps disabled (exit code 139) before being fixed.

The fourth round covered the Python-to-PHP synchronous callback boundary. phpy converts Python keyword arguments into Zend named parameters and stops immediately after any positional or named argument conversion fails, so a PHP callable that only received part of its arguments is never executed. PHPX generates and manages Zend `arg_info` parameter name metadata for AOT native closures, so Python kwargs can be bound to TypePHP closures by name rather than depending on parameter positions or degrading to a string callable. phpy PHPUnit, PHPX unit tests, and TypePHP PHPT respectively cover conversion failure, Zend named binding, and the complete Python→TypePHP callback chain.

The fifth round covered exceptions and ownership when Python strings cross the Zend boundary. When a Python lone surrogate character cannot be encoded as UTF-8, `phpy.String`, dynamic PHP class names, dictionary keys, `PyObject::__toString()`, and Python exception message formatting no longer use null pointers or uninitialized lengths; before the fix, the relevant isolation tests would exit with 139 or attempt to allocate abnormally large memory. `StrObject` now has an explicit valid state, and all callers must check the conversion result before accessing the pointer; stringification of exception messages is only best-effort auxiliary information, preserving the original Python error/type/value on failure and cleaning up the temporary CPython error indicator. `new_string()` also completes Zend carrier destruction registration, and fixed-length strings directly obtain a unique `zend_string` reference, eliminating the leak and uninitialized zval on the success path.

The sixth reference audit fixed the new-reference leak in `PySequence::slice()`. After the slice is wrapped as a Zend `PyObject`, the original ownership returned by the CPython API is released while the wrapper keeps its own reference; a `sys.getrefcount()` stress test verifies that repeatedly creating and destroying slices does not continue to increase the element reference count. Slice creation failure is also converted to `PyError` before touching a null pointer.

The seventh round covered the Python operator protocol of dynamic PHP. The phpy opcode handler now uses CPython `PyNumber_*` / `PyObject_RichCompareBool()`, `/` and `/=` use true division, compound assignment updates the Zend lvalue with the object returned by the in-place API, and correctly handles immutable Python objects, Zend reference variables, expression results, and the lvalue state after failure. `===` / `!==` use Python object identity, bool cast, `!`, and conditional branches use the Python truth protocol; new references on the PHP operand conversion, result, and exception paths are uniformly managed by RAII guards. TypePHP's `operators.phpt` runs with both ZendPHP + opcode handler and AOT + `operator` module, so it is the output-consistency gate for the two implementations.

Unary positive/negative operations in dynamic code have been confirmed as a compatibility boundary: PHP compiles `-$value` / `+$value` into multiplication by `-1` / `1`, and the opcode handler cannot distinguish this from explicit multiplication in the source. Dynamic ZendVM code no longer attempts to rewrite the AST, but explicitly retains the `$value * -1` / `$value * 1` behavior. Python builtin numerics and common objects such as NumPy usually produce consistent results, but a custom object's `__neg__()` / `__pos__()` may differ from its `__mul__()`. AOT TypePHP retains the original AST and still lowers to `operator.neg()` / `operator.pos()` respectively, so the semantics are unaffected. The external user document `python.md` has listed this difference as a compatibility limitation.

The performance benchmarks separately cover module properties, operator module calls, existing `PyObject` arguments, and PHP scalar argument conversion. In the current unoptimized build, module properties are about 0.7–0.9 μs/op, and operator calls about 1.8–2.5 μs/op; whether arguments are pre-wrapped as `PyObject` shows no stable difference. After attempting to eliminate the per-access reference count of the module map with an indirect zval wrapper, the A/B medians remained within the same noise range, so the more lifetime-sensitive optimization with unproven benefit was not retained.

phpy's CMake Python-extension target also completed the out-of-tree build fix, and is compatible with the `int` ABI of older PHP and the `zend_result` ABI of newer PHP by deriving the return type from Zend's standard cast handler. Both PHP 8.1 and PHP 8.4 have completed clean-build verification.

The memory gate uses Valgrind Memcheck. Tests disable the Zend allocator and PCRE JIT, and in a minimal independent process loop 100 times each over PHP Closure kwargs callbacks, callable PHP object kwargs callbacks, sequence slice creation/destruction, object stringification of invalid Unicode, dictionary key conversion, and exception formatting. The result is 0 invalid-access, 0 definite leak, 0 indirect leak; the 493,106 bytes retained by PHP/CPython at process exit are all still-reachable and not counted as leaks. The ASan extension cannot be safely `dlopen`-ed into the current non-ASan PHP with `RTLD_DEEPBIND` enabled, so this round uses Valgrind, which does not require recompiling PHP, as the memory checking tool.

Phase 6 final gate results: phpy PHPUnit 135 tests / 469 assertions passed (1 existing warning, 1 environment-related skip), pytest 26/26 passed, TypePHP Python PHPT 14/14 passed, TypePHP PHPUnit 1103 tests / 2729 assertions passed; the TypePHP compiler full PHPT has 934 items in total, of which 932 PASS, 2 SKIP, 0 FAIL, 0 WARN. The dynamic operator stress test under Valgrind is 0 invalid access, 0 definite leak, 0 indirect leak.

## Phase Gates

- Do not start implementation before the current phase's failing tests are established.
- Do not enter the next phase before all tests of the current phase pass.
- phpy behavior changes must first add PHPUnit/pytest tests in the phpy repository.
- Every fixed bug must retain an independent regression test.
- Do not mask implementation differences by modifying third-party test expectations.