Python requirements install fail - ERROR: Could not find a version that satisfies the requirement

[isc-patrick]

Note from @isc-tleavitt: most likely the need here is to make the --only-binary=:all: flag configurable in IPM.

---

The specific error is:

56.61 ERROR: Could not find a version that satisfies the requirement pyiceberg==0.8.0 (from pyiris-iceberg) (from versions: 0.0.1, 0.1.0rc1, 0.1.0rc2, 0.1.0, 0.2.0rc0, 0.2.0rc1, 0.2.0, 0.2.1rc0, 0.2.1, 0.3.0rc1, 0.3.0rc2, 0.3.0, 0.4.0rc1, 0.4.0rc2, 0.4.0)
56.71 ERROR: No matching distribution found for pyiceberg==0.8.0

The problem is in the flags provided to pip:

50.32 Running command=$lb("python3","-m","pip","install","-r","requirements.txt","-t","/usr/irissys/mgr/python/","--python-version","3.10.12","--only-binary=:all:")

I have solved by not installing via IPM. It does seem like there are wheels for all platforms and versions, (https://pypi.org/project/pyiceberg/0.8.0/#files). I'm not sure that this is a bug in IPM, but can't quite figure out why pip can only see up to version 0.4.0, which is from july of 2023. I upgraded pip, but that did not fix.

[isc-tleavitt]

I also find this mystifying. @isc-shuliu could you take a quick look when you get some spare cycles?

[isc-tleavitt]

Repo for this: https://github.com/isc-patrick/pyiris-iceberg

[isc-tleavitt]

Hunch: I think the issue is probably the --only-binary=:all: flag; https://github.com/apache/iceberg-python/actions/workflows/python-release.yml?page=4 seems to indicate that the build process started as of 0.5.x which lines up with the point where the list of versions you see ends.

It would be helpful to know if pip install, not via IPM, with the --only-binary=:all: flag hits the same issue on the environment where you see this. Looking at the actual contents of the artifact from https://github.com/apache/iceberg-python/blob/main/.github/workflows/python-release.yml there are fewer wheels than I'd expect from looking at the CI workflow's definition - macosx_10 and macosx_11 (nothing higher) and no arm64 linux.

From an IPM perspective, it would be nice if the --only-binary=:all: flag was configurable.

[isc-patrick]

pip install outside of IPM still has the same problems, if using the --only-binary=:all:. I should have been more clear in my initial post. By args to pip, I meant the --only-binary=:all: and --python-version 3.10.12 args, and by solved by not installing with IPM I meant removing those args.

[isc-shuliu]

This is tricky. We usually need the --python-version flag to ensure we're installing a distribution compatible with the embedded python. Yet, one can't specify --python-version without setting --only-binary.

I'll make them optional for now but a better solution is required in the future.

[isc-shuliu]

Some notes:

• When IPM calls out to pip install, it doesn't know the python version associated with the pip executable. By default, pip looks for packages with its own python version, which can be incompatible with the embedded Python of IRIS.
• pip supports --python /path/to/python/venv/ or --python /path/to/python/interpreter to manage environment of a given python interpreter. However, while IRIS doesn't comply with Python interpreter interface (meaning you can't run a python script with irisdb myscript.py).
• pip also supports --python-version=<version>. But this must be used together with --only-binary=:all:. More importantly, --only-binary=:all: does what it says. It won't default back to source distribution even if binary distribution isn't found.
• pip also supports --no-binary=:all:. This will force compilation of C extensions, which is almost always a bad idea:
  • The compilation tools and libraries may not be found on target systems
  • The build processes may fail if the build scripts are not well-maintained
  • Even if it works, compilation is very time consuming and may lack many optimizations

Ultimately, pip was not developed to support IRIS Embedded Python. Here are some ways to try:

1. Run pip install from embedded python: import pip; pip.main(["install", "-r", "requirements.txt", ...]). Caveat: a) This is discouraged by pip developers. b) We'll need to add a [ language = python ] method
2. Check pip's python version against embedded python version. If they match, the --python-version and --only-binary flags are no longer needed.
3. Allow users to choose among 3 options --python-version=3.y.z --only-binary=:all:, --no-binary=:all:, or no flags at all. I'm not sure which is the best default.

I think a combination of 2 and 3 is probably the way to go.