logo
Splunk Team

namespace packages and pip install -e

  • Last Update :
  • Techknowledgy :

Project that needs that namespace package can only install packages using pip and I keep getting this error when I run python3 pip install git+ssh://link_to_package_on_github@main#egg=package_name:

ERROR: Could not find a version that satisfies the requirement {
   package - name
}(unavailable)(from versions: 1.0)
ERROR: No matching distribution found
for {
   package - name
}(unavailable)

pyproject.toml looks something like this:

[project]
name = project - name
version = {
   use_scm = true
}

requires - python = ">=3.9"
dynamic = [
   "classifiers",
   "version"
]

[project.optional - dependencies]
[build - system]
requires = ["pdm-pep517"]
build - backend = "pdm.pep517.api"

[tool]
[tool.pdm]
package - dir = "src"
includes = ["calc"]

Suggestion : 2

Published Oct 11, 2020 by Timothée Mazzucotelli 

my_namespace_package /
   my_module_a.py

my_namespace_package /
   my_module_b.py
# main package
package /
   level1 /
   level2 /
   builtin_module.py
subpackage /
   __init__.py
sub.py
a.py

# plugin
package /
   level1 /
   level2 /
   plugin_module.py
# main package
package /
   __init__.py
level1 /
   __init__.py
level2 /
   __init__.py
builtin_module.py

# plugin
package /
   __init__.py
level1 /
   __init__.py
level2 /
   __init__.py
plugin_module.py
from pkgutil
import extend_path
__path__ = extend_path(__path__, __name__)
# main package
package /
   __init__.py
level1 /
   __init__.py
level2 /
   __init__.py
builtin_module.py

# plugin
package /
   level1 /
   level2 /
   plugin_module.py
# main package
package /
   __init__.py
level1 /
   __init__.py
level2 /
   __init__.py
builtin_module.py
plugin_module.py

Suggestion : 3

Namespace packages allow you to split the sub-packages and modules within a single package across multiple, separate distribution packages , Each sub-package can now be separately installed, used, and versioned. , Then you can break these sub-packages into two separate distributions: ,Python - Namespace Package About Articles Related Example Management Packaging

1._
mynamespace /
   __init__.py
subpackage_a /
   __init__.py
   ...
   subpackage_b /
   __init__.py
   ...
   module_b.py
setup.py

And you use this package in your code like so: 

from mynamespace
import subpackage_a
from mynamespace
import subpackage_b

Then you can break these sub-packages into two separate distributions: 

mynamespace - subpackage - a /
   setup.py
mynamespace /
   subpackage_a /
   __init__.py

mynamespace - subpackage - b /
   setup.py
mynamespace /
   subpackage_b /
   __init__.py
module_b.py

In the Python - setup.py, 

packages = find_packages(),
   namespace_packages = ['subPackage1', 'subPackage2', '...', 'subPackage3'],

Suggestion : 4

To explain why the init.py isn't there, since that seems to be a point of confusion, there isn't a flaw with piston's packaging. When you use pip to install piston, because of its use of namespace packaging, pip actually ignores the __init__.py in the package and uses an nspkg pth file to handle the namespacing: , I don't think the problem is entirely the use of __file__, per se, but in particular this pattern of locating __file__ that seems problematic with namespace packages: , I agree with Carl -- given Django's pattern of looking for various files in well known locations within application, the best we can do is document applications cannot be namespace packages. , As carljm said, the bug is that Django should *not* be relying upon file being there for its i18n functions. I'll evaluate the backward-incompatible b0rkage that would happen by making piston not a namespaced package, but Django should also do its part to DTRT.

I'll check it out on another box or two in a bit, but piston has no file attribute when i install it with pip in a virtualenv:

animal$ cd /tmp/
animal$ virtualenv env
New python executable in env/bin/python
Installing distribute.................................................................................................................................................................................done.
animal$ source env/bin/activate
(env)animal$ pip install django-piston
Downloading/unpacking django-piston
Downloading django-piston-0.2.3.tar.gz
Running setup.py egg_info for package django-piston

Installing collected packages: django-piston
Running setup.py install for django-piston

Skipping installation of /tmp/env/lib/python2.7/site-packages/piston/__init__.py (namespace package)
Installing /tmp/env/lib/python2.7/site-packages/django_piston-0.2.3-py2.7-nspkg.pth
Successfully installed django-piston
Cleaning up...
(env)animal$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import piston
>>> piston.__file__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
      AttributeError: 'module' object has no attribute '__file__'
      >>>

piston doesn't seem to have an init.py file in it's top-level directory, causing it not to have a file attribute.

(env)animal$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import piston
>>> piston
<module 'piston' (built-in)>

To explain why the init.py isn't there, since that seems to be a point of confusion, there isn't a flaw with piston's packaging. When you use pip to install piston, because of its use of namespace packaging, pip actually ignores the __init__.py in the package and uses an nspkg pth file to handle the namespacing:

$ pip install django - piston
Downloading / unpacking django - piston
Downloading django - piston - 0.2 .3.tar.gz
Running setup.py egg_info
for package django - piston
Installing collected packages: django - piston
Running setup.py install
for django - piston
Skipping installation of $VIRTUAL_ENV / lib / python2 .6 / site - packages / piston / __init__.py(namespace package)
Installing $VIRTUAL_ENV / lib / python2 .6 / site - packages / django_piston - 0.2 .3 - py2 .6 - nspkg.pth
Successfully installed django - piston
Cleaning up...

I don't think the problem is entirely the use of __file__, per se, but in particular this pattern of locating __file__ that seems problematic with namespace packages:

    if settings.SETTINGS_MODULE is not None:
       parts = settings.SETTINGS_MODULE.split('.')
    project = import_module(parts[0])
    projectpath = os.path.join(os.path.dirname(project.__file__), 'locale')

Suggestion : 5

Because any objects defined in the __init__.py get bound to the package’s namespace upon import, the __version__ variable is accessible from our package’s namespace as we saw earlier.,When building a package, it’s important to select appropriate names for your package and its modules. Python package naming guidelines and conventions are described in Python Enhancement Proposal (PEP) 8 and PEP 423. The fundamental guidelines are:,When a package is imported, the __init__.py file is executed, and any objects it defines are bound to the package’s namespace. As an example, in Python packaging, it’s convention to define a package’s version in two places:,So the question is, how do we provide the {package} and {package}-{version}.dist-info directories necessary to install our package? There are two options:

span.prompt1: before {
   content: ">>> "
}
a = 1
type(a)
int
def hello_world(name):
   print(f "Hello world! My name is {name}.")
type(hello_world)
function
def hello_world():
   print("Hello World!")

def hello_world_squamish():
   print("I chen tl'iḵ!")
import greetings
type(greetings)

Suggestion : 6

Namespace packages installed in eggs have __init__.py files, but namespace packages installed by pip don’t, in an effort to conform to pep420. According to pep420 means that the namespaces installed by pip should be excluded., Namespace packages installed in eggs have __init__.py files, but namespace packages installed by pip don’t, in an effort to conform to pep420. According to pep420 means that the namespaces installed by pip should be excluded. ,Pip appears to collapse all the name-space packages together into the one tree whilst easy_install keeps them separated in there respective egg packages. That may be an ok thing to todo considering we now have implicit name spaces from PEP 420,Solution 1: get setuptools to modify installed eggs striping the __init__.py from namespace modules for newer versions of python.

Pip installs zc.buildout with the following structure:

$ cd py / lib / python3 .6 / site - packages /
   $ ls
easy_install.py pip - 9.0 .1.dist - info __pycache__ setuptools - 38.5 .1.dist - info zc.buildout - 2.11 .0.dist - info
pip pkg_resources setuptools zc zc.buildout - 2.11 .0 - py2 .7 - nspkg.pth
$ cd zc
$ ls
buildout
$ cd buildout /
   $ ls
allowhosts.txt configparser.py download.py __init__.py runsetup.txt testrecipes.py
bootstrap_cl_settings.test configparser.test download.txt pep425tags.py setup.txt tests.py
bootstrap.txt debugging.txt easy_install.py __pycache__ testing_bugfix.txt update.txt
buildout.py dependencylinks.txt easy_install.txt repeatable.txt testing.py windows.txt
buildout.txt downloadcache.txt extends - cache.txt rmtree.py testing.txt

Whilst easy_install installs zc.buildout as:

$ cd py / lib / python3 .6 / site - packages /
   $ ls
easy - install.pth pip pkg_resources setuptools zc.buildout - 2.11 .0 - py3 .6.egg
easy_install.py pip - 9.0 .1.dist - info __pycache__ setuptools - 38.5 .1.dist - info
$ cd zc.buildout - 2.11 .0 - py3 .6.egg /
   $ ls
EGG - INFO zc zc.buildout - 2.11 .0 - py2 .7 - nspkg.pth

Now if I remove the __init__.py out of the zc.catalog egg and also make it an implicit name space things work again…

$ rm eggs / zc.catalog - 2.0 .1 - py3 .6.egg / zc / __init__.py
$ bin / py >>>
   import zc.catalog >>>
   print(zc.catalog.__path__)['/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg/zc/catalog']

Using just easy_install has problems…

$ py/bin/easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Downloading https://pypi.python.org/packages/e9/c3/5986db56819bd88e1a250cad2a97249211686b1b7b5d95f9ab64d403a2cb/setuptools-38.2.5.zip#md5=abfd02fba07b381c3a9682a32d765cc6
Best match: setuptools 38.2.5
etc...

$ py/bin/easy_install -U zc.buildout
Traceback (most recent call last):
File "py/bin/easy_install", line 11, in <module>
   load_entry_point('setuptools==38.2.5', 'console_scripts', 'easy_install')()
   File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 565, in load_entry_point
   return get_distribution(dist).load_entry_point(group, name)
   File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
   return ep.load()
   File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2291, in load
   return self.resolve()
   File "/tmp/bo/py/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2297, in resolve
   module = __import__(self.module_name, fromlist=['__name__'], level=0)
   File "/tmp/bo/py/lib/python3.6/site-packages/setuptools-38.2.5-py3.6.egg/setuptools/command/easy_install.py", line 47, in <module>
      File "/tmp/bo/py/lib/python3.6/site-packages/setuptools-38.2.5-py3.6.egg/setuptools/sandbox.py", line 15, in <module>
         ModuleNotFoundError: No module named 'pkg_resources.py31compat'

But if you use a combination of pip and easy_install everything works…

$ py / bin / pip install - U setuptools
etc...

   $ py / bin / easy_install - U zc.buildout
Searching
for zc.buildout
Reading https: //pypi.python.org/simple/zc.buildout/
   Downloading https: //pypi.python.org/packages/6f/e0/6d49af55c9ef2893429e668d88cd54b955f804869e703ac241c0ea5b49c7/zc.buildout-2.10.0-py2.py3-none-any.whl#md5=c6c6bfe595d83be2e29c1c7b17df0268
   Best match: zc.buildout 2.10 .0
etc...

   $ py / bin / buildout
Updating zc.catalog.

$ bin / py >>>
   import zc.catalog

for the record, with Ubuntu 20.04 that is using virtualenv 20.0.17-1ubuntu0.4 we get same problem while starting bin/zeoserver:

Traceback (most recent call last):
File "/srv/instance/bin/zeoserver", line 408, in <module>
   import plone.recipe.zeoserver.ctl
   File "/srv/cache/eggs/plone.recipe.zeoserver-1.3.1-py2.7.egg/plone/recipe/zeoserver/__init__.py", line 8, in <module>
      import zc.buildout
      ImportError: No module named buildout

@arterrey i used

virtualenv--version
15.2 .0

console:

$ python3.6 -m venv py
...
$ py/bin/pip install -U pip
...
$ py/bin/pip install -U setuptools zc.buildout
...
$ py/bin/buildout
...
$ bin/py
>>> import zc.catalog
Traceback (most recent call last):
File "<console>", line 1, in <module>
      ModuleNotFoundError: No module named 'zc.catalog'

After pdb debugging, it looks like the implicit zc namespace package is setup right from the beginning of the execution of the bin/py script before the sys.path is updated.

(pdb) sys.modules['zc']
<module 'zc' (namespace)>
   (pdb) sys.modules['zc'].__path__
   _NamespacePath(['/tmp/bug/py/lib/python3.6/site-packages/zc'])

After updating the sys.path you can reload the module and you get the pkg_resources version of the name space packages (I think before implicit name space packages pgk_resources emulated the namespaces etc.) by using importlib.reload(). After which our buildout installed name space packages work.

(Pdb) import importlib
(Pdb) importlib.reload(sys.modules['zc'])
<module 'zc' from '/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg/zc/__init__.py'>
   (Pdb) sys.modules['zc'].__path__
   ['/tmp/bug/eggs/zc.lockfile-1.2.1-py3.6.egg/zc', '/tmp/bug/eggs/zc.catalog-2.0.1-py3.6.egg/zc']
   (Pdb) import zc.catalog
   (Pdb)

But now we can’t iimport zc.buildout 😦 …

(Pdb) import zc.buildout **
   * ModuleNotFoundError: No module named 'zc.buildout'


Trending Technology


Most popular in python