python __init__.py vs sys.path.append/insert

The __init__.py files are required to make Python treat directories containing the file as packages. This prevents directories with a common name, such as string, unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.,Packages support one more special attribute, __path__. This is initialized to be a list containing the name of the directory holding the package’s __init__.py before the code in that file is executed. This variable can be modified; doing so affects future searches for modules and subpackages contained in the package.,When importing the package, Python searches through the directories on sys.path looking for the package subdirectory.,The directory containing the input script (or the current directory when no file is specified).

# Fibonacci numbers module

def fib(n): # write Fibonacci series up to n
a, b = 0, 1
while a < n:
   print(a, end = ' ')
a, b = b, a + b
print()

def fib2(n): #
return Fibonacci series up to n
result = []
a, b = 0, 1
while a < n:
   result.append(a)
a, b = b, a + b
return result

>>>
import fibo

>>> fibo.fib(1000)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
   >>>
   fibo.fib2(100)[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] >>>
   fibo.__name__ 'fibo'

>>> fib = fibo.fib >>>
   fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377

>>> from fibo
import fib, fib2
   >>>
   fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377

>>> from fibo
import *
>>>
fib(500)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377

Here’s a list of scenarios when using prepend or append import modes where pytest needs to change sys.path in order to import test modules or conftest.py files, and the issues users might encounter because of that.,pytest import mechanisms and sys.path/PYTHONPATH Import modes prepend and append import modes scenarios Test modules / conftest.py files inside packages Standalone test modules / conftest.py files Invoking pytest versus python -m pytest ,append: the directory containing each module is appended to the end of sys.path if not already there, and imported with __import__.,Same as prepend, requires test module names to be unique when the test directory tree is not arranged in packages, because the modules will put in sys.modules after importing.

testing / __init__.py
testing / test_pkg_under_test.py
pkg_under_test /

root /
   |
   -foo /
   |
   -__init__.py |
   -conftest.py |
   -bar /
   |
   -__init__.py |
   -tests /
   |
   -__init__.py |
   -test_foo.py

pytest root /

root /
   |
   -foo /
   |
   -conftest.py |
   -bar /
   |
   -tests /
   |
   -test_foo.py

When you create folders you should add __init__.py file as empty files in your folders. While in Python 3 there is a concept of implicit namespaces where you do not have to add those files to folder, Airflow expects that the files are added to all packages you added.,Best practices for module loading Use unique top package name Don’t use relative imports Add __init__.py in package folders ,Create the file __init__.py inside the package and add following code:,Often you want to use your own python code in your Airflow deployment, for example common code, libraries, you might want to generate DAGs using shared python code and have several DAG python files.

>>>
import sys
   >>>
   from pprint
import pprint
   >>>
   pprint(sys.path)['',
      '/home/arch/.pyenv/versions/3.7.4/lib/python37.zip',
      '/home/arch/.pyenv/versions/3.7.4/lib/python3.7',
      '/home/arch/.pyenv/versions/3.7.4/lib/python3.7/lib-dynload',
      '/home/arch/venvs/airflow/lib/python3.7/site-packages']

<DIRECTORY ON PYTHONPATH>
   | .airflowignore -- only needed in ``dags`` folder, see below
   | -- my_company
   | __init__.py
   | common_package
   | | __init__.py
   | | common_module.py
   | | subpackage
   | | __init__.py
   | | subpackaged_util_module.py
   |
   | my_custom_dags
   | __init__.py
   | my_dag1.py
   | my_dag2.py
   | base_dag.py

from my_company.common_package.common_module
import SomeClass
from my_company.common_package.subpackage.subpackaged_util_module
import AnotherClass
from my_company.my_custom_dags.base_dag
import BaseDag

my_company / common_package / .*
   my_company / my_custom_dags / base_dag\.py

my_company / common_package /
   my_company / my_custom_dags / base_dag.py

from.base_dag
import BaseDag # NEVER DO THAT!!!!

When a module is imported, Python runs all of the code in the module file. When a package is imported, Python runs all of the code in the package's __init__.py file, if such a file exists. All of the objects defined in the module or the package's __init__.py file are made available to the importer.,Importing a package is conceptually equivalent to importing the package's __init__.py file as a module. Indeed, this is what Python treats the package as:,The first time that a package or one of its modules is imported, Python will execute the __init__.py file in the root folder of the package if the file exists. All objects and functions defined in __init__.py are considered part of the package namespace.,importing a package is conceptually the same as importing that package's __init__.py file

test / # root folder
packA / # package packA
subA / # subpackage subA
__init__.py
sa1.py
sa2.py
__init__.py
a1.py
a2.py
packB / # package packB(implicit namespace package)
b1.py
b2.py
math.py
random.py
other.py
start.py

import pkgutil
search_path = '.'
# set to None to see all modules importable from sys.path
all_modules = [x[1]
   for x in pkgutil.iter_modules(path = search_path)
]
print(all_modules)

import sys
print(sys.path)

>>> import packB
>>> packB
<module 'packB' (namespace)>

def a1_func():
   print("running a1_func()")

# # this
import makes a1_func directly accessible from packA.a1_func
from packA.a1
import a1_func

def packA_func():
   print("running packA_func()")