__getstate__ method not being called when pickling a subclass of set

list does not implement __reduce__(), whereas set does:

>>> list().__reduce__()
...
TypeError: can't pickle list objects
>>> set().__reduce__()
(<type 'set'>, ([],), None)

Suggestion : 2

If __getstate__() returns a false value, the __setstate__() method will not be called upon unpickling.,Refer to the section Handling Stateful Objects for more information about how to use the methods __getstate__() and __setstate__().,instances of such classes whose __dict__ or the result of calling __getstate__() is picklable (see section Pickling Class Instances for details).,This method serves a similar purpose as __getnewargs_ex__(), but supports only positional arguments. It must return a tuple of arguments args which will be passed to the __new__() method upon unpickling.

class Foo:
   attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

def save(obj):
   return (obj.__class__, obj.__dict__)

def restore(cls, attributes):
   obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

class MyPickler(pickle.Pickler):
   dispatch_table = copyreg.dispatch_table.copy()
dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)

Suggestion : 3

with cls beeing the class for which the object has to be unhickled and state beeing the already unhickled dictionary which on hickling has been returned by call to obj.__getstate__(),And concerning pickling of any classobjects, function objects etc the about last three paragraphs under the following link indicate how pickle handles them. They are basically non picklable and just stored as named or fully qualified named references. https://docs.python.org/3.8/library/pickle.html#what-can-be-pickled-and-unpickled,The first two items state what function must be called with what arguments to set the initial state of the array. The third item in the tuple above is the object that is provided to the __setstate__ dunder method of the newly created NumPy array to repopulate the object.,But when dumping the resulting V4 File still contains objects as pickle strings and not as hdf5 groups representing object state dictionary. Looking at the code neither __getstate__ nor __setstate__ are called yet. Can this be cause my git upstream settings are not proper

import numpy as np
class with_state():
   def __init__(self):
   self.a = 12
sef.b = {
   'love': np.ones([12, 7]),
   'hatred': np.zeros([4, 9])
}
def __getstate__(self):
   return dict(
      a = self.a b = self.b
   )
def __setstate__(self, state):
   self.a = state['a']
self.b = state['b']

def __getitem__(self, index):
   if index == 0:
   return self.a
if index < 2:
   return b['hatred']
if index > 2:
   raise ValueError("index unknown")
return b['love']

 object_self = cls.__new__(cls)
 object_self.__setstate__(state)

    def test_load_complex_hickle():
    starttime = time.perf_counter()
    testfile = glob.glob(os.path.join(datapath,filename))[0]
    print(testfile)
    > dataset = hkl.load(testfile)


    <git-working-copies>/hickle/hickle/tests/test_stateful.py:20:
       _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
       <git-working-copyies>/hickle/hickle/hickle.py:616: in load
          py_container = _load(py_container, h_root_group['data'])
          h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
          ???
          h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
          ???
          /usr/local/lib/python3.6/dist-packages/h5py/_hl/group.py:264: in __getitem__
          oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
          h5py/_objects.pyx:54: in h5py._objects.with_phil.wrapper
          ???
          h5py/_objects.pyx:55: in h5py._objects.with_phil.wrapper
          ???
          _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

          > ???
          E KeyError: "Unable to open object (object 'data' doesn't exist)"

          h5py/h5o.pyx:190: KeyError

git branch
   *
   dev
master
git remote--verbose
origin git @github.com: hernot / hickle.git(fetch)
origin git @github.com: hernot / hickle.git(push)
upstream git @github.com: telegraphic / hickle.git(fetch)
upstream git @github.com: telegraphic / hickle.git(push)

1) test
if numpy array or alike numpy item - > serialize as dedicated numpy dataset / group
2) test
if dict item - > serialize as dict dateset
3) test
if iterable sequence - > serializes either as array or
if contains complex objects as dataset containing all items as subdatasets with counter appended to their name
4) otherwise save as pickled string

getobjstate = getattr(obj, '__getstate__', None)
if getobjstate is not None and getattr(obj, '__getstate__', None) is not None:

   # recall _dump on state with special attributes set for __class__ and __module__ and __loadstate__ which would read True unless returned state is false

Suggestion : 4

On the oposite side is __setstate__: it will receive what __getstate__ created and has to initialize the object.,What will be pickled can be defined in __getstate__ method. This method must return something that is picklable.,Constructor is not called! Note that in the previous example instance a2 was created in pickle.loads without ever calling A.__init__, so A.__setstate__ had to initialize everything that __init__ would have initialized if it were called.,The implementation here pikles a list with one value: [self.important_data]. That was just an example, __getstate__ could have returned anything that is picklable, as long as __setstate__ knows how to do the oppoisite. A good alternative is a dictionary of all values: {'important_data': self.important_data}.

On the oposite side is __setstate__: it will receive what __getstate__ created and has to initialize the object.

class A(object):
   def __init__(self, important_data):
   self.important_data = important_data

# Add data which cannot be pickled:
   self.func = lambda: 7

# Add data which should never be pickled, because it expires quickly:
   self.is_up_to_date = False

def __getstate__(self):
   return [self.important_data] # only this is needed

def __setstate__(self, state):
   self.important_data = state[0]

self.func = lambda: 7 # just some hard - coded unpicklable
function

self.is_up_to_date = False # even
if it was before pickling

Now, this can be done:

>>> a1 = A('very important') >>>
   >>>
   s = pickle.dumps(a1) # calls a1.__getstate__() >>>
   >>>
   a2 = pickle.loads(s) # calls a1.__setstate__(['very important']) >>>
   a2 <
   __main__.A object at 0x0000000002742470 >
   >>>
   a2.important_data 'very important' >>>
   a2.func()
7

Suggestion : 5

The pickle module implements binary protocols for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” 1 or “flattening”; however, to avoid confusion, the terms used here are “pickling” and “unpickling”.,Return the reconstituted object hierarchy of the pickled representation data of an object. data must be a bytes-like object.,Read the pickled representation of an object from the open file object file and return the reconstituted object hierarchy specified therein. This is equivalent to Unpickler(file).load().,If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.

class Foo:
   attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

def save(obj):
   return (obj.__class__, obj.__dict__)

def load(cls, attributes):
   obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

class MyPickler(pickle.Pickler):
   dispatch_table = copyreg.dispatch_table.copy()
dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)

Suggestion : 6

Last Updated : 01 Jun, 2021,GATE CS 2021 Syllabus

Output :

WRITING: pickle(elkcip)
WRITING: cPickle(elkciPc)
WRITING: last(tsal)

class pickle.Pickler(file, protocol = None, *, fix_imports = True)
This class takes a binary file for writing a pickle data stream.
1. dump(obj) – This function is used to write a pickled representation of obj to the open file object given in the constructor.
2. persistent_id(obj) – If persistent_id() returns None, obj is pickled as usual. This does nothing by default and exists so that any subclass can override it.
3. Dispatch_table – A pickler object’s dispatch table is a mapping whose keys are classes and whose values are reduction functions.
  By default, a pickler object will not have a dispatch_table attribute, and it will instead use the global dispatch table managed by the copyreg module.
  Example : The below code creates an instance of pickle.Pickler with a private dispatch table that handles the SomeClass class especially.

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

Output :

0: hi geeks!, this is line 1.
0: This is line 2.
0: hi geeks!, this is line 1.

Suggestion : 7

Everyone knows the most basic magic method, __init__. It's the way that we can define the initialization behavior of an object. However, when I call x = SomeClass(), __init__ is not the first thing to get called. Actually, it's a method called __new__, which actually creates the instance, then passes any arguments at creation on to the initializer. At the other end of the object's lifespan, there's __del__. Let's take a closer look at these 3 magic methods:,Context managers allow setup and cleanup actions to be taken for objects when their creation is wrapped with a with statement. The behavior of the context manager is determined by two magic methods:,__enter__ and __exit__ can be useful for specific classes that have well-defined and common behavior for setup and cleanup. You can also use these methods to create generic context managers that wrap other objects. Here's an example:,Python has a whole slew of magic methods designed to implement intuitive comparisons between objects using operators, not awkward method calls. They also provide a way to override the default Python behavior for comparisons of objects (by reference). Here's the list of those methods and what they do:

Putting it all together, here's an example of __init__ and __del__ in action:

from os.path
import join

class FileObject:
   ''
'Wrapper for file objects to make sure the file gets closed on deletion.'
''

def __init__(self, filepath = '~', filename = 'sample.txt'):
   # open a file filename in filepath in read and write mode
self.file = open(join(filepath, filename), 'r+')

def __del__(self):
   self.file.close()
del self.file

One of the biggest advantages of using Python's magic methods is that they provide a simple way to make objects behave like built-in types. That means you can avoid ugly, counter-intuitive, and nonstandard ways of performing basic operators. In some languages, it's common to do something like this:

if instance.equals(other_instance):
   # do something

You could certainly do this in Python, too, but this adds confusion and is unnecessarily verbose. Different libraries might use different names for the same operations, making the client do way more work than necessary. With the power of magic methods, however, we can define one method (__eq__, in this case), and say what we mean instead:

if instance == other_instance:
   #do something

You know how I said I would get to reflected arithmetic in a bit? Some of you might think it's some big, scary, foreign concept. It's actually quite simple. Here's an example:

some_object + other

That was "normal" addition. The reflected equivalent is the same thing, except with the operands switched around:

other + some_object

Suggestion : 8

The pickle module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.,To serialize an object hierarchy, you simply call the dumps() function. Similarly, to de-serialize a data stream, you call the loads() function. However, if you want more control over serialization and de-serialization, you can create a Pickler or an Unpickler object, respectively.,The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again. marshal doesn’t do this.,As our examples shows, you have to be careful with what you allow to be unpickled. Therefore if security is a concern, you may want to consider alternatives such as the marshalling API in xmlrpc.client or third-party solutions.

class Foo:
   attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

def save(obj):
   return (obj.__class__, obj.__dict__)

def load(cls, attributes):
   obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

class MyPickler(pickle.Pickler):
   dispatch_table = copyreg.dispatch_table.copy()
dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)