os.link() vs. os.rename() vs. os.replace() for writing atomic write files. what is the best approach?

os.link isn't really suitable for this function unless you can guarantee that the target file does not exist (as os.link will error):

$ touch a b
$ link a b
link: cannot create link 'b' to 'a': File exists
$ python3 -c 'import os; os.link("a", "b")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
      FileExistsError: [Errno 17] File exists: 'a' -> 'b'

Here is an example of a common case when some data should be extracted from an existing file or otherwise created in such a file:

import time
filename = 'data'
# file with data
temp_filename = 'data.temp'
# temp used to create 'data'

def get_data():
   while True:
   try:
   os.remove(temp_filename)
except FileNotFoundError: #
if no data.temp
try: # check
if data already exists:
   with open(filename, 'rt', encoding = 'utf8') as f:
   return f.read() #
return data here
except FileNotFoundError:
   pass # create data
except PermissionError: #
if another process / thread is creating data.temp right now
time.sleep(0.1) # wait
for it
continue
else:
   pass # something went wrong and it 's better to create all again

# data creation:
   excl_access = 'xt'
# raises error
if file exists - used as a primitive lock
try:
with open(temp_filename, excl_access, encoding = 'utf8') as f:
   # process can be interrupted here
f.write('Hello ') # or here
f.write('world!') # or here
except FileExistsError: # another one is creating it now
time.sleep(0.1) # wait
for it
continue
except Exception: # something went wrong
continue
try:
os.replace(temp_filename, filename) # not sure this would be atomic in 100 %
   except FileNotFoundError:
   continue #
try again

Suggestion : 2

Does os.link() use more memmory? What are anycodings_python the benefits of each and are all of them anycodings_python atomic? ,I am struggling to figure out what would be anycodings_python the best action to do in line 3. Should I anycodings_python use os.replace(), os.rename() or should I anycodings_python create a hard link between tempfile and anycodings_python target file using os.link()? ,The only difference is os.replace uses anycodings_python is_replace=1 which has no effect on anycodings_python posix but sets MOVEFILE_REPLACE_EXISTING anycodings_python flag on windows:,os.rename / os.replace are both anycodings_python implemented using this function

Hi am trying to write an atomic write anycodings_python function like so...

with tempfile.NamedTemporaryFile(mode = "w", dir = target_directory) as f:
   #perform file writing operation
os.replace(f.name, target_file_name)

os.link isn't really suitable for this anycodings_python function unless you can guarantee that anycodings_python the target file does not exist (as anycodings_python os.link will error):

$ touch a b
$ link a b
link: cannot create link 'b' to 'a': File exists
$ python3 -c 'import os; os.link("a", "b")'
Traceback (most recent call last):
File "<string>", line 1, in <module>
      FileExistsError: [Errno 17] File exists: 'a' -> 'b'

Here is an example of a common case when anycodings_python some data should be extracted from an anycodings_python existing file or otherwise created in anycodings_python such a file:

import time
filename = 'data'
# file with data
temp_filename = 'data.temp'
# temp used to create 'data'

def get_data():
   while True:
   try:
   os.remove(temp_filename)
except FileNotFoundError: #
if no data.temp
try: # check
if data already exists:
   with open(filename, 'rt', encoding = 'utf8') as f:
   return f.read() #
return data here
except FileNotFoundError:
   pass # create data
except PermissionError: #
if another process / thread is creating data.temp right now
time.sleep(0.1) # wait
for it
continue
else:
   pass # something went wrong and it 's better to create all again

# data creation:
   excl_access = 'xt'
# raises error
if file exists - used as a primitive lock
try:
with open(temp_filename, excl_access, encoding = 'utf8') as f:
   # process can be interrupted here
f.write('Hello ') # or here
f.write('world!') # or here
except FileExistsError: # another one is creating it now
time.sleep(0.1) # wait
for it
continue
except Exception: # something went wrong
continue
try:
os.replace(temp_filename, filename) # not sure this would be atomic in 100 %
   except FileNotFoundError:
   continue #
try again

Suggestion : 3

I'm using this lib on Windows 10 with no issues so far, but I'm affraid os.rename, though not well documented for Windows, isn't atomic. As per the documentation:,On Windows: That's not really possible for the case where atomicwrites should fail when the target file already exists. For that you need to use the winapi directly.,The overwrite functionality isn't a big deal for me, though. The main reason I'm using this lib is because it's the only one that does the job right.,This library doesn't use os.rename for Windows.

def _replace_atomic(src, dst):
   try:
   os.replace(src, dst)
except:
   os.rename(src, dst)
_sync_directory(os.path.normpath(os.path.dirname(dst)))

Suggestion : 4

All functions accepting path or file names accept both bytes and string objects, and result in an object of the same type, if a path or file name is returned.,All functions in this module raise OSError (or subclasses thereof) in the case of invalid or inaccessible file names and paths, or other arguments that have the correct type, but are not accepted by the operating system.,Return an open file object connected to the file descriptor fd. This is an alias of the open() built-in function and accepts the same arguments. The only difference is that the first argument of fdopen() must always be an integer.,path may be a path-like object. If path is of type bytes (directly or indirectly through the PathLike interface), the filenames returned will also be of type bytes; in all other circumstances, they will be of type str.

for fd in range(fd_low, fd_high):
   try:
   os.close(fd)
except OSError:
   pass

if os.access("myfile", os.R_OK):
   with open("myfile") as fp:
   return fp.read()
return "some default data"

try:
fp = open("myfile")
except PermissionError:
   return "some default data"
else:
   with fp:
   return fp.read()

with os.scandir(path) as it:
   for entry in it:
   if not entry.name.startswith('.') and entry.is_file():
   print(entry.name)

>>>
import os
   >>>
   statinfo = os.stat('somefile.txt') >>>
   statinfo
os.stat_result(st_mode = 33188, st_ino = 7876932, st_dev = 234881026,
      st_nlink = 1, st_uid = 501, st_gid = 501, st_size = 264, st_atime = 1297230295,
      st_mtime = 1297230027, st_ctime = 1297230027) >>>
   statinfo.st_size
264

os.stat in os.supports_dir_fd

Suggestion : 5

This issue tracker has been migrated to GitHub, and is currently read-only. For more information, see the GitHub FAQs in the Python's Developer Guide., Administration User List Committer List

os.rename() is atomic on Linux, but on Windows it raises an error
if the destination does already exist.

Not atomic pseudo - code
for Windows:
   if exists(b):
   unlink(b)
rename(a, b)

Windows offers different functions depending on the version:
   -MoveFileTransacted(): atomic!version >= (Windows Vista, Windows Server 2008) -
   ReplaceFile(): version >= Windows 2000 -
   MoveFileEx() with MOVEFILE_REPLACE_EXISTING and MOVEFILE_WRITE_THROUGH flags: not atomic(eg.
      "If the file is to be moved to a different volume, the function simulates the move by using the CopyFile and DeleteFile functions."), version >= Windows 2000

I don 't think that it'
s possible to write an atomic rename(file)
function
for any OS, so it 's only a "best effort" atomic function. The documentation will give a list of OS on which the operation *is* atomic (eg. Linux).

Note: os.rename() uses MoveFileW() on Windows.

A first implementation can be:

   if os.name in ('nt', 'ce'):
   def atomic_rename(a, b):
   if os.path.exists(b):
   unlink(b)
rename(a, b)
else:
   atomic_rename = os.rename

This implementation is atomic on POSIX, and not atomic on Windows.Tell me
if I am wrong.

It can be improved later by adding the support of better Windows functions.

See issue #8604: proposal of a new "with atomic_write(filename) as fp: ..." context manager.

Begin by removing the dest file is maybe not the safer approach: -) Here is a new
try: begin by renaming the dest file to a new file.

   -- -- --
# use maybe a PRNG instead of a dummy counter or tempfile
def _create_old_filename(filename):
   old = filename + '.old'
index = 2
while os.path.exists(old):
   old = filename + '-%s.old' % index
index += 1
return old

if os.name in ('nt', 'ce'):
   def atomic_rename(src, dst):
   if os.path.exists(dst):
   old = _create_old_filename(dst)
rename(dst, old)
rename(src, dst)
unlink(old)
else:
   rename(src, dst)
else:
   atomic_rename = os.rename
   -- -- --

What can we do
   if "rename(src, dst)" fails ?

> This implementation is atomic on POSIX, ...

Wrong: -)

"On how rename is broken in Mac OS X"
http: //www.weirdnet.nl/apple/rename.html

   "Update January 8, 2010: ... the original bug (5398777) was resolved in Snow Leopard but the issue I reported was not fixed. ..."

We have to think about network file systems like NFS.Gnome(nautilus) had a bug on rename because NFS emitted a delete notification on a rename:
   http: //linux-nfs.org/pipermail/nfsv4/2009-March/010134.html
   https: //bugzilla.gnome.org/show_bug.cgi?id=575684

   It looks like rename is atomic, it 's just a bug about notification. But other virtual file systems may not implement atomic rename (eg. is rename atomic with sshfs?). Should Python detect the file system type to choose the algorithm? I would like to say no, because I consider that as a file system (or kernel) bug, not a Python bug.

   --

Should we also implement a atomic version of shutil.move() ? Support rename
if the source and the destination are on different file systems.Or is shutil.move() already atomic ?

   Note : this issue is only about renaming a * file * .Atomic rename of a directory is much more complex: -)

Suggestion : 6

This module provides a portable way of using operating system dependent functionality. If you just want to read or write a file see open(), if you want to manipulate paths, see the os.path module, and if you want to read all the lines in all the files on the command line see the fileinput module. For creating temporary files and directories see the tempfile module, and for high-level file and directory handling see the shutil module.,All functions in this module raise OSError in the case of invalid or inaccessible file names and paths, or other arguments that have the correct type, but are not accepted by the operating system.,Extensions peculiar to a particular operating system are also available through the os module, but using them is of course a threat to portability.,This is implemented using subprocess.Popen; see that class’s documentation for more powerful ways to manage and communicate with subprocesses.

for fd in range(fd_low, fd_high):
   try:
   os.close(fd)
except OSError:
   pass

if os.access("myfile", os.R_OK):
   with open("myfile") as fp:
   return fp.read()
return "some default data"

try:
fp = open("myfile")
except PermissionError:
   return "some default data"
else:
   with fp:
   return fp.read()

with os.scandir(path) as it:
   for entry in it:
   if not entry.name.startswith('.') and entry.is_file():
   print(entry.name)

>>>
import os
   >>>
   statinfo = os.stat('somefile.txt') >>>
   statinfo
os.stat_result(st_mode = 33188, st_ino = 7876932, st_dev = 234881026,
      st_nlink = 1, st_uid = 501, st_gid = 501, st_size = 264, st_atime = 1297230295,
      st_mtime = 1297230027, st_ctime = 1297230027) >>>
   statinfo.st_size
264

os.stat in os.supports_dir_fd

Suggestion : 7

Note that this method is intended for simple cases where it is convenient to read all lines in a single operation. It is not intended for reading in large files., Depending on the implementation this method may require to access the file system to determine if the file is considered hidden., If an IOException is thrown when accessing the directory after returned from this method, it is wrapped in an UncheckedIOException which will be thrown from the method that caused the access to take place., If an IOException is thrown when accessing the directory after this method has returned, it is wrapped in an UncheckedIOException which will be thrown from the method that caused the access to take place.

public final class Files
extends Object

newInputStream

public static InputStream newInputStream(Path path,
   OpenOption...options)
throws IOException

newOutputStream

public static OutputStream newOutputStream(Path path,
   OpenOption...options)
throws IOException

Usage Examples:

     Path path = ...

        // truncate and overwrite an existing file, or create the file if
        // it doesn't initially exist
        OutputStream out = Files.newOutputStream(path);

     // append to an existing file, fail if the file does not exist
     out = Files.newOutputStream(path, APPEND);

     // append to an existing file, create file if it doesn't initially exist
     out = Files.newOutputStream(path, CREATE, APPEND);

     // always create new file, failing if it already exists
     out = Files.newOutputStream(path, CREATE_NEW);

Usage Examples:

     Path path = ...

        // truncate and overwrite an existing file, or create the file if
        // it doesn't initially exist
        OutputStream out = Files.newOutputStream(path);

     // append to an existing file, fail if the file does not exist
     out = Files.newOutputStream(path, APPEND);

     // append to an existing file, create file if it doesn't initially exist
     out = Files.newOutputStream(path, CREATE, APPEND);

     // always create new file, failing if it already exists
     out = Files.newOutputStream(path, CREATE_NEW);

newByteChannel

public static SeekableByteChannel newByteChannel(Path path,
   Set < ? extends OpenOption > options,
   FileAttribute < ? > ...attrs)
throws IOException

Usage Examples:

     Path path = ...

     // open file for reading
     ReadableByteChannel rbc = Files.newByteChannel(path, EnumSet.of(READ)));

     // open file for writing to the end of an existing file, creating
     // the file if it doesn't already exist
     WritableByteChannel wbc = Files.newByteChannel(path, EnumSet.of(CREATE,APPEND));

     // create file with initial permissions, opening it for both reading and writing
     FileAttribute<Set<PosixFilePermission>> perms = ...
        SeekableByteChannel sbc = Files.newByteChannel(path, EnumSet.of(CREATE_NEW,READ,WRITE), perms);

Usage Examples:

     Path path = ...

     // open file for reading
     ReadableByteChannel rbc = Files.newByteChannel(path, EnumSet.of(READ)));

     // open file for writing to the end of an existing file, creating
     // the file if it doesn't already exist
     WritableByteChannel wbc = Files.newByteChannel(path, EnumSet.of(CREATE,APPEND));

     // create file with initial permissions, opening it for both reading and writing
     FileAttribute<Set<PosixFilePermission>> perms = ...
        SeekableByteChannel sbc = Files.newByteChannel(path, EnumSet.of(CREATE_NEW,READ,WRITE), perms);