Prevent parallel downloads of the same file with a lock.

Since the file is written to the filesystem, a filesystem lock is the way to go here. Add a dependency on the third-party filelock library to get cross-platform functionality.
2019-12-20 20:56:59 +01:00
parent b670c26684
commit a4c9338b83
2 changed files with 50 additions and 40 deletions
--- a/setup.py
+++ b/setup.py
@@ -59,6 +59,7 @@ setup(
                                    "tests.*", "tests"]),
    install_requires=['numpy',
                      'boto3',
+                      'filelock',
                      'requests',
                      'tqdm',
                      'regex != 2019.12.17',