v2.0.3-rc (#32 )

# Changelog ## v2.0.3-rc ### Issues * \#28 \[bug\] Address slow directory responses in S3 mounts for deeply nested directories * \#29 \[bug\] S3 error responses are not being logged * \#30 \[bug\] Sia provider error responses are not logged * \#31 \[bug\] S3 provider should limit max key size to 1024 ### Changes from v2.0.2-rc * Always use direct for read-only providers * Fixed externally removed files not being processed during cleanup * Fixed http headers not being added for requests * Fixed incorrect `stat` values for remote mounts * Fixed invalid directory nullptr error on remote mounts * Fixed memory leak in event system * Refactored application shutdown * Refactored event system * Updated build system to Alpine 3.21.0 * Updated build system to MinGW-w64 12.0.0 * Updated copyright to 2018-2025 Reviewed-on: #32
2025-02-11 17:26:24 -06:00
10 changed files with 121 additions and 49 deletions
--- a/.cspell/words.txt
+++ b/.cspell/words.txt
@ -128,6 +128,7 @@ libcurl
 libdsm
 libevent
 libexample
+libexpat
 libfuse3
 libgmock
 libgtest
@ -140,10 +141,12 @@ libuuid
 libuuid_include_dirs
 libvlc
 linkflags
+localappdata
 lptr
 lpwstr
 markdownlint
 mbig
+minio
 msvc
 msvcr120
 msys2
@ -200,6 +203,7 @@ stduuid_project
 strequal
 ularge_integer
 uring
+url
 userenv
 utimens_impl
 utimensat
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -4,10 +4,6 @@

 ### Issues

-* ~~\#12 \[Unit Test\] Complete all providers unit tests~~
-* ~~\#20 Add support to evict open files~~
-* ~~\#21 \[Unit Test \] Complete WinFSP unit tests~~
-* ~~\#22 \[Unit Test\] Complete FUSE unit tests~~
 * \#28 \[bug\] Address slow directory responses in S3 mounts for deeply nested directories
 * \#29 \[bug\] S3 error responses are not being logged
 * \#30 \[bug\] Sia provider error responses are not logged
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@ -3,6 +3,10 @@ cmake_minimum_required(VERSION 3.27)
 cmake_policy(SET CMP0135 NEW)
 cmake_policy(SET CMP0144 NEW)

+if (NOT PROJECT_INTERFACE) 
+  message(FATAL_ERROR "Project must be compiled via 'make_win32.cmd'/'make_win32.sh' or 'make_unix.sh' build scripts. Invoking 'cmake' directly is not supported.")
+endif()
+
 project(${PROJECT_NAME}
  DESCRIPTION ${PROJECT_DESC}
  HOMEPAGE_URL ${PROJECT_URL}
@ -158,6 +162,7 @@ endif()
      -DPROJECT_FUSE=${PROJECT_FUSE}
      -DPROJECT_FUSE_INCLUDE_DIRS=${PROJECT_FUSE_INCLUDE_DIRS}
      -DPROJECT_GIT_REV=${PROJECT_GIT_REV}
+      -DPROJECT_INTERFACE=1
      -DPROJECT_IS_ALPINE=${PROJECT_IS_ALPINE}
      -DPROJECT_IS_ARM64=${PROJECT_IS_ARM64}
      -DPROJECT_IS_MINGW=${PROJECT_IS_MINGW}
--- a/README.md
+++ b/README.md
@ -1,23 +1,22 @@
 # Repertory

-Repertory allows you to mount AWS S3 and Sia via FUSE on Linux ~~/OS X~~ or via WinFSP
+Repertory allows you to mount S3 and Sia via FUSE on Linux or via WinFSP
 on Windows.

 ## Details and Features

 * Optimized for [Plex Media Server](https://www.plex.tv/)
-* Single application to mount AWS S3 and/or Sia
-* Remote mounting of Repertory instances on Linux ~~, OS X~~ and Windows
-  * Securely share your mounts over TCP/IP (`XChaCha20-Poly1305` stream cipher)
-* Cross-platform support (Linux 64-bit, Linux arm64/aarch64, ~~OS X,~~ Windows 64-bit)
+* Single application to mount S3 and/or Sia
+* Remote mounting of Repertory instances on Linux and Windows
+  * Securely share your mounts over TCP/IP via `XChaCha20-Poly1305` with other systems on your network or over the internet.
+* Cross-platform support (Linux 64-bit, Linux arm64/aarch64, Windows 64-bit)
+* Optionally encrypt file names and file data via `XChaCha20-Poly1305` in S3 mounts

 ## Minimum Requirements

 * [Sia renterd](https://github.com/SiaFoundation/renterd/releases) v0.4.0+ for Sia support
 * Only 64-bit operating systems are supported
  * By default, Linux requires `fusermount3`; otherwise, `repertory` must be manually compiled with `libfuse2` support
-  * ~~OS X requires the following dependency to be installed:~~
-    * ~~[FUSE for macOS v4.5.0](https://github.com/osxfuse/osxfuse/releases/download/macfuse-4.5.0/macfuse-4.5.0.dmg)~~
  * Windows requires the following dependencies to be installed:
    * [WinFSP 2023](https://github.com/winfsp/winfsp/releases/download/v2.0/winfsp-2.0.23075.msi)

@ -25,47 +24,103 @@ on Windows.

 * Linux `arm64/aarch64`
 * Linux `amd64`
-* ~~OS X Mojave and above~~
 * Windows 64-bit 10, 11

 ## Usage

+### Sia
+
+* Initial Configuration
+  * Sia steps:
+    * Set the appropriate bucket name and `renterd` API password in `repertory` configuration:
+      * To use `default` as the bucket name and configuration name:
+        * `repertory -set HostConfig.ApiPassword '<my password>'`
+      * To use a different bucket name with `default` as the configuration name:
+        * `repertory -set HostConfig.ApiPassword '<my password>'`
+        * `repertory -set SiaConfig.Bucket '<my bucket>'`
+      * For all other configurations:
+        * `repertory --name '<my config name>' -set HostConfig.ApiPassword '<my password>'`
+        * `repertory --name '<my config name>' -set SiaConfig.Bucket '<my bucket name>'`
+  * To verify/view all configuration options:
+    * `repertory -dc`
+    * `repertory --name '<my config name>' -dc`
+      * Example:
+        * `repertory --name default -dc`
+* Mounting on Linux:
+  * `repertory /mnt/location`
+  * `repertory --name '<my config name>' /mnt/location`
+    * Example:
+      * `repertory --name default /mnt/location`
+* Mounting on Windows:
+  * `repertory t:`
+  * `repertory --name '<my config name>' t:`
+    * Example:
+      * `repertory --name default t:`
+
+### S3
+
+* Initial Configuration
+  * S3 steps:
+    * Set the appropriate base URL:
+      * `repertory -s3 --name '<my config name>' -set S3Config.URL '<my url>'`
+        * Example:
+          * `repertory -s3 --name minio -set S3Config.URL 'http://localhost:9000'`
+    * Set the appropriate bucket name:
+      * `repertory -s3 --name '<my config name>' -set S3Config.Bucket '<my bucket name>'`
+    * Set the appropriate access key:
+      * `repertory -s3 --name '<my config name>' -set S3Config.AccessKey '<my access key>'`
+    * Set the appropriate secret key:
+      * `repertory -s3 --name '<my config name>' -set S3Config.SecretKey '<my secret key>'`
+    * For Sia and most local S3 gateway instances, enable path style URL's:
+      * `repertory -s3 --name '<my config name>' -set S3Config.UsePathStyle true`
+    * Optional steps:
+      * Set an appropriate region. Default is set to `any`:
+        * `repertory -s3 --name '<my config name>' -set S3Config.Region '<my region>'`
+      * Enable encrypted file names and file data. Set a strong, random encryption token and be sure to store it in a secure backup location:
+        * `repertory -s3 --name '<my config name>' -set S3Config.EncryptionToken '<my strong password>'`
+  * To verify/view all configuration options:
+    * `repertory -s3 --name '<my config name>' -dc`
+      * Example:
+        * `repertory -s3 --name minio -dc`
+* Mounting on Linux:
+  * `repertory -s3 --name '<my config name>' /mnt/location`
+    * Example:
+      * `repertory -s3 --name minio /mnt/location`
+* Mounting on Windows:
+  * `repertory -s3 --name '<my config name>' t:`
+    * Example:
+      * `repertory -s3 --name minio t:`
+
 ### Notable Options

-* `-dc`
-  * Display mount configuration.
-  * For Sia, `--name` is optional
-  * For S3, the `-s3` option is required along with `--name`
 * `--help`
-  * Display all mount utility options.
+  * Display all mount utility options
 * `--name, -na [name]`
  * The `--name` option can be set to any valid value allowed as a file name for your filesystem.
  * For Sia, the bucket name will be set to the same value if it is empty in the configuration file.
    * If the `--name` option is not specified, `default` will be used.
  * For S3, the `--name` option is required and does not affect the bucket name.
-* `-set SiaConfig.Bucket`
-  * Set Sia bucket name for the mount.
-  * Can be used in combination with `--name` to target a unique configuration.
-* `-set S3Config.Bucket`
-  * S3 bucket name for the mount.
-  * Must be used in combination with `--name` to target a unique configuration.
-  * Must be used in combination with `-s3`.
+* `-dc`
+  * Display mount configuration
+  * For Sia, `--name` is optional
+  * For S3, the `-s3` option is required along with `--name`

-### Sia
+### Data Directories

 * Linux
-  * `repertory /mnt/location`
-  * `repertory --name default /mnt/location`
+  * `~/.local/repertory2`
 * Windows
-  * `repertory.exe t:`
-  * `repertory.exe --name default t:`
+  * `%LOCALAPPDATA%\repertory2`
+    * Example:
+      * `C:\Users\Tom\AppData\Local\repertory2`
+    * IMPORTANT:
+      * It is highly recommended to exclude this folder from any anti-virus/anti-malware applications as severe performance issues may arise.
+      * Excluding the mounted drive letter is also highly recommended.

-### S3
+## Remote Mounting

-* Linux 
-  * `repertory --name storj -s3 /mnt/location`
-* Windows
-  * `repertory.exe --name storj -s3 t:`
+`Repertory` allows local mounts to be shared with other computers on your network.
+This option is referred to as remote mounting. Instructions TBD.

 ## Compiling

@ -94,17 +149,26 @@ on Windows.

 ## Credits

+* [binutils](https://www.gnu.org/software/binutils/)
 * [boost c++ libraries](https://www.boost.org/)
 * [cpp-httplib](https://github.com/yhirose/cpp-httplib)
 * [curl](https://curl.haxx.se/)
-* ~~[FUSE for macOS](https://osxfuse.github.io/)~~
+* [docker](https://www.docker.com/)
 * [Google Test](https://github.com/google/googletest)
+* [ICU](https://icu.unicode.org/)
 * [JSON for Modern C++](https://github.com/nlohmann/json)
+* [libexpat](https://github.com/libexpat/libexpat)
 * [libfuse](https://github.com/libfuse/libfuse)
 * [libsodium](https://doc.libsodium.org/)
+* [mingw-w64](https://www.mingw-w64.org/)
+* [MSYS2](https://www.msys2.org)
 * [OpenSSL](https://www.openssl.org/)
+* [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/)
+* [pugixml](https://pugixml.org/)
+* [RocksDB](https://rocksdb.org)
 * [ScPrime](https://scpri.me/)
 * [Sia Decentralized Cloud Storage](https://sia.tech/)
+* [spdlog](https://github.com/gabime/spdlog)
 * [SQLite](https://www.sqlite.org)
 * [stduuid](https://github.com/mariusbancila/stduuid)
 * [Storj](https://storj.io/)
--- a/cmake/versions.cmake
+++ b/cmake/versions.cmake
@ -1,15 +1,15 @@
 set(BINUTILS_VERSION 2.43)
-set(BOOST2_MAJOR_VERSION 1)
-set(BOOST2_MINOR_VERSION 76)
-set(BOOST2_PATCH_VERSION 0)
 set(BOOST_MAJOR_VERSION 1)
 set(BOOST_MINOR_VERSION 87)
 set(BOOST_PATCH_VERSION 0)
+set(BOOST2_MAJOR_VERSION 1)
+set(BOOST2_MINOR_VERSION 76)
+set(BOOST2_PATCH_VERSION 0)
 set(CPP_HTTPLIB_VERSION 0.18.1)
-set(CURL2_VERSION 8_11_0)
 set(CURL_VERSION 8.11.0)
-set(EXPAT2_VERSION 2_6_4)
+set(CURL2_VERSION 8_11_0)
 set(EXPAT_VERSION 2.6.4)
+set(EXPAT2_VERSION 2_6_4)
 set(GCC_VERSION 14.2.0)
 set(GTEST_VERSION 1.15.2)
 set(ICU_VERSION 76-1)
@ -22,7 +22,7 @@ set(PKG_CONFIG_VERSION 0.29.2)
 set(PUGIXML_VERSION 1.14)
 set(ROCKSDB_VERSION 9.7.4)
 set(SPDLOG_VERSION 1.15.0)
-set(SQLITE2_VERSION 3.46.1)
 set(SQLITE_VERSION 3460100)
+set(SQLITE2_VERSION 3.46.1)
 set(STDUUID_VERSION 1.2.3)
 set(ZLIB_VERSION 1.3.1)
--- a/repertory/librepertory/include/types/repertory.hpp
+++ b/repertory/librepertory/include/types/repertory.hpp
@ -43,7 +43,8 @@ constexpr const auto max_orphaned_file_retention_days{std::uint16_t(31U)};
 constexpr const auto max_ring_buffer_file_size{std::uint16_t(1024U)};
 constexpr const auto max_s3_object_name_length{1024U};
 constexpr const auto min_cache_size_bytes{
-    std::uint64_t(100UL * 1024UL * 1024UL)};
+    std::uint64_t(100UL * 1024UL * 1024UL),
+};
 constexpr const auto min_download_timeout_secs{std::uint8_t(5U)};
 constexpr const auto min_online_check_retry_secs{std::uint16_t(15U)};
 constexpr const auto min_orphaned_file_retention_days{std::uint16_t(1U)};
--- a/repertory/repertory/include/cli/get_version.hpp
+++ b/repertory/repertory/include/cli/get_version.hpp
@ -27,7 +27,7 @@
 namespace repertory::cli::actions {
 template <typename drive> inline void version(std::vector<const char *> args) {
  std::cout << "Repertory core version: " << project_get_version() << std::endl;
-  std::cout << "Repertory Git revision: " << project_get_git_rev() << std::endl;
+  std::cout << "Repertory git revision: " << project_get_git_rev() << std::endl;
  drive::display_version_information(args);
 }
 } // namespace repertory::cli::actions
--- a/repertory/repertory/include/cli/mount.hpp
+++ b/repertory/repertory/include/cli/mount.hpp
@ -41,7 +41,7 @@ using repertory_drive = repertory::winfsp_drive;
 using remote_client = repertory::remote_winfsp::remote_client;
 using remote_drive = repertory::remote_winfsp::remote_winfsp_drive;
 using remote_instance = repertory::remote_winfsp::i_remote_instance;
-#else
+#else // !defined(_WIN32)
 #include "drives/fuse/fuse_drive.hpp"
 #include "drives/fuse/remotefuse/remote_client.hpp"
 #include "drives/fuse/remotefuse/remote_fuse_drive.hpp"
@ -50,7 +50,7 @@ using repertory_drive = repertory::fuse_drive;
 using remote_client = repertory::remote_fuse::remote_client;
 using remote_drive = repertory::remote_fuse::remote_fuse_drive;
 using remote_instance = repertory::remote_fuse::i_remote_instance;
-#endif
+#endif // defined(_WIN32)

 namespace repertory::cli::actions {
 [[nodiscard]] inline auto
@ -77,7 +77,8 @@ mount(std::vector<const char *> args, std::string data_directory,
        config.set_remote_config(cfg);
      } else if (prov == provider_type::sia &&
                 config.get_sia_config().bucket.empty()) {
-        config.set_value_by_name("SiaConfig.Bucket", unique_id);
+        [[maybe_unused]] auto bucket =
+            config.set_value_by_name("SiaConfig.Bucket", unique_id);
      }

      std::cout << "Generated " << app_config::get_provider_display_name(prov)
@ -157,7 +158,8 @@ mount(std::vector<const char *> args, std::string data_directory,
      } else {
        if (prov == provider_type::sia &&
            config.get_sia_config().bucket.empty()) {
-          config.set_value_by_name("SiaConfig.Bucket", unique_id);
+          [[maybe_unused]] auto bucket =
+              config.set_value_by_name("SiaConfig.Bucket", unique_id);
        }

        try {
--- a/repertory/repertory_test/src/file_db_test.cpp
+++ b/repertory/repertory_test/src/file_db_test.cpp
@ -21,7 +21,6 @@
 */

 #include "fixtures/file_db_fixture.hpp"
-#include <filesystem>

 namespace {
 const auto get_stop_requested = []() -> bool { return false; };
@ -345,7 +344,7 @@ TYPED_TEST(file_db_test, can_enumerate_item_list) {
  this->file_db->clear();

  EXPECT_EQ(api_error::success,
-            this->file_db->add_directory("/", std::filesystem::current_path()));
+            this->file_db->add_directory("/", std::filesystem::current_path().string()));
  EXPECT_EQ(api_error::success, this->file_db->add_or_update_file({
                                    "/file",
                                    0U,
--- a/scripts/make_common.sh
+++ b/scripts/make_common.sh
@ -23,6 +23,7 @@ cmake "${PROJECT_SOURCE_DIR}" \
  -DPROJECT_COMPANY_NAME="${PROJECT_COMPANY_NAME}" \
  -DPROJECT_COPYRIGHT="${PROJECT_COPYRIGHT}" \
  -DPROJECT_DESC="${PROJECT_DESC}" \
+  -DPROJECT_INTERFACE=1 \
  -DPROJECT_URL="${PROJECT_URL}" \
  ${PROJECT_CMAKE_OPTS} || exit 1