BlockStorage/repertory
1
0
Fork 0
Code Issues 8 Pull Requests Releases 6 Wiki Activity

2.0.0-rc (#9)
Some checks failed
BlockStorage/repertory_osx/pipeline/head This commit looks good
Details
BlockStorage/repertory_windows/pipeline/head This commit looks good
Details
BlockStorage/repertory/pipeline/head There was a failure building this commit
Details
BlockStorage/repertory_linux_builds/pipeline/head This commit looks good
Details
BlockStorage/repertory_osx_builds/pipeline/head There was a failure building this commit
Details

Browse Source 
### Issues

* \#1 \[bug\] Unable to mount S3 due to 'item_not_found' exception
* \#2 Require bucket name for S3 mounts
* \#3 \[bug\] File size is not being updated in S3 mount
* \#4 Upgrade to libfuse-3.x.x
* \#5 Switch to renterd for Sia support
* \#6 Switch to cpp-httplib to further reduce dependencies
* \#7 Remove global_data and calculate used disk space per provider
* \#8 Switch to libcurl for S3 mount support

### Changes from v1.x.x

* Added read-only encrypt provider
  * Pass-through mount point that transparently encrypts source data using `XChaCha20-Poly1305`
* Added S3 encryption support via `XChaCha20-Poly1305`
* Added replay protection to remote mounts
* Added support base64 writes in remote FUSE
* Created static linked Linux binaries for `amd64` and `aarch64` using `musl-libc`
* Removed legacy Sia renter support
* Removed Skynet support
* Fixed multiple remote mount WinFSP API issues on \*NIX servers
* Implemented chunked read and write
  * Writes for non-cached files are performed in chunks of 8Mib
* Removed `repertory-ui` support
* Removed `FreeBSD` support
* Switched to `libsodium` over `CryptoPP`
* Switched to `XChaCha20-Poly1305` for remote mounts
* Updated `GoogleTest` to v1.14.0
* Updated `JSON for Modern C++` to v3.11.2
* Updated `OpenSSL` to v1.1.1w
* Updated `RocksDB` to v8.5.3
* Updated `WinFSP` to 2023
* Updated `boost` to v1.78.0
* Updated `cURL` to v8.3.0
* Updated `zlib` to v1.3
* Use `upload_manager` for all providers
  * Adds a delay to uploads to prevent excessive API calls
  * Supports re-upload after mount restart for incomplete uploads
  * NOTE: Uploads for all providers are full file (no resume support)
    * Multipart upload support is planned for S3

Reviewed-on: #9
This commit is contained in:
Scott E. Graves
2023-10-29 06:55:59 +00:00
parent 3ff46723b8
commit f43c41f88a
839 changed files with 98214 additions and 92959 deletions
Download Patch File Download Diff File Expand all files Collapse all files

198
include/providers/base_provider.hpp
View File

@ -1,30 +1,34 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_BASE_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_BASE_PROVIDER_HPP_
#include "common.hpp"
#include "db/meta_db.hpp"
#include "providers/i_provider.hpp"
namespace repertory {
class app_config;
class i_file_manager;
class base_provider : public i_provider {
public:
explicit base_provider(app_config &config);
@ -33,90 +37,168 @@ public:
private:
app_config &config_;
std::atomic<std::uint64_t> used_space_{0U};
protected:
api_item_added_callback api_item_added_;
meta_db meta_db_;
std::unique_ptr<meta_db> meta_db_;
mutable std::recursive_mutex notify_added_mutex_;
i_open_file_table *oft_ = nullptr;
i_file_manager *fm_ = nullptr;
stop_type stop_requested_ = false;
protected:
app_config &get_config() { return config_; }
void calculate_used_drive_space(bool add_missing);
app_config &get_config() const { return config_; }
[[nodiscard]] virtual auto
check_file_exists(const std::string &api_path) const -> api_error = 0;
virtual std::string format_api_path(std::string api_path) const { return api_path; }
void cleanup();
virtual std::string &restore_api_path(std::string &api_path) const { return api_path; }
[[nodiscard]] auto get_config() -> app_config & { return config_; }
void notify_directory_added(const std::string &api_path, const std::string &api_parent) const {
const_cast<base_provider *>(this)->notify_directory_added(api_path, api_parent);
[[nodiscard]] auto get_config() const -> app_config & { return config_; }
[[nodiscard]] virtual auto
handle_rename_file(const std::string &from_api_path,
const std::string &to_api_path,
const std::string &source_path) -> api_error = 0;
[[nodiscard]] auto notify_directory_added(const std::string &api_path,
const std::string &api_parent) const
-> api_error {
return const_cast<base_provider *>(this)->notify_directory_added(
api_path, api_parent);
}
virtual void notify_directory_added(const std::string &api_path, const std::string &api_parent);
[[nodiscard]] virtual auto
notify_directory_added(const std::string &api_path,
const std::string &api_parent) -> api_error;
api_error notify_file_added(const std::string &api_path, const std::string &api_parent,
const std::uint64_t &size) const {
return const_cast<base_provider *>(this)->notify_file_added(api_path, api_parent, size);
[[nodiscard]] auto notify_file_added(const std::string &api_path,
const std::string &api_parent,
std::uint64_t size) const -> api_error {
return const_cast<base_provider *>(this)->notify_file_added(
api_path, api_parent, size);
}
virtual api_error notify_file_added(const std::string &api_path, const std::string &api_parent,
const std::uint64_t &size) = 0;
[[nodiscard]] virtual auto notify_file_added(const std::string &api_path,
const std::string &api_parent,
std::uint64_t size)
-> api_error = 0;
void remove_item_meta(const std::string &api_path) {
return meta_db_.remove_item_meta(format_api_path(api_path));
[[nodiscard]] virtual auto
populate_directory_items(const std::string &api_path,
directory_item_list &list) const -> api_error = 0;
[[nodiscard]] virtual auto populate_file(const std::string &api_path,
api_file &file) const
-> api_error = 0;
auto processed_orphaned_file(const std::string &source_path,
const std::string &api_path = "") const -> bool;
void remove_deleted_files();
void remove_expired_orphaned_files();
void remove_unknown_source_files();
[[nodiscard]] auto remove_item_meta(const std::string &api_path)
-> api_error {
return meta_db_->remove_item_meta(api_path);
}
void update_filesystem_item(const bool &directory, const api_error &error,
const std::string &api_path, filesystem_item &fsi) const;
void update_filesystem_item(bool directory, const api_error &error,
const std::string &api_path,
filesystem_item &fsi) const;
public:
api_error create_directory_clone_source_meta(const std::string &source_api_path,
const std::string &api_path) override;
[[nodiscard]] auto
create_directory_clone_source_meta(const std::string &source_api_path,
const std::string &api_path)
-> api_error override;
api_error create_file(const std::string &api_path, api_meta_map &meta) override;
[[nodiscard]] auto create_file(const std::string &api_path,
api_meta_map &meta) -> api_error override;
api_error get_api_path_from_source(const std::string &source_path,
std::string &api_path) const override;
[[nodiscard]] auto get_api_path_from_source(const std::string &source_path,
std::string &api_path) const
-> api_error override;
api_error get_filesystem_item(const std::string &api_path, const bool &directory,
filesystem_item &fsi) const override;
[[nodiscard]] auto get_directory_items(const std::string &api_path,
directory_item_list &list) const
-> api_error override;
api_error get_filesystem_item_and_file(const std::string &api_path, api_file &file,
filesystem_item &fsi) const override;
[[nodiscard]] auto get_file(const std::string &api_path, api_file &file) const
-> api_error override;
api_error get_filesystem_item_from_source_path(const std::string &source_path,
filesystem_item &fsi) const override;
[[nodiscard]] auto get_file_size(const std::string &api_path,
std::uint64_t &file_size) const
-> api_error override;
api_error get_item_meta(const std::string &api_path, api_meta_map &meta) const override;
[[nodiscard]] auto get_filesystem_item(const std::string &api_path,
bool directory,
filesystem_item &fsi) const
-> api_error override;
api_error get_item_meta(const std::string &api_path, const std::string &key,
std::string &value) const override;
[[nodiscard]] auto get_filesystem_item_and_file(const std::string &api_path,
api_file &file,
filesystem_item &fsi) const
-> api_error override;
std::vector<std::string> get_pinned_files() const override { return meta_db_.get_pinned_files(); }
[[nodiscard]] auto
get_filesystem_item_from_source_path(const std::string &source_path,
filesystem_item &fsi) const
-> api_error override;
std::uint64_t get_used_drive_space() const override;
[[nodiscard]] auto get_item_meta(const std::string &api_path,
api_meta_map &meta) const
-> api_error override;
bool is_file_writeable(const std::string &) const override { return true; }
[[nodiscard]] auto get_item_meta(const std::string &api_path,
const std::string &key,
std::string &value) const
-> api_error override;
api_error remove_item_meta(const std::string &api_path, const std::string &key) override {
return meta_db_.remove_item_meta(format_api_path(api_path), key);
[[nodiscard]] auto get_pinned_files() const
-> std::vector<std::string> override {
return meta_db_->get_pinned_files();
}
api_error set_item_meta(const std::string &api_path, const std::string &key,
const std::string &value) override {
return meta_db_.set_item_meta(format_api_path(api_path), key, value);
[[nodiscard]] auto get_used_drive_space() const -> std::uint64_t override;
[[nodiscard]] auto is_file_writeable(const std::string &) const
-> bool override {
return true;
}
api_error set_item_meta(const std::string &api_path, const api_meta_map &meta) override {
return meta_db_.set_item_meta(format_api_path(api_path), meta);
[[nodiscard]] auto remove_item_meta(const std::string &api_path,
const std::string &key)
-> api_error override {
return meta_db_->remove_item_meta(api_path, key);
}
api_error set_source_path(const std::string &api_path, const std::string &source_path) override {
return meta_db_.set_source_path(format_api_path(api_path), source_path);
[[nodiscard]] auto rename_file(const std::string &from_api_path,
const std::string &to_api_path)
-> api_error override;
[[nodiscard]] auto set_item_meta(const std::string &api_path,
const std::string &key,
const std::string &value)
-> api_error override {
return meta_db_->set_item_meta(api_path, key, value);
}
bool start(api_item_added_callback api_item_added, i_open_file_table *oft) override;
[[nodiscard]] auto set_item_meta(const std::string &api_path,
const api_meta_map &meta)
-> api_error override {
return meta_db_->set_item_meta(api_path, meta);
}
[[nodiscard]] auto start(api_item_added_callback api_item_added,
i_file_manager *fm) -> bool override;
void stop() override;
};
} // namespace repertory

218
include/providers/encrypt/encrypt_provider.hpp Normal file
View File

@ -0,0 +1,218 @@
/*
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_ENCRYPT_ENCRYPT_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_ENCRYPT_ENCRYPT_PROVIDER_HPP_
#include "app_config.hpp"
#include "providers/i_provider.hpp"
#include "utils/encrypting_reader.hpp"
namespace repertory {
class encrypt_provider final : public i_provider {
public:
explicit encrypt_provider(app_config &config);
~encrypt_provider() override = default;
private:
struct reader_info final {
std::chrono::system_clock::time_point last_access_time =
std::chrono::system_clock::now();
std::unique_ptr<utils::encryption::encrypting_reader> reader{};
std::mutex reader_mtx;
};
private:
app_config &config_;
std::unique_ptr<rocksdb::DB> db_;
rocksdb::ColumnFamilyHandle *dir_family_{};
rocksdb::ColumnFamilyHandle *file_family_{};
rocksdb::ColumnFamilyHandle *source_family_{};
const std::string DB_NAME = "meta_db";
private:
i_file_manager *fm_ = nullptr;
std::unordered_map<std::string, std::shared_ptr<reader_info>>
reader_lookup_{};
std::recursive_mutex reader_lookup_mtx_{};
private:
static auto create_api_file(const std::string api_path, bool directory,
const std::string &source_path) -> api_file;
static void create_item_meta(api_meta_map &meta, bool directory,
const api_file &file);
[[nodiscard]] auto
process_directory_entry(const std::filesystem::directory_entry &dir_entry,
const encrypt_config &cfg,
std::string &api_path) const -> bool;
void remove_deleted_files();
public:
[[nodiscard]] auto create_directory(const std::string &api_path,
api_meta_map &meta) -> api_error override;
[[nodiscard]] auto
create_directory_clone_source_meta(const std::string & /*source_api_path*/,
const std::string & /*api_path*/)
-> api_error override {
return api_error::not_implemented;
}
[[nodiscard]] auto create_file(const std::string & /*api_path*/,
api_meta_map & /*meta*/)
-> api_error override {
return api_error::not_implemented;
}
[[nodiscard]] auto
get_api_path_from_source(const std::string & /*source_path*/,
std::string & /*api_path*/) const
-> api_error override;
[[nodiscard]] auto get_directory_item_count(const std::string &api_path) const
-> std::uint64_t override;
[[nodiscard]] auto get_directory_items(const std::string &api_path,
directory_item_list &list) const
-> api_error override;
[[nodiscard]] auto get_file(const std::string &api_path, api_file &file) const
-> api_error override;
[[nodiscard]] auto get_file_list(api_file_list &list) const
-> api_error override;
[[nodiscard]] auto get_file_size(const std::string &api_path,
std::uint64_t &file_size) const
-> api_error override;
[[nodiscard]] auto get_filesystem_item(const std::string &api_path,
bool directory,
filesystem_item &fsi) const
-> api_error override;
[[nodiscard]] auto get_filesystem_item_and_file(const std::string &api_path,
api_file &file,
filesystem_item &fsi) const
-> api_error override;
[[nodiscard]] auto
get_filesystem_item_from_source_path(const std::string &source_path,
filesystem_item &fsi) const
-> api_error override;
[[nodiscard]] auto get_pinned_files() const
-> std::vector<std::string> override;
[[nodiscard]] auto get_item_meta(const std::string &api_path,
api_meta_map &meta) const
-> api_error override;
[[nodiscard]] auto get_item_meta(const std::string &api_path,
const std::string &key,
std::string &value) const
-> api_error override;
[[nodiscard]] auto get_total_drive_space() const -> std::uint64_t override;
[[nodiscard]] auto get_total_item_count() const -> std::uint64_t override;
[[nodiscard]] auto get_provider_type() const -> provider_type override {
return provider_type::encrypt;
}
[[nodiscard]] auto get_used_drive_space() const -> std::uint64_t override;
[[nodiscard]] auto is_direct_only() const -> bool override { return true; }
[[nodiscard]] auto is_directory(const std::string &api_path,
bool &exists) const -> api_error override;
[[nodiscard]] auto is_file(const std::string &api_path, bool &exists) const
-> api_error override;
[[nodiscard]] auto is_file_writeable(const std::string &api_path) const
-> bool override;
[[nodiscard]] auto is_online() const -> bool override;
[[nodiscard]] auto is_rename_supported() const -> bool override;
[[nodiscard]] auto read_file_bytes(const std::string &api_path,
std::size_t size, std::uint64_t offset,
data_buffer &data,
stop_type &stop_requested)
-> api_error override;
[[nodiscard]] auto remove_directory(const std::string & /*api_path*/)
-> api_error override {
return api_error::not_implemented;
}
[[nodiscard]] auto remove_file(const std::string & /*api_path*/)
-> api_error override {
return api_error::not_implemented;
}
[[nodiscard]] auto remove_item_meta(const std::string & /*api_path*/,
const std::string & /*key*/)
-> api_error override {
return api_error::success;
}
[[nodiscard]] auto rename_file(const std::string & /*from_api_path*/,
const std::string & /*to_api_path*/)
-> api_error override {
return api_error::not_implemented;
}
[[nodiscard]] auto set_item_meta(const std::string & /*api_path*/,
const std::string & /*key*/,
const std::string & /*value*/)
-> api_error override {
return api_error::success;
}
[[nodiscard]] auto set_item_meta(const std::string & /*api_path*/,
const api_meta_map & /*meta*/)
-> api_error override {
return api_error::success;
}
[[nodiscard]] auto start(api_item_added_callback api_item_added,
i_file_manager *fm) -> bool override;
void stop() override;
[[nodiscard]] auto upload_file(const std::string & /*api_path*/,
const std::string & /*source_path*/,
const std::string & /*encryption_token*/,
stop_type & /*stop_requested*/)
-> api_error override {
return api_error::not_implemented;
}
};
} // namespace repertory
#endif // INCLUDE_PROVIDERS_ENCRYPT_ENCRYPT_PROVIDER_HPP_

162
include/providers/i_provider.hpp
View File

@ -1,114 +1,156 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_I_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_I_PROVIDER_HPP_
#include "common.hpp"
#include "types/repertory.hpp"
namespace repertory {
class i_open_file_table;
class i_file_manager;
class i_provider {
INTERFACE_SETUP(i_provider);
public:
virtual api_error create_directory(const std::string &api_path, const api_meta_map &meta) = 0;
[[nodiscard]] virtual auto create_directory(const std::string &api_path,
api_meta_map &meta)
-> api_error = 0;
virtual api_error create_directory_clone_source_meta(const std::string &source_api_path,
const std::string &api_path) = 0;
[[nodiscard]] virtual auto
create_directory_clone_source_meta(const std::string &source_api_path,
const std::string &api_path)
-> api_error = 0;
virtual api_error create_file(const std::string &api_path, api_meta_map &meta) = 0;
[[nodiscard]] virtual auto create_file(const std::string &api_path,
api_meta_map &meta) -> api_error = 0;
virtual api_error get_api_path_from_source(const std::string &source_path,
std::string &api_path) const = 0;
[[nodiscard]] virtual auto
get_api_path_from_source(const std::string &source_path,
std::string &api_path) const -> api_error = 0;
virtual api_error get_file_list(api_file_list &list) const = 0;
[[nodiscard]] virtual auto
get_directory_item_count(const std::string &api_path) const
-> std::uint64_t = 0;
virtual std::uint64_t get_directory_item_count(const std::string &api_path) const = 0;
[[nodiscard]] virtual auto
get_directory_items(const std::string &api_path,
directory_item_list &list) const -> api_error = 0;
virtual api_error get_directory_items(const std::string &api_path,
directory_item_list &list) const = 0;
[[nodiscard]] virtual auto get_file(const std::string &api_path,
api_file &file) const -> api_error = 0;
virtual api_error get_file(const std::string &api_path, api_file &file) const = 0;
[[nodiscard]] virtual auto get_file_list(api_file_list &list) const
-> api_error = 0;
virtual api_error get_file_size(const std::string &api_path, std::uint64_t &file_size) const = 0;
[[nodiscard]] virtual auto get_file_size(const std::string &api_path,
std::uint64_t &file_size) const
-> api_error = 0;
virtual api_error get_filesystem_item(const std::string &api_path, const bool &directory,
filesystem_item &fsi) const = 0;
[[nodiscard]] virtual auto get_filesystem_item(const std::string &api_path,
bool directory,
filesystem_item &fsi) const
-> api_error = 0;
virtual api_error get_filesystem_item_and_file(const std::string &api_path, api_file &file,
filesystem_item &fsi) const = 0;
[[nodiscard]] virtual auto
get_filesystem_item_and_file(const std::string &api_path, api_file &file,
filesystem_item &fsi) const -> api_error = 0;
virtual api_error get_filesystem_item_from_source_path(const std::string &source_path,
filesystem_item &fsi) const = 0;
[[nodiscard]] virtual auto
get_filesystem_item_from_source_path(const std::string &source_path,
filesystem_item &fsi) const
-> api_error = 0;
virtual std::vector<std::string> get_pinned_files() const = 0;
[[nodiscard]] virtual auto get_item_meta(const std::string &api_path,
api_meta_map &meta) const
-> api_error = 0;
virtual api_error get_item_meta(const std::string &api_path, api_meta_map &meta) const = 0;
[[nodiscard]] virtual auto get_item_meta(const std::string &api_path,
const std::string &key,
std::string &value) const
-> api_error = 0;
virtual api_error get_item_meta(const std::string &api_path, const std::string &key,
std::string &value) const = 0;
[[nodiscard]] virtual auto get_pinned_files() const
-> std::vector<std::string> = 0;
virtual std::uint64_t get_total_drive_space() const = 0;
[[nodiscard]] virtual auto get_provider_type() const -> provider_type = 0;
virtual std::uint64_t get_total_item_count() const = 0;
[[nodiscard]] virtual auto get_total_drive_space() const -> std::uint64_t = 0;
virtual provider_type get_provider_type() const = 0;
[[nodiscard]] virtual auto get_total_item_count() const -> std::uint64_t = 0;
virtual std::uint64_t get_used_drive_space() const = 0;
[[nodiscard]] virtual auto get_used_drive_space() const -> std::uint64_t = 0;
virtual bool is_directory(const std::string &api_path) const = 0;
[[nodiscard]] virtual auto is_direct_only() const -> bool = 0;
virtual bool is_file(const std::string &api_path) const = 0;
[[nodiscard]] virtual auto is_directory(const std::string &api_path,
bool &exists) const -> api_error = 0;
virtual bool is_file_writeable(const std::string &api_path) const = 0;
[[nodiscard]] virtual auto is_file(const std::string &api_path,
bool &exists) const -> api_error = 0;
virtual bool is_online() const = 0;
[[nodiscard]] virtual auto
is_file_writeable(const std::string &api_path) const -> bool = 0;
virtual bool is_rename_supported() const = 0;
[[nodiscard]] virtual auto is_online() const -> bool = 0;
virtual api_error read_file_bytes(const std::string &api_path, const std::size_t &size,
const std::uint64_t &offset, std::vector<char> &data,
const bool &stop_requested) = 0;
[[nodiscard]] virtual auto is_rename_supported() const -> bool = 0;
virtual api_error remove_directory(const std::string &api_path) = 0;
[[nodiscard]] virtual auto
read_file_bytes(const std::string &api_path, std::size_t size,
std::uint64_t offset, data_buffer &data,
stop_type &stop_requested) -> api_error = 0;
virtual api_error remove_file(const std::string &api_path) = 0;
[[nodiscard]] virtual auto remove_directory(const std::string &api_path)
-> api_error = 0;
virtual api_error remove_item_meta(const std::string &api_path, const std::string &key) = 0;
[[nodiscard]] virtual auto remove_file(const std::string &api_path)
-> api_error = 0;
virtual api_error rename_file(const std::string &fromApiPath, const std::string &toApiPath) = 0;
[[nodiscard]] virtual auto remove_item_meta(const std::string &api_path,
const std::string &key)
-> api_error = 0;
virtual api_error set_item_meta(const std::string &api_path, const std::string &key,
const std::string &value) = 0;
[[nodiscard]] virtual auto rename_file(const std::string &from_api_path,
const std::string &to_api_path)
-> api_error = 0;
virtual api_error set_item_meta(const std::string &api_path, const api_meta_map &meta) = 0;
[[nodiscard]] virtual auto set_item_meta(const std::string &api_path,
const std::string &key,
const std::string &value)
-> api_error = 0;
virtual api_error set_source_path(const std::string &api_path,
const std::string &source_path) = 0;
[[nodiscard]] virtual auto set_item_meta(const std::string &api_path,
const api_meta_map &meta)
-> api_error = 0;
virtual bool start(api_item_added_callback api_item_added, i_open_file_table *oft) = 0;
[[nodiscard]] virtual auto start(api_item_added_callback api_item_added,
i_file_manager *fm) -> bool = 0;
virtual void stop() = 0;
virtual api_error upload_file(const std::string &api_path, const std::string &source_path,
const std::string &encryption_token) = 0;
[[nodiscard]] virtual auto
upload_file(const std::string &api_path, const std::string &source_path,
const std::string &encryption_token, stop_type &stop_requested)
-> api_error = 0;
};
} // namespace repertory

115
include/providers/passthrough/passthroughprovider.hpp
View File

@ -1,115 +0,0 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_PASSTHROUGH_PASSTHROUGHPROVIDER_HPP_
#define INCLUDE_PROVIDERS_PASSTHROUGH_PASSTHROUGHPROVIDER_HPP_
#if defined(REPERTORY_TESTING_NEW)
#include "common.hpp"
#include "providers/basebase_provider.hpp"
namespace repertory {
class app_config;
class CPassthroughProvider final : public base_provider {
public:
explicit CPassthroughProvider(app_config &config)
: base_provider(config), passthroughLocation_("" /*config.GetPassthroughConfig().Location*/) {
}
~CPassthroughProvider() override = default;
private:
const std::string passthroughLocation_;
private:
std::string ConstructFullPath(const std::string &api_path) const;
protected:
api_error notify_file_added(const std::string &api_path, const std::string &api_parent,
const std::uint64_t &size) override;
public:
api_error CreateDirectory(const std::string &api_path, const api_meta_map &metaMap) override;
api_error get_file_list(ApiFileList &fileList) const override;
std::uint64_t get_directory_item_count(const std::string &api_path) const override;
api_error get_directory_items(const std::string &api_path,
directory_item_list &list) const override;
api_error GetFile(const std::string &api_path, ApiFile &file) const override;
api_error GetFileSize(const std::string &api_path, std::uint64_t &fileSize) const override;
api_error get_filesystem_item(const std::string &api_path, const bool &directory,
FileSystemItem &fileSystemItem) const override;
api_error get_filesystem_item_from_source_path(const std::string &sourceFilePath,
FileSystemItem &fileSystemItem) const override;
api_error get_item_meta(const std::string &api_path, api_meta_map &meta) const override;
api_error get_item_meta(const std::string &api_path, const std::string &key,
std::string &value) const override;
provider_type get_provider_type() const override { return provider_type::passthrough; }
std::uint64_t get_total_drive_space() const override;
std::uint64_t get_total_item_count() const override;
std::uint64_t get_used_drive_space() const override;
bool IsDirectory(const std::string &api_path) const override;
bool IsFile(const std::string &api_path) const override;
bool IsOnline() const override { return true; }
bool is_rename_supported() const override { return true; }
api_error read_file_bytes(const std::string &apiFilepath, const std::size_t &size,
const std::uint64_t &offset, std::vector<char> &data,
const bool &stop_requested) override;
api_error RemoveDirectory(const std::string &api_path) override;
api_error RemoveFile(const std::string &api_path) override;
api_error RenameFile(const std::string &fromApiPath, const std::string &toApiPath) override;
api_error remove_item_meta(const std::string &api_path, const std::string &key) override;
api_error set_item_meta(const std::string &api_path, const std::string &key,
const std::string &value) override;
api_error set_item_meta(const std::string &api_path, const api_meta_map &meta) override;
api_error set_source_path(const std::string &api_path, const std::string &sourcePath) override;
bool Start(api_item_added_callback apiItemAdded, i_open_file_table *openFileTable) override;
void Stop() override;
api_error upload_file(const std::string &api_path, const std::string &sourcePath,
const std::string &encryptionToken) override;
};
} // namespace repertory
#endif // REPERTORY_TESTING_NEW
#endif // INCLUDE_PROVIDERS_PASSTHROUGH_PASSTHROUGHPROVIDER_HPP_

32
include/providers/provider.hpp
View File

@ -1,31 +1,35 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_PROVIDER_HPP_
#include "common.hpp"
#include "types/repertory.hpp"
namespace repertory {
class app_config;
class i_provider;
std::unique_ptr<i_provider> create_provider(const provider_type &pt, app_config &config);
[[nodiscard]] auto create_provider(const provider_type &pt, app_config &config)
-> std::unique_ptr<i_provider>;
} // namespace repertory
#endif // INCLUDE_PROVIDERS_PROVIDER_HPP_

140
include/providers/s3/s3_provider.hpp
View File

@ -1,112 +1,144 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_S3_S3_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_S3_S3_PROVIDER_HPP_
#if defined(REPERTORY_ENABLE_S3)
#include "common.hpp"
#include "db/directory_db.hpp"
#include "providers/base_provider.hpp"
#include "upload/upload_manager.hpp"
#include "types/repertory.hpp"
namespace repertory {
class app_config;
class i_file_manager;
class i_s3_comm;
class s3_provider final : public base_provider {
public:
s3_provider(app_config &config, i_s3_comm &s3_comm);
~s3_provider() override = default;
private:
i_s3_comm &s3_comm_;
directory_db directory_db_;
upload_manager upload_manager_;
std::uint64_t total_item_count_ = 0u;
private:
void build_directories();
std::unique_ptr<directory_db> directory_db_;
std::unique_ptr<std::thread> background_thread_;
void update_item_meta(directory_item &di, const bool &format_path) const;
private:
void create_directories();
void upload_completed(const std::string &, const std::string &, const json &) {}
void create_parent_directories(const api_file &file, bool directory);
api_error upload_handler(const upload_manager::upload &upload, json &, json &);
void update_item_meta(directory_item &di) const;
protected:
std::string format_api_path(std::string api_path) const override;
[[nodiscard]] auto check_file_exists(const std::string &api_path) const
-> api_error override;
void notify_directory_added(const std::string &api_path, const std::string &api_parent) override;
[[nodiscard]] auto handle_rename_file(const std::string &from_api_path,
const std::string &to_api_path,
const std::string &source_path)
-> api_error override;
api_error notify_file_added(const std::string &api_path, const std::string &api_parent,
const std::uint64_t &size) override;
[[nodiscard]] auto notify_directory_added(const std::string &api_path,
const std::string &api_parent)
-> api_error override;
std::string &restore_api_path(std::string &api_path) const override;
[[nodiscard]] auto notify_file_added(const std::string &api_path,
const std::string &api_parent,
std::uint64_t size)
-> api_error override;
[[nodiscard]] auto populate_directory_items(const std::string &api_path,
directory_item_list &list) const
-> api_error override;
[[nodiscard]] auto populate_file(const std::string &api_path,
api_file &file) const -> api_error override;
public:
api_error create_directory(const std::string &api_path, const api_meta_map &meta) override;
#ifdef REPERTORY_TESTING
void set_callback(api_item_added_callback cb) { api_item_added_ = cb; }
#endif // REPERTORY_TESTING
api_error create_file(const std::string &api_path, api_meta_map &meta) override;
[[nodiscard]] auto create_directory(const std::string &api_path,
api_meta_map &meta) -> api_error override;
api_error get_file_list(api_file_list &list) const override;
[[nodiscard]] auto create_file(const std::string &api_path,
api_meta_map &meta) -> api_error override;
std::uint64_t get_directory_item_count(const std::string &api_path) const override;
[[nodiscard]] auto get_directory_item_count(const std::string &api_path) const
-> std::uint64_t override;
api_error get_directory_items(const std::string &api_path,
directory_item_list &list) const override;
[[nodiscard]] auto get_file_list(api_file_list &list) const
-> api_error override;
api_error get_file(const std::string &api_path, api_file &file) const override;
[[nodiscard]] auto get_provider_type() const -> provider_type override {
return provider_type::s3;
}
api_error get_file_size(const std::string &api_path, std::uint64_t &file_size) const override;
provider_type get_provider_type() const override { return provider_type::s3; }
std::uint64_t get_total_drive_space() const override {
[[nodiscard]] auto get_total_drive_space() const -> std::uint64_t override {
return std::numeric_limits<std::int64_t>::max() / std::int64_t(2);
}
std::uint64_t get_total_item_count() const override { return total_item_count_; }
[[nodiscard]] auto get_total_item_count() const -> std::uint64_t override;
bool is_directory(const std::string &api_path) const override;
[[nodiscard]] auto is_direct_only() const -> bool override { return false; }
bool is_file(const std::string &api_path) const override;
[[nodiscard]] auto is_directory(const std::string &api_path,
bool &exists) const -> api_error override;
bool is_online() const override;
[[nodiscard]] auto is_file(const std::string &api_path, bool &exists) const
-> api_error override;
bool is_processing(const std::string &api_path) const;
[[nodiscard]] auto is_online() const -> bool override;
bool is_rename_supported() const override { return false; }
[[nodiscard]] auto is_rename_supported() const -> bool override {
return false;
}
api_error read_file_bytes(const std::string &api_path, const std::size_t &size,
const std::uint64_t &offset, std::vector<char> &data,
const bool &stop_requested) override;
[[nodiscard]] auto read_file_bytes(const std::string &api_path,
std::size_t size, std::uint64_t offset,
data_buffer &data,
stop_type &stop_requested)
-> api_error override;
api_error remove_directory(const std::string &api_path) override;
[[nodiscard]] auto remove_directory(const std::string &api_path)
-> api_error override;
api_error remove_file(const std::string &api_path) override;
[[nodiscard]] auto remove_file(const std::string &api_path)
-> api_error override;
api_error rename_file(const std::string &from_api_path, const std::string &to_api_path) override;
bool start(api_item_added_callback api_item_added, i_open_file_table *oft) override;
[[nodiscard]] auto start(api_item_added_callback api_item_added,
i_file_manager *fm) -> bool override;
void stop() override;
api_error upload_file(const std::string &api_path, const std::string &source_path,
const std::string &encryption_token) override;
[[nodiscard]] auto
upload_file(const std::string &api_path, const std::string &source_path,
const std::string &encryption_token, stop_type &stop_requested)
-> api_error override;
};
} // namespace repertory

228
include/providers/sia/sia_provider.hpp
View File

@ -1,136 +1,194 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_SIA_SIAPROVIDER_HPP_
#define INCLUDE_PROVIDERS_SIA_SIAPROVIDER_HPP_
#ifndef INCLUDE_PROVIDERS_SIA_SIA_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_SIA_SIA_PROVIDER_HPP_
#include "common.hpp"
#include "providers/base_provider.hpp"
#include "providers/i_provider.hpp"
#include "types/repertory.hpp"
namespace repertory {
class app_config;
class i_comm;
class sia_provider : public base_provider {
class i_file_manager;
class i_http_comm;
class sia_provider : public i_provider {
public:
sia_provider(app_config &config, i_comm &comm);
sia_provider(app_config &config, i_http_comm &comm);
~sia_provider() override = default;
private:
i_comm &comm_;
std::uint64_t total_drive_space_ = 0u;
bool stop_requested_ = false;
std::mutex start_stop_mutex_;
std::condition_variable start_stop_notify_;
std::unique_ptr<std::thread> drive_space_thread_;
app_config &config_;
i_http_comm &comm_;
private:
api_item_added_callback &get_api_item_added() { return api_item_added_; }
i_comm &get_comm() { return comm_; }
i_comm &get_comm() const { return comm_; } // TODO Fix non-const reference return
const std::string DB_NAME = "meta_db";
api_item_added_callback api_item_added_;
std::unique_ptr<rocksdb::DB> db_;
i_file_manager *fm_ = nullptr;
private:
void calculate_total_drive_space();
[[nodiscard]] static auto create_api_file(std::string path,
std::uint64_t size) -> api_file;
api_error calculate_used_drive_space(std::uint64_t &used_space);
[[nodiscard]] static auto create_api_file(std::string path,
std::uint64_t size,
api_meta_map &meta) -> api_file;
void drive_space_thread();
[[nodiscard]] auto get_object_info(const std::string &api_path,
json &object_info) const -> api_error;
bool processed_orphaned_file(const std::string &source_path,
const std::string &api_path = "") const;
[[nodiscard]] auto get_object_list(const std::string api_path,
nlohmann::json &object_list) const -> bool;
void remove_deleted_files();
void remove_expired_orphaned_files();
void remove_unknown_source_files();
protected:
api_error check_file_exists(const std::string &api_path) const;
bool check_directory_found(const json &error) const;
bool check_not_found(const json &error) const;
virtual void cleanup();
void create_api_file(const json &json_file, const std::string &path_name, api_file &file) const;
api_error get_directory(const std::string &api_path, json &result, json &error) const;
api_error notify_file_added(const std::string &api_path, const std::string &api_parent,
const std::uint64_t &size) override;
void set_api_file_dates(const json &file, api_file &apiFile) const;
public:
api_error create_directory(const std::string &api_path, const api_meta_map &meta) override;
[[nodiscard]] auto create_directory(const std::string &api_path,
api_meta_map &meta) -> api_error override;
std::uint64_t get_directory_item_count(const std::string &api_path) const override;
[[nodiscard]] auto
create_directory_clone_source_meta(const std::string &source_api_path,
const std::string &api_path)
-> api_error override;
api_error get_directory_items(const std::string &api_path,
directory_item_list &list) const override;
[[nodiscard]] auto get_directory_item_count(const std::string &api_path) const
-> std::uint64_t override;
api_error get_file(const std::string &api_path, api_file &file) const override;
[[nodiscard]] auto create_file(const std::string &api_path,
api_meta_map &meta) -> api_error override;
api_error get_file_list(api_file_list &list) const override;
[[nodiscard]] auto get_api_path_from_source(const std::string &source_path,
std::string &api_path) const
-> api_error override;
api_error get_file_size(const std::string &api_path, std::uint64_t &file_size) const override;
[[nodiscard]] auto get_directory_items(const std::string &api_path,
directory_item_list &list) const
-> api_error override;
api_error get_filesystem_item(const std::string &api_path, const bool &directory,
filesystem_item &fsi) const override;
[[nodiscard]] auto get_file(const std::string &api_path, api_file &file) const
-> api_error override;
api_error get_filesystem_item_and_file(const std::string &api_path, api_file &apiFile,
filesystem_item &fsi) const override;
[[nodiscard]] auto get_file_list(api_file_list &list) const
-> api_error override;
provider_type get_provider_type() const override { return provider_type::sia; }
[[nodiscard]] auto get_file_size(const std::string &api_path,
std::uint64_t &file_size) const
-> api_error override;
std::uint64_t get_total_drive_space() const override { return total_drive_space_; }
[[nodiscard]] auto get_filesystem_item(const std::string &api_path,
bool directory,
filesystem_item &fsi) const
-> api_error override;
std::uint64_t get_total_item_count() const override;
[[nodiscard]] auto get_filesystem_item_and_file(const std::string &api_path,
api_file &f,
filesystem_item &fsi) const
-> api_error override;
bool is_directory(const std::string &api_path) const override;
[[nodiscard]] auto
get_filesystem_item_from_source_path(const std::string &source_path,
filesystem_item &fsi) const
-> api_error override;
bool is_file(const std::string &api_path) const override;
[[nodiscard]] auto get_item_meta(const std::string &api_path,
api_meta_map &meta) const
-> api_error override;
bool is_online() const override;
[[nodiscard]] auto get_item_meta(const std::string &api_path,
const std::string &key,
std::string &value) const
-> api_error override;
bool is_rename_supported() const override { return true; }
[[nodiscard]] auto get_pinned_files() const
-> std::vector<std::string> override;
api_error read_file_bytes(const std::string &api_path, const std::size_t &size,
const std::uint64_t &offset, std::vector<char> &buffer,
const bool &stop_requested) override;
[[nodiscard]] auto get_provider_type() const -> provider_type override {
return provider_type::sia;
}
api_error remove_directory(const std::string &api_path) override;
[[nodiscard]] auto get_total_drive_space() const -> std::uint64_t override;
api_error remove_file(const std::string &api_path) override;
[[nodiscard]] auto get_total_item_count() const -> std::uint64_t override;
api_error rename_file(const std::string &from_api_path, const std::string &to_api_path) override;
[[nodiscard]] auto get_used_drive_space() const -> std::uint64_t override;
bool start(api_item_added_callback api_item_added, i_open_file_table *oft) override;
[[nodiscard]] auto is_direct_only() const -> bool override { return false; }
[[nodiscard]] auto is_directory(const std::string &api_path,
bool &exists) const -> api_error override;
[[nodiscard]] auto is_file(const std::string &api_path, bool &exists) const
-> api_error override;
[[nodiscard]] auto is_file_writeable(const std::string &api_path) const
-> bool override;
[[nodiscard]] auto is_online() const -> bool override;
[[nodiscard]] auto is_rename_supported() const -> bool override {
return false;
}
[[nodiscard]] auto read_file_bytes(const std::string &api_path,
std::size_t size, std::uint64_t offset,
data_buffer &buffer,
stop_type &stop_requested)
-> api_error override;
[[nodiscard]] auto remove_directory(const std::string &api_path)
-> api_error override;
[[nodiscard]] auto remove_file(const std::string &api_path)
-> api_error override;
[[nodiscard]] auto remove_item_meta(const std::string &api_path,
const std::string &key)
-> api_error override;
[[nodiscard]] auto rename_file(const std::string &from_api_path,
const std::string &to_api_path)
-> api_error override;
[[nodiscard]] auto set_item_meta(const std::string &api_path,
const std::string &key,
const std::string &value)
-> api_error override;
[[nodiscard]] auto set_item_meta(const std::string &api_path,
const api_meta_map &meta)
-> api_error override;
[[nodiscard]] auto start(api_item_added_callback api_item_added,
i_file_manager *fm) -> bool override;
void stop() override;
api_error upload_file(const std::string &api_path, const std::string &source_path,
const std::string &) override;
[[nodiscard]] auto upload_file(const std::string &api_path,
const std::string &source_path,
const std::string & /* encryption_token */,
stop_type &stop_requested)
-> api_error override;
};
} // namespace repertory
#endif // INCLUDE_PROVIDERS_SIA_SIAPROVIDER_HPP_
#endif // INCLUDE_PROVIDERS_SIA_SIA_PROVIDER_HPP_

139
include/providers/skynet/skynet_provider.hpp
View File

@ -1,139 +0,0 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
#ifndef INCLUDE_PROVIDERS_SKYNET_SKYNET_PROVIDER_HPP_
#define INCLUDE_PROVIDERS_SKYNET_SKYNET_PROVIDER_HPP_
#if defined(REPERTORY_ENABLE_SKYNET)
#include "common.hpp"
#include "db/directory_db.hpp"
#include "db/meta_db.hpp"
#include "events/event_system.hpp"
#include "providers/base_provider.hpp"
#include "upload/upload_manager.hpp"
namespace repertory {
class app_config;
class i_comm;
class skynet_provider final : public base_provider {
E_CONSUMER();
public:
skynet_provider(app_config &config, i_comm &comm);
~skynet_provider() override { E_CONSUMER_RELEASE(); }
private:
i_comm &comm_;
directory_db directory_db_;
upload_manager upload_manager_;
bool stop_requested_ = false;
std::atomic<std::size_t> next_download_index_;
std::atomic<std::size_t> next_upload_index_;
std::shared_ptr<std::vector<host_config>> download_list_;
std::shared_ptr<std::vector<host_config>> upload_list_;
mutable std::mutex portal_mutex_;
private:
std::size_t get_retry_count() const;
void populate_api_file(api_file &file) const;
void process_export(json &result, const std::string &api_path) const;
void upload_completed(const std::string &api_path, const std::string &, const json &data);
api_error upload_handler(const upload_manager::upload &upload, json &data, json &error);
protected:
void notify_directory_added(const std::string &api_path, const std::string &api_parent) override;
api_error notify_file_added(const std::string &, const std::string &,
const std::uint64_t &) override;
public:
api_error create_directory(const std::string &api_path, const api_meta_map &meta) override;
api_error create_file(const std::string &api_path, api_meta_map &meta) override;
json export_all() const;
json export_list(const std::vector<std::string> &api_path_list) const;
std::uint64_t get_directory_item_count(const std::string &api_path) const override;
api_error get_directory_items(const std::string &api_path,
directory_item_list &list) const override;
api_error get_file(const std::string &api_path, api_file &file) const override;
api_error get_file_list(api_file_list &list) const override;
api_error get_file_size(const std::string &api_path, std::uint64_t &file_size) const override;
host_config get_host_config(const bool &upload);
provider_type get_provider_type() const override { return provider_type::skynet; }
api_error get_skynet_metadata(const std::string &skylink, json &json_meta);
std::uint64_t get_total_drive_space() const override {
return std::numeric_limits<std::int64_t>::max() / std::int64_t(2);
}
std::uint64_t get_total_item_count() const override {
return directory_db_.get_total_item_count();
}
api_error import_skylink(const skylink_import &si);
bool is_directory(const std::string &api_path) const override;
bool is_file(const std::string &api_path) const override;
bool is_file_writeable(const std::string &api_path) const override;
bool is_online() const override { return true; }
bool is_processing(const std::string &api_path) const;
bool is_rename_supported() const override { return true; }
api_error read_file_bytes(const std::string &api_path, const std::size_t &size,
const std::uint64_t &offset, std::vector<char> &data,
const bool &stop_requested) override;
api_error remove_directory(const std::string &api_path) override;
api_error remove_file(const std::string &api_path) override;
api_error rename_file(const std::string &from_api_path, const std::string &to_api_path) override;
bool start(api_item_added_callback api_item_added, i_open_file_table *oft) override;
void stop() override;
bool update_portal_list();
api_error upload_file(const std::string &api_path, const std::string &source_path,
const std::string &encryption_token) override;
};
} // namespace repertory
#endif // REPERTORY_ENABLE_SKYNET
#endif // INCLUDE_PROVIDERS_SKYNET_SKYNET_PROVIDER_HPP_