BlockStorage/repertory
1
0
Fork 0
Code Issues 8 Pull Requests Releases 6 Wiki Activity

2.0.0-rc (#9)
Some checks failed
BlockStorage/repertory_osx/pipeline/head This commit looks good
Details
BlockStorage/repertory_windows/pipeline/head This commit looks good
Details
BlockStorage/repertory/pipeline/head There was a failure building this commit
Details
BlockStorage/repertory_linux_builds/pipeline/head This commit looks good
Details
BlockStorage/repertory_osx_builds/pipeline/head There was a failure building this commit
Details

Browse Source 
### Issues

* \#1 \[bug\] Unable to mount S3 due to 'item_not_found' exception
* \#2 Require bucket name for S3 mounts
* \#3 \[bug\] File size is not being updated in S3 mount
* \#4 Upgrade to libfuse-3.x.x
* \#5 Switch to renterd for Sia support
* \#6 Switch to cpp-httplib to further reduce dependencies
* \#7 Remove global_data and calculate used disk space per provider
* \#8 Switch to libcurl for S3 mount support

### Changes from v1.x.x

* Added read-only encrypt provider
  * Pass-through mount point that transparently encrypts source data using `XChaCha20-Poly1305`
* Added S3 encryption support via `XChaCha20-Poly1305`
* Added replay protection to remote mounts
* Added support base64 writes in remote FUSE
* Created static linked Linux binaries for `amd64` and `aarch64` using `musl-libc`
* Removed legacy Sia renter support
* Removed Skynet support
* Fixed multiple remote mount WinFSP API issues on \*NIX servers
* Implemented chunked read and write
  * Writes for non-cached files are performed in chunks of 8Mib
* Removed `repertory-ui` support
* Removed `FreeBSD` support
* Switched to `libsodium` over `CryptoPP`
* Switched to `XChaCha20-Poly1305` for remote mounts
* Updated `GoogleTest` to v1.14.0
* Updated `JSON for Modern C++` to v3.11.2
* Updated `OpenSSL` to v1.1.1w
* Updated `RocksDB` to v8.5.3
* Updated `WinFSP` to 2023
* Updated `boost` to v1.78.0
* Updated `cURL` to v8.3.0
* Updated `zlib` to v1.3
* Use `upload_manager` for all providers
  * Adds a delay to uploads to prevent excessive API calls
  * Supports re-upload after mount restart for incomplete uploads
  * NOTE: Uploads for all providers are full file (no resume support)
    * Multipart upload support is planned for S3

Reviewed-on: #9
This commit is contained in:
Scott E. Graves
2023-10-29 06:55:59 +00:00
parent 3ff46723b8
commit f43c41f88a
839 changed files with 98214 additions and 92959 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
src/utils/action_queue.cpp Normal file
View File

@ -0,0 +1,21 @@
#include "utils/action_queue.hpp"
#include "types/repertory.hpp"
namespace repertory::utils::action_queue {
action_queue::action_queue(const std::string &id,
std::uint8_t max_concurrent_actions)
: single_thread_service_base("action_queue_" + id),
id_(id),
max_concurrent_actions_(max_concurrent_actions) {}
void action_queue::service_function() {
//
}
void action_queue::push(std::function<void()> action) {
unique_mutex_lock queue_lock(queue_mtx_);
queue_.emplace_back(action);
queue_notify_.notify_all();
}
} // namespace repertory::utils::action_queue

158
src/utils/cli_utils.cpp
View File

@ -1,22 +1,26 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/cli_utils.hpp"
#include "app_config.hpp"
#include "utils/file_utils.hpp"
#include "utils/path_utils.hpp"
@ -24,18 +28,18 @@
#include "utils/utils.hpp"
namespace repertory::utils::cli {
void get_api_authentication_data(std::string &user, std::string &password, std::uint16_t &port,
const provider_type &pt, const std::string &data_directory) {
void get_api_authentication_data(std::string &user, std::string &password,
std::uint16_t &port, const provider_type &pt,
const std::string &data_directory) {
const auto cfg_file_path = utils::path::combine(
data_directory.empty() ? app_config::default_data_directory(pt) : data_directory,
data_directory.empty() ? app_config::default_data_directory(pt)
: data_directory,
{"config.json"});
json data;
auto success = false;
for (auto i = 0; not(success = utils::file::read_json_file(cfg_file_path, data)) && (i < 20);
i++) {
std::this_thread::sleep_for(100ms);
}
const auto success = utils::retryable_action([&]() -> bool {
return utils::file::read_json_file(cfg_file_path, data);
});
if (success) {
if (user.empty() && password.empty()) {
@ -46,29 +50,24 @@ void get_api_authentication_data(std::string &user, std::string &password, std::
}
}
provider_type get_provider_type_from_args(const int &argc, char *argv[]) {
auto pt = provider_type::unknown;
auto get_provider_type_from_args(int argc, char *argv[]) -> provider_type {
#if defined(REPERTORY_ENABLE_S3)
if ((pt == provider_type::unknown) && (has_option(argc, argv, options::s3_option))) {
pt = provider_type::s3;
if (has_option(argc, argv, options::s3_option)) {
return provider_type::s3;
}
#endif // defined(REPERTORY_ENABLE_S3)
#if defined(REPERTORY_ENABLE_SKYNET)
if ((pt == provider_type::unknown) && (has_option(argc, argv, options::skynet_option))) {
pt = provider_type::skynet;
if (has_option(argc, argv, options::remote_mount_option)) {
return provider_type::remote;
}
#endif // defined(REPERTORY_ENABLE_SKYNET)
if ((pt == provider_type::unknown) && (has_option(argc, argv, options::remote_mount_option))) {
pt = provider_type::remote;
if (has_option(argc, argv, options::encrypt_option)) {
return provider_type::encrypt;
}
if (pt == provider_type::unknown) {
pt = provider_type::sia;
}
return pt;
return provider_type::sia;
}
bool has_option(const int &argc, char *argv[], const std::string &option_name) {
auto has_option(int argc, char *argv[], const std::string &option_name)
-> bool {
auto ret = false;
for (int i = 0; not ret && (i < argc); i++) {
ret = (option_name == argv[i]);
@ -76,12 +75,12 @@ bool has_option(const int &argc, char *argv[], const std::string &option_name) {
return ret;
}
bool has_option(const int &argc, char *argv[], const option &opt) {
auto has_option(int argc, char *argv[], const option &opt) -> bool {
return has_option(argc, argv, opt[0u]) || has_option(argc, argv, opt[1u]);
}
std::vector<std::string> parse_option(const int &argc, char *argv[], const std::string &option_name,
std::uint8_t count) {
auto parse_option(int argc, char *argv[], const std::string &option_name,
std::uint8_t count) -> std::vector<std::string> {
std::vector<std::string> ret;
auto found = false;
for (auto i = 0; not found && (i < argc); i++) {
@ -97,14 +96,16 @@ std::vector<std::string> parse_option(const int &argc, char *argv[], const std::
return ret;
}
exit_code parse_string_option(const int &argc, char **argv, const option &opt, std::string &value) {
auto parse_string_option(int argc, char **argv, const option &opt,
std::string &value) -> exit_code {
auto ret = exit_code::success;
if (has_option(argc, argv, opt[0u]) || has_option(argc, argv, opt[1u])) {
auto data = parse_option(argc, argv, opt[0u], 1u);
if (data.empty()) {
data = parse_option(argc, argv, opt[1u], 1u);
if (data.empty()) {
std::cerr << "Invalid syntax for '" << opt[0u] << "," << opt[1u] << '\'' << std::endl;
std::cerr << "Invalid syntax for '" << opt[0u] << "," << opt[1u] << '\''
<< std::endl;
ret = exit_code::invalid_syntax;
}
}
@ -117,16 +118,19 @@ exit_code parse_string_option(const int &argc, char **argv, const option &opt, s
return ret;
}
std::vector<std::string> parse_drive_options(const int &argc, char **argv, provider_type &pt,
std::string &data_directory) {
auto parse_drive_options(int argc, char **argv,
[[maybe_unused]] provider_type &pt,
[[maybe_unused]] std::string &data_directory)
-> std::vector<std::string> {
// Strip out options from command line
const auto &option_list = options::option_list;
std::vector<std::string> drive_args;
for (int i = 0; i < argc; i++) {
const auto &a = argv[i];
if (std::find_if(option_list.begin(), option_list.end(), [&a](const auto &pair) -> bool {
return ((pair[0] == a) || (pair[1] == a));
}) == option_list.end()) {
if (std::find_if(option_list.begin(), option_list.end(),
[&a](const auto &pair) -> bool {
return ((pair[0] == a) || (pair[1] == a));
}) == option_list.end()) {
drive_args.emplace_back(argv[i]);
} else if ((std::string(argv[i]) == options::remote_mount_option[0]) ||
(std::string(argv[i]) == options::remote_mount_option[1])) {
@ -160,17 +164,14 @@ std::vector<std::string> parse_drive_options(const int &argc, char **argv, provi
pt = provider_type::s3;
}
#endif // defined(REPERTORY_ENABLE_S3)
#if defined(REPERTORY_ENABLE_SKYNET)
if ((fuse_option.find("sk") == 0) || (fuse_option.find("skynet") == 0)) {
pt = provider_type::skynet;
}
#endif // defined(REPERTORY_ENABLE_SKYNET)
if ((fuse_option.find("dd") == 0) || (fuse_option.find("data_directory") == 0)) {
if ((fuse_option.find("dd") == 0) ||
(fuse_option.find("data_directory") == 0)) {
const auto data = utils::string::split(fuse_option, '=');
if (data.size() == 2) {
data_directory = data[1];
} else {
std::cerr << "Invalid syntax for '-dd,--data_directory'" << std::endl;
std::cerr << "Invalid syntax for '-dd,--data_directory'"
<< std::endl;
exit(3);
}
} else {
@ -197,25 +198,44 @@ std::vector<std::string> parse_drive_options(const int &argc, char **argv, provi
}
}
#if FUSE_USE_VERSION < 30
{
auto it = std::remove_if(
fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool { return opt.find("hard_remove") == 0; });
if (it == fuse_flags_list.end()) {
fuse_flags_list.emplace_back("hard_remove");
}
}
#endif
#ifdef __APPLE__
auto it = std::remove_if(fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool { return opt.find("volname") == 0; });
if (it != fuse_flags_list.end()) {
fuse_flags_list.erase(it, fuse_flags_list.end());
}
fuse_flags_list.emplace_back("volname=Repertory" + app_config::get_provider_display_name(pt));
{
auto it = std::remove_if(
fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool { return opt.find("volname") == 0; });
if (it != fuse_flags_list.end()) {
fuse_flags_list.erase(it, fuse_flags_list.end());
}
fuse_flags_list.emplace_back("volname=Repertory" +
app_config::get_provider_display_name(pt));
it = std::remove_if(fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool { return opt.find("daemon_timeout") == 0; });
if (it != fuse_flags_list.end()) {
fuse_flags_list.erase(it, fuse_flags_list.end());
}
fuse_flags_list.emplace_back("daemon_timeout=300");
it = std::remove_if(fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool {
return opt.find("daemon_timeout") == 0;
});
if (it != fuse_flags_list.end()) {
fuse_flags_list.erase(it, fuse_flags_list.end());
}
fuse_flags_list.emplace_back("daemon_timeout=300");
it = std::remove_if(fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool { return opt.find("noappledouble") == 0; });
if (it == fuse_flags_list.end()) {
fuse_flags_list.emplace_back("noappledouble");
it = std::remove_if(fuse_flags_list.begin(), fuse_flags_list.end(),
[](const auto &opt) -> bool {
return opt.find("noappledouble") == 0;
});
if (it == fuse_flags_list.end()) {
fuse_flags_list.emplace_back("noappledouble");
}
}
#endif // __APPLE__

222
src/utils/encrypting_reader.cpp
View File

@ -1,31 +1,38 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/encrypting_reader.hpp"
#include "events/events.hpp"
#include "events/event_system.hpp"
#include "events/events.hpp"
#include "types/repertory.hpp"
#include "utils/encryption.hpp"
#include "utils/error_utils.hpp"
#include "utils/utils.hpp"
namespace repertory::utils::encryption {
class encrypting_streambuf final : public encrypting_reader::streambuf {
public:
explicit encrypting_streambuf(const encrypting_reader &reader) : reader_(reader) {
explicit encrypting_streambuf(const encrypting_reader &reader)
: reader_(reader) {
setg(reinterpret_cast<char *>(0), reinterpret_cast<char *>(0),
reinterpret_cast<char *>(reader_.get_total_size()));
}
@ -36,9 +43,10 @@ private:
encrypting_reader reader_;
protected:
pos_type seekoff(off_type off, std::ios_base::seekdir dir,
std::ios_base::openmode which = std::ios_base::out |
std::ios_base::in) override {
auto seekoff(off_type off, std::ios_base::seekdir dir,
std::ios_base::openmode which = std::ios_base::out |
std::ios_base::in)
-> pos_type override {
if ((which & std::ios_base::in) != std::ios_base::in) {
throw std::runtime_error("output is not supported");
}
@ -46,9 +54,11 @@ protected:
const auto set_position = [this](char *next) -> pos_type {
if ((next <= egptr()) && (next >= eback())) {
setg(eback(), next, reinterpret_cast<char *>(reader_.get_total_size()));
return pos_type(reinterpret_cast<std::uintptr_t>(gptr()));
return static_cast<std::streamoff>(
reinterpret_cast<std::uintptr_t>(gptr()));
}
return pos_type(traits_type::eof());
return {traits_type::eof()};
};
switch (dir) {
@ -65,12 +75,14 @@ protected:
return encrypting_reader::streambuf::seekoff(off, dir, which);
}
pos_type seekpos(pos_type pos, std::ios_base::openmode which = std::ios_base::out |
std::ios_base::in) override {
auto seekpos(pos_type pos,
std::ios_base::openmode which = std::ios_base::out |
std::ios_base::in)
-> pos_type override {
return seekoff(pos, std::ios_base::beg, which);
}
int_type uflow() override {
auto uflow() -> int_type override {
auto ret = underflow();
if (ret == traits_type::eof()) {
return ret;
@ -81,12 +93,13 @@ protected:
return ret;
}
int_type underflow() override {
auto underflow() -> int_type override {
if (gptr() == egptr()) {
return traits_type::eof();
}
reader_.set_read_position(static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(gptr())));
reader_.set_read_position(
static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(gptr())));
char c{};
const auto res = reader_.reader_function(&c, 1u, 1u, &reader_);
@ -97,12 +110,13 @@ protected:
return c;
}
std::streamsize xsgetn(char *ptr, std::streamsize count) override {
auto xsgetn(char *ptr, std::streamsize count) -> std::streamsize override {
if (gptr() == egptr()) {
return traits_type::eof();
}
reader_.set_read_position(static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(gptr())));
reader_.set_read_position(
static_cast<std::uint64_t>(reinterpret_cast<std::uintptr_t>(gptr())));
const auto res = reader_.reader_function(ptr, 1u, count, &reader_);
if ((res == reader_.get_error_return()) ||
@ -110,14 +124,16 @@ protected:
return traits_type::eof();
}
setg(eback(), gptr() + res, reinterpret_cast<char *>(reader_.get_total_size()));
setg(eback(), gptr() + res,
reinterpret_cast<char *>(reader_.get_total_size()));
return res;
}
};
class encrypting_reader_iostream final : public encrypting_reader::iostream {
public:
explicit encrypting_reader_iostream(std::unique_ptr<encrypting_streambuf> buffer)
explicit encrypting_reader_iostream(
std::unique_ptr<encrypting_streambuf> buffer)
: encrypting_reader::iostream(buffer.get()), buffer_(std::move(buffer)) {}
~encrypting_reader_iostream() override = default;
@ -127,42 +143,101 @@ private:
};
const std::size_t encrypting_reader::header_size_ = ([]() {
CryptoPP::XChaCha20Poly1305::Encryption enc;
return enc.IVSize() + enc.DigestSize();
return crypto_aead_xchacha20poly1305_IETF_NPUBBYTES +
crypto_aead_xchacha20poly1305_IETF_ABYTES;
})();
const std::size_t encrypting_reader::data_chunk_size_ = (8u * 1024u * 1024u);
const std::size_t encrypting_reader::encrypted_chunk_size_ = data_chunk_size_ + header_size_;
const std::size_t encrypting_reader::encrypted_chunk_size_ =
data_chunk_size_ + header_size_;
encrypting_reader::encrypting_reader(const std::string &file_name, const std::string &source_path,
const bool &stop_requested, const std::string &token,
const size_t error_return)
encrypting_reader::encrypting_reader(
const std::string &file_name, const std::string &source_path,
stop_type &stop_requested, const std::string &token,
std::optional<std::string> relative_parent_path, const size_t error_return)
: key_(utils::encryption::generate_key(token)),
stop_requested_(stop_requested),
error_return_(error_return) {
const auto res = native_file::open(source_path, source_file_);
const auto res = native_file::open(
source_path, not relative_parent_path.has_value(), source_file_);
if (res != api_error::success) {
throw std::runtime_error("file open failed: " + source_path + ':' + api_error_to_string(res));
throw std::runtime_error("file open failed|src|" + source_path + '|' +
api_error_to_string(res));
}
std::vector<char> result;
data_buffer result;
utils::encryption::encrypt_data(key_, file_name.c_str(),
strnlen(file_name.c_str(), file_name.size()), result);
strnlen(file_name.c_str(), file_name.size()),
result);
encrypted_file_name_ = utils::to_hex_string(result);
if (relative_parent_path.has_value()) {
for (const auto &part :
std::filesystem::path(relative_parent_path.value())) {
utils::encryption::encrypt_data(
key_, part.string().c_str(),
strnlen(part.string().c_str(), part.string().size()), result);
encrypted_file_path_ += '/' + utils::to_hex_string(result);
}
encrypted_file_path_ += '/' + encrypted_file_name_;
}
std::uint64_t file_size{};
if (not utils::file::get_file_size(source_path, file_size)) {
throw std::runtime_error("get file size failed: " + source_path + ':' +
throw std::runtime_error("get file size failed|src|" + source_path + '|' +
std::to_string(utils::get_last_error_code()));
}
const auto total_chunks = static_cast<std::size_t>(
utils::divide_with_ceiling(file_size, static_cast<std::uint64_t>(data_chunk_size_)));
total_size_ = file_size + (total_chunks * encrypting_reader::get_header_size());
const auto total_chunks = static_cast<std::size_t>(utils::divide_with_ceiling(
file_size, static_cast<std::uint64_t>(data_chunk_size_)));
total_size_ =
file_size + (total_chunks * encrypting_reader::get_header_size());
last_data_chunk_ = total_chunks - 1u;
last_data_chunk_size_ =
static_cast<std::size_t>((file_size <= data_chunk_size_) ? file_size
: (file_size % data_chunk_size_) ? file_size % data_chunk_size_
: data_chunk_size_);
last_data_chunk_size_ = static_cast<std::size_t>(
(file_size <= data_chunk_size_) ? file_size
: (file_size % data_chunk_size_) ? file_size % data_chunk_size_
: data_chunk_size_);
iv_list_.resize(total_chunks);
for (auto &iv : iv_list_) {
randombytes_buf(iv.data(), iv.size());
}
}
encrypting_reader::encrypting_reader(
const std::string &encrypted_file_path, const std::string &source_path,
stop_type &stop_requested, const std::string &token,
std::vector<
std::array<unsigned char, crypto_aead_xchacha20poly1305_IETF_NPUBBYTES>>
iv_list,
const size_t error_return)
: key_(utils::encryption::generate_key(token)),
stop_requested_(stop_requested),
error_return_(error_return) {
const auto res = native_file::open(source_path, false, source_file_);
if (res != api_error::success) {
throw std::runtime_error("file open failed|src|" + source_path + '|' +
api_error_to_string(res));
}
encrypted_file_path_ = encrypted_file_path;
encrypted_file_name_ =
std::filesystem::path(encrypted_file_path_).filename().string();
std::uint64_t file_size{};
if (not utils::file::get_file_size(source_path, file_size)) {
throw std::runtime_error("get file size failed|src|" + source_path + '|' +
std::to_string(utils::get_last_error_code()));
}
const auto total_chunks = static_cast<std::size_t>(utils::divide_with_ceiling(
file_size, static_cast<std::uint64_t>(data_chunk_size_)));
total_size_ =
file_size + (total_chunks * encrypting_reader::get_header_size());
last_data_chunk_ = total_chunks - 1u;
last_data_chunk_size_ = static_cast<std::size_t>(
(file_size <= data_chunk_size_) ? file_size
: (file_size % data_chunk_size_) ? file_size % data_chunk_size_
: data_chunk_size_);
iv_list_ = std::move(iv_list);
}
encrypting_reader::encrypting_reader(const encrypting_reader &r)
@ -171,6 +246,8 @@ encrypting_reader::encrypting_reader(const encrypting_reader &r)
error_return_(r.error_return_),
chunk_buffers_(r.chunk_buffers_),
encrypted_file_name_(r.encrypted_file_name_),
encrypted_file_path_(r.encrypted_file_path_),
iv_list_(r.iv_list_),
last_data_chunk_(r.last_data_chunk_),
last_data_chunk_size_(r.last_data_chunk_size_),
read_offset_(r.read_offset_),
@ -183,23 +260,41 @@ encrypting_reader::~encrypting_reader() {
}
}
std::uint64_t encrypting_reader::calculate_decrypted_size(const std::uint64_t &total_size) {
auto encrypting_reader::calculate_decrypted_size(std::uint64_t total_size)
-> std::uint64_t {
return total_size - (utils::divide_with_ceiling(
total_size, static_cast<std::uint64_t>(get_encrypted_chunk_size())) *
total_size, static_cast<std::uint64_t>(
get_encrypted_chunk_size())) *
get_header_size());
}
std::shared_ptr<encrypting_reader::iostream> encrypting_reader::create_iostream() const {
auto encrypting_reader::calculate_encrypted_size(const std::string &source_path)
-> std::uint64_t {
std::uint64_t file_size{};
if (not utils::file::get_file_size(source_path, file_size)) {
throw std::runtime_error("get file size failed|src|" + source_path + '|' +
std::to_string(utils::get_last_error_code()));
}
const auto total_chunks = static_cast<std::size_t>(utils::divide_with_ceiling(
file_size, static_cast<std::uint64_t>(data_chunk_size_)));
return file_size + (total_chunks * encrypting_reader::get_header_size());
}
auto encrypting_reader::create_iostream() const
-> std::shared_ptr<encrypting_reader::iostream> {
return std::make_shared<encrypting_reader_iostream>(
std::make_unique<encrypting_streambuf>(*this));
}
size_t encrypting_reader::reader_function(char *buffer, size_t size, size_t nitems) {
const auto read_size = static_cast<std::size_t>(
std::min(static_cast<std::uint64_t>(size * nitems), total_size_ - read_offset_));
auto encrypting_reader::reader_function(char *buffer, size_t size,
size_t nitems) -> size_t {
const auto read_size = static_cast<std::size_t>(std::min(
static_cast<std::uint64_t>(size * nitems), total_size_ - read_offset_));
auto chunk = static_cast<std::size_t>(read_offset_ / encrypted_chunk_size_);
auto chunk_offset = static_cast<std::size_t>(read_offset_ % encrypted_chunk_size_);
auto chunk_offset =
static_cast<std::size_t>(read_offset_ % encrypted_chunk_size_);
std::size_t total_read = 0u;
auto ret = false;
@ -210,21 +305,25 @@ size_t encrypting_reader::reader_function(char *buffer, size_t size, size_t nite
while (not stop_requested_ && ret && remain) {
if (chunk_buffers_.find(chunk) == chunk_buffers_.end()) {
auto &chunk_buffer = chunk_buffers_[chunk];
std::vector<char> file_data(chunk == last_data_chunk_ ? last_data_chunk_size_
: data_chunk_size_);
data_buffer file_data(chunk == last_data_chunk_
? last_data_chunk_size_
: data_chunk_size_);
chunk_buffer.resize(file_data.size() + get_header_size());
std::size_t bytes_read{};
if ((ret = source_file_->read_bytes(&file_data[0u], file_data.size(),
chunk * data_chunk_size_, bytes_read))) {
utils::encryption::encrypt_data(key_, file_data, chunk_buffer);
chunk * data_chunk_size_,
bytes_read))) {
utils::encryption::encrypt_data(iv_list_.at(chunk), key_, file_data,
chunk_buffer);
}
} else if (chunk) {
chunk_buffers_.erase(chunk - 1u);
}
auto &chunk_buffer = chunk_buffers_[chunk];
const auto to_read = std::min(chunk_buffer.size() - chunk_offset, remain);
const auto to_read =
std::min(chunk_buffer.size() - chunk_offset, remain);
std::memcpy(buffer + total_read, &chunk_buffer[chunk_offset], to_read);
total_read += to_read;
remain -= to_read;
@ -233,12 +332,13 @@ size_t encrypting_reader::reader_function(char *buffer, size_t size, size_t nite
read_offset_ += to_read;
}
} catch (const std::exception &e) {
event_system::instance().raise<repertory_exception>(__FUNCTION__,
e.what() ? e.what() : "unkown error");
utils::error::raise_error(__FUNCTION__, e, "exception occurred");
ret = false;
}
}
return stop_requested_ ? CURL_READFUNC_ABORT : ret ? total_read : error_return_;
return stop_requested_ ? CURL_READFUNC_ABORT
: ret ? total_read
: error_return_;
}
} // namespace repertory::utils::encryption

125
src/utils/encryption.cpp
View File

@ -1,91 +1,136 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/encryption.hpp"
#include "events/events.hpp"
#include "events/event_system.hpp"
#include "events/events.hpp"
#include "types/repertory.hpp"
#include "utils/encrypting_reader.hpp"
#include "utils/utils.hpp"
namespace repertory::utils::encryption {
api_error decrypt_file_name(const std::string &encryption_token, std::string &file_name) {
std::vector<char> buffer;
auto decrypt_file_path(const std::string &encryption_token,
std::string &file_path) -> api_error {
std::string decrypted_file_path{};
for (const auto &part : std::filesystem::path(file_path)) {
auto file_name = part.string();
if (file_name == "/") {
continue;
}
auto res = decrypt_file_name(encryption_token, file_name);
if (res != api_error::success) {
return res;
}
decrypted_file_path += '/' + file_name;
}
file_path = decrypted_file_path;
return api_error::success;
}
auto decrypt_file_name(const std::string &encryption_token,
std::string &file_name) -> api_error {
data_buffer buffer;
if (not utils::from_hex_string(file_name, buffer)) {
return api_error::error;
}
file_name.clear();
if (not utils::encryption::decrypt_data(encryption_token, buffer, file_name)) {
if (not utils::encryption::decrypt_data(encryption_token, buffer,
file_name)) {
return api_error::decryption_error;
}
return api_error::success;
}
CryptoPP::SecByteBlock generate_key(const std::string &encryption_token) {
CryptoPP::SecByteBlock key(CryptoPP::SHA256::DIGESTSIZE);
CryptoPP::SHA256().CalculateDigest(
reinterpret_cast<CryptoPP::byte *>(&key[0u]),
reinterpret_cast<const CryptoPP::byte *>(encryption_token.c_str()),
strnlen(encryption_token.c_str(), encryption_token.size()));
auto generate_key(const std::string &encryption_token) -> key_type {
crypto_hash_sha256_state state{};
auto res = crypto_hash_sha256_init(&state);
if (res != 0) {
throw std::runtime_error("failed to initialize sha256|" +
std::to_string(res));
}
return key;
if ((res = crypto_hash_sha256_update(
&state,
reinterpret_cast<const unsigned char *>(encryption_token.c_str()),
strnlen(encryption_token.c_str(), encryption_token.size()))) != 0) {
throw std::runtime_error("failed to update sha256|" + std::to_string(res));
}
key_type ret{};
if ((res = crypto_hash_sha256_final(&state, ret.data())) != 0) {
throw std::runtime_error("failed to finalize sha256|" +
std::to_string(res));
}
return ret;
}
api_error read_encrypted_range(
const http_range &range, const CryptoPP::SecByteBlock &key,
const std::function<api_error(std::vector<char> &ct, const std::uint64_t &start_offset,
const std::uint64_t &end_offset)> &reader,
const std::uint64_t &total_size, std::vector<char> &data) {
static constexpr const auto &encrypted_chunk_size =
auto read_encrypted_range(
const http_range &range, const key_type &key,
const std::function<api_error(data_buffer &ct, std::uint64_t start_offset,
std::uint64_t end_offset)> &reader,
std::uint64_t total_size, data_buffer &data) -> api_error {
const auto encrypted_chunk_size =
utils::encryption::encrypting_reader::get_encrypted_chunk_size();
static constexpr const auto &data_chunk_size =
const auto data_chunk_size =
utils::encryption::encrypting_reader::get_data_chunk_size();
static constexpr const auto &header_size =
const auto header_size =
utils::encryption::encrypting_reader::get_header_size();
const auto start_chunk = static_cast<std::size_t>(range.begin / data_chunk_size);
const auto start_chunk =
static_cast<std::size_t>(range.begin / data_chunk_size);
const auto end_chunk = static_cast<std::size_t>(range.end / data_chunk_size);
auto remain = range.end - range.begin + 1u;
auto source_offset = static_cast<std::size_t>(range.begin % data_chunk_size);
for (std::size_t chunk = start_chunk; chunk <= end_chunk; chunk++) {
std::vector<char> ct;
data_buffer ct;
const auto start_offset = chunk * encrypted_chunk_size;
const auto end_offset =
std::min(start_offset + (total_size - (chunk * data_chunk_size)) + header_size - 1u,
static_cast<std::uint64_t>(start_offset + encrypted_chunk_size - 1u));
const auto end_offset = std::min(
start_offset + (total_size - (chunk * data_chunk_size)) + header_size -
1u,
static_cast<std::uint64_t>(start_offset + encrypted_chunk_size - 1u));
const auto result = reader(ct, start_offset, end_offset);
if (result != api_error::success) {
return result;
}
std::vector<char> source_buffer;
data_buffer source_buffer;
if (not utils::encryption::decrypt_data(key, ct, source_buffer)) {
return api_error::decryption_error;
}
ct.clear();
const auto data_size = static_cast<std::size_t>(
std::min(remain, static_cast<std::uint64_t>(data_chunk_size - source_offset)));
const auto data_size = static_cast<std::size_t>(std::min(
remain, static_cast<std::uint64_t>(data_chunk_size - source_offset)));
std::copy(source_buffer.begin() + source_offset,
source_buffer.begin() + source_offset + data_size, std::back_inserter(data));
source_buffer.begin() + source_offset + data_size,
std::back_inserter(data));
remain -= data_size;
source_offset = 0u;
}

172
src/utils/error_utils.cpp Normal file
View File

@ -0,0 +1,172 @@
/*
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/error_utils.hpp"
#include "events/event_system.hpp"
#include "events/events.hpp"
#include "types/repertory.hpp"
namespace repertory::utils::error {
void raise_error(std::string_view function, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function), static_cast<std::string>(msg));
}
void raise_error(std::string_view function, const api_error &e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|err|" + api_error_to_string(e));
}
void raise_error(std::string_view function, const std::exception &e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|err|" +
(e.what() ? e.what() : "unknown error"));
}
void raise_error(std::string_view function, const json &e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|err|" + e.dump(2));
}
void raise_error(std::string_view function, std::int64_t e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|err|" + std::to_string(e));
}
void raise_error(std::string_view function, const api_error &e,
std::string_view file_path, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|sp|" +
static_cast<std::string>(file_path) + "|err|" +
api_error_to_string(e));
}
void raise_error(std::string_view function, std::int64_t e,
std::string_view file_path, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|sp|" +
static_cast<std::string>(file_path) + "|err|" + std::to_string(e));
}
void raise_error(std::string_view function, const std::exception &e,
std::string_view file_path, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|sp|" +
static_cast<std::string>(file_path) + "|err|" +
(e.what() ? e.what() : "unknown error"));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
const api_error &e, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|err|" +
api_error_to_string(e));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
std::int64_t e, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|err|" + std::to_string(e));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
const std::exception &e, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|err|" +
(e.what() ? e.what() : "unknown error"));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
std::string_view source_path, const api_error &e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|sp|" +
static_cast<std::string>(source_path) + "|err|" +
api_error_to_string(e));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
std::string_view source_path, std::int64_t e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|sp|" +
static_cast<std::string>(source_path) + "|err|" + std::to_string(e));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
const json &e, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|err|" + e.dump(2));
}
void raise_api_path_error(std::string_view function, std::string_view api_path,
std::string_view source_path, const std::exception &e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|ap|" +
static_cast<std::string>(api_path) + "|sp|" +
static_cast<std::string>(source_path) + "|err|" +
(e.what() ? e.what() : "unknown error"));
}
void raise_url_error(std::string_view function, std::string_view url,
CURLcode e, std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|url|" + static_cast<std::string>(url) +
"|err|" + curl_easy_strerror(e));
}
void raise_url_error(std::string_view function, std::string_view url,
std::string_view source_path, const std::exception &e,
std::string_view msg) {
event_system::instance().raise<repertory_exception>(
static_cast<std::string>(function),
static_cast<std::string>(msg) + "|url|" + static_cast<std::string>(url) +
"|sp|" + static_cast<std::string>(source_path) + "|err|" +
(e.what() ? e.what() : "unknown error"));
}
} // namespace repertory::utils::error

410
src/utils/file_utils.cpp
View File

@ -1,44 +1,34 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/file_utils.hpp"
#include "types/repertory.hpp"
#include "utils/error_utils.hpp"
#include "utils/path_utils.hpp"
#include "utils/string_utils.hpp"
#include "utils/utils.hpp"
namespace repertory::utils::file {
api_error assign_and_get_native_file(filesystem_item &fi, native_file_ptr &nf) {
recur_mutex_lock l(*fi.lock);
if (fi.handle == REPERTORY_INVALID_HANDLE) {
const auto res = native_file::create_or_open(fi.source_path, nf);
if (res != api_error::success) {
return res;
}
fi.handle = nf->get_handle();
} else {
nf = native_file::attach(fi.handle);
}
return api_error::success;
}
std::uint64_t calculate_used_space(std::string path, const bool &recursive) {
auto calculate_used_space(std::string path, bool recursive) -> std::uint64_t {
path = utils::path::absolute(path);
std::uint64_t ret = 0u;
#ifdef _WIN32
@ -50,7 +40,8 @@ std::uint64_t calculate_used_space(std::string path, const bool &recursive) {
const auto file_name = std::string(fd.cFileName);
if (fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) {
if (recursive && (file_name != ".") && (file_name != "..")) {
ret += calculate_used_space(utils::path::combine(path, {file_name}), recursive);
ret += calculate_used_space(utils::path::combine(path, {file_name}),
recursive);
}
} else {
std::uint64_t file_size{};
@ -67,12 +58,15 @@ std::uint64_t calculate_used_space(std::string path, const bool &recursive) {
struct dirent *de{};
while ((de = readdir(root)) != nullptr) {
if (de->d_type == DT_DIR) {
if (recursive && (strcmp(de->d_name, ".") != 0) && (strcmp(de->d_name, "..") != 0)) {
ret += calculate_used_space(utils::path::combine(path, {de->d_name}), recursive);
if (recursive && (strcmp(de->d_name, ".") != 0) &&
(strcmp(de->d_name, "..") != 0)) {
ret += calculate_used_space(utils::path::combine(path, {de->d_name}),
recursive);
}
} else {
std::uint64_t file_size{};
if (get_file_size(utils::path::combine(path, {de->d_name}), file_size)) {
if (get_file_size(utils::path::combine(path, {de->d_name}),
file_size)) {
ret += file_size;
}
}
@ -88,7 +82,8 @@ void change_to_process_directory() {
#ifdef _WIN32
std::string file_name;
file_name.resize(MAX_PATH);
::GetModuleFileNameA(nullptr, &file_name[0u], static_cast<DWORD>(file_name.size()));
::GetModuleFileNameA(nullptr, &file_name[0u],
static_cast<DWORD>(file_name.size()));
std::string path = file_name.c_str();
::PathRemoveFileSpecA(&path[0u]);
@ -106,19 +101,18 @@ void change_to_process_directory() {
#endif
}
bool copy_file(std::string from_path, std::string to_path) {
auto copy_file(std::string from_path, std::string to_path) -> bool {
from_path = utils::path::absolute(from_path);
to_path = utils::path::absolute(to_path);
if (is_file(from_path) && not is_directory(to_path)) {
boost::system::error_code ec{};
boost::filesystem::copy_file(from_path, to_path, ec);
return not ec.failed();
return std::filesystem::copy_file(from_path, to_path);
}
return false;
}
bool copy_directory_recursively(std::string from_path, std::string to_path) {
auto copy_directory_recursively(std::string from_path, std::string to_path)
-> bool {
from_path = utils::path::absolute(from_path);
to_path = utils::path::absolute(to_path);
auto ret = create_full_directory_path(to_path);
@ -131,9 +125,11 @@ bool copy_directory_recursively(std::string from_path, std::string to_path) {
ret = true;
do {
if (fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) {
if ((std::string(fd.cFileName) != ".") && (std::string(fd.cFileName) != "..")) {
ret = copy_directory_recursively(utils::path::combine(from_path, {fd.cFileName}),
utils::path::combine(to_path, {fd.cFileName}));
if ((std::string(fd.cFileName) != ".") &&
(std::string(fd.cFileName) != "..")) {
ret = copy_directory_recursively(
utils::path::combine(from_path, {fd.cFileName}),
utils::path::combine(to_path, {fd.cFileName}));
}
} else {
ret = copy_file(utils::path::combine(from_path, {fd.cFileName}),
@ -149,9 +145,11 @@ bool copy_directory_recursively(std::string from_path, std::string to_path) {
struct dirent *de{};
while (ret && (de = readdir(root))) {
if (de->d_type == DT_DIR) {
if ((strcmp(de->d_name, ".") != 0) && (strcmp(de->d_name, "..") != 0)) {
ret = copy_directory_recursively(utils::path::combine(from_path, {de->d_name}),
utils::path::combine(to_path, {de->d_name}));
if ((strcmp(de->d_name, ".") != 0) &&
(strcmp(de->d_name, "..") != 0)) {
ret = copy_directory_recursively(
utils::path::combine(from_path, {de->d_name}),
utils::path::combine(to_path, {de->d_name}));
}
} else {
ret = copy_file(utils::path::combine(from_path, {de->d_name}),
@ -167,15 +165,16 @@ bool copy_directory_recursively(std::string from_path, std::string to_path) {
return ret;
}
bool create_full_directory_path(std::string path) {
auto create_full_directory_path(std::string path) -> bool {
#ifdef _WIN32
const auto unicode_path = utils::string::from_utf8(utils::path::absolute(path));
const auto unicode_path =
utils::string::from_utf8(utils::path::absolute(path));
return is_directory(path) ||
(::SHCreateDirectory(nullptr, unicode_path.c_str()) == ERROR_SUCCESS);
#else
auto ret = true;
const auto paths =
utils::string::split(utils::path::absolute(path), utils::path::directory_seperator[0u]);
const auto paths = utils::string::split(utils::path::absolute(path),
utils::path::directory_seperator[0u]);
std::string current_path;
for (std::size_t i = 0u; ret && (i < paths.size()); i++) {
if (paths[i].empty()) { // Skip root
@ -191,21 +190,22 @@ bool create_full_directory_path(std::string path) {
#endif
}
bool delete_directory(std::string path, const bool &recursive) {
auto delete_directory(std::string path, bool recursive) -> bool {
if (recursive) {
return delete_directory_recursively(path);
}
path = utils::path::absolute(path);
#ifdef _WIN32
return (not is_directory(path) ||
utils::retryable_action([&]() -> bool { return !!::RemoveDirectoryA(path.c_str()); }));
return (not is_directory(path) || utils::retryable_action([&]() -> bool {
return !!::RemoveDirectoryA(path.c_str());
}));
#else
return not is_directory(path) || (rmdir(path.c_str()) == 0);
#endif
}
bool delete_directory_recursively(std::string path) {
auto delete_directory_recursively(std::string path) -> bool {
path = utils::path::absolute(path);
#ifdef _WIN32
@ -213,14 +213,16 @@ bool delete_directory_recursively(std::string path) {
const auto search = utils::path::combine(path, {"*.*"});
auto find = ::FindFirstFile(search.c_str(), &fd);
if (find != INVALID_HANDLE_VALUE) {
auto res = false;
auto res = true;
do {
if (fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) {
if ((std::string(fd.cFileName) != ".") && (std::string(fd.cFileName) != "..")) {
res = delete_directory_recursively(utils::path::combine(path, {fd.cFileName}));
if ((std::string(fd.cFileName) != ".") &&
(std::string(fd.cFileName) != "..")) {
res = delete_directory_recursively(
utils::path::combine(path, {fd.cFileName}));
}
} else {
res = delete_file(utils::path::combine(path, {fd.cFileName}));
res = retry_delete_file(utils::path::combine(path, {fd.cFileName}));
}
} while (res && (::FindNextFile(find, &fd) != 0));
@ -234,10 +236,11 @@ bool delete_directory_recursively(std::string path) {
while (res && (de = readdir(root))) {
if (de->d_type == DT_DIR) {
if ((strcmp(de->d_name, ".") != 0) && (strcmp(de->d_name, "..") != 0)) {
res = delete_directory_recursively(utils::path::combine(path, {de->d_name}));
res = delete_directory_recursively(
utils::path::combine(path, {de->d_name}));
}
} else {
res = delete_file(utils::path::combine(path, {de->d_name}));
res = retry_delete_file(utils::path::combine(path, {de->d_name}));
}
}
@ -248,7 +251,7 @@ bool delete_directory_recursively(std::string path) {
return delete_directory(path, false);
}
bool delete_file(std::string path) {
auto delete_file(std::string path) -> bool {
path = utils::path::absolute(path);
#ifdef _WIN32
return (not is_file(path) || utils::retryable_action([&]() -> bool {
@ -264,18 +267,55 @@ bool delete_file(std::string path) {
#endif
}
std::string generate_sha256(const std::string &file_path) {
std::string value;
CryptoPP::SHA256 hash;
CryptoPP::FileSource(
file_path.c_str(), true,
new CryptoPP::HashFilter(hash, new CryptoPP::HexEncoder(new CryptoPP::StringSink(value))));
return value;
auto generate_sha256(const std::string &file_path) -> std::string {
crypto_hash_sha256_state state{};
auto res = crypto_hash_sha256_init(&state);
if (res != 0) {
throw std::runtime_error("failed to initialize sha256|" +
std::to_string(res));
}
native_file_ptr nf;
if (native_file::open(file_path, nf) != api_error::success) {
throw std::runtime_error("failed to open file|" + file_path);
}
{
data_buffer buffer(1048576u);
std::uint64_t read_offset = 0u;
std::size_t bytes_read = 0u;
while (
nf->read_bytes(buffer.data(), buffer.size(), read_offset, bytes_read)) {
if (not bytes_read) {
break;
}
read_offset += bytes_read;
res = crypto_hash_sha256_update(
&state, reinterpret_cast<const unsigned char *>(buffer.data()),
bytes_read);
if (res != 0) {
nf->close();
throw std::runtime_error("failed to update sha256|" +
std::to_string(res));
}
}
nf->close();
}
std::array<unsigned char, crypto_hash_sha256_BYTES> out{};
res = crypto_hash_sha256_final(&state, out.data());
if (res != 0) {
throw std::runtime_error("failed to finalize sha256|" +
std::to_string(res));
}
return utils::to_hex_string(out);
}
std::uint64_t get_available_drive_space(const std::string &path) {
auto get_free_drive_space(const std::string &path) -> std::uint64_t {
#ifdef _WIN32
ULARGE_INTEGER li = {0};
ULARGE_INTEGER li{};
::GetDiskFreeSpaceEx(path.c_str(), &li, nullptr, nullptr);
return li.QuadPart;
#endif
@ -290,12 +330,33 @@ std::uint64_t get_available_drive_space(const std::string &path) {
#if __APPLE__
struct statvfs st {};
statvfs(path.c_str(), &st);
return st.f_bavail * st.f_frsize;
return st.f_bfree * st.f_frsize;
#endif
}
std::deque<std::string> get_directory_files(std::string path, const bool &oldest_first,
const bool &recursive) {
auto get_total_drive_space(const std::string &path) -> std::uint64_t {
#ifdef _WIN32
ULARGE_INTEGER li{};
::GetDiskFreeSpaceEx(path.c_str(), nullptr, &li, nullptr);
return li.QuadPart;
#endif
#ifdef __linux__
std::uint64_t ret = 0;
struct statfs64 st {};
if (statfs64(path.c_str(), &st) == 0) {
ret = st.f_blocks * st.f_bsize;
}
return ret;
#endif
#if __APPLE__
struct statvfs st {};
statvfs(path.c_str(), &st);
return st.f_blocks * st.f_frsize;
#endif
}
auto get_directory_files(std::string path, bool oldest_first, bool recursive)
-> std::deque<std::string> {
path = utils::path::absolute(path);
std::deque<std::string> ret;
std::unordered_map<std::string, std::uint64_t> lookup;
@ -305,20 +366,28 @@ std::deque<std::string> get_directory_files(std::string path, const bool &oldest
auto find = ::FindFirstFile(search.c_str(), &fd);
if (find != INVALID_HANDLE_VALUE) {
do {
if ((fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != FILE_ATTRIBUTE_DIRECTORY) {
ULARGE_INTEGER li{};
li.HighPart = fd.ftLastWriteTime.dwHighDateTime;
li.LowPart = fd.ftLastWriteTime.dwLowDateTime;
const auto full_path = utils::path::combine(path, {fd.cFileName});
lookup[full_path] = li.QuadPart;
ret.emplace_back(full_path);
const auto full_path = utils::path::combine(path, {fd.cFileName});
if ((fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) ==
FILE_ATTRIBUTE_DIRECTORY) {
if (recursive) {
const auto sub_files =
get_directory_files(full_path, oldest_first, recursive);
ret.insert(ret.end(), sub_files.begin(), sub_files.end());
} else {
ULARGE_INTEGER li{};
li.HighPart = fd.ftLastWriteTime.dwHighDateTime;
li.LowPart = fd.ftLastWriteTime.dwLowDateTime;
lookup[full_path] = li.QuadPart;
ret.emplace_back(full_path);
}
}
} while (::FindNextFile(find, &fd) != 0);
::FindClose(find);
std::sort(ret.begin(), ret.end(), [&](const auto &p1, const auto &p2) -> bool {
return (oldest_first != 0) == (lookup[p1] < lookup[p2]);
});
std::sort(ret.begin(), ret.end(),
[&](const auto &p1, const auto &p2) -> bool {
return (oldest_first != 0) == (lookup[p1] < lookup[p2]);
});
}
#else
auto *root = opendir(path.c_str());
@ -327,9 +396,10 @@ std::deque<std::string> get_directory_files(std::string path, const bool &oldest
while ((de = readdir(root)) != nullptr) {
if (de->d_type == DT_DIR) {
if (recursive) {
const auto subFiles = get_directory_files(utils::path::combine(path, {de->d_name}),
oldest_first, recursive);
ret.insert(ret.end(), subFiles.begin(), subFiles.end());
const auto sub_files =
get_directory_files(utils::path::combine(path, {de->d_name}),
oldest_first, recursive);
ret.insert(ret.end(), sub_files.begin(), sub_files.end());
}
} else {
ret.emplace_back(utils::path::combine(path, {de->d_name}));
@ -343,25 +413,28 @@ std::deque<std::string> get_directory_files(std::string path, const bool &oldest
stat(lookup_path.c_str(), &st);
#ifdef __APPLE__
lookup[lookup_path] = static_cast<std::uint64_t>(
(st.st_mtimespec.tv_sec * NANOS_PER_SECOND) + st.st_mtimespec.tv_nsec);
(st.st_mtimespec.tv_sec * NANOS_PER_SECOND) +
st.st_mtimespec.tv_nsec);
#else
lookup[lookup_path] =
static_cast<std::uint64_t>((st.st_mtim.tv_sec * NANOS_PER_SECOND) + st.st_mtim.tv_nsec);
lookup[lookup_path] = static_cast<std::uint64_t>(
(st.st_mtim.tv_sec * NANOS_PER_SECOND) + st.st_mtim.tv_nsec);
#endif
}
};
std::sort(ret.begin(), ret.end(), [&](const auto &p1, const auto &p2) -> bool {
add_to_lookup(p1);
add_to_lookup(p2);
return (oldest_first != 0) == (lookup[p1] < lookup[p2]);
});
std::sort(ret.begin(), ret.end(),
[&](const auto &p1, const auto &p2) -> bool {
add_to_lookup(p1);
add_to_lookup(p2);
return (oldest_first != 0) == (lookup[p1] < lookup[p2]);
});
}
#endif
return ret;
}
bool get_accessed_time(const std::string &path, std::uint64_t &accessed) {
auto get_accessed_time(const std::string &path, std::uint64_t &accessed)
-> bool {
auto ret = false;
accessed = 0;
#ifdef _WIN32
@ -372,10 +445,11 @@ bool get_accessed_time(const std::string &path, std::uint64_t &accessed) {
struct stat st {};
if (stat(path.c_str(), &st) != -1) {
#ifdef __APPLE__
accessed = static_cast<uint64_t>(st.st_atimespec.tv_nsec +
(st.st_atimespec.tv_sec * NANOS_PER_SECOND));
accessed = static_cast<uint64_t>(
st.st_atimespec.tv_nsec + (st.st_atimespec.tv_sec * NANOS_PER_SECOND));
#else
accessed = static_cast<uint64_t>(st.st_atim.tv_nsec + (st.st_atim.tv_sec * NANOS_PER_SECOND));
accessed = static_cast<uint64_t>(st.st_atim.tv_nsec +
(st.st_atim.tv_sec * NANOS_PER_SECOND));
#endif
#endif
ret = true;
@ -384,7 +458,8 @@ bool get_accessed_time(const std::string &path, std::uint64_t &accessed) {
return ret;
}
bool get_modified_time(const std::string &path, std::uint64_t &modified) {
auto get_modified_time(const std::string &path, std::uint64_t &modified)
-> bool {
auto ret = false;
modified = 0u;
#ifdef _WIN32
@ -395,10 +470,11 @@ bool get_modified_time(const std::string &path, std::uint64_t &modified) {
struct stat st {};
if (stat(path.c_str(), &st) != -1) {
#ifdef __APPLE__
modified = static_cast<uint64_t>(st.st_mtimespec.tv_nsec +
(st.st_mtimespec.tv_sec * NANOS_PER_SECOND));
modified = static_cast<uint64_t>(
st.st_mtimespec.tv_nsec + (st.st_mtimespec.tv_sec * NANOS_PER_SECOND));
#else
modified = static_cast<uint64_t>(st.st_mtim.tv_nsec + (st.st_mtim.tv_sec * NANOS_PER_SECOND));
modified = static_cast<uint64_t>(st.st_mtim.tv_nsec +
(st.st_mtim.tv_sec * NANOS_PER_SECOND));
#endif
#endif
ret = true;
@ -407,7 +483,7 @@ bool get_modified_time(const std::string &path, std::uint64_t &modified) {
return ret;
}
bool get_file_size(std::string path, std::uint64_t &file_size) {
auto get_file_size(std::string path, std::uint64_t &file_size) -> bool {
file_size = 0u;
path = utils::path::finalize(path);
@ -433,7 +509,7 @@ bool get_file_size(std::string path, std::uint64_t &file_size) {
return (st.st_size >= 0);
}
bool is_directory(const std::string &path) {
auto is_directory(const std::string &path) -> bool {
#ifdef _WIN32
return ::PathIsDirectory(path.c_str()) != 0;
#else
@ -442,36 +518,43 @@ bool is_directory(const std::string &path) {
#endif
}
bool is_file(const std::string &path) {
auto is_file(const std::string &path) -> bool {
#ifdef _WIN32
return (::PathFileExists(path.c_str()) && not ::PathIsDirectory(path.c_str()));
return (::PathFileExists(path.c_str()) &&
not ::PathIsDirectory(path.c_str()));
#else
struct stat st {};
return (not stat(path.c_str(), &st) && not S_ISDIR(st.st_mode));
#endif
}
bool is_modified_date_older_than(const std::string &path, const std::chrono::hours &hours) {
auto is_modified_date_older_than(const std::string &path,
const std::chrono::hours &hours) -> bool {
auto ret = false;
std::uint64_t modified = 0;
if (get_modified_time(path, modified)) {
const auto seconds = std::chrono::duration_cast<std::chrono::seconds>(hours);
const auto seconds =
std::chrono::duration_cast<std::chrono::seconds>(hours);
#ifdef _WIN32
return (std::chrono::system_clock::from_time_t(modified) + seconds) <
std::chrono::system_clock::now();
#else
return (modified + (seconds.count() * NANOS_PER_SECOND)) < utils::get_time_now();
return (modified + (seconds.count() * NANOS_PER_SECOND)) <
utils::get_time_now();
#endif
}
return ret;
}
bool move_file(std::string from, std::string to) {
auto move_file(std::string from, std::string to) -> bool {
from = utils::path::finalize(from);
to = utils::path::finalize(to);
const auto directory = utils::path::remove_file_name(to);
create_full_directory_path(directory);
if (not create_full_directory_path(directory)) {
return false;
}
#ifdef _WIN32
const bool ret = ::MoveFile(from.c_str(), to.c_str()) != 0;
#else
@ -481,24 +564,7 @@ bool move_file(std::string from, std::string to) {
return ret;
}
api_error read_from_source(filesystem_item &fi, const std::size_t &read_size,
const std::uint64_t &read_offset, std::vector<char> &data) {
native_file_ptr nf;
const auto res = assign_and_get_native_file(fi, nf);
if (res != api_error::success) {
return res;
}
data.resize(read_size);
std::size_t bytes_read = 0u;
if (not nf->read_bytes(&data[0u], data.size(), read_offset, bytes_read)) {
return api_error::os_error;
}
return api_error::success;
}
std::vector<std::string> read_file_lines(const std::string &path) {
auto read_file_lines(const std::string &path) -> std::vector<std::string> {
std::vector<std::string> ret;
if (is_file(path)) {
std::ifstream fs(path);
@ -512,30 +578,39 @@ std::vector<std::string> read_file_lines(const std::string &path) {
return ret;
}
bool read_json_file(const std::string &path, json &data) {
auto ret = false;
auto read_json_file(const std::string &path, json &data) -> bool {
if (not utils::file::is_file(path)) {
return true;
}
try {
std::ifstream file_stream(path.c_str());
if (file_stream.is_open()) {
std::stringstream ss;
ss << file_stream.rdbuf();
std::string json_text = ss.str();
if (json_text.empty()) {
try {
std::stringstream ss;
ss << file_stream.rdbuf();
std::string json_text = ss.str();
if (not json_text.empty()) {
data = json::parse(json_text.c_str());
}
file_stream.close();
} else {
data = json::parse(json_text.c_str());
ret = true;
return true;
} catch (const std::exception &e) {
file_stream.close();
throw e;
}
}
} catch (const std::exception &) {
} catch (const std::exception &e) {
utils::error::raise_error(__FUNCTION__, e, path,
"failed to read json file");
}
return ret;
return false;
}
bool reset_modified_time(const std::string &path) {
auto reset_modified_time(const std::string &path) -> bool {
auto ret = false;
#ifdef _WIN32
SYSTEMTIME st{};
@ -543,9 +618,10 @@ bool reset_modified_time(const std::string &path) {
FILETIME ft{};
if ((ret = !!::SystemTimeToFileTime(&st, &ft))) {
auto handle = ::CreateFileA(path.c_str(), FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES,
FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr,
OPEN_EXISTING, 0, nullptr);
auto handle = ::CreateFileA(
path.c_str(), FILE_READ_ATTRIBUTES | FILE_WRITE_ATTRIBUTES,
FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr,
OPEN_EXISTING, 0, nullptr);
if ((ret = (handle != INVALID_HANDLE_VALUE))) {
ret = !!::SetFileTime(handle, nullptr, &ft, &ft);
::CloseHandle(handle);
@ -561,21 +637,27 @@ bool reset_modified_time(const std::string &path) {
return ret;
}
api_error truncate_source(filesystem_item &fi, const std::uint64_t &size) {
native_file_ptr nf;
auto res = assign_and_get_native_file(fi, nf);
if (res != api_error::success) {
return res;
auto retry_delete_directory(const std::string &dir) -> bool {
auto deleted = false;
for (std::uint8_t i = 0u; not(deleted = delete_directory(dir)) && (i < 200u);
i++) {
std::this_thread::sleep_for(10ms);
}
if (not nf->truncate(size)) {
return api_error::os_error;
}
return api_error::success;
return deleted;
}
bool write_json_file(const std::string &path, const json &j) {
auto retry_delete_file(const std::string &file) -> bool {
auto deleted = false;
for (std::uint8_t i = 0u; not(deleted = delete_file(file)) && (i < 200u);
i++) {
std::this_thread::sleep_for(10ms);
}
return deleted;
}
auto write_json_file(const std::string &path, const json &j) -> bool {
std::string data;
{
std::stringstream ss;
@ -586,27 +668,11 @@ bool write_json_file(const std::string &path, const json &j) {
auto ret = (native_file::create_or_open(path, nf) == api_error::success);
if (ret) {
std::size_t bytes_written = 0u;
ret = nf->truncate(0) && nf->write_bytes(&data[0u], data.size(), 0u, bytes_written);
ret = nf->truncate(0) &&
nf->write_bytes(&data[0u], data.size(), 0u, bytes_written);
nf->close();
}
return ret;
}
api_error write_to_source(filesystem_item &fi, const std::uint64_t &write_offset,
const std::vector<char> &data, std::size_t &bytes_written) {
bytes_written = 0u;
native_file_ptr nf;
auto res = assign_and_get_native_file(fi, nf);
if (res != api_error::success) {
return res;
}
if (not nf->write_bytes(&data[0u], data.size(), write_offset, bytes_written)) {
return api_error::os_error;
}
return api_error::success;
}
} // namespace repertory::utils::file

40
src/utils/global_data.cpp
View File

@ -1,40 +0,0 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/
#include "utils/global_data.hpp"
namespace repertory {
global_data global_data::instance_;
void global_data::update_used_space(const std::uint64_t &file_size,
const std::uint64_t &new_file_size, const bool &cache_only) {
if (file_size > new_file_size) {
const auto diff = (file_size - new_file_size);
used_cache_space_ -= diff;
if (not cache_only) {
used_drive_space_ -= diff;
}
} else if (file_size < new_file_size) {
const auto diff = (new_file_size - file_size);
used_cache_space_ += diff;
if (not cache_only) {
used_drive_space_ += diff;
}
}
}
} // namespace repertory

153
src/utils/native_file.cpp
View File

@ -1,26 +1,41 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/native_file.hpp"
#include "types/repertory.hpp"
#include "utils/string_utils.hpp"
#include "utils/unix/unix_utils.hpp"
#include "utils/utils.hpp"
namespace repertory {
native_file_ptr native_file::clone(const native_file_ptr &nf) {
auto native_file::get_handle() -> native_handle { return handle_; }
native_file::~native_file() {
if (auto_close) {
close();
}
}
auto native_file::clone(const native_file_ptr &nf) -> native_file_ptr {
std::string source_path;
#ifdef _WIN32
@ -31,53 +46,79 @@ native_file_ptr native_file::clone(const native_file_ptr &nf) {
source_path.resize(PATH_MAX + 1);
#ifdef __APPLE__
fcntl(nf->get_handle(), F_GETPATH, &source_path[0u]);
a
#else
readlink(("/proc/self/fd/" + utils::string::from_int64(nf->get_handle())).c_str(),
readlink(("/proc/self/fd/" + std::to_string(nf->get_handle())).c_str(),
&source_path[0u], source_path.size());
#endif
#endif
source_path = source_path.c_str();
native_file_ptr clone;
if (native_file::open(source_path, clone) != api_error::success) {
throw std::runtime_error("unable to clone file " + source_path);
auto res = native_file::open(source_path, clone);
if (res != api_error::success) {
throw std::runtime_error("unable to open file|sp|" + source_path + "|err|" +
api_error_to_string(res));
}
return clone;
}
api_error native_file::create_or_open(const std::string &source_path, native_file_ptr &nf) {
auto native_file::create_or_open(const std::string &source_path,
[[maybe_unused]] bool should_chmod,
native_file_ptr &nf) -> api_error {
#ifdef _WIN32
auto handle = ::CreateFileA(source_path.c_str(), GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, OPEN_ALWAYS,
FILE_FLAG_RANDOM_ACCESS, nullptr);
FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr,
OPEN_ALWAYS, FILE_FLAG_RANDOM_ACCESS, nullptr);
#else
auto handle = ::open(source_path.c_str(), O_CREAT | O_RDWR | O_CLOEXEC, 0600u);
chmod(source_path.c_str(), 0600u);
auto handle = should_chmod ? ::open(source_path.c_str(),
O_CREAT | O_RDWR | O_CLOEXEC, 0600u)
: ::open(source_path.c_str(), O_RDWR | O_CLOEXEC);
if (should_chmod) {
chmod(source_path.c_str(), 0600u);
}
#endif
nf = native_file::attach(handle);
return ((handle == REPERTORY_INVALID_HANDLE) ? api_error::os_error : api_error::success);
return ((handle == REPERTORY_INVALID_HANDLE) ? api_error::os_error
: api_error::success);
}
api_error native_file::open(const std::string &source_path, native_file_ptr &nf) {
auto native_file::create_or_open(const std::string &source_path,
native_file_ptr &nf) -> api_error {
return create_or_open(source_path, true, nf);
}
auto native_file::open(const std::string &source_path, native_file_ptr &nf)
-> api_error {
return open(source_path, true, nf);
}
auto native_file::open(const std::string &source_path,
[[maybe_unused]] bool should_chmod, native_file_ptr &nf)
-> api_error {
#ifdef _WIN32
auto handle = ::CreateFileA(source_path.c_str(), GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr, OPEN_EXISTING,
FILE_FLAG_RANDOM_ACCESS, nullptr);
FILE_SHARE_READ | FILE_SHARE_WRITE, nullptr,
OPEN_EXISTING, FILE_FLAG_RANDOM_ACCESS, nullptr);
#else
auto handle = ::open(source_path.c_str(), O_RDWR | O_CLOEXEC, 0600u);
chmod(source_path.c_str(), 0600u);
auto handle = should_chmod
? ::open(source_path.c_str(), O_RDWR | O_CLOEXEC, 0600u)
: ::open(source_path.c_str(), O_RDONLY | O_CLOEXEC);
if (should_chmod) {
chmod(source_path.c_str(), 0600u);
}
#endif
nf = native_file::attach(handle);
return ((handle == REPERTORY_INVALID_HANDLE) ? api_error::os_error : api_error::success);
return ((handle == REPERTORY_INVALID_HANDLE) ? api_error::os_error
: api_error::success);
}
bool native_file::allocate(const std::uint64_t &file_size) {
auto native_file::allocate(std::uint64_t file_size) -> bool {
#ifdef _WIN32
LARGE_INTEGER li{};
li.QuadPart = file_size;
return (::SetFilePointerEx(handle_, li, nullptr, FILE_BEGIN) && ::SetEndOfFile(handle_));
return (::SetFilePointerEx(handle_, li, nullptr, FILE_BEGIN) &&
::SetEndOfFile(handle_));
#endif
#ifdef __linux__
return (fallocate(handle_, 0, 0, file_size) >= 0);
@ -98,16 +139,17 @@ void native_file::close() {
}
}
bool native_file::copy_from(const native_file_ptr &nf) {
auto native_file::copy_from(const native_file_ptr &nf) -> bool {
auto ret = false;
std::uint64_t file_size = 0u;
if ((ret = nf->get_file_size(file_size))) {
std::vector<char> buffer;
data_buffer buffer;
buffer.resize(65536u * 2u);
std::uint64_t offset = 0u;
while (ret && (file_size > 0u)) {
std::size_t bytes_read{};
if ((ret = nf->read_bytes(&buffer[0u], buffer.size(), offset, bytes_read))) {
if ((ret = nf->read_bytes(&buffer[0u], buffer.size(), offset,
bytes_read))) {
std::size_t bytes_written = 0u;
ret = write_bytes(&buffer[0u], bytes_read, offset, bytes_written);
file_size -= bytes_read;
@ -120,7 +162,7 @@ bool native_file::copy_from(const native_file_ptr &nf) {
return ret;
}
bool native_file::copy_from(const std::string &path) {
auto native_file::copy_from(const std::string &path) -> bool {
auto ret = false;
native_file_ptr nf;
if (native_file::create_or_open(path, nf) == api_error ::success) {
@ -140,7 +182,7 @@ void native_file::flush() {
#endif
}
bool native_file::get_file_size(std::uint64_t &file_size) {
auto native_file::get_file_size(std::uint64_t &file_size) -> bool {
auto ret = false;
#ifdef _WIN32
LARGE_INTEGER li{};
@ -164,8 +206,9 @@ bool native_file::get_file_size(std::uint64_t &file_size) {
}
#ifdef _WIN32
bool native_file::read_bytes(char *buffer, const std::size_t &read_size,
const std::uint64_t &read_offset, std::size_t &bytes_read) {
auto native_file::read_bytes(char *buffer, std::size_t read_size,
std::uint64_t read_offset, std::size_t &bytes_read)
-> bool {
recur_mutex_lock l(read_write_mutex_);
auto ret = false;
@ -176,7 +219,8 @@ bool native_file::read_bytes(char *buffer, const std::size_t &read_size,
DWORD current_read = 0u;
do {
current_read = 0u;
ret = !!::ReadFile(handle_, &buffer[bytes_read], static_cast<DWORD>(read_size - bytes_read),
ret = !!::ReadFile(handle_, &buffer[bytes_read],
static_cast<DWORD>(read_size - bytes_read),
&current_read, nullptr);
bytes_read += current_read;
} while (ret && (bytes_read < read_size) && (current_read != 0));
@ -189,13 +233,14 @@ bool native_file::read_bytes(char *buffer, const std::size_t &read_size,
return ret;
}
#else
bool native_file::read_bytes(char *buffer, const std::size_t &read_size,
const std::uint64_t &read_offset, std::size_t &bytes_read) {
auto native_file::read_bytes(char *buffer, std::size_t read_size,
std::uint64_t read_offset, std::size_t &bytes_read)
-> bool {
bytes_read = 0u;
ssize_t result = 0;
do {
result =
pread64(handle_, &buffer[bytes_read], read_size - bytes_read, read_offset + bytes_read);
result = pread64(handle_, &buffer[bytes_read], read_size - bytes_read,
read_offset + bytes_read);
if (result > 0) {
bytes_read += static_cast<size_t>(result);
}
@ -204,20 +249,22 @@ bool native_file::read_bytes(char *buffer, const std::size_t &read_size,
return (result >= 0);
}
#endif
bool native_file::truncate(const std::uint64_t &file_size) {
auto native_file::truncate(std::uint64_t file_size) -> bool {
#ifdef _WIN32
recur_mutex_lock l(read_write_mutex_);
LARGE_INTEGER li{};
li.QuadPart = file_size;
return (::SetFilePointerEx(handle_, li, nullptr, FILE_BEGIN) && ::SetEndOfFile(handle_));
return (::SetFilePointerEx(handle_, li, nullptr, FILE_BEGIN) &&
::SetEndOfFile(handle_));
#else
return (ftruncate(handle_, file_size) >= 0);
#endif
}
#ifdef _WIN32
bool native_file::write_bytes(const char *buffer, const std::size_t &write_size,
const std::uint64_t &write_offset, std::size_t &bytes_written) {
auto native_file::write_bytes(const char *buffer, std::size_t write_size,
std::uint64_t write_offset,
std::size_t &bytes_written) -> bool {
recur_mutex_lock l(read_write_mutex_);
bytes_written = 0u;
@ -229,7 +276,8 @@ bool native_file::write_bytes(const char *buffer, const std::size_t &write_size,
do {
DWORD current_write = 0u;
ret = !!::WriteFile(handle_, &buffer[bytes_written],
static_cast<DWORD>(write_size - bytes_written), &current_write, nullptr);
static_cast<DWORD>(write_size - bytes_written),
&current_write, nullptr);
bytes_written += current_write;
} while (ret && (bytes_written < write_size));
}
@ -237,13 +285,14 @@ bool native_file::write_bytes(const char *buffer, const std::size_t &write_size,
return ret;
}
#else
bool native_file::write_bytes(const char *buffer, const std::size_t &write_size,
const std::uint64_t &write_offset, std::size_t &bytes_written) {
auto native_file::write_bytes(const char *buffer, std::size_t write_size,
std::uint64_t write_offset,
std::size_t &bytes_written) -> bool {
bytes_written = 0;
ssize_t result = 0;
do {
result = pwrite64(handle_, &buffer[bytes_written], write_size - bytes_written,
write_offset + bytes_written);
result = pwrite64(handle_, &buffer[bytes_written],
write_size - bytes_written, write_offset + bytes_written);
if (result > 0) {
bytes_written += static_cast<size_t>(result);
}

133
src/utils/path_utils.cpp
View File

@ -1,31 +1,33 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/path_utils.hpp"
#include "types/repertory.hpp"
#include "utils/error_utils.hpp"
#include "utils/string_utils.hpp"
#include "utils/utils.hpp"
namespace repertory::utils::path {
static std::string &format_path(std::string &path, const std::string &sep,
const std::string &not_sep);
std::string absolute(std::string path) {
auto absolute(std::string path) -> std::string {
#ifdef _WIN32
if (not path.empty() && ::PathIsRelative(&path[0u])) {
std::string temp;
@ -51,35 +53,61 @@ std::string absolute(std::string path) {
}
#endif
return path;
return finalize(path);
}
std::string combine(std::string path, const std::vector<std::string> &paths) {
auto combine(std::string path, const std::vector<std::string> &paths)
-> std::string {
return finalize(
std::accumulate(paths.begin(), paths.end(), path, [](std::string path, const auto &pathPart) {
path += (directory_seperator + pathPart);
return path;
}));
std::accumulate(paths.begin(), paths.end(), path,
[](std::string next_path, const auto &path_part) {
if (not next_path.empty()) {
next_path += (directory_seperator + path_part);
}
return next_path;
}));
}
std::string create_api_path(std::string path) {
if (path.empty() || (path == ".")) {
path = "/";
} else {
format_path(path, "/", "\\");
if (path.find("./") == 0) {
path = path.substr(1);
}
if (path[0u] != '/') {
path = "/" + path;
}
auto create_api_path(std::string path) -> std::string {
if (path.empty() || (path == ".") || (path == "/")) {
return "/";
}
format_path(path, "/", "\\");
if (path.find("./") == 0) {
path = path.substr(1);
}
if (path[0u] != '/') {
path = "/" + path;
}
if ((path != "/") && utils::string::ends_with(path, "/")) {
utils::string::right_trim(path, '/');
}
return path;
}
static std::string &format_path(std::string &path, const std::string &sep,
const std::string &not_sep) {
auto finalize(std::string path) -> std::string {
format_path(path, not_directory_seperator, directory_seperator);
format_path(path, directory_seperator, not_directory_seperator);
if ((path.size() > 1u) &&
(path[path.size() - 1u] == directory_seperator[0u])) {
path = path.substr(0u, path.size() - 1u);
}
#ifdef _WIN32
if ((path.size() >= 2u) && (path[1u] == ':')) {
path[0u] = utils::string::to_lower(std::string(1u, path[0u]))[0u];
}
#endif
return path;
}
auto format_path(std::string &path, const std::string &sep,
const std::string &not_sep) -> std::string & {
std::replace(path.begin(), path.end(), not_sep[0u], sep[0u]);
while (utils::string::contains(path, sep + sep)) {
@ -89,16 +117,7 @@ static std::string &format_path(std::string &path, const std::string &sep,
return path;
}
std::string finalize(std::string path) {
format_path(path, directory_seperator, not_directory_seperator);
if ((path.size() > 1) && (path[path.size() - 1] == directory_seperator[0u])) {
path = path.substr(0, path.size() - 1);
}
return path;
}
std::string get_parent_api_path(const std::string &path) {
auto get_parent_api_path(const std::string &path) -> std::string {
std::string ret;
if (path != "/") {
ret = path.substr(0, path.rfind('/') + 1);
@ -111,7 +130,7 @@ std::string get_parent_api_path(const std::string &path) {
}
#ifndef _WIN32
std::string get_parent_directory(std::string path) {
auto get_parent_directory(std::string path) -> std::string {
auto ret = std::string(dirname(&path[0u]));
if (ret == ".") {
ret = "/";
@ -121,7 +140,7 @@ std::string get_parent_directory(std::string path) {
}
#endif
bool is_ads_file_path(const std::string &path) {
auto is_ads_file_path([[maybe_unused]] const std::string &path) -> bool {
#ifdef _WIN32
return utils::string::contains(path, ":");
#else
@ -129,7 +148,7 @@ bool is_ads_file_path(const std::string &path) {
#endif
}
bool is_trash_directory(std::string path) {
auto is_trash_directory(std::string path) -> bool {
path = create_api_path(utils::string::to_lower(path));
if (utils::string::begins_with(path, "/.trash-") ||
utils::string::begins_with(path, "/.trashes") ||
@ -139,7 +158,7 @@ bool is_trash_directory(std::string path) {
return false;
}
std::string remove_file_name(std::string path) {
auto remove_file_name(std::string path) -> std::string {
path = finalize(path);
#ifdef _WIN32
@ -160,18 +179,20 @@ std::string remove_file_name(std::string path) {
}
#ifndef _WIN32
std::string resolve(std::string path) {
struct passwd *pw = getpwuid(getuid());
std::string home = (pw->pw_dir ? pw->pw_dir : "");
if (home.empty() || ((home == "/") && (getuid() != 0))) {
home = combine("/home", {pw->pw_name});
}
auto resolve(std::string path) -> std::string {
std::string home{};
use_getpwuid(getuid(), [&home](struct passwd *pw) {
home = (pw->pw_dir ? pw->pw_dir : "");
if (home.empty() || ((home == "/") && (getuid() != 0))) {
home = combine("/home", {pw->pw_name});
}
});
return finalize(utils::string::replace(path, "~", home));
}
#endif
std::string strip_to_file_name(std::string path) {
auto strip_to_file_name(std::string path) -> std::string {
#ifdef _WIN32
return ::PathFindFileName(&path[0u]);
#else

104
src/utils/polling.cpp
View File

@ -1,44 +1,57 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/polling.hpp"
#include "app_config.hpp"
namespace repertory {
polling polling::instance_;
void polling::frequency_thread(std::function<std::uint32_t()> get_frequency_seconds,
bool low_frequency) {
void polling::frequency_thread(
std::function<std::uint32_t()> get_frequency_seconds, frequency freq) {
while (not stop_requested_) {
std::deque<std::future<void>> futures;
unique_mutex_lock l(mutex_);
if (not stop_requested_ && notify_.wait_for(l, std::chrono::seconds(get_frequency_seconds())) ==
std::cv_status::timeout) {
for (auto &kv : items_) {
if (kv.second.low_frequency == low_frequency) {
futures.emplace_back(std::async(std::launch::async, [kv]() -> void {
event_system::instance().raise<polling_item_begin>(kv.first);
kv.second.action();
event_system::instance().raise<polling_item_end>(kv.first);
}));
if (not stop_requested_ &&
notify_.wait_for(l, std::chrono::seconds(get_frequency_seconds())) ==
std::cv_status::timeout) {
for (const auto &kv : items_) {
if (kv.second.freq == freq) {
futures.emplace_back(
std::async(std::launch::async, [this, &freq, kv]() -> void {
if (config_->get_event_level() == event_level::verbose ||
freq != frequency::second) {
event_system::instance().raise<polling_item_begin>(kv.first);
}
kv.second.action();
if (config_->get_event_level() == event_level::verbose ||
freq != frequency::second) {
event_system::instance().raise<polling_item_end>(kv.first);
}
}));
}
}
l.unlock();
while (not futures.empty()) {
futures.front().wait();
futures.pop_front();
@ -52,40 +65,57 @@ void polling::remove_callback(const std::string &name) {
items_.erase(name);
}
void polling::set_callback(const polling_item &pollingItem) {
void polling::set_callback(const polling_item &pi) {
mutex_lock l(mutex_);
items_[pollingItem.name] = pollingItem;
items_[pi.name] = pi;
}
void polling::start(app_config *config) {
mutex_lock l(start_stop_mutex_);
if (not high_frequency_thread_) {
event_system::instance().raise<service_started>("polling");
config_ = config;
stop_requested_ = false;
high_frequency_thread_ = std::make_unique<std::thread>([this]() -> void {
this->frequency_thread(
[this]() -> std::uint32_t { return config_->get_high_frequency_interval_secs(); }, false);
[this]() -> std::uint32_t {
return config_->get_high_frequency_interval_secs();
},
frequency::high);
});
low_frequency_thread_ = std::make_unique<std::thread>([this]() -> void {
this->frequency_thread(
[this]() -> std::uint32_t { return config_->get_low_frequency_interval_secs(); }, true);
[this]() -> std::uint32_t {
return config_->get_low_frequency_interval_secs();
},
frequency::low);
});
second_frequency_thread_ = std::make_unique<std::thread>([this]() -> void {
this->frequency_thread([]() -> std::uint32_t { return 1U; },
frequency::second);
});
}
}
void polling::stop() {
mutex_lock l(start_stop_mutex_);
if (high_frequency_thread_) {
event_system::instance().raise<service_shutdown>("polling");
{
mutex_lock l2(mutex_);
stop_requested_ = true;
notify_.notify_all();
event_system::instance().raise<service_shutdown_begin>("polling");
mutex_lock l(start_stop_mutex_);
if (high_frequency_thread_) {
{
stop_requested_ = true;
mutex_lock l2(mutex_);
notify_.notify_all();
}
high_frequency_thread_->join();
low_frequency_thread_->join();
second_frequency_thread_->join();
high_frequency_thread_.reset();
low_frequency_thread_.reset();
second_frequency_thread_.reset();
}
high_frequency_thread_->join();
low_frequency_thread_->join();
high_frequency_thread_.reset();
low_frequency_thread_.reset();
event_system::instance().raise<service_shutdown_end>("polling");
}
}
} // namespace repertory

55
src/utils/rocksdb_utils.cpp
View File

@ -1,25 +1,30 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/rocksdb_utils.hpp"
#include "app_config.hpp"
#include "events/event_system.hpp"
#include "types/startup_exception.hpp"
#include "utils/error_utils.hpp"
#include "utils/path_utils.hpp"
namespace repertory::utils::db {
@ -30,35 +35,37 @@ void create_rocksdb(const app_config &config, const std::string &name,
options.db_log_dir = config.get_log_directory();
options.keep_log_file_num = 10;
rocksdb::DB *db_ptr = nullptr;
rocksdb::DB *db_ptr{};
const auto status = rocksdb::DB::Open(
options, utils::path::combine(config.get_data_directory(), {name}), &db_ptr);
options, utils::path::combine(config.get_data_directory(), {name}),
&db_ptr);
if (status.ok()) {
db.reset(db_ptr);
} else {
event_system::instance().raise<repertory_exception>(__FUNCTION__, status.ToString());
utils::error::raise_error(__FUNCTION__, status.ToString());
throw startup_exception(status.ToString());
}
}
void create_rocksdb(const app_config &config, const std::string &name,
const std::vector<rocksdb::ColumnFamilyDescriptor> &families,
std::vector<rocksdb::ColumnFamilyHandle *> &handles,
std::unique_ptr<rocksdb::DB> &db) {
void create_rocksdb(
const app_config &config, const std::string &name,
const std::vector<rocksdb::ColumnFamilyDescriptor> &families,
std::vector<rocksdb::ColumnFamilyHandle *> &handles,
std::unique_ptr<rocksdb::DB> &db) {
rocksdb::Options options{};
options.create_if_missing = true;
options.create_missing_column_families = true;
options.db_log_dir = config.get_log_directory();
options.keep_log_file_num = 10;
rocksdb::DB *db_ptr = nullptr;
const auto status =
rocksdb::DB::Open(options, utils::path::combine(config.get_data_directory(), {name}),
families, &handles, &db_ptr);
rocksdb::DB *db_ptr{};
const auto status = rocksdb::DB::Open(
options, utils::path::combine(config.get_data_directory(), {name}),
families, &handles, &db_ptr);
if (status.ok()) {
db.reset(db_ptr);
} else {
event_system::instance().raise<repertory_exception>(__FUNCTION__, status.ToString());
utils::error::raise_error(__FUNCTION__, status.ToString());
throw startup_exception(status.ToString());
}
}

65
src/utils/single_thread_service_base.cpp Normal file
View File

@ -0,0 +1,65 @@
/*
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/single_thread_service_base.hpp"
#include "events/event_system.hpp"
#include "events/events.hpp"
#include "types/repertory.hpp"
namespace repertory {
void single_thread_service_base::notify_all() const {
mutex_lock lock(get_mutex());
notify_.notify_all();
}
void single_thread_service_base::start() {
mutex_lock lock(mtx_);
if (not thread_) {
stop_requested_ = false;
on_start();
thread_ = std::make_unique<std::thread>([this]() {
event_system::instance().raise<service_started>(service_name_);
while (not stop_requested_) {
service_function();
}
});
}
}
void single_thread_service_base::stop() {
if (thread_) {
event_system::instance().raise<service_shutdown_begin>(service_name_);
unique_mutex_lock lock(mtx_);
if (thread_) {
stop_requested_ = true;
notify_.notify_all();
lock.unlock();
thread_->join();
thread_.reset();
on_stop();
}
event_system::instance().raise<service_shutdown_end>(service_name_);
}
}
} // namespace repertory

188
src/utils/string_utils.cpp
View File

@ -1,121 +1,129 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/string_utils.hpp"
#include "common.hpp"
namespace repertory::utils::string {
bool begins_with(const std::string &str, const std::string &val) { return (str.find(val) == 0); }
bool contains(const std::string &str, const std::string &search) {
return (str.find(search) != std::string::npos);
}
bool ends_with(const std::string &str, const std::string &val) {
/* constexpr c++20 */ auto ends_with(std::string_view str, std::string_view val)
-> bool {
if (val.size() > str.size()) {
return false;
}
return std::equal(val.rbegin(), val.rend(), str.rbegin());
}
std::string from_bool(const bool &val) { return std::to_string(val); }
auto from_bool(bool val) -> std::string { return std::to_string(val); }
std::string from_double(const double &value) { return std::to_string(value); }
std::string from_dynamic_bitset(const boost::dynamic_bitset<> &bitset) {
auto from_dynamic_bitset(const boost::dynamic_bitset<> &bitset) -> std::string {
std::stringstream ss;
boost::archive::text_oarchive archive(ss);
archive << bitset;
return ss.str();
}
std::string from_int32(const std::int32_t &val) { return std::to_string(val); }
std::string from_int64(const std::int64_t &val) { return std::to_string(val); }
std::string from_uint8(const std::uint8_t &val) { return std::to_string(val); }
std::string from_uint16(const std::uint16_t &val) { return std::to_string(val); }
std::string from_uint32(const std::uint32_t &val) { return std::to_string(val); }
std::string from_uint64(const std::uint64_t &val) { return std::to_string(val); }
std::wstring from_utf8(const std::string &str) {
return str.empty() ? L""
: std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t>().from_bytes(str);
auto from_utf8(const std::string &str) -> std::wstring {
return str.empty()
? L""
: std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t>()
.from_bytes(str);
}
bool is_numeric(const std::string &s) {
static const auto r = std::regex(R"(^(\+|\-)?(([0-9]*)|(([0-9]*)\.([0-9]*)))$)");
return std::regex_match(s, r);
/* constexpr c++20 */ auto is_numeric(std::string_view s) -> bool {
if ((s.length() > 1u) && (s[0u] == '+' || s[0u] == '-')) {
s = s.substr(1u);
}
if (s.empty()) {
return false;
}
auto has_decimal = false;
return std::find_if(
s.begin(), s.end(),
[&has_decimal](const std::string_view::value_type &c) -> bool {
if (has_decimal && c == '.') {
return true;
}
if ((has_decimal = has_decimal || c == '.')) {
return false;
}
return not std::isdigit(c);
}) == s.end();
}
std::string join(const std::vector<std::string> &arr, const char &delim) {
auto join(const std::vector<std::string> &arr, const char &delim)
-> std::string {
if (arr.empty()) {
return "";
}
return std::accumulate(std::next(arr.begin()), arr.end(), arr[0u],
[&delim](auto s, const auto &v) { return s + delim + v; });
return std::accumulate(
std::next(arr.begin()), arr.end(), arr[0u],
[&delim](auto s, const auto &v) { return s + delim + v; });
}
std::string &left_trim(std::string &s) { return left_trim(s, ' '); }
auto left_trim(std::string &s) -> std::string & { return left_trim(s, ' '); }
std::string &left_trim(std::string &s, const char &c) {
auto left_trim(std::string &s, const char &c) -> std::string & {
s.erase(0, s.find_first_not_of(c));
return s;
}
std::string &replace(std::string &src, const char &character, const char &with) {
auto replace(std::string &src, const char &character, const char &with)
-> std::string & {
std::replace(src.begin(), src.end(), character, with);
return src;
}
std::string &replace(std::string &src, const std::string &find, const std::string &with,
size_t startPos) {
if (!src.empty() && (startPos < src.size())) {
while ((startPos = src.find(find, startPos)) != std::string::npos) {
src.replace(startPos, find.size(), with);
startPos += with.size();
auto replace(std::string &src, const std::string &find, const std::string &with,
size_t start_pos) -> std::string & {
if (!src.empty() && (start_pos < src.size())) {
while ((start_pos = src.find(find, start_pos)) != std::string::npos) {
src.replace(start_pos, find.size(), with);
start_pos += with.size();
}
}
return src;
}
std::string replace_copy(std::string src, const char &character, const char &with) {
auto replace_copy(std::string src, const char &character, const char &with)
-> std::string {
std::replace(src.begin(), src.end(), character, with);
return src;
}
std::string replace_copy(std::string src, const std::string &find, const std::string &with,
size_t startPos) {
return replace(src, find, with, startPos);
auto replace_copy(std::string src, const std::string &find,
const std::string &with, size_t start_pos) -> std::string {
return replace(src, find, with, start_pos);
}
std::string &right_trim(std::string &s) { return right_trim(s, ' '); }
auto right_trim(std::string &s) -> std::string & { return right_trim(s, ' '); }
std::string &right_trim(std::string &s, const char &c) {
auto right_trim(std::string &s, const char &c) -> std::string & {
s.erase(s.find_last_not_of(c) + 1);
return s;
}
std::vector<std::string> split(const std::string &str, const char &delim, const bool &should_trim) {
auto split(const std::string &str, const char &delim, bool should_trim)
-> std::vector<std::string> {
std::vector<std::string> ret;
std::stringstream ss(str);
std::string item;
@ -125,7 +133,7 @@ std::vector<std::string> split(const std::string &str, const char &delim, const
return ret;
}
bool to_bool(std::string val) {
auto to_bool(std::string val) -> bool {
auto b = false;
trim(val);
@ -142,9 +150,9 @@ bool to_bool(std::string val) {
return b;
}
double to_double(const std::string &str) { return std::stod(str); }
auto to_double(const std::string &str) -> double { return std::stod(str); }
boost::dynamic_bitset<> to_dynamic_bitset(const std::string &val) {
auto to_dynamic_bitset(const std::string &val) -> boost::dynamic_bitset<> {
std::stringstream ss(val);
boost::dynamic_bitset<> bitset;
boost::archive::text_iarchive archive(ss);
@ -152,44 +160,64 @@ boost::dynamic_bitset<> to_dynamic_bitset(const std::string &val) {
return bitset;
}
std::int32_t to_int32(const std::string &val) { return std::stoi(val); }
auto to_int32(const std::string &val) -> std::int32_t { return std::stoi(val); }
std::int64_t to_int64(const std::string &val) { return std::stoll(val); }
auto to_int64(const std::string &val) -> std::int64_t {
return std::stoll(val);
}
std::string to_lower(std::string str) {
auto to_lower(std::string str) -> std::string {
std::transform(str.begin(), str.end(), str.begin(), ::tolower);
return str;
}
std::uint8_t to_uint8(const std::string &val) { return static_cast<std::uint8_t>(std::stoul(val)); }
auto to_size_t(const std::string &val) -> std::size_t {
return static_cast<std::size_t>(std::stoull(val));
}
std::uint16_t to_uint16(const std::string &val) {
auto to_uint8(const std::string &val) -> std::uint8_t {
return static_cast<std::uint8_t>(std::stoul(val));
}
auto to_uint16(const std::string &val) -> std::uint16_t {
return static_cast<std::uint16_t>(std::stoul(val));
}
std::uint32_t to_uint32(const std::string &val) {
auto to_uint32(const std::string &val) -> std::uint32_t {
return static_cast<std::uint32_t>(std::stoul(val));
}
std::uint64_t to_uint64(const std::string &val) { return std::stoull(val); }
auto to_uint64(const std::string &val) -> std::uint64_t {
return std::stoull(val);
}
std::string to_upper(std::string str) {
auto to_upper(std::string str) -> std::string {
std::transform(str.begin(), str.end(), str.begin(), ::toupper);
return str;
}
const std::string &to_utf8(const std::string &str) { return str; }
auto to_utf8(std::string str) -> std::string { return str; }
std::string to_utf8(const std::wstring &str) {
return str.empty() ? ""
: std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t>().to_bytes(str);
auto to_utf8(const std::wstring &str) -> std::string {
return str.empty()
? ""
: std::wstring_convert<std::codecvt_utf8<wchar_t>, wchar_t>()
.to_bytes(str);
}
std::string &trim(std::string &str) { return right_trim(left_trim(str)); }
auto trim(std::string &str) -> std::string & {
return right_trim(left_trim(str));
}
std::string &trim(std::string &str, const char &c) { return right_trim(left_trim(str, c), c); }
auto trim(std::string &str, const char &c) -> std::string & {
return right_trim(left_trim(str, c), c);
}
std::string trim_copy(std::string str) { return right_trim(left_trim(str)); }
auto trim_copy(std::string str) -> std::string {
return right_trim(left_trim(str));
}
std::string trim_copy(std::string str, const char &c) { return right_trim(left_trim(str, c), c); }
auto trim_copy(std::string str, const char &c) -> std::string {
return right_trim(left_trim(str, c), c);
}
} // namespace repertory::utils::string

28
src/utils/throttle.cpp
View File

@ -1,22 +1,26 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/throttle.hpp"
#include "types/repertory.hpp"
namespace repertory {

47
src/utils/timeout.cpp
View File

@ -1,22 +1,26 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/timeout.hpp"
#include "types/repertory.hpp"
namespace repertory {
@ -24,15 +28,16 @@ timeout::timeout(std::function<void()> timeout_callback,
const std::chrono::system_clock::duration &duration)
: timeout_killed_(duration == 0s) {
if (not timeout_killed_) {
timeout_thread_ = std::make_unique<std::thread>([this, duration, timeout_callback]() {
unique_mutex_lock lock(timeout_mutex_);
if (not timeout_killed_) {
timeout_notify_.wait_for(lock, duration);
if (not timeout_killed_) {
timeout_callback();
}
}
});
timeout_thread_ =
std::make_unique<std::thread>([this, duration, timeout_callback]() {
unique_mutex_lock lock(timeout_mutex_);
if (not timeout_killed_) {
timeout_notify_.wait_for(lock, duration);
if (not timeout_killed_) {
timeout_callback();
}
}
});
}
}

202
src/utils/unix/unix_utils.cpp
View File

@ -1,53 +1,39 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifndef _WIN32
#include "utils/unix/unix_utils.hpp"
#include "utils/error_utils.hpp"
#include "utils/utils.hpp"
namespace repertory::utils {
#ifndef __APPLE__
std::uint64_t convert_to_uint64(const pthread_t &t) { return static_cast<std::uint64_t>(t); }
#endif
int get_last_error_code() { return errno; }
std::uint64_t get_thread_id() { return convert_to_uint64(pthread_self()); }
bool is_uid_member_of_group(const uid_t &uid, const gid_t &gid) {
auto *pw = getpwuid(uid);
std::vector<gid_t> groups;
int group_count = 0;
if (getgrouplist(pw->pw_name, pw->pw_gid, nullptr, &group_count) < 0) {
groups.resize(static_cast<unsigned long>(group_count));
#ifdef __APPLE__
getgrouplist(pw->pw_name, pw->pw_gid, reinterpret_cast<int *>(&groups[0]), &group_count);
#else
getgrouplist(pw->pw_name, pw->pw_gid, &groups[0], &group_count);
#endif
}
return collection_includes(groups, gid);
auto convert_to_uint64(const pthread_t &t) -> std::uint64_t {
return static_cast<std::uint64_t>(t);
}
#endif
int translate_api_error(const api_error &e) {
auto from_api_error(const api_error &e) -> int {
switch (e) {
case api_error::access_denied:
return -EACCES;
@ -62,14 +48,14 @@ int translate_api_error(const api_error &e) {
case api_error::directory_not_found:
return -ENOTDIR;
case api_error::download_failed:
#if __APPLE__
#ifdef __APPLE__
return -EBADMSG;
#else
return -EREMOTEIO;
#endif
case api_error::error:
return -EIO;
case api_error::file_exists:
case api_error::item_exists:
return -EEXIST;
case api_error::file_in_use:
return -EBUSY;
@ -77,8 +63,10 @@ int translate_api_error(const api_error &e) {
return -EINVAL;
case api_error::item_not_found:
return -ENOENT;
case api_error::item_is_file:
return -ENOTDIR;
case api_error::out_of_memory:
return -ENOMEM;
case api_error::no_disk_space:
return -ENOSPC;
case api_error::os_error:
return -errno;
case api_error::permission_denied:
@ -90,17 +78,11 @@ int translate_api_error(const api_error &e) {
case api_error::not_implemented:
return -ENOSYS;
case api_error::upload_failed:
#if __APPLE__
return -EBADMSG;
#else
return -EREMOTEIO;
#endif
return -ENETDOWN;
case api_error::xattr_buffer_small:
return -ERANGE;
case api_error::xattr_exists:
return -EEXIST;
case api_error::xattr_invalid_namespace:
return -ENOTSUP;
case api_error::xattr_not_found:
#ifdef __APPLE__
return -ENOATTR;
@ -112,19 +94,109 @@ int translate_api_error(const api_error &e) {
return -ENAMETOOLONG;
#else
return -E2BIG;
#endif
#ifdef __APPLE__
case api_error::XAttrOSXInvalid:
return -EINVAL;
#endif
default:
return -EIO;
}
}
auto get_last_error_code() -> int { return errno; }
auto get_thread_id() -> std::uint64_t {
return convert_to_uint64(pthread_self());
}
auto is_uid_member_of_group(const uid_t &uid, const gid_t &gid) -> bool {
static const auto function_name = __FUNCTION__;
std::vector<gid_t> groups{};
use_getpwuid(uid, [&groups](struct passwd *pw) {
int group_count{};
if (getgrouplist(pw->pw_name, pw->pw_gid, nullptr, &group_count) < 0) {
groups.resize(static_cast<std::size_t>(group_count));
#ifdef __APPLE__
getgrouplist(pw->pw_name, pw->pw_gid,
reinterpret_cast<int *>(groups.data()), &group_count);
#else
getgrouplist(pw->pw_name, pw->pw_gid, groups.data(), &group_count);
#endif
}
});
return collection_includes(groups, gid);
}
auto to_api_error(int e) -> api_error {
switch (abs(e)) {
case 0:
return api_error::success;
case EBADF:
return api_error::invalid_handle;
case EACCES:
return api_error::access_denied;
case EFAULT:
return api_error::bad_address;
case EOF:
return api_error::directory_end_of_files;
case EISDIR:
return api_error::directory_exists;
case ENOTEMPTY:
return api_error::directory_not_empty;
case ENOTDIR:
return api_error::directory_not_found;
#ifdef __APPLE__
case EBADMSG:
return api_error::download_failed;
#else
case EREMOTEIO:
return api_error::download_failed;
#endif
case EIO:
return api_error::error;
case EEXIST:
return api_error::item_exists;
case EBUSY:
return api_error::file_in_use;
case EINVAL:
return api_error::invalid_operation;
case ENOENT:
return api_error::item_not_found;
case ENOMEM:
return api_error::out_of_memory;
case EPERM:
return api_error::permission_denied;
case ENOSPC:
return api_error::no_disk_space;
case ENOTSUP:
return api_error::not_supported;
case ENOSYS:
return api_error::not_implemented;
case ENETDOWN:
return api_error::upload_failed;
case ERANGE:
return api_error::xattr_buffer_small;
#ifdef __APPLE__
case ENOATTR:
return api_error::xattr_not_found;
#else
case ENODATA:
return api_error::xattr_not_found;
#endif
#ifdef __APPLE__
case ENAMETOOLONG:
return api_error::xattr_too_big;
#else
case E2BIG:
return api_error::xattr_too_big;
#endif
default:
return api_error::error;
}
}
void set_last_error_code(int error_code) { errno = error_code; }
std::int32_t unix_error_to_windows(const int &e) {
auto unix_error_to_windows(int e) -> std::int32_t {
switch (e) {
case 0:
return STATUS_SUCCESS;
@ -146,7 +218,7 @@ std::int32_t unix_error_to_windows(const int &e) {
case EIO:
return STATUS_UNEXPECTED_IO_ERROR;
case EISDIR:
return STATUS_FILE_IS_A_DIRECTORY;
return STATUS_OBJECT_NAME_EXISTS;
case EMFILE:
return STATUS_INSUFFICIENT_RESOURCES;
case ENOENT:
@ -158,18 +230,31 @@ std::int32_t unix_error_to_windows(const int &e) {
case ENOSPC:
return STATUS_DEVICE_INSUFFICIENT_RESOURCES;
case ENOTDIR:
return STATUS_OBJECT_PATH_INVALID;
return STATUS_OBJECT_NAME_NOT_FOUND;
default:
return STATUS_INTERNAL_ERROR;
}
}
UINT64 unix_time_to_windows_time(const remote::file_time &ts) {
auto unix_time_to_windows_time(const remote::file_time &ts) -> UINT64 {
return (ts / 100ull) + 116444736000000000ull;
}
void windows_create_to_unix(const UINT32 &create_options, const UINT32 &granted_access,
std::uint32_t &flags, remote::file_mode &mode) {
void use_getpwuid(uid_t uid, std::function<void(struct passwd *pw)> fn) {
static std::mutex mtx{};
mutex_lock lock{mtx};
auto *pw = getpwuid(uid);
if (not pw) {
utils::error::raise_error(__FUNCTION__, "'getpwuid' returned nullptr");
return;
}
fn(pw);
}
void windows_create_to_unix(const UINT32 &create_options,
const UINT32 &granted_access, std::uint32_t &flags,
remote::file_mode &mode) {
mode = S_IRUSR | S_IWUSR;
flags = O_CREAT | O_RDWR;
if (create_options & FILE_DIRECTORY_FILE) {
@ -177,13 +262,14 @@ void windows_create_to_unix(const UINT32 &create_options, const UINT32 &granted_
flags = O_DIRECTORY;
}
if ((granted_access & GENERIC_EXECUTE) || (granted_access & FILE_GENERIC_EXECUTE) ||
if ((granted_access & GENERIC_EXECUTE) ||
(granted_access & FILE_GENERIC_EXECUTE) ||
(granted_access & FILE_EXECUTE)) {
mode |= (S_IXUSR);
}
}
remote::file_time windows_time_to_unix_time(const std::uint64_t &t) {
auto windows_time_to_unix_time(std::uint64_t t) -> remote::file_time {
return (t - 116444736000000000ull) * 100ull;
}
} // namespace repertory::utils

206
src/utils/utils.cpp
View File

@ -1,27 +1,30 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#include "utils/utils.hpp"
#include "app_config.hpp"
#include "events/events.hpp"
#include "events/event_system.hpp"
#include "events/events.hpp"
#include "providers/i_provider.hpp"
#include "types/skynet.hpp"
#include "types/startup_exception.hpp"
#include "utils/com_init_wrapper.hpp"
#include "utils/native_file.hpp"
@ -29,18 +32,9 @@
#include "utils/string_utils.hpp"
namespace repertory::utils {
hastings api_currency_to_hastings(const api_currency &currency) {
ttmath::Parser<api_currency> parser;
parser.Parse(currency.ToString() + " * (10 ^ 24)");
ttmath::Conv conv;
conv.scient_from = 256;
conv.base = 10u;
conv.round = 0;
return parser.stack[0u].value.ToString(conv);
}
void calculate_allocation_size(const bool &directory, const std::uint64_t &file_size,
UINT64 allocation_size, std::string &allocation_meta_size) {
void calculate_allocation_size(bool directory, std::uint64_t file_size,
UINT64 allocation_size,
std::string &allocation_meta_size) {
if (directory) {
allocation_meta_size = "0";
return;
@ -49,20 +43,24 @@ void calculate_allocation_size(const bool &directory, const std::uint64_t &file_
if (file_size > allocation_size) {
allocation_size = file_size;
}
allocation_size = ((allocation_size == 0u) ? WINFSP_ALLOCATION_UNIT : allocation_size);
allocation_size =
utils::divide_with_ceiling(allocation_size, WINFSP_ALLOCATION_UNIT) * WINFSP_ALLOCATION_UNIT;
((allocation_size == 0u) ? WINFSP_ALLOCATION_UNIT : allocation_size);
allocation_size =
utils::divide_with_ceiling(allocation_size, WINFSP_ALLOCATION_UNIT) *
WINFSP_ALLOCATION_UNIT;
allocation_meta_size = std::to_string(allocation_size);
}
std::size_t calculate_read_size(const uint64_t &total_size, const std::size_t &read_size,
const uint64_t &offset) {
return static_cast<std::size_t>(((offset + read_size) > total_size)
? ((offset < total_size) ? total_size - offset : 0u)
: read_size);
auto calculate_read_size(const uint64_t &total_size, std::size_t read_size,
const uint64_t &offset) -> std::size_t {
return static_cast<std::size_t>(
((offset + read_size) > total_size)
? ((offset < total_size) ? total_size - offset : 0u)
: read_size);
}
int compare_version_strings(std::string version1, std::string version2) {
auto compare_version_strings(std::string version1, std::string version2)
-> int {
if (utils::string::contains(version1, "-")) {
version1 = utils::string::split(version1, '-')[0u];
}
@ -94,44 +92,33 @@ int compare_version_strings(std::string version1, std::string version2) {
return 0;
}
std::uint64_t convert_api_date(const std::string &date) {
//"2019-02-21T02:24:37.653091916-06:00"
const auto parts = utils::string::split(date, '.');
const auto dt = parts[0];
const auto nanos = utils::string::to_uint64(utils::string::split(parts[1u], '-')[0u]);
auto convert_api_date(const std::string &date) -> std::uint64_t {
// 2009-10-12T17:50:30.000Z
const auto date_parts = utils::string::split(date, '.');
const auto date_time = date_parts[0U];
const auto nanos =
utils::string::to_uint64(utils::string::split(date_parts[1U], 'Z')[0U]);
struct tm tm1 {};
#ifdef _WIN32
auto convert_time = [](time_t t) -> std::uint64_t {
const auto ll = Int32x32To64(t, 10000000) + 116444736000000000ull;
ULARGE_INTEGER ft{};
ft.LowPart = static_cast<DWORD>(ll);
ft.HighPart = static_cast<DWORD>(ll >> 32);
return ft.QuadPart;
};
const auto parts2 = utils::string::split(dt, 'T');
const auto date_parts = utils::string::split(parts2[0u], '-');
const auto time_parts = utils::string::split(parts2[1u], ':');
tm1.tm_year = utils::string::to_int32(date_parts[0u]) - 1900;
tm1.tm_mon = utils::string::to_int32(date_parts[1u]) - 1;
tm1.tm_mday = utils::string::to_int32(date_parts[2u]);
tm1.tm_hour = utils::string::to_uint32(time_parts[0u]);
tm1.tm_min = utils::string::to_uint32(time_parts[1u]);
tm1.tm_sec = utils::string::to_uint32(time_parts[2u]);
tm1.tm_wday = -1;
tm1.tm_yday = -1;
tm1.tm_isdst = -1;
return (nanos / 100) + convert_time(mktime(&tm1));
utils::strptime(date_time.c_str(), "%Y-%m-%dT%T", &tm1);
#else
strptime(&dt[0], "%Y-%m-%dT%T", &tm1);
return nanos + (mktime(&tm1) * NANOS_PER_SECOND);
strptime(date_time.c_str(), "%Y-%m-%dT%T", &tm1);
#endif
return nanos + (mktime(&tm1) * NANOS_PER_SECOND);
}
CURL *create_curl() { return reset_curl(curl_easy_init()); }
auto create_curl() -> CURL * {
static std::recursive_mutex mtx;
std::string create_uuid_string() {
unique_recur_mutex_lock l(mtx);
curl_global_init(CURL_GLOBAL_DEFAULT);
l.unlock();
return reset_curl(curl_easy_init());
}
auto create_uuid_string() -> std::string {
#ifdef _WIN32
UUID guid{};
UuidCreate(&guid);
@ -161,11 +148,13 @@ std::string create_uuid_string() {
#endif
}
std::string create_volume_label(const provider_type &pt) {
auto create_volume_label(const provider_type &pt) -> std::string {
return "repertory_" + app_config::get_provider_name(pt);
}
download_type download_type_from_string(std::string type, const download_type &default_type) {
auto download_type_from_string(std::string type,
const download_type &default_type)
-> download_type {
type = utils::string::to_lower(utils::string::trim(type));
if (type == "direct") {
return download_type::direct;
@ -178,7 +167,7 @@ download_type download_type_from_string(std::string type, const download_type &d
return default_type;
}
std::string download_type_to_string(const download_type &type) {
auto download_type_to_string(const download_type &type) -> std::string {
switch (type) {
case download_type::direct:
return "direct";
@ -193,7 +182,7 @@ std::string download_type_to_string(const download_type &type) {
#ifdef _WIN32
// https://www.frenk.com/2009/12/convert-filetime-to-unix-timestamp/
remote::file_time filetime_to_unix_time(const FILETIME &ft) {
auto filetime_to_unix_time(const FILETIME &ft) -> remote::file_time {
LARGE_INTEGER date{};
date.HighPart = ft.dwHighDateTime;
date.LowPart = ft.dwLowDateTime;
@ -209,7 +198,7 @@ void unix_time_to_filetime(const remote::file_time &ts, FILETIME &ft) {
}
#endif
std::string generate_random_string(const std::uint16_t &length) {
auto generate_random_string(std::uint16_t length) -> std::string {
srand(static_cast<unsigned int>(get_time_now()));
std::string ret;
@ -217,13 +206,18 @@ std::string generate_random_string(const std::uint16_t &length) {
for (std::uint16_t i = 0u; i < length; i++) {
do {
ret[i] = static_cast<char>(rand() % 74 + 48);
} while (((ret[i] >= 91) && (ret[i] <= 96)) || ((ret[i] >= 58) && (ret[i] <= 64)));
} while (((ret[i] >= 91) && (ret[i] <= 96)) ||
((ret[i] >= 58) && (ret[i] <= 64)));
}
return ret;
}
std::string get_environment_variable(const std::string &variable) {
auto get_attributes_from_meta(const api_meta_map &meta) -> DWORD {
return static_cast<DWORD>(utils::string::to_uint32(meta.at(META_ATTRIBUTES)));
}
auto get_environment_variable(const std::string &variable) -> std::string {
#ifdef _WIN32
std::string value;
auto sz = ::GetEnvironmentVariable(&variable[0], nullptr, 0);
@ -239,7 +233,7 @@ std::string get_environment_variable(const std::string &variable) {
#endif
}
std::uint64_t get_file_time_now() {
auto get_file_time_now() -> std::uint64_t {
#ifdef _WIN32
SYSTEMTIME st{};
::GetSystemTime(&st);
@ -254,7 +248,8 @@ std::uint64_t get_file_time_now() {
void get_local_time_now(struct tm &local_time) {
memset(&local_time, 0, sizeof(local_time));
const auto now = std::chrono::system_clock::to_time_t(std::chrono::system_clock::now());
const auto now =
std::chrono::system_clock::to_time_t(std::chrono::system_clock::now());
#ifdef _WIN32
localtime_s(&local_time, &now);
#else
@ -265,10 +260,8 @@ void get_local_time_now(struct tm &local_time) {
#endif
}
#ifdef _WIN32
#endif
bool get_next_available_port(std::uint16_t first_port, std::uint16_t &available_port) {
auto get_next_available_port(std::uint16_t first_port,
std::uint16_t &available_port) -> bool {
using namespace boost::asio;
using ip::tcp;
boost::system::error_code ec;
@ -285,28 +278,25 @@ bool get_next_available_port(std::uint16_t first_port, std::uint16_t &available_
return not ec;
}
std::uint64_t get_time_now() {
auto get_time_now() -> std::uint64_t {
#ifdef _WIN32
return static_cast<std::uint64_t>(
std::chrono::system_clock::to_time_t(std::chrono::system_clock::now()));
#else
#if __APPLE__
return std::chrono::nanoseconds(std::chrono::system_clock::now().time_since_epoch()).count();
return std::chrono::nanoseconds(
std::chrono::system_clock::now().time_since_epoch())
.count();
#else
return static_cast<std::uint64_t>(
std::chrono::nanoseconds(std::chrono::high_resolution_clock::now().time_since_epoch())
std::chrono::nanoseconds(
std::chrono::high_resolution_clock::now().time_since_epoch())
.count());
#endif
#endif
}
api_currency hastings_string_to_api_currency(const std::string &amount) {
ttmath::Parser<api_currency> parser;
parser.Parse(amount + " / (10 ^ 24)");
return parser.stack[0].value;
}
CURL *reset_curl(CURL *curl_handle) {
auto reset_curl(CURL *curl_handle) -> CURL * {
curl_easy_reset(curl_handle);
#if __APPLE__
curl_easy_setopt(curl_handle, CURLOPT_NOSIGNAL, 1);
@ -314,56 +304,40 @@ CURL *reset_curl(CURL *curl_handle) {
return curl_handle;
}
bool retryable_action(const std::function<bool()> &action) {
auto retryable_action(const std::function<bool()> &action) -> bool {
auto succeeded = false;
for (auto i = 0; not(succeeded = action()) && (i < 10); i++) {
for (std::uint8_t i = 0u; not(succeeded = action()) && (i < 20u); i++) {
std::this_thread::sleep_for(100ms);
}
return succeeded;
}
/* bool parse_url(const std::string &url, HostConfig &hc) { */
/* static const auto pathRegex = std::regex( */
/* R"(^((\w+):)?(\/\/((\w+)?(:(\w+))?@)?([^\/\?:]+)(:(\d+))?)?(\/?([^\/\?#][^\?#]*)?)?(\?([^#]+))?(#(\w*))?$)");
*/
/* */
/* std::smatch results; */
/* if (std::regex_search(url, results, pathRegex)) { */
/* hc.HostNameOrIp = results[8u].str(); */
/* hc.Path = utils::path::create_api_path(results[11u].str()); */
/* hc.Protocol = utils::string::to_lower(results[2u].str()); */
/* hc.ApiPort = results[10u].str().empty() ? ((hc.Protocol == "https") ? 443u : 80u) */
/* : utils::string::to_uint16(results[10u].str()); */
/* return not hc.HostNameOrIp.empty() && not hc.Protocol.empty() && */
/* ((hc.Protocol == "http") || (hc.Protocol == "https")); */
/* } */
/* */
/* return false; */
/* } */
void spin_wait_for_mutex(std::function<bool()> complete, std::condition_variable &cv,
std::mutex &mtx, const std::string &text) {
void spin_wait_for_mutex(std::function<bool()> complete,
std::condition_variable &cv, std::mutex &mtx,
const std::string &text) {
while (not complete()) {
unique_mutex_lock l(mtx);
if (not complete()) {
if (not text.empty()) {
/* event_system::instance().raise<DebugLog>(__FUNCTION__, "spin_wait_for_mutex", text); */
/* event_system::instance().raise<DebugLog>(__FUNCTION__,
* "spin_wait_for_mutex", text); */
}
cv.wait_for(l, 5s);
cv.wait_for(l, 1s);
}
l.unlock();
}
}
void spin_wait_for_mutex(bool &complete, std::condition_variable &cv, std::mutex &mtx,
const std::string &text) {
void spin_wait_for_mutex(bool &complete, std::condition_variable &cv,
std::mutex &mtx, const std::string &text) {
while (not complete) {
unique_mutex_lock l(mtx);
if (not complete) {
if (not text.empty()) {
/* event_system::instance().raise<DebugLog>(__FUNCTION__, "spin_wait_for_mutex", text); */
/* event_system::instance().raise<DebugLog>(__FUNCTION__,
* "spin_wait_for_mutex", text); */
}
cv.wait_for(l, 5s);
cv.wait_for(l, 1s);
}
l.unlock();
}

194
src/utils/windows/windows_utils.cpp
View File

@ -1,73 +1,43 @@
/*
Copyright <2018-2022> <scott.e.graves@protonmail.com>
Copyright <2018-2023> <scott.e.graves@protonmail.com>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/
#ifdef _WIN32
#include "utils/windows/windows_utils.hpp"
#include "types/startup_exception.hpp"
#include "utils/com_init_wrapper.hpp"
#include "utils/string_utils.hpp"
#ifndef STATUS_DEVICE_INSUFFICIENT_RESOURCES
#define STATUS_DEVICE_INSUFFICIENT_RESOURCES ((NTSTATUS)0xC0000468L)
#endif
namespace repertory::utils {
DWORD get_last_error_code() { return ::GetLastError(); }
const std::string &get_local_app_data_directory() {
static std::string app_data = ([]() -> std::string {
com_init_wrapper cw;
PWSTR local_app_data{};
if (SUCCEEDED(::SHGetKnownFolderPath(FOLDERID_LocalAppData, 0, nullptr, &local_app_data))) {
auto app_data = utils::string::to_utf8(local_app_data);
::CoTaskMemFree(local_app_data);
return app_data;
}
throw startup_exception("unable to detect local application data folder");
})();
return app_data;
}
std::uint64_t get_accessed_time_from_meta(const api_meta_map &meta) {
return utils::string::to_uint64(meta.at(META_ACCESSED));
}
DWORD get_attributes_from_meta(const api_meta_map &meta) {
return static_cast<DWORD>(utils::string::to_uint32(meta.at(META_ATTRIBUTES)));
}
std::uint64_t get_changed_time_from_meta(const api_meta_map &meta) {
return utils::string::to_uint64(meta.at(META_MODIFIED));
}
std::uint64_t get_creation_time_from_meta(const api_meta_map &meta) {
return utils::string::to_uint64(meta.at(META_CREATION));
}
std::uint64_t get_written_time_from_meta(const api_meta_map &meta) {
return utils::string::to_uint64(meta.at(META_WRITTEN));
}
std::uint64_t get_thread_id() { return static_cast<std::uint64_t>(::GetCurrentThreadId()); }
NTSTATUS translate_api_error(const api_error &e) {
auto from_api_error(const api_error &e) -> NTSTATUS {
switch (e) {
case api_error::access_denied:
return FspNtStatusFromWin32(ERROR_ACCESS_DENIED);
return STATUS_ACCESS_DENIED;
case api_error::bad_address:
return STATUS_INVALID_ADDRESS;
case api_error::buffer_too_small:
return STATUS_BUFFER_TOO_SMALL;
case api_error::buffer_overflow:
@ -84,20 +54,24 @@ NTSTATUS translate_api_error(const api_error &e) {
return FspNtStatusFromWin32(ERROR_INTERNAL_ERROR);
case api_error::error:
return FspNtStatusFromWin32(ERROR_INTERNAL_ERROR);
case api_error::file_exists:
case api_error::invalid_handle:
return STATUS_INVALID_HANDLE;
case api_error::invalid_operation:
return STATUS_INVALID_PARAMETER;
case api_error::item_exists:
return FspNtStatusFromWin32(ERROR_FILE_EXISTS);
case api_error::file_in_use:
return FspNtStatusFromWin32(ERROR_BUSY);
return STATUS_DEVICE_BUSY;
case api_error::incompatible_version:
return STATUS_CLIENT_SERVER_PARAMETERS_INVALID;
case api_error::invalid_operation:
return FspNtStatusFromWin32(ERROR_INTERNAL_ERROR);
case api_error::item_not_found:
return STATUS_OBJECT_NAME_NOT_FOUND;
case api_error::item_is_file:
return STATUS_ACCESS_DENIED;
case api_error::no_disk_space:
return STATUS_DEVICE_INSUFFICIENT_RESOURCES;
case api_error::os_error:
return FspNtStatusFromWin32(::GetLastError());
case api_error::out_of_memory:
return STATUS_NO_MEMORY;
case api_error::permission_denied:
return FspNtStatusFromWin32(ERROR_ACCESS_DENIED);
case api_error::success:
@ -112,7 +86,47 @@ NTSTATUS translate_api_error(const api_error &e) {
return FspNtStatusFromWin32(ERROR_INTERNAL_ERROR);
}
}
bool is_process_elevated() {
auto get_last_error_code() -> DWORD { return ::GetLastError(); }
auto get_local_app_data_directory() -> const std::string & {
static std::string app_data = ([]() -> std::string {
com_init_wrapper cw;
PWSTR local_app_data{};
if (SUCCEEDED(::SHGetKnownFolderPath(FOLDERID_LocalAppData, 0, nullptr,
&local_app_data))) {
auto app_data = utils::string::to_utf8(local_app_data);
::CoTaskMemFree(local_app_data);
return app_data;
}
throw startup_exception("unable to detect local application data folder");
})();
return app_data;
}
auto get_accessed_time_from_meta(const api_meta_map &meta) -> std::uint64_t {
return utils::string::to_uint64(meta.at(META_ACCESSED));
}
auto get_changed_time_from_meta(const api_meta_map &meta) -> std::uint64_t {
return utils::string::to_uint64(meta.at(META_MODIFIED));
}
auto get_creation_time_from_meta(const api_meta_map &meta) -> std::uint64_t {
return utils::string::to_uint64(meta.at(META_CREATION));
}
auto get_written_time_from_meta(const api_meta_map &meta) -> std::uint64_t {
return utils::string::to_uint64(meta.at(META_WRITTEN));
}
auto get_thread_id() -> std::uint64_t {
return static_cast<std::uint64_t>(::GetCurrentThreadId());
}
auto is_process_elevated() -> bool {
auto ret = false;
HANDLE token = INVALID_HANDLE_VALUE;
if (::OpenProcessToken(GetCurrentProcess(), TOKEN_QUERY, &token)) {
@ -130,7 +144,7 @@ bool is_process_elevated() {
return ret;
}
int run_process_elevated(int argc, char *argv[]) {
auto run_process_elevated(int argc, char *argv[]) -> int {
std::cout << "Elevating Process" << std::endl;
std::string parameters = "-hidden";
for (int i = 1; i < argc; i++) {
@ -161,7 +175,19 @@ int run_process_elevated(int argc, char *argv[]) {
void set_last_error_code(DWORD error_code) { ::SetLastError(error_code); }
int unix_access_mask_to_windows(std::int32_t mask) {
// https://stackoverflow.com/questions/321849/strptime-equivalent-on-windows
auto strptime(const char *s, const char *f, struct tm *tm) -> const char * {
std::istringstream input(s);
input.imbue(std::locale(setlocale(LC_ALL, nullptr)));
input >> std::get_time(tm, f);
if (input.fail()) {
return nullptr;
}
return reinterpret_cast<const char *>(s + input.tellg());
}
auto unix_access_mask_to_windows(std::int32_t mask) -> int {
if (mask & 1) {
mask -= 1;
if (not mask) {
@ -172,25 +198,43 @@ int unix_access_mask_to_windows(std::int32_t mask) {
return mask & 6;
}
int unix_open_flags_to_flags_and_perms(const remote::file_mode &mode,
const remote::open_flags &flags, std::int32_t &perms) {
auto unix_open_flags_to_flags_and_perms(const remote::file_mode & /*mode*/,
const remote::open_flags &flags,
std::int32_t &perms) -> int {
auto ret = _O_BINARY | _O_RANDOM;
ret |= (((flags & remote::open_flags::create) == remote::open_flags::create) ? _O_CREAT : 0);
ret |= (((flags & remote::open_flags::excl) == remote::open_flags::excl) ? _O_EXCL : 0);
ret |= (((flags & remote::open_flags::truncate) == remote::open_flags::truncate) ? _O_TRUNC : 0);
ret |= (((flags & remote::open_flags::append) == remote::open_flags::append) ? _O_APPEND : 0);
ret |= (((flags & remote::open_flags::temp_file) == remote::open_flags::temp_file) ? _O_TEMPORARY
: 0);
ret |= ((flags & remote::open_flags::write_only) == remote::open_flags::write_only) ? _O_WRONLY
: ((flags & remote::open_flags::read_write) == remote::open_flags::read_write) ? _O_RDWR
: _O_RDONLY;
ret |= (((flags & remote::open_flags::create) == remote::open_flags::create)
? _O_CREAT
: 0);
ret |= (((flags & remote::open_flags::excl) == remote::open_flags::excl)
? _O_EXCL
: 0);
ret |=
(((flags & remote::open_flags::truncate) == remote::open_flags::truncate)
? _O_TRUNC
: 0);
ret |= (((flags & remote::open_flags::append) == remote::open_flags::append)
? _O_APPEND
: 0);
ret |= (((flags & remote::open_flags::temp_file) ==
remote::open_flags::temp_file)
? _O_TEMPORARY
: 0);
ret |= ((flags & remote::open_flags::write_only) ==
remote::open_flags::write_only)
? _O_WRONLY
: ((flags & remote::open_flags::read_write) ==
remote::open_flags::read_write)
? _O_RDWR
: _O_RDONLY;
perms = _S_IREAD | _S_IWRITE;
return ret;
}
remote::file_time time64_to_unix_time(const __time64_t &t) { return t * NANOS_PER_SECOND; }
auto time64_to_unix_time(const __time64_t &t) -> remote::file_time {
return t * NANOS_PER_SECOND;
}
} // namespace repertory::utils
#endif // _WIN32