Now, before you tell me this is a lot of work: I know, and am working on it (and almost done). Ideally, I would like to have my changes merged here, so I have a few questions and concerns for my current port.

Questions

Should I target a specific C++ standard?

Currently, my code depends on a few C++11 features, which can be easily removed with a few macros. This makes the code less readable, however, if C++03 support is desired, I will gladly change my implementation to conform to an older standard.

How to handle Unicode filesystem support?

Currently, LevelDB uses char-based (narrow) strings for for all filesystem operations, which does not translate well for Windows systems (since narrow strings use the ANSI, or OEM legacy codepages, and not UTF-8, for backwards compatibility). This means paths using international characters, or emojis, are therefore not supported with a simple port, something I consider to be an undesirable solution for a modern library. All the current forks of levelDB do not solve this fundamental issue, leading me to create my own implementation. Possible solutions include:

1. A narrow (UTF-8) API on *Nix, and a wide (UTF-16) API on Windows, using a typedef to determine the proper path type.
2. Converting all narrow strings from UTF-8 to UTF-16 before calling WinAPI functions.
3. Providing both a narrow (ANSI) and wide (UTF-16) API on Windows.

The 2nd option, although the least amount of work, is the least amenable for me since the expected encoding for paths from levelDB would then conflict with the entirety of the WinAPI. The 3rd option, however, duplicates code to support both the narrow and wide WinAPI, which would increase the amount of work required to maintain levelDB. The first option is a happy median: it minimizes redundancy and is consistent with expectations about *Nix and Windows paths. I am, however, amenable to any suggestions the levelDB authors may have.

Intellectual Property

To emulate the behavior of mmap on Windows, I used a very lightweight library (<250 lines of code) from Steven Lee, mman-win32. However, looking over your contributor license agreement, it seems that my port would not satisfy Google's CLA until I remove this code from my implementation. If this is the case, I could easily use the raw WinAPI functions rather than the emulated mmap in my Windows port. Please notify me if I should remove this code prior to submitting a pull request.

Other Changes

CMake Build System

I introduced a CMake build system, which retains most of the same logic as the existing Makefile. The existing Makefile has not been deprecated.

AppVeyor Continual Integration

To ensure builds do not break the Windows builds, I am planning to add an AppVeyor configuration, which allows continual integration on Windows using MSVC.

Summary

If there is still interest for native Windows support, and the proposed changes are amenable to the levelDB authors, I would gladly submit a pull request.

Original issue 27 created by quadrispro on 2011-08-09T12:57:55.000Z:

Please add a target into the Makefile to compile a shared library object.

Thanks in advance for any reply.

Hi, @cmumford

Does it make sense if we add cmake support to leveldb? if the answer is YES, I will try to do it.

There are some useful LLVM tools likes clang-tidy/woboq that need cmake support. We will get code format automatically, static check and online code browser if here is CMakeLists.txt.

any comments are appreciated. thx

I also read the issue 181

LevelDB's above a certain size (about 40 GB) seems to cause leveldb to open every single file in the database without closing anything in between.

Also, it seems it opens every file twice, for some reason.

My problem is almost the same.

OS: FreeBSD 10.1-RELEASE amd64 Leveldb: master branch, ( also test 1.18,1.17,...1.14 ) Dataset: 99G with snappy compressed, 58612 *.sst files. ulimit -n: 706995 kern.maxfiles: 785557 kern.maxfilesperproc: 706995

The dataset was generated by leveldb 1.8.0 , running several months. Last week , I restart the server , then the issue occurred.

It seems open every *.sst file twice,and not close them.

$ fstat -m|grep leveldb|wc
  117223 1055007 8668825

58612 * 2 ~= 117223 < 706995 (system limit)

$ fstat -m|grep leveldb
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     leveldb-tools 67098   67 /         92326 -rw-r--r--  1594319  r
root     leveldb-tools 67098   68 /         92326 -rw-r--r--  1594319  r
root     leveldb-tools 67098   69 /         45578 -rw-r--r--  2124846  r
root     leveldb-tools 67098   70 /         45578 -rw-r--r--  2124846  r
root     leveldb-tools 67098   71 /         45579 -rw-r--r--  2123789  r
root     leveldb-tools 67098   72 /         45579 -rw-r--r--  2123789  r
root     leveldb-tools 67098   73 /         45580 -rw-r--r--  2125455  r
root     leveldb-tools 67098   74 /         45580 -rw-r--r--  2125455  r
root     leveldb-tools 67098   75 /         45581 -rw-r--r--  2123795  r
root     leveldb-tools 67098   76 /         45581 -rw-r--r--  2123795  r
root     leveldb-tools 67098   77 /         45582 -rw-r--r--  2122645  r
root     leveldb-tools 67098   78 /         45582 -rw-r--r--  2122645  r
root     leveldb-tools 67098   79 /         45583 -rw-r--r--  2119487  r
root     leveldb-tools 67098   80 /         45583 -rw-r--r--  2119487  r
root     leveldb-tools 67098   81 /         45584 -rw-r--r--  2117737  r
root     leveldb-tools 67098   82 /         45584 -rw-r--r--  2117737  r
... more ....

as above , each file open twice (the same inode num: 92326,92326,45578,45578,...)

$ tail -f LOG
2016/08/10-11:17:48.121149 802006400 Recovering log #18223888
2016/08/10-11:17:48.329778 802006400 Delete type=2 #18223889
2016/08/10-11:17:48.333491 802006400 Delete type=3 #18223887
2016/08/10-11:17:48.333993 802006400 Delete type=0 #18223888
2016/08/10-11:17:48.388989 802007400 Compacting 58608@0 + 0@1 files
2016/08/10-11:20:14.324576 802007400 compacted to: files[ 58608 0 0 0 0 0 0 ]
2016/08/10-11:20:14.325108 802007400 Compaction error: IO error: ..../leveldb/18223891.ldb: Too many open files

After the IO error, the open files reduse to 87580

fstat -m | grep leveldb | wc
   87580  788220 6476498

And the program cost 100% CPU

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
67098 root          6  35    0  5293M  3607M uwait   4  48:55 100.00% leveldb-tools

But there is no disk io at all

  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
67098 root            0    216      0      0      0      0   0.00% leveldb-tools

Then ,can't seek, can't get, can't put .....

I'v tried change leveldb_options_set_max_open_files() , 100, 1024, 400000, but it not worked.

Original issue 176 created by shri314 on 2013-06-10T20:03:38.000Z:

Can the designers of leveldb explain the rational behind the design decision of not supporting multiple processes in leveldb implementation?

The documentation clearly says, under Concurrency section that: "A database may only be opened by one process at a time. The leveldb implementation acquires a lock from the operating system to prevent misuse."

Currently I can see that when one process opens level db, it uses fcntl with RW lock (exclusive lock). However this is a severely limiting, as no other process can ever open the same database even if it wants to just inspect the database contents for RDONLY purposes.

The use case for example is - one process exclusively opens leveldb database and fills up the database, then closes it. Then n different processes start reading that database.

Original issue 75 created by [email protected] on 2012-03-13T10:14:38.000Z:

Static initializers are totally fine in 99% of the projects. However in Chrome we are trying to remove them as they significantly slow down startup due to disk seeks.

There is only one static initializer generated by leveldb: $ nm libleveldb.a|grep _GLOBAL__I 0000000000000050 t _GLOBAL__I__ZN7leveldb10ComparatorD2Ev $

A global instance of BytewiseComparatorImpl is created at static initialization time in util/comparator.cc:

// Intentionally not destroyed to prevent destructor racing // with background threads. static const Comparator* bytewise = new BytewiseComparatorImpl;

const Comparator* BytewiseComparator() { return bytewise; }

I tried to make BytewiseComparator() CreateBytewiseComparator() instead so that it returns a new instance every time it is called. But then I'm encountering some ownership issues when it is used in the Options class. I initially made Options call CreateBytewiseComparator() in its constructor and delete it in its destructor (I also provided the correct implementations of copy constructor/assignment operator). The thing is that the comparator must live longer than the Options instance which owns it since the client seems to still use the pointer after Options goes out of scope.

Therefore I was also thinking about a totally different approach and wanted to add atomicops and CallOnce (GoogleOnceInit) from V8 to leveldb. That way we can keep BytewiseComparator() as it is and initialize the global instance the first time it is used. Adding all these dependencies might seem overkill. This is why I'm not directly sending a CL to you. They might serve you later though.

What do you think?

Original issue 184 created by chirino on 2013-07-01T13:33:37.000Z:

If an application wants to take a consistent backup of the leveldb data files, it needs to ensure that the background compaction threads are not modifying those files.

There are 3 warnings when building a project in Xcode 9 with Swift 4.

Two warnings are the same, for lines 274 and 275: Possible misuse of comma operator here - Cast expression to void to silence warning

and on line 1350: Code will never be executed

This prevents file descriptors from leaking to child processes.

When compiled for older (pre-2.6.23) kernels which lack support for O_CLOEXEC, there is no change in behavior. With newer kernels, child processes will no longer inherit leveldb's file handles, which reduces the changes of accidentally corrupting the database.

Fixes #623

Getting this Error while compiling IOS project. Looks like some CPP code is there in project which is not being complied properly.

/Users/cvi/Desktop/Ritesh/quintessence-learning/iOSApp/Pods/leveldb-library/include/leveldb/slice.h:21:10: error: 'string' file not found #include ^ :0: error: could not build Objective-C module 'CoreFoundation'

Hi, i used MSYS2 and mingw compiler , if this pull request interesting for you please merge if not simple reject. I tested code on Windows/Linux/MacOSX , used in FastoNoSQL application.

Description

We run a simple workload on LevelDB that inserts two key-value pairs. The two inserts end up going to different log files, and the first insert is set as asynchronous.

The file system trace we observed is shown below:

1 append("3.log") # first insert
2 create("4.log")
3 close("3.log")
4 append("4.log") # second insert
5 fdatasync("4.log")

When deployed on GlusterFS, the first append (line 1) may return successfully, but the data fails to persist to disk. This is due to a common approach in distributed file system for write optimization, which delays write submission to server, and lie to application that write has finished without error.

When any failure happens during the write submission, GlusterFS will make close (line 3) return with -1 to propagate the error. However, since LevelDB doesn't check any error returned by close, it's not aware about any error happens during the first insert.

In GlusterFS, fdatasync("4.log") will only persist data on 4.log but not 3.log, therefore, if any crash happens after fsync (line 5), LevelDB will not recover the first insert after reboot.

As a consequence, there is data loss on the first insert, but not second insert, which violates the ordering guarantee provided by LevelDB.

Fix

To fix the problem, we could add error handling logic for close operation. Basically, when error happens, we should consider previous append as failed, and either redo it or call fsync on that specific log file to force the file system persist the write.

I tried to build the source shortly after i clone the repo:

[ 71%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark.dir/complexity.cc.o
leveldb/third_party/benchmark/src/complexity.cc:85:10: error: variable 'sigma_gn' set but not used [-Werror,-Wunused-but-set-variable]
  double sigma_gn = 0.0;
         ^
1 error generated.
make[2]: *** [third_party/benchmark/src/CMakeFiles/benchmark.dir/complexity.cc.o] Error 1
make[1]: *** [third_party/benchmark/src/CMakeFiles/benchmark.dir/all] Error 2
make: *** [all] Error 2

It seems that the sigma_gn did not get used at any place.

And the solution is to simply remove this two lines:

LeastSq MinimalLeastSq(const std::vector<int64_t>& n,
                       const std::vector<double>& time,
                       BigOFunc* fitting_curve) {
-  double sigma_gn = 0.0;
+ //  double sigma_gn = 0.0;
  double sigma_gn_squared = 0.0;
  double sigma_time = 0.0;
  double sigma_time_gn = 0.0;

  // Calculate least square fitting parameter
  for (size_t i = 0; i < n.size(); ++i) {
    double gn_i = fitting_curve(n[i]);
-     sigma_gn += gn_i;
+ //    sigma_gn += gn_i;
    sigma_gn_squared += gn_i * gn_i;
    sigma_time += time[i];
    sigma_time_gn += time[i] * gn_i;
  }

I noticed that there are many assert in the programme which may cause levelDB to crash. May I know whether there are specific reasons we only assert instead of using specific exceptions?

For example, in write_batch.cc

void WriteBatchInternal::SetContents(WriteBatch* b, const Slice& contents) {
  assert(contents.size() >= kHeader);
  b->rep_.assign(contents.data(), contents.size());
}

Thank you.