Resizing of LMDB map

[Daniel Kraft <domob1812@gmail.com>]

Hi!

I'm trying to use LMDB in a way that automatically resizes the map
whenever it becomes too small, i.e. a function fails with MDB_MAP_FULL.
I've created and attached a program that resembles my real usecase in a
very simplified fashion:  It just writes data to the database, and tries
to double the map size whenever MDB_MAP_FULL is received.

However, for me this program (as well as my real code) produces a
segfault in the mdb_put call that follows after the first resize.  What
am I doing wrong here, and how can I correctly implement a resize when full?

According to the docs [1], it is ok to call mdb_env_set_mapsize for an
open environment as long as no transactions are active.  My code ensures
this by aborting the current transaction before resizing the map.  Also,
there seems to be other code out there [2] that does the resize in a way
that looks like what I'm trying to do, and presumably works fine.

  [1]
http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
  [2] https://github.com/BVLC/caffe/pull/3731

Yours,
Daniel

-- 
https://www.domob.eu/
OpenPGP: 1142 850E 6DFF 65BA 63D6  88A8 B249 2AC4 A733 0737
Namecoin: id/domob -> https://nameid.org/?name=domob
--
3.6.0: Bar-Pri-Ran-Rog-Sam-Val-Wiz
To go: Arc-Cav-Hea-Kni-Mon-Tou


-- 
https://www.domob.eu/
OpenPGP: 1142 850E 6DFF 65BA 63D6  88A8 B249 2AC4 A733 0737
Namecoin: id/domob -> https://nameid.org/?name=domob
--
3.6.0: Bar-Pri-Ran-Rog-Sam-Val-Wiz
To go: Arc-Cav-Hea-Kni-Mon-Tou

#include <lmdb.h>

#include <glog/logging.h>

#include <cstdlib>

void
Check (const int code)
{
  if (code == 0)
    return;
  LOG (FATAL) << "LMDB error: " << mdb_strerror (code);
}

void
Resize (MDB_env* env, MDB_txn* txn)
{
  mdb_txn_abort (txn);

  MDB_envinfo stat;
  Check (mdb_env_info (env, &stat));
  const size_t newSize = (stat.me_mapsize << 1);
  LOG (INFO)
      << "Map is full, resizing from " << stat.me_mapsize
      << " to " << newSize;

  Check (mdb_env_set_mapsize (env, newSize));
}

int
main ()
{
  MDB_env* env;
  Check (mdb_env_create (&env));
  Check (mdb_env_open (env, "/tmp/db", 0, 0644));

  MDB_envinfo stat;
  Check (mdb_env_info (env, &stat));
  LOG (INFO) << "Map size: " << stat.me_mapsize;

  const std::string dataStr(256, '0');
  MDB_val data;
  data.mv_size = dataStr.size ();
  data.mv_data = const_cast<void*> (static_cast<const void*> (&dataStr[0]));

  for (int i = 0; i < (1 << 20); ++i)
    {
      MDB_txn* txn;
      Check (mdb_txn_begin (env, nullptr, 0, &txn));
      MDB_dbi dbi;
      Check (mdb_dbi_open (txn, nullptr, 0, &dbi));

      MDB_val key;
      key.mv_size = sizeof (i);
      key.mv_data = &i;

      int code = mdb_put (txn, dbi, &key, &data, 0);
      if (code == MDB_MAP_FULL)
        {
          Resize (env, txn);
          continue;
        }
      Check (code);

      code = mdb_txn_commit (txn);
      if (code == MDB_MAP_FULL)
        {
          Resize (env, txn);
          continue;
        }
      Check (code);
    }

  mdb_env_close (env);

  return EXIT_SUCCESS;
}

Attachment: signature.asc
Description: OpenPGP digital signature