This project is read-only.
MFC, SQLite, and Boost C++ demo

Concept/Reason for project:
  • It is a common enough business case: you have a massive number of files to process from a given directory and you find the process time is slow.
  • Or, you need to import all the files from a given directory into SQLite. Perhaps as a backup.
  • Or, you need a significant set of data to work with in SQLite to test some bits of text processing code in SQLite and C++.

Beyond that this is a half decent sample for using,
  • MFC with a resizable dialog
  • SQLite with the C Api
  • boost
    • signals & slots
    • bind
    • memory mapped files.

Initial tests are done using the uncompressed Project Gutenberg Catalog - https://www.gutenberg.org/wiki/Gutenberg:Feeds

The RDF/XML Feed when extracted contains just under 50,000 files ranging from 2-240KB totaling around 800MB.
Files are read into a SQLite database using a boost directory iterator and a boost memory mapped file using the raw SQLite C Api. Since the files are not read into memory by the sample there is very little overhead other than the embedded SQLite database and the read is rather quick.

Project status / Todo
  • Test cases are needed

Highlights / Performance
  • The 48,953 files are read into the database in a little less than 8 seconds.
  • The resulting database is a little under 820 MB
  • Little more is planned in terms of performance tuning as 100MB/second is pretty close to my disk read|write max.

Enviroment
Visual studio 2010
Boost 1.58
SQLite 3.8.5

Project type
MFC Dialog

Project Settings
  • C/C++, Additional IncludeDirectories : V:\boost\;V:\Common\Data\Sqlite\
    • Update this to where you have boost and SQLite
  • Linker, Additional Library Directories : V:\boost\stage\lib\
    • Update this to where you have boost compiled



Last edited May 18, 2015 at 4:14 PM by gaccettola, version 25