My question is whether this is an efficient way to process the data or
would it be better to use a database such as SQLITE3?
Using something like SQLite will almost certainly be faster, and probably significantly so. But of course you lose the ability to simply load the text files into a text editor. Whether the benefits outweigh the costs is a decision only you can make.
Given that the data set is quite large (i.e. not just a 100-line config file), I'd move to SQLite if I were in your position. The command line sqlite program allows you to modify the data quite easily.
