• 0 Posts
  • 1.74K Comments
Joined 2 years ago
cake
Cake day: June 12th, 2023

help-circle








  • Really depends on what data it is and whether you want to search it regularly or just as a one time thing.

    You could load them into an rdbms (MySQL/Postgres) and have it handle the indexing, or use python tools to process the files. Something like elasticsearch could work too.

    If it’s just a one time thing grep is probably fine tho.

    Aleph could work as well but I have no experience with it.

    I guess it depends on how much time you want to invest in setting something up versus how much time you’d lose waiting for grep to finish (if you only need to search a certain column, you can create an index with just that column using awk, search that index file, then extract the full line from the source file based on that result, but at that point you’re basically creating a new database engine).