+3 votes
in Programming Languages by (51.1k points)
I have many files in a directory that I want to use as inputs. Should I combine them first and then read or is there any other way to read data from those files?

1 Answer

+2 votes
by (40.7k points)
selected by
 
Best answer

You can combine all files and can create one file for the processing, however, there is another way to read data from all files present in a directory. You can create a list of filenames and then open/read one file at a time for the processing. Here is an example I used for the same scenario:

>>> filelist = [fl for fl in next(os.walk('classifier_outputs'))[2] if 'metavisits_probs' in fl[:len('metavisits_probs')]]
>>> filelist
['metavisits_probs1.txt', 'metavisits_probs11.txt', 'metavisits_probs21.txt', 'metavisits_probs31.txt', 'metavisits_probs41.txt', 'metavisits_probs51.txt', 'metavisits_probs61.txt', 'metavisits_probs71.txt', 'metavisits_probs81.txt', 'metavisits_probs91.txt', 'metavisits_probs101.txt', 'metavisits_probs111.txt']
>>> for filename in filelist:
...     filepath = wv.metavisits_probs_file.split('/')[0] + '/' + filename
...     with open(filepath) as mpf:
...             for line in mpf:
...                     print(filepath)
...                     break

...
classifier_outputs/metavisits_probs1.txt
classifier_outputs/metavisits_probs11.txt
classifier_outputs/metavisits_probs21.txt
classifier_outputs/metavisits_probs31.txt
classifier_outputs/metavisits_probs41.txt
classifier_outputs/metavisits_probs51.txt
classifier_outputs/metavisits_probs61.txt
classifier_outputs/metavisits_probs71.txt
classifier_outputs/metavisits_probs81.txt
classifier_outputs/metavisits_probs91.txt
classifier_outputs/metavisits_probs101.txt
>>>

You need to use 'import os' and replace 'print(filepath)' and 'break' with your statements.


...