Programming for Everybody: Assignment 11.1 Regular Expressions - edorlando07/datasciencecoursera GitHub Wiki

###Using Python to Access Web Data

Finding Numbers in a Haystack

In this assignment you will read through and parse a file with text and numbers. You will extract all the numbers in the file and compute the sum of the numbers.

Data Files

We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.

Sample data: http://python-data.dr-chuck.net/regex_sum_42.txt 
(There are 87 values with a sum=445822)

Actual data: http://python-data.dr-chuck.net/regex_sum_340267.txt 
(There are 82 values and the sum ends with 826)

These links open in a new window. Make sure to save the file into the same folder as you will be writing your Python program. Note: Each student will have a distinct data file for the assignment - so only use your own data file for analysis.

Data Format The file contains much of the text from the introduction of the textbook except that random numbers are inserted throughout the text. Here is a sample of the output you might see:

Why should you learn to write programs? 7746
12 1929 8827
Writing programs (or programming) is a very creative 
7 and rewarding activity.  You can write programs for 
many reasons, ranging from making your living to solving
8837 a difficult data analysis problem to having fun to helping 128
someone else solve a problem.  This book assumes that 
everyone needs to know how to program ...

The sum for the sample text above is 27486. The numbers can appear anywhere in the line. There can be any number of numbers in each line (including none).

Handling The Data

The basic outline of this problem is to read the file, look for integers using the re.findall(), looking for a regular expression of '[0-9]+' and then converting the extracted strings to integers and summing up the integers.

import re         #imports the regular expression library

hand = open("regex_sum_340267.txt", "r")
numlist = []      #To create a list, put a number of expressions in square brackets     

for line in hand:
    line = line.rstrip()
    integers = re.findall('([0-9]+)', line)  #regex that reads all
                                             #integers in each line
                                             #[0-9] means select any digit
                                             #between 0 and 9 and find
                                             #the + means one or more
                                             #findall will read multiple
                                             #sets of numbers in same line.
    for number in integers:
        numlist.append(int(number))          #each number is an
                                             #element within the list

If numlist was printed now, the numbers in the list include the following:

[9273, 5081, 1662, 22, 2677, 1573, 7665, 9262, 4334, 5196, 3151, 633, 7356, 3571, 3989, 335, 8380, 5565, 3143, 5051,    
8091, 430, 5644, 7130, 9463, 5034, 7398, 3941, 5166, 6939, 2986, 5983, 149, 5099, 7151, 7720, 2274, 1112, 9050, 8795,     
2293, 3724, 2982, 3881, 8617, 7120, 5741, 7815, 4813, 232, 9337, 7600, 6649, 9612, 3424, 2182, 739, 2472, 3832, 6915, 
2069, 3936, 2057, 7725, 566, 1124, 4350, 5675, 2048, 4778, 841, 9986, 1591, 5516, 2546, 7419, 8203, 1175, 833, 7859,    
2033, 42]

The rest of the code is listed below:

print sum(numlist)                           #aggregates the numbers
                                             #found in the list

###The final output of the code is the following: