Search Gear #17 – TEST the TWITTER COLLECTOR

First run
Starts at: Wed Aug 27 03:44:15 +0000 2014
Ends at: Wed Aug 27 07:49:42 +0000 2014
19 records at 09:21 August 27th 2014

Removed file.Sync() after each tweet is written to ‘disk’ (buffer I expect) which makes things much quicker.

Second run
Starts at: Wed Aug 27 10:33:48 +0000 2014
Ends at: Wed Aug 27 13:38:09 +0000 2014
20 records at 14:51 August 27th 2014

DO NOT APPEND

changed file.Seek(0, 0) to file.Seek(0, 2)
BUT could also be the file.Sync removal

sync that after testing if no sync then no appending at the beginning only adding at the end. otherwise will add at front.

try again.
Starts at: Wed Aug 27 10:56:58 +0000 2014
Ends at: Wed Aug 27 14:18:41 +0000 2014
20 records at 15:27 August 27th 2014

So in reverse chronological chunks is best – so will need to filter by time on input.

==

Now, I also need a function to process wildcards out of hashtags maybe creating multiple requests from a single tag.

Then a function to loop ‘collect’ for each individual. Also need a clear up function to scrub old files by date.

Gunna be a slow coding week too with meetings getting in the way of gophering!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s