Using python to check data since that last time it was checked…

follow-upIn my last post on time, I mentioned that I wanted to be checking in on whether features in GIS data had been updated or not. If it had, I wanted to copy the data that had been modified and record it in an archive. Originally I thought I would just check in every 5 minutes, or 30 minutes, or whatever the client wanted. Then I started thinking about all the things that could go wrong. If it was every minute, and the process of archiving the data took longer than one minute, the script wouldn’t run again until the first process had finished and I could potentially miss the changes that happened during that slack time. Or what if it failed entirely and then I lost track of when the data had last been checked? Or… I’m sure my mind wandered off into all the things that could go wrong.

So I thought a better way to handle this was to record the date and time the the data had been checked in a little text file and then read it back the next time the script ran. The file acts as kind of safety net in case things don’t go as planned.

Now the script can be run as often or as little as needed and all the data should be caught.

Here’s how I did it…

#RightNow =  # if you need local time
DateTextFile = "c:\your\path\yourTimeKeeper.txt"
UTCNow = datetime.datetime.utcnow()  # if you need UTC time

# read the last search date and use in the search string below  
# if using last_edit and last_create dates, use UTC time
f = open(DateTextFile, 'r')
searchDate = f.readline()

expression = "last_edited_date > '" + str(searchDate) + "' OR created_date > '" + str(searchDate) +"'"

# write a text file storing the last search date to be referenced in the next go-round
# write it as a UTC time
f = open(DateTextFile, 'w')
f.write(UTCNow.strftime("%Y-%m-%d %H:%M:%S"))

# archive changes made throughout the day
arcpy.MakeFeatureLayer_management(MyFeatureclass,"MyFC_lyr", expression)
result = int(arcpy.GetCount_management("MyFC_lyr").getOutput(0))
print "There were %s points selected." % str(result)

#write the results to an archive feature class or table

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s