Submitting and retrieving reports

[JasonCrowe]

I have implemented several programs that download different reports.

Here is my process so far...

1. Submit the requests for several reports
2. Sleep 3-5 minutes
3. Request the report list, and cycle through it looking for the submitted reports
4. When found, download the data. Otherwise, wait 60 seconds and try again.

This is not a problem until I have several reports and they don't all come back with data, essentially blocking the process. The only way I have found around this is to check for the latest report first, then submit the report for the next iteration.

Another idea I have thought of is to skip the submission step by scheduling the report, but not every report can be scheduled, so it ends up being a partial solution.

How do you guys handle report submissions? Am I missing something?

[Bobspadger]

Could you request the reports, lodge the report ID in a database, then just poll every minute to see if reports that are not completed are ready, download and mark as complete in the db?

I've only done some basic work with the reports API so this would be my go to solution to begin with.

[JasonCrowe]

Thanks Bob. That's close to where I started from. I have been working on a new project moving data to Mysql tables & the client has 12 reports to gather.

I am adding this here in case it helps someone else.

In the process, I found that polling for the id's in the list of finished reports won't end well... I didn't know this before yesterday, but after submitting a report, Amazon will cancel it if the data hasn't been updated since your last request. At least I am assuming that's the reason they cancel it. Also, some reports may not have data to return.

This causes big problems when you expect 12 reports and only get 1 or 2.

Here is my code for submitting the reports:

reports_api = mws.Reports(config['access_key'], config['secret_key'], config['seller_id'], region='US')

def submit_reports(report_list):
    if len(report_list) == 0:
        return True

    reports_api.request_report(report_list[0])
    return submit_reports(report_list[1:])

What I have done to get around the problem is request submitted report list from get_report_request_list and save that to the database.

reports_api.get_report_request_list(max_count=str(len(reports))

This will work as a log file of the submitted reports. This way the client won't be calling me twice a week asking me to check on report 'xyz' because it's not in the database.

Next, I will need to write code to poll the database and download the reports with a 'DONE' status and save them to the correct table.

[Mitalee]

Hi, the amazon MWS documentation recommends using the push notifications service instead of polling. I believe we can use the Subscriptions API for the same.

It seems to me that this feature was implemented, although I cannot find it in the code. Would @Bobspadger be able to tell us where it is? I would be happy to document it once I'm able to try it.

[JasonCrowe]

This portion of the page shows the polling method that should be used. This could be different for subscriptions, but the 'Report' endpoint doesn't offer push notification that I know of.

[Bobspadger]

Looks like they do support Feeds and Reports in notifications

http://docs.developer.amazonservices.com/en_UK/subscriptions/Subscriptions_NotificationType.html

The subscriptions part of this project is in the develop branch here:

https://github.com/python-amazon-mws/python-amazon-mws/blob/develop/mws/apis/subscriptions.py

[Mitalee]

That's quite a bit of effort! Thank you! I'll try it out.

[Bobspadger]

for the subscriptions you will need to install and configure boto to receive the messages.
Also take note of the fact they are not necessarily in order, and can be duplicated so you need to keep and check the message ID's to make sure you have not already actioned them.

I find a Redis db for this is fantastic.

[Mitalee]

i have redis configured in my application - is both still required? can i set redis as the queue system instead?

[Bobspadger]

@Mitalee sadly the queue system has to be amazon SQS ( their docs are helpful on this)

I use Redis because of the Key/Value aspect, I lodge the ASIN : message_time in there so I know if I have used a message before, along with ASIN_TIME: time_of_message_from_message to make sure I don't action an old message that came in late.

Redis works superbly for this :)