I wonder if they use IMAP's EXAMINE or SELECT commands when iterating through folders. The former is read-only, although the end user has no control or guarantee that this is how they'll go through folders.

Did you ever try to contact someone at Google for higher quota? (There's a form linked from the developer site.) I'm not surprised the default quotas were low, it is still Beta...

Our primary issue isn't the quota limits, it's the actual speed of the API. The best we were able to achieve with the Gmail API was 3000 messages/minute whereas IMAP easily lets us get 20k messages/minute.

Are you limiting yourself to one thread or something? If you're not getting limited by quota then why not use batching + more threads until you do get to those limits?

We are getting limited by the quota. As the documentation says, quotas can be temporarily exceeded and in our tests we could fetch about 3000 messages in those bursts. After that, we would be 403'd for one to two minutes before requests would succeed again with quotas strictly enforced for an unknown amount of time past that.

How did you make javamail fast? My impression was that you could batch request meta data info, but not the actual body.

Gmail's been way faster for me

We don't request message bodies, just headers.

When you do your tests are you using the new-ish message.get(format=METADATA)?

No, I wasn't aware of that option.

In our tests we did list requests for 100 messages and then bulk requests for each messages' headers (messages.get("me", message.getId).setFields("historyId,payload/headers")).

I'll look into that option and re-run the tests.

gotcha - thanks!

Would they have preferred to just have someone at Google to contact about this?

You can speak to a human at Google regarding their API limitations? I'd be shocked if that was the case.

One can request an increase in quota as it says here: https://developers.google.com/gmail/api/v1/reference/quota

Perhaps my comment was a bit opaque... b^)

Not sure if it was you, but someone from Google did reach out to us. If it was you, I'll be replying this morning once I get to the office! If not, thanks anyways!

No I'm not at Google. Congratulations on finding a contact, and good luck with a resolution for the issue you describe. It might be interesting if you did a followup after the whole process plays out.

I work at Google. We don't typically monitor HN, reddit, twitter, etc but we have plenty of people watching Stack Overflow (the one we direct people to from our docs). The default quotas are conservative during this Beta period but we're happy to increase when needed--the service is certainly not provisioned as high as IMAP at this point. Submitting the quota increase request (from the developers console) will send us an email and we'll either approve it or if we need to, we'll start an email thread to ask for more info/suggest improvements. Hope that helps!