Tuesday, December 2, 2014

Implementing a Large Scale Document Storage Service

In this post, we explain why we decided to work on a document storage service for Genius Scan and how we are approaching this engineering challenge. We think that the process is worthy of being shared and we also hope to get feedback from the community.


A little background

At The Grizzly Labs, we create helpful productivity apps. Our main product, Genius Scan, is a scanner for iOS and Android. It’s one of the most popular mobile apps, netting more than 15 million downloads since its debut in June 2010 — we never invested a cent in advertising — and constantly receiving great feedback.

Genius Scan is at the interface between the physical and the digital world: it enables you to snap a photo of a paper document, to correct any perspective distortion and to process it like a real scanner. With several scans, you can build multi-page PDF documents. Genius Scan is also particularly good at exporting them wherever you want (Box, Dropbox, Email, Evernote, Expensify, Google Drive, OneNote, OneDrive, FTP, WebDAV…)



While we expected users to use Genius Scan to scan and export documents, we noticed that a lot of them are using Genius Scan to store their scans for the long term.

Solving problems for our users

Genius Scan stores documents on the phone in the local app folder. Due to the app sandboxing, this folder is only accessible to Genius Scan and documents are not synced between different devices. They also never leave the phone unless you decide to export them (the exception being your automatic iOS backups). This has two advantages: simplicity of implementation and privacy.

Yet, after 4 years of answering support emails, we are clearly seeing patterns in user requests:
  • Backup. Users can lose documents for many reasons: accidental deletions, improper install of new OS updates, phone lost or stolen. The unsatisfying workaround is to restore a complete phone backup from iCloud, provided that users have any.
  • Device migration (transfer of documents from an older device to a newer one). The problem arises during cross-platform transfers, typically between iOS and Android. There is no obvious way for users to access the documents they created on the previous platform. The workaround is to export the documents to a cloud service such as Dropbox. Users can later access them within the Dropbox app on the new device. We experimented with a backup and restore system for the Android app, but it’s more a patch than a satisfactory solution. This issue can arise as well when upgrading iOS devices if the user doesn’t have device backups setup.
  • Synchronization (accessing documents from any device). Users scan documents on their phones and want to read them afterwards on a tablet. The only way to do this currently is to export the documents to a service such as Dropbox and view them on the tablet within the Dropbox app. Likewise, users would want the documents to magically appear on their computer.
  • Extra features which require processing power or a backend. Examples of such features are: OCR, OCR indexing (similar to what Evernote does), document signing and sharing.
All this feedback keeps showing us the need for a document storage backend with synchronization logic.

Self-imposed requirements

One of the main challenges we face is that Genius Scan is already at large scale. Our millions of monthly active users scan about 250,000 new documents every day and have already generated more than 200 million scans. While we can find ways to stagger the launch, we can’t build a simple architecture that we would subsequently scale.

Looks like Mondays are scanning days, and that nobody scans turkeys :)

Our backend needs to meet the following requirements:
  • Security and privacy. Obviously, the first feature of this backend.
  • Data integrity. Users store very important documents in Genius Scan (losing receipts can translate into thousands of dollars that cannot be expensed.) There cannot be any loss of document as a consequence of users turning on synchronization.
  • Ability to handle network conditions. Mobile devices can go offline for a long time, and then pop back online (eg.: a user scans documents during a flight)
  • Cross-platform. We need to support iOS, Android and potentially other platforms.
  • Optionality. Users don’t have to use the synchronization, particularly if they have privacy concerns.
  • Financial sustainability. we plan to be able to cover the costs associated with this backend for the long run. A simple spreadsheet and a few estimations are enough to understand the cost structure. In our case, the monthly storage cost (S3) will dominate.

Different options

Synchronization is a typical problem and there are various alternatives:
  • With iCloud, iCloud Drive, no server-side work would be required. However, backup, device migration are not addressed, and synchronization only partially. One of our worries with iCloud was its instability, but it overcame this bad reputation with iOS 7 and 8.
  • CloudKit is also a potential solution. Given our scale, though, it won’t be a good fit due to the storage limits. On top of that, it’s limited to iOS.
  • Cross-platform cloud storage services: Dropbox, Box, OneDrive… Using these services as a backend tackles backup, device migration and synchronization and we wouldn’t be reinventing the wheel. There are several drawbacks: we need to adapt to their synchronization model — with Dropbox: the sync unit is the file — and we also become dependent on a third-party platform. Last but not least, any extra features (indexing, search, OCR) will be harder to offer since we won’t manage the document storage.
  • Ensembles developed by Drew McComack is an impressive project and could be an appropriate solution. However, it only synchronizes objects stored in Core Data: it wouldn’t be trivial to synchronize the scans themselves. It also doesn’t support Android (that being said, Ensembles 1.x is an open source project and could be extended.) Last, the cloud is used for temporary transfer, and not for data storage (peer-to-peer synchronization model) so it wouldn’t be usable for backup and extra features.
  • Our own service. This would address the different feature requests although the main drawback is that we would be reinventing the wheel.
After meeting with several top enterprise companies in San Francisco and in France, we decided to settle on implementing our own service due to the drawbacks of other solutions.

Steps we’ve taken

It’s crucial for us to identify trouble early. However, it’s easy to keep listing potential issues and corner cases and postpone the implementation, so we decided to take iterative steps towards our goal.

Metrics

We are estimating the load that our millions of MAU would generate on our backend through a couple channels: metrics and surveys. They are a lot of options for metrics. As for surveys, we can query users by displaying an in-house banner ad in Genius Scan. We also ask people who contact us for support.

An important metric is how many documents a typical user (vs. a power user) has. We are also interested in estimating the peak rate of document scanning during a day.

On top of that, surveying users enables us to understand the financial sustainability: Are users ready to pay for such a service? Should we offer a limited trial to everyone? etc.

Using the Dropbox SDK in our Android app

To get started, we wanted to put together a version of Genius Scan with some kind of synchronization. We first implemented a simple prototype by modifying the Android version of Genius Scan to use the Dropbox Sync API. This was fairly quick and took a couple day for a rough implementation.

Having such a prototype had several benefits: first, it gave us some self-confidence regarding the feasibility of this project. Second, it let us play with something concrete in order to think about the UX of the client apps; and finally, this was a first step in identifying potential issues on Android and in general (what are the typical synchronization conflicts, what behavior would we expect as users).

Custom implementation

Our next step was to develop our own client-server solution. This time, we focused on the iOS app. We developed a straightforward backend using Sinatra. We covered this implementation with extensive specs in iOS and Ruby for the server.

We based our architecture on Evernote’s synchronization. The brain of the synchronization is located in the client and the server holds the state. In some ways, the model structure is similar: In Genius Scan, documents contains pages, and can be optionally tagged. In Evernote, notebooks contain notes and can be optionally tagged.



Our client implementation fits nicely with the existing Core Data code Genius Scan relies on. It’s also surprisingly not intrusive: all the synchronization happens in a different managed object context and the contexts use decoupled notifications to communicate.

The goal of this iOS prototype was to deepen our understanding of the synchronization. The first iteration of this implementation took a bit less than a week to develop without the conflict handling.

We are refining this early implementation (we plan to write a more technical blog post on the subject). Note that it still focuses a lot on the client side and that our server component is running on fast local network. One of the next moves will be to test this implementation under real network conditions including latency and errors.

Conclusion

The two of us are super excited to be working on this project that we had postponed for a while. We want to provide this service to our users as soon as we can; however, we are not in a hurry. Since we don’t have any pressure, we can take our time to get it right.

If you’ve read until here, we are also very interested in your feedback (Hacker News). In any case, rest assured that we will keep you updated on our progress.

Tuesday, September 23, 2014

Genius Scan 3.6 with iOS 8 compatibility

We keep making scanning and exporting your documents easier with Genius Scan 3.6.



Specifically, for iOS 8, we are excited to introduce a number of new features:

Export your documents to iCloud Drive

With iOS 8 (and soon MacOS X Yosemite), Apple introduced iCloud Drive, an easy way to save all your documents in the same place, wherever they come from. To export to iCloud Drive, just use the Export feature as usual:

Export to iCloud Drive


Import scans from iCloud Drive

It's also possible to import scans from iCloud Drive in Genius Scan to process them and use them as part of a PDF file. To import a scan from iCloud Drive, perform a long press on the Photos scan button and choose your image document:

A long press on the Photos
button triggers the import
from iCloud Drive.

Compatibility with the new iPhone 6 and iPhone 6 Plus

The new iPhone 6 and iPhone 6 Plus offer larger screens with more real-estate. By default, Apple scales app so that they use the full screen, but it looks a bit blurry. Being compatible with the iPhone 6 and iPhone 6 Plus, Genius Scan will look great on those phones. Genius Scan takes advantage of this larger screens to display more of your documents.

***


We think that iOS 8 is a great update, and as of iOS 8.0.2, there are no known issues that should prevent you from upgrading. But if you decided not to upgrade (or if you can't), we have a load of new features coming for everyone:

Support for choosing a location when exporting to OneNote (notebook and section)

Your OneNote account is organized in notebooks, and each notebook is subdivided in sections. With Genius Scan, you can now choose in which section of which notebook you want to export your documents.

Select the destination as you
export to OneNote.

Ability to scan from the Photos or iCloud Drive when adding a page to a document

Before this version of Genius Scan, it was easy to add a new scan to a document from the camera, but not from your photos. Now, when you tap the + button in an existing document, you will have the choice to import from the camera, your photos or iCloud Drive.

Tap on the add button at the
bottom of the document view

Select the source when
adding a page to an existing
document.

Ability to use self-signed certificates with WebDAV.

Sometimes, when you host your own WebDAV server, you don't want Genius Scan to reject self-signed certificates. You can then turn on the self-signed certificate option (use with caution !)

The self-signed certificate option
is located in the WebDAV settings.


Updated self-help system

Also available at help.thegrizzlylabs.com, our updated help system offers great tutorials and answers to all your questions. And if you can't find an answer, we're always an email away.

The new help system


And of course, we fixed a bunch of nasty bugs.

Tuesday, September 2, 2014

Genius Fax now available on Android

Today we take a new step to make our genius tools available to more users and small businesses. We are excited to announce that Genius Fax is now available for Android.

We originally created Genius Fax for iOS because there wasn't any good application to send faxes. It's annoying enough to have to send a fax, so why should it be painful because the applications are badly designed or have hidden costs.

Android users will now benefit from the same hassle-free features our iOS users have enjoyed for more than a year:
  • Faxing documents located in any app
  • Subscribing to a fax number to be able to receive faxes directly on your phone
Our pricing stays simple: one page costs one fax credit (for international, long distance or local #). You purchase credits for as low as 39c if purchased in bulk. The cover page generated by Genius Fax is always free.


As always, we will quickly reply to your feedback. Email us at android.support@geniusfax.com.

Below are a few screenshots of the app to show you how simple and friendly it is:

A friendly UI to start with.

Your fax history clearly
displays the status of your
faxes.


Sending a fax is as easy
as entering a fax number
and a few optional details.

The status of your faxes is
always clearly displayed.

Get a notification when you
receive a new fax, or when
your faxes are sent.

Monday, August 4, 2014

Introducing Genius Scan 3.5 featuring Auto-Upload, Importing and Page Swipe

For our last release before iOS 8, we are excited to introduce a lot of cool features. As always, our focus has been to simplify your workflow, so to remove unnecessary steps.

Swiping between the pages of a document

This should have been available in Genius Scan since the beginning but early design decisions made that harder. Well, it’s now fixed!

Swipe left and right to
browse your documents.

Auto Upload

Your documents are important and you want to export them as soon as you can. This feature lets you upload your documents as soon as they changee if you are connected to Wifi. To configure the Auto Upload, open the settings of your favorite service  (Box, Dropbox, Google Drive, Evernote, FTP, OneDrive, OneNote, WebDAV), turn on the auto-uploading and choose the destination folder.

A simple checkbox turns
the auto-upload on


Two small notes:

  • This will only work when you are connected to a wifi network to avoid heavy data usage
  • In some cases, making several changes to a document will result in multiple copies. This is true for notes services such as Evernote or OneNote. We consciously decided to still enable it for these services because we think it’s more important that your documents are backed up.

Improved notification system

Genius Scan notifies you of noteworthy events: your document was safely uploaded, there was an error exporting your document, and a few other notifications. The new notification system is more visible and friendlier.
Tap the notification if you
need more time to read it.


Import from other apps

You can now import documents (PDF, JPEG) from other apps into Genius Scan+. This is very powerful because it enables you to mix scanned documents into existing PDFs.

From any app, export a PDF
to Genius Scan

Importing takes only a few
seconds even for large documents

This 37-pages PDF just appears as
a regular document in Genius Scan.

And a lot more …


  • In March, we partnered with Microsoft to let you export your scans to OneNote. We keep improving on this: When exporting to OneNote, PDF documents are now embedded in the note.
  • When adding pages to a document, the Save button directly saves in the current document
  • Ability to select a custom FTP port
  • FTP and WebDAV folders are now sorted alphabetically
  • Adjusted magnifier position so that it's more visible
  • When naming your documents, Genius Scan avoids duplicate names by appending the seconds to the document name if necessary.
  • Genius Scan warns you if your documents become too big and may cause issues when exporting.
  • Magnifier is 5x smoother on iPad and older phones
  • Smoother, more natural animations
  • Improved help system
  • Last but not least, if you are a developer, you might be interested in the following in the option to open Genius Scan and Genius Scan+ directly from a URL scheme

We are convinced all these tweaks will improve your experience with Genius Scan. Of course, always feel free to contact us at support@thegrizzlylabs.com.

As always, the free version of Genius Scan is fully functional and received a lot of improvements with this upgrade. You will notice that some of the new features (auto-upload, import) are only available in Genius Scan+. For the cost of two latt├ęs, you will benefit from lots of new features and make it possible for us to keep improving our product as we have done over the past 4 years.

Thursday, July 17, 2014

Backup and Restore your Data in Genius Scan for Android

In the latest version of Genius Scan for Android (1.4.2), we added a new feature that was often suggested by our users: being able to backup and restore Genius Scan data.

The most common case for using this feature is when you get a new device and you want to transfer your documents. It can also be useful when you were using Genius Scan and you purchase the Genius Scan+ app to transfer your documents from one version to the other (though the most practical way to upgrade to Genius Scan+ is to do it from the free version with an in-app purchase).

And of course, you can also regularly backup your Genius Scan data if you work with important documents and you don't want to lose them in case your device breaks or gets stolen.

So, how does it work?

Backup Genius Scan data

To create a backup, open the Genius Scan settings and select Backup



Then, select a folder where the backup file will be saved



The backup can take several minutes or more, depending on the number of documents you have. Do not close the app during the backup operation.


After the backup is complete, a file named with today's date and a ".gs-bck" extension is created in the folder you selected. You can move this file to your computer or to another device.

Restore a backup

Similarly, open the Genius Scan settings and select Restore a backup



Select the backup file you want to restore


The restore operations can take up to 10 minutes if you have a lot of documents. Do not close the app or data could be corrupted.



When the Restore operation is completed, the app will restart and all your documents will be ready to use :)


If you need any help backing up your data or restoring, feel free to contact us with the Help section of the app or directly at android.support@thegrizzlylabs.com.





























Wednesday, April 16, 2014

Receive Faxes with Genius Fax

With Genius Fax 2.0, the day has come to get rid of your fax machine.

Having the ability of receiving a fax generally means the following: buy a fax machine and reserve a line, or subscribe to an expensive service with ridiculous and incomprehensible fees.

With Genius Fax 2.0, you just pay as you go. You reserve a fax number for a very small fee (as low $3.49 per month) and you only need credits to receive pages. These are the same credits that you use to send faxes.

Update main screen displaying
your fax number.

As you receive a fax, you will get a push notification and an email and the received fax will appear in your app:


A received fax. Tap on "View document"
to display your document and
export it to other apps.

This feature is available now in Genius Fax 2.0 on the App Store.

Thursday, April 3, 2014

How to Print your Genius Scan Documents on Android

The Android version of Genius Scan+ now supports printing via Cloud Print.
Cloud Print is a technology from Google that allows you to print to any printer wirelessly.

The first step is to setup your printer to be used with Cloud Print. You can find on this page the list of Cloud Ready Printers and the instructions to add such a printer to Cloud Print. If your printer is not in this list, you will simply need to connect it in Google Chrome settings, as explained in details here.

Printer added to Google Cloud Print

Then, install the official Cloud Print Android app on your device, open Genius Scan+ and export any of your scan, as JPG or PDF, to the Cloud Print app.

Select Cloud Print on the Export screen

A cloud Print dialog appears and enables you to choose your printer and edit the printing settings.

Edit printing settings


Your document will print as soon as your printer is online! 

Printing requires Genius Scan+ ($6.99).