Add-Ons make it easy for anyone to add and share additional features within the DocumentCloud platform, ranging from automating repetitive tasks to integrating machine learning and data visualization techniques.
For end users, using an Add-On is as simple as selecting documents or executing a search, picking which feature they’d like to apply to the results, and then submitting.
On the backend, Add-Ons execute Python scripts organized in a standard way, hosted and processed right on GitHub.
Add-Ons can take advantage of the full DocumentCloud API as well as the ability to call other third-party services and a few Add-On specific functions such as the ability to store arbitrary files, send a user emails, and track the progress of an Add-Ons' execution and display messages to the user.
In addition to being able to execute Add-Ons via the DocumentCloud user interface, these extensions are also designed to run smoothly on your local computer — simply clone the repository to your local device, install the DocumentCloud Python wrapper, and then invoke the main.py file of the Add-On. Invocation requires your DocumentCloud username and password if
the add-on requires authentication, which is used to fetch a refresh and access
token. They can be passed in as command line arguments (--username
and
--password
), or as environment variables (DC_USERNAME
and DC_PASSWORD
).
You can also pass in a list of document IDs (--documents
), a search query
(--query
), and JSON parameters for your Add-On (--data
) - be sure to
properly quote your JSON at the command line.
Example invocation:
python main.py --documents 123 --data '{"name": "World"}'
We have an Add-On template hosted on GitHub that demonstrates basic features, as well as a variety of other example Add-Ons that might serve as a useful base for your own work.
Some Add-Ons require AI credits to run as they use paid services to perform operations like OCR, document translation or the Add-On uses AI tools which have costs.
DocumentCloud Premium comes with AI credits for both professional and organizational accounts on MuckRock. You can upgrade your account or organization on the MuckRock Select Plan page. You can also upgrade your plan by clicking on the drop-down menu named "Premium" once you log in. It will link you to the same upgrade plan page.
To check your AI credit balance, you can:
AI Credit usage is logged. If you want to know how your AI credits were used, contact us.
At this time, there are four premium Add-Ons:
There are several different types of Add-Ons, including ones that use AI, perform bulk operations, others that specialize in data extraction, ones that calculate DocumentCloud statistics, some are used to export documents or data contained in documents, some monitor websites for changes or for newly uploaded documents, and others transform other types of files DocumentCloud doesn't natively support into more readily analyzable documents.
michael-morisy-658
)New Add-Ons are being added all the time. Under the Add-Ons dialog, click "Browse All Add-Ons" to explore and activate or deactivate Add-Ons. Register for the DocumentCloud newsletter to get updates on additional features and other announcements.
If you write your own Add-On, you can run it from with DocumentCloud's user interface through a few simple steps.
First, install the Github DocumentCloud App. Note that for this to work properly, you must have your primary Github and MuckRock accounts set to use the same email address. You can set your primary MuckRock account email here.
As you add the Github DocumentCloud App, give it access to only those repositories in your Github account that are Add-Ons you want to run. You can modify this from this page once you have the app installed in Github.
Then your Add-On will appear for you under "Browse All Add-Ons" and you can activate it there.
Currently, the DocumentCloud team reviews and vets each Add-On that's integrated directly within the site (i.e., the ones you see in the Add-On dropdown). Add-Ons that a user downloads and runs locally, however, are not necessarily vetted or reviewed by the DocumentCloud team and you should only run Add-Ons that are published by individuals you trust.
Currently, Add-Ons are essentially given full access to your user account, and can do anything you can while logged in, including reading all of your documents, deleting or modifying them, sharing documents with other users, and much more.
For Add-Ons run through the site, they do not see your account credentials, just a unique token granted to that Add-On. For Add-Ons run through a GitHub Action or run locally, there is the potential for a maliciously written Add-On to obtain your credentials so it is particularly important to understand and trust the source of the Add-On before you run it.
As we open up Add-Ons to additional third-party contributions, we'll begin to offer more limited access tokens that constrain permissions to just the documents and actions explicitly granted to them, as well as defining certain time scopes for that access.
When you run an Add-On via the DocumentCloud web interface, it will take one of four options for what documents to act on:
Note that currently, these options determine what specific document IDs are sent to the Add-On, but the Add-On still has permissions to your entire document collection. In the future, as we better understand Add-On use cases, we plan to restrict access permissions to only the subset of documents that an Add-On requires to successfully run.
DocumentCloud Add-Ons have deep linking enabled, meaning you can share the link to a useful Add-On to others with ease. You will notice when clicking on an Add-On it pulls up the configuration menu and change the URL as well. For example, clicking on the PII Detector Add-On allows me to link to the Add-On directly like so: https://www.documentcloud.org/app?q=%2B#add-ons/MuckRock/PII-Detector
Add-Ons can also be shared with parameters pre-filled by modifying the URL. For example, to share a URL to the PII Detector Add-On with the Detect SSNs field pre-selected, one can do so like this:
https://www.documentcloud.org/app?q=%2B&ssn=true#add-ons/MuckRock/PII-Detector
Properties defined in the Add-On's config.yaml can continue to be chained one after another and deep linked, like this one that specifies both the site to monitor along with the * (all) CSS selector for Klaxon.
https://www.documentcloud.org/app?q=%2B&site=https://muckrock.com&selector=*#add-ons/MuckRock/Klaxon
Hourly, daily, or weekly event options for scheduled Add-Ons (like Klaxon and Scraper) can be passed as parameters as well.
https://www.documentcloud.org/app?q=%2B&site=https://muckrock.com&selector=*&event=hourly#add-ons/MuckRock/Klaxon
You can submit your Add-On for review to share with all DocumentCloud users by filling out this form.
If you have other questions, suggestions, or feedback, please email info@documentcloud.org — we’re excited to see what you do with Add-Ons!