Getting bulk data into CiviCRM

CiviCRM’s features for manual data entry are obvious: direct entry through the back office area, standard data collection on contribution pages and event registrations, and custom configured forms using profiles or Form Builder.

But bulk data is different. A recent client inquiry reminds me that there's more than one way to get large amounts of data into CiviCRM.

When we talk about getting large amounts of data into CiviCRM, we’re usually talking about one of three things: imports, integrations, and migrations.

1. Imports

Now and then you might get some data in a spreadsheet that you'd like to import into your existing system. CiviCRM offers some standard features for CSV imports of contacts, contributions, and memberships, and with a little patience you can make it work.

But you have to think carefully about what you're importing, the process can be a little tedious. And you may run into server timeout issues with larger data sets; depending on your site and server configuration, those issues might show up with 100 rows or with 5,000 rows.

Still, the documentation is pretty good, and it's a good place to start.

If you're looking to solve some of the problems mentioned above, you can have developer work on a more streamlined process. The newly approved extension Advanced Import offers a pretty good framework for making that development work a lot easier.

The CiviCRM Extensions Directory also lists some other great extensions that can, in the right hands, make frequent imports much easier.

2. Integrations

You might instead be looking to synchronize data from some external system into CiviCRM, on a daily basis, or in real time.

Some good integrations already exist for certain systems, including MailChimp and QuickBooks.

For other external systems, there might already be an integration that does exactly what you need. Check the CiviCRM Extensions Directory to find out.

Or talk to a developer about building a custom integration that suits your needs. This can make for a huge time saver for your staff.

3. Migrations

If you're moving to CiviCRM from some other CRM system, you'll want to bring in all the relevant data from your legacy system. This is undoubtedly the most complex and difficult of the three topics mentioned here.

You'll almost certainly want to hire a specialist for this. Specialized data manipulation tools such as Pentaho will make the whole thing much easier, assuming you're familiar with the tool itself.

Jon Goldberg of Megaphone Tech has done a great job sharing his insights on migrations from Raiser's Edge. If you're moving from another CRM, you may find some information online from folks who have managed such a migration.

Either way, this is a large project that involves extracting the data from the legacy system, and significant work cleaning that data and mapping it to the appropriate data entities in CiviCRM.

I recommend hiring an experienced pro to do the heavy lifting on this one. And of course, that experienced pro will be relying on your frequent input, because this is not a one-size-fits-all operation.

Here's the thing:

Getting bulk data into CiviCRM could mean a lot of different things, some of them fairly easy, others very complicated.

For more frequent tasks, the big win is usually in making it as easy as possible.

For one time operations like a migration, the win lies in good planning, and careful testing to make sure everything is done right the first time.

All the best,
A.