CiviCRM upgrade oddities: file overrides

One of the best things about CiviCRM is that it's customizable — you can modify it in just about any way you want.

But this great and wonderful thing can make your upgrades awful and painful, if your customizations rely on custom file overrides of PHP or Smarty *.tpl files. This is an officially supported but lightly documented way to customize CiviCRM behavior. It’s falling out of favor for all the reasons I mention here, but it still might affect you.

By the way, this doesn’t even cover any direct modifications of CiviCRM code files that you or other developers may have made on your site. That’s totally unsupported and officially a Bad Idea™.

Why is this an upgrade oddity?

If you have customized files like this, you'll need developer skills to sort it out. This is not a task for a non-technical site owner.

So can you skip this step?

No, unfortunately.

How can you know if this affects you?

It's pretty easy to know, with these steps:

• In CiviCRM, navigate to Administer > System Settings > Directories. This takes you to the Upload Directories page

• Notice these two settings:

  • Custom Templates Directory

  • Custom PHP Directory

• If both of those are blank, you can stop here. This issue doesn't affect you.

• But if there's a value in either of those, more the directory that each one is set to.

• In your hosting system file browser, navigate to those directories and examine their contents.

• If the directories are empty, to can stop here. This issue doesn't affect you.

• But if there are files in either of these directories, this needs your attention. somebody with developer skills needs to examine them and ensure they stay compatible with the upgraded CiviCRM version.

“This sucks. How can I just avoid this?”

Yeah, it’s a mess.

If you're not already in this situation, yes you can avoid it. Just don't use file overrides. And if you hire a developer for custom features, insist that they should not use file overrides either.

If you're already in this situation, there's probably a way to remove those file overrides completely. Whatever customizations they're providing, there probably another way to do it - without file overrides.

A good CiviCRM developer should be able to help you with that. You should probably explain that you're trying to keep your customizations while making upgrades as easy as possible.

Your pre-upgrade homework:

Before your next upgrade, follow the steps above to see if this issue affects you. If it does, ask your CiviCRM coach or a developer for help.

You'll want to get this sorted out before it's time to upgrade.

All the best,
A.

CiviCRM upgrade oddities: backups

One of the first and most important steps in a CiviCRM upgrade is to make sure you have a backup of your entire site.

Why is this an oddity?

This can be a fairly technical task for a non-technical site owner to undertake.

And there is no “one right way” for all CiviCRM sites. The steps to back up your site will vary greatly depending on your hosting platform, your CMS, and your database configuration.

If you've never done it before, you'll need the answers to several important questions first.

So can you skip this step?

No. Absolutely not.

Any upgrade can go wrong, CiviCRM or otherwise. If you don't have a pre-upgrade backup that you can revert to, you may be left with a permanently broken system.

Take the backup.

So how can you handle this oddity?

First, you'll need to answer all of these questions for yourself:

1. Which mySQL database is storing your CMS data?

2. Which mySQL database is storing your CiviCRM data?

3. Where are your site files stored (i.e., on your hosting platform, which directory has your CMS index.php file)?

4. Where are your backups going to be stored?

Once you have those answers, you'll need to select or formulate a process for creating and restoring backups.

CMS-based backup and restore options:

Drupal, WordPress, and other CMSs offer extensions that can back up your site and then restore those backups if needed.

But those methods, by default, won't include your CiviCRM database in the backup. You'll need to see whether they can be configured to do that.

Hosting-based backup options:

Many hosting providers do offer control-panel-based tools to backup your site, and to restore the site from a backup.

Check with your hosting provider to see what your options are.

Testing your backups:

A backup that's incomplete is about as good as no backup at all.

You'll want to test your backup system to ensure it covers everything.

The challenge is this: If you're not certain that your backup is complete (which is why you're testing it), do you really want to test it on your live site?

Probably not.

The answer is this: Whatever backup method you're using (CMS-based, or hosting-based), it should offer some way to restore your backup to a different location, such as your sandbox site.

That way you can take a backup of your live site, restore it into your sandbox site, and peruse the sandbox site to be sure the backup and restore both worked well.

Getting help:

Pre-upgrade backups are not something you want to skip. Getting this wrong can ruin years of priceless CRM data.

If you can't figure out on your own how to backup and restore all databases and files, get some help.

A developer or a CiviCRM coach should be able to help you get this working, so that you have reliable and reusable tools to manage your backups.

Don't take chances with your priceless constituent data.

All the best,
A.

CiviCRM upgrade oddities: the sandbox site

CiviCRM's upgrade documentation instructs you to test your upgrade first on a copy of the live site, and only after that to actually upgrade the live site.

Why is this an oddity?

Your average non-technical site owner will not be able to create a working copy of their live site without professional help.

So is this a step you can skip?

I'll admit, I often skip it. But I've been doing CiviCRM upgrades for 15 years.

For your first several times applying a CiviCRM upgrade, I cannot in good conscience recommend that you skip this step.

If you have a high risk tolerance, you might ignore that advice and skip this step anyway. That's fine. Just don't tell anyone I recommended it.

So how can you handle this oddity?

You have a couple of options:

1. Learn how to create a working copy of your live site. You may find a CiviCRM coach who can teach you how to do it. Or, you may enjoy learning this on your own.

2. Hire a developer to create a tool that will clone your live site to a working “sandbox” copy. This is a software development task, and perfectly suitable for outsourcing (because you don't care how they build it, you just care that it works whenever you need to use it).

Are there other reasons you would want to have a sandbox site?

Yes, there probably are.

A sandbox site is a place where you can test any number of things:

• CiviCRM upgrades

• CMS upgrades

• New configurations and features

• Mew content and designs

• Data imports

• Lots more …

If you try that stuff on your live site and it goes poorly, you may have some big headaches. But if that happens on your sandbox site, you can just throw it all away by rebuilding the sandbox site from the live site.

Your pre-upgrade homework:

If you’re interested in handling CiviCRM upgrades in-house, you’ll need to put some thought into this part of it.

Find out what it would take to build a sandbox copy of your life site. Can you do it yourself? Could you have developer automate it, so you can rebuild the sandbox site whenever you want?

To sum up:

Building a working “sandbox” copy of your live site is something you may never have thought of before. But the CiviCRM documentation recommends it for good reason. It's also nice to have for other things, not just CiviCRM upgrades.

All the best,
A.

CiviCRM upgrades: why do it in-house

CiviCRM releases several new versions every month, and there’s a security release every few months.

You could hire an outside provider to do the upgrade for you.

But there are advantages to doing it in-house.

1. You control the timing.

2. You know what needs to be tested afterward (and there's always something to test afterward).

3. If your post upgrade testing shows that something is broken, and you need to revert to backup, then you also control the timing on that.

4. Upgrading CiviCRM can mean upgrading or disabling certain extensions. You, more than any outsider, are in the position to know which extensions you do or don't need.

Naturally, there's nothing that says you have to do this in-house.

It's just a question of how much you want to be in control of the systems that keep your organization running smoothly.

All the best,
A.

CiviCRM upgrade WTFs

I'm on a mission to make it easy for non-technical site owners to master and maintain CiviCRM.

Nowhere does this become more difficult than in the matter of CiviCRM upgrades.

A CiviCRM upgrade “should be” straightforward. But too often it's plagued by problems that simply flummox non-technical site owners.

This email starts a series which aims to prepare you, as a non-technical site owner, to handle the surprises that may come up during your next upgrade of CiviCRM.

We'll cover the following topics over the next few days:

• Backups require some technical knowledge

• Code customizations make everything harder

• "Core hacks" are the devil

• Extensions must support your new core version

• Message Templates modifications

• In Joomla: Upgrades can fail to cleanup old files

• Kooky extra steps in pre-upgrade and post-upgrade instructions (SQL queries, file patches, etc.)

Depending on your appetite for adventure, these are all things you can master.

But it can help to know ahead of time what you might be getting yourself into.

All the best,
A.

One thing at a time

If you noticed that your kitchen sink tap was dripping, and then a few days later noticed that your bathtub drain was slow, would you assume the two problems were related?

Could the two of them together indicate that there's some larger problem, somewhere, that you should be worried about? What else might be about to go wrong?

Your answers will depend on how much you know about household plumbing.

But whatever you might believe, one thing is for sure:

If you want them fixed, you'll need to pick one and work on that first. Then fix the other.

Here's the thing:

In complex systems like CiviCRM, data and functionality can be connected in ways that are sometimes surprising.

And when you encounter a few surprises in a short time, it's tempting to feel overwhelmed, as if "everything" is going wrong at once.

Chances are, not everything is going wrong.

But either way, if you want to address those surprises, the only way is to pick one and deal with it first. Then pick another.

Even if they are interconnected, you still have to deal with them one at a time.

All the best,
A.

Every funnel is a filter

A kitchen funnel has one main purpose: making it easier to transfer material from one container to another.

But it has two important side effects:

• It's a filter. Anything that won't fit through the small end won't make it into that container.

• It's a speed regulator: The size of the small end dictates how quickly you can pour without overflowing the funnel.

Your primary CRM expert is a funnel, of sorts.

You count on them to make it easier to get data (and your own ideas for improvement) into your CRM.

But that process there will always imvolve some filtering of the ideas, and some slow-down in the turnaround time.

So here's the question:

Do you want that funneling, filtering, and slowing to happen in-house (where you have more control over it), or in the hands of an outside service provider (where you have very little control)?

All the best,
A.

Comparing yourself

Comparing yourself to others is usually not helpful.

But comparing yourself to yourself — from last month, or from last year — is a way to measure growth.

And if you're not measuring, how can you know you're improving?

All the best,
A.

Scale

As hard as you might work to “get things done” yourself, you’re going to hit a pretty hard limit of scale:

There are only 24 hours in a day.

The only way you can scale beyond that limit is by leading others to multiply your effort.

Leadership is scale.

All the best,
A.

It depends

For any goal you want to achieve, there's no One Right Way to get there.

There's also no Wrong Way.

Consider traveling from Maine to Georgia. How best to travel?

• By private jet? If you have the budget, and the preference, go ahead.

• By commercial air? Probably the most popular choice, with all of its pros and cons.

• By car? You can take the highways, or the back roads.

• By foot? Lots of people do it, with pleasure. It’s the Apalachian Trail.

You just need to know what you want, and what it’s worth to you.

(Hint: What you want is often not “the fastest, cheapest, or shortest route”.)

All the best,
A.

Getting organized

If you've ever tried to organize your home library, or your garage, or anything, you know there are always choices.

• Do I put my East Asian fiction with my East Asian nonfiction, or should I separate fiction and nonfiction entirely?

• Does this Japanese cookbook go in the East Asian section or in the cookbook section?

You have to organize your CRM data too.

• is this person's meal preference part of his individual contact record, or just his participation at this event?

• We're asking personal interest questions on this contribution page, but should that be stored as part of the contribution or the contact?

Here's the thing:

Whether it's books or data, there is no One Right Answer.

It totally depends on how you plan to use the thing later.

Where will it be easiest to find? Where will it make the most sense a month from now?

Of course it's hard to predict the future, so you make your best guess now.

It's also hard to reorganize the whole thing later, so it's worth putting some thought into it at the beginning.

Either way, with a little experience you can get it right in the beginning most of the time.

And with a little more experience, you'll understand that sometimes you guessed wrong and need to reorganize.

That's the way it goes.

All the best,
A.

Zoom out

Can you tell me what this is?

Probably not.

It's an everyday object, but this picture is zoomed in too tightly to provide enough context.

To understand what's going on, you'd have to zoom out — a lot.

Then it’s easy:

Here’s the thing:

When a problem is big and in your face, the immediate symptoms can take up all of your attention.

So it can be hard to understand what could possibly be causing it.

When that happens, take a moment and mentally “zoom out”. Consider all the elements — human and technological — that could contribute to the undesirable situation, or to a more desirable outcome.

Gather clues. Check your assumptions. Step back and try to take in the whole picture.

You’ll have a much better chance at identifying the cause, and then working toward a solution.

All the best,
A.

P.S. If you like these photos, check out Lenstore’s “Close-Up” challenge, from whence they came.

What’s wrong with this report?

You’re staring at a report that shows 300 contacts meeting certain criteria; but you know it should be 303.

Three contacts are missing.

What's wrong with this report?

The short answer is: it's a trick question. There's not yet enough information to know.

• Are the criteria really correct?

• Is there a bug in the report?

• Are those three contacts missing important data?

• Are you just using the wrong report?

I hear from clients with problems like this pretty often.

Yesterday, for example, a client reached out with just this situation.

The difference was, this time she had already thought it through carefully, and identified the three contacts who were missing important data.

Now we can focus on the real problem. Not, “What's wrong with this report?” but, “Why is that data missing?”

Here's the thing:

Problems will come up. That's life.

And the symptoms won't always show up exactly where the problem is.

By stepping back and thinking through the possible causes of those symptoms, you can avoid wasting a lot of time trying to fix a problem that doesn't exist.

All the best,
A.

The one-click unsubscribe requirement

One-click unsubscribe is one of the new requirements that Google and Yahoo are starting to enforce.

If you want your mass emails to land in the in-box for recipients at those providers, you’ll need to make sure you’re supporting this unsubscribe method.

Google says they’re requiring it for senders who send over 5,000 messages each day, but Yahoo just says they’re requiring it, period.

Fortunately, CiviCRM supports this, in versions 5.69.3 and higher.

This could be a good reason to upgrade CiviCRM, even though the latest security release is 5.65.0.

One you’re running CiviCRM 5.69.3 or higher, you can easily enable one-click unsubscribe like so:

Navigate to "Administer => CiviMail => Mailer Settings". This page displays the option "Unsubscribe Methods". Go ahead and enable all available methods:

It’s also a good idea to test this after you’ve made the changes. CiviCRM has provided online documentation covering steps to test this. It should just take a couple of minutes for you to verify.

Here’s the thing:

Crafting the very best email content for your subscribers is a usually well-placed investment, because it help you leverage the interest and good-will of your most ardent supporters.

But if you’re not landing in their in-box, they may just never see your messages.

That’s why it’s just as important — and valuable — to take the extra steps to make sure you’re in compliance with these new mass-mailing requirements.

All the best,
A.

Broomstick challenge

My 8th grade health teacher walked in and said, “You guys are being stupid.”

He pulled out a 5-foot long broomstick, pointed at me and said, “Allen, come over here.”

He drew a mark on the stick about an inch away from one end.

He said, “I want you to balance this stick vertically on your hand while looking at this mark.” And he handed me the stick, holding it vertically, with the marked end at the bottom.

I placed that end on my palm, stared at the mark — and at my palm — and gave it a try.

I couldn't keep it balanced for more than a second or two.

Nobody else in the class could either.

He said, “Here, I'll show you.”

Then held the stick vertically on his palm, with the marked end at the top, and began balancing.

He kept his eyes fixed on that mark, and within a few seconds we understood his trick.

It's a lot easier to balance that stick when you're looking at the far end.

He said, “Very good. But you're still being stupid. Too many of you are living life by staring at things that are just an inch in front of you. You'll never get what you want that way. Why aren't you thinking about where you'll be in a year, or two, or five? Pick a point in the future, keep your eye on it, and adjust as needed to stay on track.”

Here's the thing:

You're probably dealing every day with challenges to make your CRM — and your other systems, both human and technological — do what you need them to do right now.

Naturally, you need those systems to work for you today.

But at the same time: Where do you want to be with them a year from now, or two, or five?

Hopefully you're thinking about that on a regular basis. Hopefully you're taking steps to shape those systems into resources for long-term value for your organization and your mission.

It's a lot easier to balance all of that when you're looking at the far end.

All the best,
A.

Memorizing Jeopardy

Say I record an episode of Jeopardy, and memorize all the answers.

If I then watch that episode with friends, they might be impressed.

But working as a bartender based only on few memorized "Potent Potables" answers? Nobody will be impressed.

In the same way...

Memorizing a set of steps in your CRM is great for quick results the next time you're in the exact same situation.

But life is not Groundhog Day and you're not Bill Murray. You're almost never in the exact same situation twice.

All the best,
A.

Leadership is lasting

Taking action yourself is quick.

Leading others to action is lasting.

As a leader, your job is to make sure the important stuff gets done.

But it's also to make sure the work has a lasting effect, even when you're not around.

All the best,
A.

Steps vs concepts

Watching my high-school daughter study chemistry is a little window into the process of learning.

Her textbook explains the steps to solve a particular type of problem.

So the practice questions are easy.

But at the end of the chapter, there are questions for which the solution isn't obvious.

At that point, what matters is whether she's mastered the underlying concepts.

Does she really know the difference between “resonance structures” and “isomers”?

Sometimes she's curious enough to ask focused questions, and I can help.

Sometimes she's so stressed that she just stares at it in frustration. I don't blame her. This is hard stuff.

So is your CRM.

Having someone explain the steps to get a particular result is one thing. And it's useful enough, for what it's worth.

But if you don’t understand the underlying concepts, the steps are just a quick fix.

To work through the hard problems, to figure out what's going on when things don't seem to add up, you need the underlying concepts:

• What is this data?

• Where did it come from?

• Why does it appear this way in this report, but another way in that search result?

• How is all of this information connected together?

When you're stressed, you might just have to stare at it in frustration.

But when you're curious, you can ask your CiviCRM specialist or coach to help you understand what's going on.

Those conversations are a little slower, and don't always produce the immediate gratification that comes with a set of easy steps.

But they give you the understanding that will empower you at the times when you really do need answers “right now.”

All the best,
A.

Ready for changes at Gmail?

Outbound email is getting a little harder.

Starting in April 2024, Gmail and Yahoo! will enforce new requirements for what mail they’ll accept and transmit to users’ inboxes.

SendGrid and other outbound email providers are doing their best to remind folks about these changes.

Here’s a summary of some tips they sent me this week: 

• All senders: set up SPF or DKIM email authentication for your domain and verify your configuration.

• Senders of 5,000+ daily emails across any email service: set up DMARC email authentication for your sending domain; this can be set to ‘none’.

• If you don’t have this in place already (or make sure it gets done), any email you send to Gmail and Yahoo! inboxes can be temporarily deferred now, and fully rejected in April 2024.

Whether you’re on SendGrid or not, these two guides could be useful in helping you sort things out:

• SendGrid: Gmail and Yahoo’s New Requirements Guide

• SendGrid: Gmail and Yahoo’s Latest Updates FAQ

If you’re feeling a little lost or need some help, please shoot me an email, and I’ll be glad to help you get it sorted.

Either way, don’t put this off.

Your supporters do want to hear from you, and failing to meet these requirements will make that a lot harder than it should be.

All the best,
-A.

Too many contacts?

Your CiviCRM system is never going to limit the number of contacts you can have, or charge you more for adding more users.

(Some CRM providers do that.)

So there's almost never a reason to delete a contact just because you "don't need them anymore."

Duplicate contacts? Yes, merge those.

Meaningless spam contacts like "Mr. 12345"? Yes, go ahead and get rid of those.

But contacts you haven't heard from in a couple of years? Or contacts who made it into your system because of a very brief one-time interaction?

There's little expense in keeping them, and potential value in retaining them:

• If they do come around again, it will be nice to know something about your history with them.

• In the aggregate, such records contain information that could be useful in analysis of trends and common behaviors, especially if you’d like to convert more such people into active supporters. (And you would, woudn’t you?)

Yes, you might want to tag these old contacts somehow as “historical”, and you'll certainly want to maintain your mailing groups so that your mass emails are only going out to contacts who still want to hear from you.

But if you're actively segmenting your contacts, as I hope you are, then there's not much reason to completely delete their history with you just because they're not so active anymore.

All the best,
A.

P.S. To be fair, some hosted CiviCRM services (for example, CiviCRM Spark) do put a limit on the as well, but I don't think many of my leaders are using such a service — or are you?