Adding a dev database to Firebase

I've just put SquadWOD into beta testing and made the annoying error of not properly preparing the backend before onboarding the first few users, I just kept the same database that I had been using during development. The obvious ramification of this mistake is that my own development testing can't now be against the same database as I'd risk making my users very unhappy, potentially corrupting their data.

The solution is to have at least a dev and production database, in Firebase this means creating a new project and making all the same setup choices as the first database. Simple enough, but I now had some real life useful data (not created by me) in that production database that I wanted in my new, empty, dev database.

Exporting and importing with Google Cloud Platform isn't particularly difficult but it is specific and I lost a couple of hours following the wrong documentation and misunderstand some concepts, so here's a walk through for future me (and you) on what to do as of June 2020.

Prerequisites
  1. You have an existing Firebase project with a populated database in it
  2. You have a new Firebase project with an empty database created
  3. You have the Firebase: CLI installed
  4. You have the Cloud SDK: CLI installed
  5. You have billing enabled on both projects

We are going to use gcloud's export command to get the data out of the source database and into a storage bucket that we will create using Google Cloud Platform Console in the same project.

Creating the export bucket
  1. Go to Google Cloud Platform Console and click on the projects drop down at the top.
  2. Make sure your source project (ie where the data currently is) is selected, if you can't see it then check the 'All' tab.
  3. Click the hamburger and find 'Storage'.
  4. Click 'Create Bucket' at the top, call it whatever you like. Choose a region near or the same as your project, I choose uniform access control and leave the rest and default and click 'Create'.
  5. You'll now see the bucket in the list of buckets.
Exporting the data to the bucket
  1. Switch in to the project in the Cloud CLI with gcloud config set project [your-project-name]
  2. Run the export command into that new bucket gcloud firestore export gs://[bucket-name], if you get a permissions error then it is highly likely you have created the bucket in a different project to the one your Firebase project is in, this is what tripped me up for a while.
  3. Once that completes go back to Google Cloud Platform Console and you should see a document in your bucket named simlar to '2020-06-05T14:59:17_93348/' - you'll need that later
Importing the data from the bucket
  1. Run gcloud config set project [destination-project-id] to change into the destination project.
  2. Google Cloud Platform has Service Accounts, these are accounts for each project that can have permissions to do stuff in other projects as required. You get a service account as default for each project that is in this format:
    [project-name]v@appspot.gserviceaccount.com:admin.
  3. Run the following to add read privileges for that account in you source project with gsutil iam ch serviceAccount:[project-name]@appspot.gserviceaccount.com:admin gs://[source-bucket-name].
  4. Run gcloud firestore import gs://[bucket-name]/2020-06-05T14:59:17_93348 (changing the part after the / to match yours) to import that source bucket straight into your Cloud Firestore database.
  5. Once finished head over to the Database in the Firebase console and you will see it fully populated!
User accounts

Now the data is in, you need to bring across the user accounts so that the data ties to the correct user (assuming you have users). This is very straightforward and uses the Firebase CLI instead.

  1. Run firebase projects:list to get a handy list of all your projects.
  2. Run firebase auth:export accounts.json --format=json --project=[source-project-id] to export the users to a file on your local machine.
  3. Run firebase auth:import accounts.json --project=[destination-project-id] to import them to the destination project. Now all the user created data will tie to their information should you need it.

Done! Not too difficult but fiddly at points and you need to be very aware of what account you're in ans what you're trying to do at that moment.