Migrate from Firebase Firestore to Neon Postgres
newLearn how to migrate your data from Firebase Firestore to Neon Postgres using a custom Python script
This guide describes how to migrate data from Firebase Firestore to Neon Postgres.
We'll use a custom Python script to export data from Firestore to a local file, and then import the data into Neon Postgres. This approach allows us to handle Firestore's document-based structure and convert it into the relational database format suitable for Postgres.
Prerequisites
-
A Firebase project containing the Firestore data you want to migrate.
-
A Neon project to move the data to.
For detailed information on creating a Neon project, see Create a project.
-
Python 3.10 or later installed on your local machine. Additionally, add the following packages to your Python virtual environment:
firebase_admin
, which is Google's python SDK for Firebase andpsycopg
, which is used to connect to Neon Postgres database.You can install them using
pip
:
Retrieve Firebase credentials
This section describes how to fetch the credentials to connect to your Firebase Firestore database.
- Log in to your Firebase Console and navigate to your project.
- Go to Project settings (the gear icon next to "Project Overview" in the left sidebar).
- Under the Service Accounts tab, click Generate new private key. This will download a JSON file containing your credentials.
- Save this JSON file securely on your local machine. We'll use it in our Python script.
For more information, please consult the Firebase documentation.
Export data from Firestore
In this step, we will use a Python script to export data from Firestore. This script will:
- Connect to Firestore
- Retrieve all collections and documents
- Save the Firestore documents to a format suitable for ingesting into Postgres later
Here's the Python script:
Save this script as firebase-download.py
. To run the script, you need to provide the path to your Firebase credentials JSON file and the output directory for the downloaded data. Run the following command in your terminal:
For each unique collection id, this script creates a line-delimited JSON file, and all documents in that collection (spanning different top-level documents) are saved to it. For example, if you have a collection with the following structure:
The script will create the following files:
users.json
: Contains all user documents, i.e.,user1
,user2
.orders.json
: Contains all order documents across all users -order1
,order2
,order3
.items.json
: Contains all item documents across all orders -item1
,item2
.
Each file contains a JSON object for each document. To illustrate, order1
gets saved to orders.json
in the following format:
This structure allows for easy reconstruction of the hierarchical relationships between users, orders, and items, while also providing a flat file structure that's easy to process and import into other systems.
Prepare your Neon destination database
This section describes how to prepare your destination Neon Postgres database to receive the imported data.
Create the Neon database
- In the Neon Console, go to your project dashboard.
- In the sidebar, click on Databases.
- Click the New Database button.
- Enter a name for your database and click Create.
For more information, see Create a database.
Retrieve Neon connection details
-
In the Neon Console, go to your project dashboard.
-
Find the Connection Details widget, and toggle to the correct
Database
option. -
Copy the connection string. It will look similar to this:
Import data into Neon
We use another python script to import the firestore data we previously downloaded into Neon.
Save this script as neon-import.py
. To run the script, you need to provide the path to the input directory containing the JSON files and the Neon connection string. Run the following command in your terminal:
This script iterates over each JSON file in the input directory, creates a table in the Neon database for each collection, and inserts the data into the table. It also handles conflicts by updating the existing data with the new data.
Verify the migration
After running both the Firestore export and the Neon import scripts, you should verify that your data has been successfully migrated:
-
Connect to your Neon database using the Neon SQL Editor or
psql
. -
List all tables in your database:
-
Run some sample queries to check that the data has been successfully imported. For example, the following query fetches all orders made by the first two customers:
Compare the results with those from your Firestore database to ensure data integrity. Note that using the
parent_id
field, we can navigate through the hierarchical structure of the original data.
Other migration options
While this guide focuses on using a custom Python script, there are other migration options available:
-
Firestore managed export/import
If you have a large volume of data to migrate, you can use the Google Cloud Firestore managed export and import service. It allows you to export your Firestore data to a Google Cloud Storage bucket, from where you can download and ingest it into Neon.
-
Open source utilities
There are also a number of open source utilities available that can help export data from Firestore to local files.
However, these utilities are not as robust as the managed export/import service. If your data size is not big, we recommend using the sample code provided above or adapting it to your specific needs.
Reference
For more information on the tools and libraries used in this guide, refer to the following documentation:
Need help?
Join our Discord Server to ask questions or see what others are doing with Neon. Users on paid plans can open a support ticket from the console. For more detail, see Getting Support.