Amazon Simple Storage Service (AWS S3) is an object storage service widely used for storing and retrieving large amounts of data, such as images, videos, backups, and application assets.
This guide demonstrates how to integrate AWS S3 with Neon by storing file metadata (like the object key and URL) in your Neon database, while using S3 for file storage.
Setup steps
Create a Neon project
- Navigate to pg.new to create a new Neon project.
- Copy the connection string by clicking the Connect button on your Project Dashboard. For more information, see Connect from any application.
Create an AWS account and S3 bucket
-
Sign up for or log in to your AWS Account.
-
Navigate to the S3 service in the AWS Management Console.
-
Click Create bucket. Provide a unique bucket name (e.g.,
my-neon-app-s3-uploads
), select an AWS Region (e.g.,us-east-1
), and configure initial settings. -
Public Access (for this example): For simplicity in accessing uploaded files via URL in this guide, we'll configure the bucket to allow public read access for objects uploaded with specific permissions. Under Block Public Access settings for this bucket, uncheck "Block all public access". Acknowledge the warning.
Public buckets
Making buckets or objects publicly readable carries security risks. For production applications, it's strongly recommended to:
- Keep buckets private (Block all public access enabled).
- Use presigned URLs not only for uploads but also for downloads (temporary read access). This guide uses public access for simplicity, but you should implement secure access controls in production.
-
After the bucket is created, navigate to the Permissions tab. Under Bucket Policy, you can set up a policy to allow public read access to objects. For example:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-neon-app-s3-uploads/*" } ] }
Replace
my-neon-app-s3-uploads
with your actual bucket name. -
Create IAM user for programmatic access:
- Navigate to the IAM service in the AWS Console.
- Go to Users and click Add users.
- Enter a username (e.g.,
neon-app-s3-user
). Select Access key - Programmatic access as the credential type. Click Next: Permissions. - Choose Attach policies directly. Search for and select
AmazonS3FullAccess
. - Click Next, then Create user.
- Click on Create access key.
- Click Other > Create access key. Copy the Access key ID and Secret access key. These will be used in your application to authenticate with AWS S3.
-
Configure CORS for client-side uploads
If your application involves uploading files directly from a web browser using the generated presigned URLs, you must configure Cross-Origin Resource Sharing (CORS) on your S3 bucket. CORS rules tell S3 which web domains are allowed to make requests (like
PUT
requests for uploads) to your bucket. Without proper CORS rules, browser security restrictions will block these direct uploads.In your S3 bucket settings, navigate to the Permissions tab and find the CORS configuration section. Add the following CORS rules:
[ { "AllowedHeaders": ["*"], "AllowedMethods": ["GET", "PUT"], "AllowedOrigins": ["*"], "ExposeHeaders": [], "MaxAgeSeconds": 9000 } ]
This configuration allows any origin (
*
) to performGET
andPUT
requests. In a production environment, you should restrictAllowedOrigins
to your application's domain(s) for security.Create a table in Neon for file metadata
We need a table in Neon to store metadata about the objects uploaded to S3.
-
Connect to your Neon database using the Neon SQL Editor or a client like psql. Create a table including the object key, URL, user ID, and timestamp:
CREATE TABLE IF NOT EXISTS s3_files ( id SERIAL PRIMARY KEY, object_key TEXT NOT NULL UNIQUE, -- Key (path/filename) in S3 file_url TEXT NOT NULL, -- Publicly accessible URL (if object is public) user_id TEXT NOT NULL, -- User associated with the file upload_timestamp TIMESTAMPTZ DEFAULT NOW() );
-
Run the SQL statement. Add other relevant columns as needed (e.g.,
content_type
,size
).
Securing metadata with RLS
If you use Neon's Row Level Security (RLS), remember to apply appropriate access policies to the
s3_files
table. This controls who can view or modify the object references stored in Neon based on your RLS rules.Note that these policies apply only to the metadata in Neon. Access control for the objects within the S3 bucket itself is managed via S3 bucket policies, IAM permissions, and object ACLs.
-
Upload files to S3 and store metadata in Neon
The recommended pattern for client-side uploads to S3 involves presigned upload URLs. Your backend generates a temporary URL that the client uses to upload the file directly to S3. Afterwards, your backend saves the file's metadata to Neon.
This requires two backend endpoints:
/presign-upload
: Generates the temporary presigned URL./save-metadata
: Records the metadata in Neon after the client confirms successful upload.
We'll use Hono for the server,
@aws-sdk/client-s3
and@aws-sdk/s3-request-presigner
for S3 interaction, and@neondatabase/serverless
for Neon.First, install the necessary dependencies:
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner @neondatabase/serverless @hono/node-server hono dotenv
Create a
.env
file:# AWS S3 Credentials & Config AWS_ACCESS_KEY_ID=your_iam_user_access_key_id AWS_SECRET_ACCESS_KEY=your_iam_user_secret_access_key AWS_REGION=your_s3_bucket_region # e.g., us-east-1 S3_BUCKET_NAME=your_s3_bucket_name # e.g., my-neon-app-s3-uploads # Neon Connection String DATABASE_URL=your_neon_database_connection_string
The following code snippet demonstrates this workflow:
import { serve } from '@hono/node-server'; import { Hono } from 'hono'; import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'; import { getSignedUrl } from '@aws-sdk/s3-request-presigner'; import { neon } from '@neondatabase/serverless'; import 'dotenv/config'; import { randomUUID } from 'crypto'; const S3_BUCKET = process.env.S3_BUCKET_NAME; const AWS_REGION = process.env.AWS_REGION; const s3 = new S3Client({ region: AWS_REGION, credentials: { accessKeyId: process.env.AWS_ACCESS_KEY_ID, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY, }, }); const sql = neon(process.env.DATABASE_URL); const app = new Hono(); // Replace this with your actual user authentication logic, by validating JWTs/Headers, etc. const authMiddleware = async (c, next) => { c.set('userId', 'user_123'); await next(); }; // 1. Generate Presigned URL for Upload app.post('/presign-upload', authMiddleware, async (c) => { try { const { fileName, contentType } = await c.req.json(); if (!fileName || !contentType) throw new Error('fileName and contentType required'); const objectKey = `${randomUUID()}-${fileName}`; const publicFileUrl = `https://${S3_BUCKET}.s3.${AWS_REGION}.amazonaws.com/${objectKey}`; const command = new PutObjectCommand({ Bucket: S3_BUCKET, Key: objectKey, ContentType: contentType, }); const presignedUrl = await getSignedUrl(s3, command, { expiresIn: 300 }); return c.json({ success: true, presignedUrl, objectKey, publicFileUrl }); } catch (error) { console.error('Presign Error:', error.message); return c.json({ success: false, error: 'Failed to prepare upload' }, 500); } }); // 2. Save Metadata after Client Upload Confirmation app.post('/save-metadata', authMiddleware, async (c) => { try { const { objectKey, publicFileUrl } = await c.req.json(); const userId = c.get('userId'); if (!objectKey) throw new Error('objectKey required'); await sql` INSERT INTO s3_files (object_key, file_url, user_id) VALUES (${objectKey}, ${publicFileUrl}, ${userId}) `; console.log(`Metadata saved for S3 object: ${objectKey}`); return c.json({ success: true }); } catch (error) { console.error('Metadata Save Error:', error.message); return c.json({ success: false, error: 'Failed to save metadata' }, 500); } }); const port = 3000; serve({ fetch: app.fetch, port }, (info) => { console.log(`Server running at http://localhost:${info.port}`); });
Explanation
- Setup: Initializes the Neon database client (
sql
), Hono (app
), and the AWS S3 client (s3
) configured with region and credentials. - Authentication: A placeholder
authMiddleware
is included. Crucially, this needs to be replaced with real authentication logic. It currently just sets a staticuserId
for demonstration. - Upload endpoints:
/presign-upload
: Generates a temporary secure URL (presignedUrl
) using@aws-sdk/s3-request-presigner
that allows uploading a file directly to S3. It returns the URL, the generatedobjectKey
, and the standard S3 public URL./save-metadata
: Called by the client after successful upload. Saves theobjectKey
,file_url
, anduserId
into thes3_files
table in Neon using@neondatabase/serverless
.
Testing the upload workflow
Testing the presigned URL flow involves multiple steps:
-
Get presigned URL: Send a
POST
request to your/presign-upload
endpoint with a JSON body containingfileName
andcontentType
. Using cURL:curl -X POST http://localhost:3000/presign-upload \ -H "Content-Type: application/json" \ -d '{"fileName": "test-s3.txt", "contentType": "text/plain"}'
You should receive a JSON response with a
presignedUrl
,objectKey
, andpublicFileUrl
:{ "success": true, "presignedUrl": "https://<BUCKET_NAME>.s3.us-east-1.amazonaws.com/.....&x-id=PutObject", "objectKey": "<OBJECT_KEY>", "publicFileUrl": "https://<BUCKET_NAME>.s3.us-east-1.amazonaws.com/<OBJECT_KEY>" }
Note the
presignedUrl
,objectKey
, andpublicFileUrl
from the response. You will use these in the next steps. -
Upload file to S3: Use the received
presignedUrl
to upload the actual file using an HTTPPUT
request. Using cURL:curl -X PUT "<PRESIGNED_URL>" \ --upload-file /path/to/your/test-s3.txt \ -H "Content-Type: text/plain"
A successful upload typically returns HTTP
200 OK
with no body. -
Save metadata: Send a
POST
request to your/save-metadata
endpoint with theobjectKey
andpublicFileUrl
obtained in step 1. Using cURL:curl -X POST http://localhost:3000/save-metadata \ -H "Content-Type: application/json" \ -d '{"objectKey": "<OBJECT_KEY>", "publicFileUrl": "<PUBLIC_URL>"}'
You should receive a JSON response indicating success:
{ "success": true }
Expected outcome:
- The file appears in your S3 bucket (check the AWS Console).
- A new row appears in your
s3_files
table in Neon containing theobject_key
andfile_url
.
You can now integrate API calls to these endpoints from various parts of your application (e.g., web clients using JavaScript's
fetch
API, mobile apps, backend services) to handle file uploads.-
Accessing file metadata and files
Storing metadata in Neon allows your application to easily retrieve references to the files hosted on S3.
Query the
s3_files
table from your application's backend when needed.Example SQL query:
Retrieve files for user 'user_123':
SELECT id, object_key, -- Key (path/filename) in S3 file_url, -- Publicly accessible S3 URL user_id, -- User associated with the file upload_timestamp FROM s3_files WHERE user_id = 'user_123'; -- Use actual authenticated user ID
Using the data:
-
The query returns metadata stored in Neon.
-
The
file_url
column contains the direct link to access the file via S3. -
Use this
file_url
in your application (e.g.,<img>
tags, download links)Private buckets
For private S3 buckets, store only the
object_key
and generate presigned read URLs on demand using a similar backend process.
This pattern effectively separates file storage and delivery concerns (handled by S3) from structured metadata management (handled by Neon), leveraging the strengths of both services.
-
Resources
Need help?
Join our Discord Server to ask questions or see what others are doing with Neon. Users on paid plans can open a support ticket from the console. For more details, see Getting Support.