5-minute tutorial

Migrate X Ads to Databricks in 60 Seconds

Learn how to copy your X Ads data to Databricks with a single command using ingestr - no code required.

One command Zero code Production ready

What you'll learn

How to install and set up ingestr in seconds
Connect to X Ads and Databricks with proper authentication
Copy entire tables or specific data with a single command
Set up incremental loading for continuous data synchronization

Prerequisites

  • Python 3.8 or higher installed
  • X Ads account with API access
  • Approved developer application
  • OAuth 1.0a credentials
  • Databricks workspace with SQL endpoint
  • Personal access token generated
  • SQL endpoint running (not terminated)
  • Appropriate permissions on catalog/schema

Step 1: Install ingestr

Install ingestr in seconds using pip. Choose the method that works best for you:

Recommended: Using uv (fastest)

# Install uv first if you haven't already
pip install uv

# Run ingestr using uvx
uvx ingestr

Alternative: Global installation

# Install globally using uv
uv pip install --system ingestr

# Or using standard pip
pip install ingestr

Verify installation: Run ingestr --version to confirm it's installed correctly.

Step 2: Your First Migration

Let's copy a table from X Ads to Databricks. This example shows a complete, working command you can adapt to your needs.

Set up your connections

X Ads connection format:

twitter-ads://api_key:api_secret@account_id?access_token=token&access_secret=secret

Parameters:

  • • api_key: X API key
  • • api_secret: X API secret
  • • account_id: Ads account identifier
  • • access_token: OAuth access token

Databricks connection format:

databricks://token@host:port/http_path

Parameters:

  • • token: Personal access token (use as username)
  • • host: Workspace URL
  • • port: Port number (usually 443)
  • • http_path: SQL endpoint HTTP path

Run your first copy

Copy the entire users table from X Ads to Databricks:

ingestr ingest \
    --source-uri 'twitter-ads://key:secret@acc_123?access_token=tok&access_secret=sec' \
    --source-table 'campaigns' \
    --dest-uri 'databricks://[email protected]:443/sql/1.0/endpoints/abc123' \
    --dest-table 'raw.campaigns'

What this does:

  • • Connects to your X Ads database
  • • Reads all data from the specified table
  • • Creates the table in Databricks if needed
  • • Copies all rows to the destination

Command breakdown:

  • --source-uri Your source database
  • --source-table Table to copy from
  • --dest-uri Your destination
  • --dest-table Where to write data

Step 3: Verify your data

After the migration completes, verify your data was copied correctly:

Check row count in Databricks:

-- Run this in Databricks
SELECT COUNT(*) as row_count 
FROM raw.campaigns;

-- Check a sample of the data
SELECT * 
FROM raw.campaigns 
LIMIT 10;

Advanced Patterns

Once you've mastered the basics, use these patterns for production workloads.

Only copy new or updated records since the last sync. Perfect for daily updates. 

ingestr ingest \
    --source-uri 'twitter-ads://key:secret@acc_123?access_token=tok&access_secret=sec' \
    --source-table 'public.orders' \
    --dest-uri 'databricks://[email protected]:443/sql/1.0/endpoints/abc123' \
    --dest-table 'raw.orders' \
    --incremental-strategy merge \
    --incremental-key updated_at \
    --primary-key order_id

How it works: The merge strategy updates existing rows and inserts new ones based on the primary key. Only rows where updated_at has changed will be processed.

Common Use Cases

Ready-to-use commands for typical X Ads to Databricks scenarios.

Daily Customer Data Sync

Keep your analytics warehouse updated with the latest customer information every night. 

# Add this to your cron job or scheduler
ingestr ingest \
    --source-uri 'twitter-ads://key:secret@acc_123?access_token=tok&access_secret=sec' \
    --source-table 'public.customers' \
    --dest-uri 'databricks://[email protected]:443/sql/1.0/endpoints/abc123' \
    --dest-table 'analytics.customers' \
    --incremental-strategy merge \
    --incremental-key updated_at \
    --primary-key customer_id

Historical Data Migration

One-time migration of all historical records to your data warehouse. 

# One-time full table copy
ingestr ingest \
    --source-uri 'twitter-ads://key:secret@acc_123?access_token=tok&access_secret=sec' \
    --source-table 'public.transactions' \
    --dest-uri 'databricks://[email protected]:443/sql/1.0/endpoints/abc123' \
    --dest-table 'warehouse.transactions_historical'

Development Environment Sync

Copy production data to your development Databricks instance (with sensitive data excluded). 

# Copy sample data to development
ingestr ingest \
    --source-uri 'twitter-ads://key:secret@acc_123?access_token=tok&access_secret=sec' \
    --source-table 'public.products' \
    --dest-uri 'databricks://[email protected]:443/sql/1.0/endpoints/abc123' \
    --dest-table 'dev.products' \
    --limit 1000  # Only copy 1000 rows for testing

Troubleshooting Guide

Solutions to common issues when migrating from X Ads to Databricks.

Connection refused or timeout errors

Check your connection details:

  • Ensure SQL endpoint is running
  • Verify personal access token is valid
  • Check workspace URL is correct
  • Confirm HTTP path matches your endpoint
Authentication failures

Common authentication issues:

  • Ensure SQL endpoint is running
  • Verify personal access token is valid
  • Check workspace URL is correct
  • Confirm HTTP path matches your endpoint
Schema or data type mismatches

Handling data type differences:

  • ingestr automatically handles most type conversions
  • Databricks: Delta tables support schema evolution
  • Databricks: Complex types (arrays, maps, structs) supported
  • Databricks: Photon acceleration for certain operations
  • Databricks: Partitioning affects query performance
Performance issues with large tables

Optimize large data transfers:

  • Use incremental loading to process data in chunks
  • Run migrations during off-peak hours
  • Split very large tables by date ranges using interval parameters

Ready to scale your data pipeline?

You've learned how to migrate data from X Ads to Databricks with ingestr. For production workloads with monitoring, scheduling, and data quality checks, explore Bruin Cloud.

Star ingestr on GitHub