Choosing a cloud data warehouse: Snowflake vs BigQuery vs Redshift for analytics
I'm a data architect at a mid-sized e-commerce company, and we're outgrowing our current PostgreSQL-based analytics setup. We're evaluating a move to a proper cloud data warehouse, specifically comparing Snowflake, BigQuery, and Redshift. Our primary needs are handling complex, join-heavy queries on terabyte-scale datasets, cost predictability, and seamless integration with our existing dbt and Looker stack. For teams that have recently made this transition, what were the decisive factors in your choice? How did you approach the migration of ETL pipelines and historical data, and what unexpected operational challenges did you face regarding performance tuning or vendor lock-in?
If I were choosing today, I’d treat the decision as a multi-factor trade-off rather than chasing the fastest benchmark. Snowflake tends to win on flexibility and cross‑cloud resilience (AWS, Azure, GCP), robust concurrency, and strong support for semi-structured data. BigQuery is a great fit if you’re already in Google Cloud and want a serverless model with tight Looker integration and strong analytics tooling. Redshift remains compelling if you’re deeply invested in AWS and want predictable egress costs with RA3 scaling, but you’ll trade some cloud-agnostic flexibility. Budget, data gravity, and how you’ll manage costs day to day often swing the decision more than pure performance numbers.
Migration plan (brief, actionable): 1) inventory datasets, ETL/ELT flows, and SLAs; 2) pick a target platform for a 6–12 week pilot on a representative 1–2 terabyte data slice; 3) adopt dbt for transformations in the warehouse and run incremental loads; 4) build Looker connections to the new warehouse and parallelize dashboards; 5) run a parallel cutover where both systems ingest the same sources for a few weeks; 6) decommission legacy pipelines once validation passes; 7) implement cost governance and alerts.
Operational realities to expect: concurrency limits and queueing can surprise on large joins; fingerprint and scene your data model to avoid churn; beware of long-running queries on BigQuery if you scan huge datasets; monitor storage vs compute spend in Snowflake—auto-suspend and multi-cluster warehouses help but need governance; don’t underestimate data quality drift during migration.
Practical dbt/Looker guidance: standardize on a single warehouse as the canonical source, use dbt docs and lineage to keep teams aligned, and configure Looker with a centralized model to avoid replication drift. If you’re multi-cloud, Snowflake makes governance and access control simpler across regions; with BigQuery/Looker, rely on You to manage GCP IAM/Looker permissions.
Quick yes/no prompts to tailor advice: what cloud are you leaning toward (AWS, Azure, or GCP)? roughly how much data (TB) and how many daily loads? what’s your tolerance for vendor lock-in and data egress costs?
Forum Jump:
Private Messages
User Control Panel
Who's Online
Search
Forum Home
Technology
-- Best Software & Apps Discussions
-- Latest Tech Gadgets & Hardware Talk
-- Programming & Coding Help Forum
-- Cybersecurity Tips and Security News
-- Artificial Intelligence & Machine Learning Insights
-- Mobile Devices Reviews & Troubleshooting
-- Operating Systems Help (Windows, Mac, Linux)
-- Tech Support & Troubleshooting Center
-- Web Hosting, Domains & Server Management
-- IT Careers, Certifications & Training Guides
-- Cloud Computing & DevOps
-- No-Code & Low-Code Platforms
-- Tech Comparisons & Benchmarks
-- Open Source Software & Communities
-- Software Bugs, Errors & Fixes
-- APIs, Integrations & Web Services
-- Data, Databases & Analytics
-- Tech Tutorials & Step-by-Step Guides
-- Emerging Technologies & Innovation
-- Tech Buying Advice & Setup Guides
Entertainment
-- Movie & TV Show Reviews and Discussions
-- Music Talk, Recommendations & News
-- PC and Console Gaming Community
-- Anime & Manga Fan Discussions
-- Book Reviews & Literature Talk
-- Podcast Recommendations & Discussions
-- Comic Books & Graphic Novel Community
-- Celebrity News, Gossip & Updates
-- Streaming Platforms Tips & Recommendations
-- Entertainment Events & Convention News
-- Upcoming Movies & TV Shows (Trailers & Leaks)
-- Best Of Lists & Rankings (Movies, Music, Games)
-- Movie & TV Show Ending Explanations
-- Soundtracks, Scores & Theme Music
-- Behind the Scenes & Production Insights
-- Reboots, Remakes & Sequels Discussions
-- Fan Theories & Easter Eggs
-- Box Office, Ratings & Viewership Stats
-- Awards, Festivals & Red Carpet Events
-- Nostalgia & Classic Entertainment
Lifestyle
-- Travel Tips, Destinations & Guides
-- Food Recipes, Cooking Tips & Culinary Talk
-- Fitness Workouts, Health Tips & Exercise Plans
-- Fashion Trends, Style Tips & Outfit Ideas
-- Home Improvement & Gardening Advice
-- Relationship Advice & Dating Discussions
-- Parenting Help, Tips & Family Life
-- Hobbies, Crafts & DIY Projects
-- Health, Wellness & Self-Improvement
-- Personal Journals & Life Stories
-- Minimalism, Decluttering & Simple Living
-- Morning Routines, Habits & Productivity
-- Biohacking, Longevity & Anti-Aging
-- Sleep, Recovery & Energy Optimization
-- Mindfulness, Meditation & Stress Relief
-- Nutrition Trends, Diets & Eating Styles
-- Smart Home, Home Tech & Automation
-- Sustainable Living & Eco Lifestyle
-- Personal Style, Grooming & Self-Care
-- Life Planning, Goals & Personal Growth
Science & Education
-- Physics Concepts & Research Discussions
-- Biology Studies, Research & Discoveries
-- Chemistry Experiments & Science Help
-- Space Exploration & Astronomy News
-- Mathematics Help, Problems & Solutions
-- Social Science Discussions & Research
-- History Facts, Events & Debates
-- Homework Help & Academic Support
-- Research Projects & Scientific Analysis
-- Latest Science News & Discoveries
-- Data Science & Statistics
-- Mathematics Explained & Problem Solving
-- Artificial Intelligence in Science
-- Medical Science & Health Education
-- Astronomy, Space Missions & Astrophysics
-- Cognitive Science & Learning Psychology
-- Engineering Principles & Technology Science
-- Scientific Experiments & DIY Science
-- Academic Writing, Research & Citations
-- Science Careers, Degrees & Academic Paths
Business & Finance
-- Entrepreneurship Tips & Startup Advice
-- Investing Strategies, Stocks & Trading Discussions
-- Cryptocurrency & Blockchain Insights
-- E-Commerce Business Tips & Platforms
-- Digital Marketing & Advertising Strategies
-- Freelancing Jobs, Tips & Client Management
-- Real Estate Investing & Property Advice
-- Career Development & Job Search Tips
-- Business Management & Leadership Skills
-- Taxes, Accounting & Financial Planning
-- New Member Introductions & Welcomes
-- Business Reputation & Trustworthiness
-- Business Mistakes, Failures & Lessons
-- Startup Validation & Idea Testing
-- Pricing, Revenue Models & Monetization
-- Cash Flow, Forecasting & Financial Planning
-- Legal Basics for Business & Freelancers
-- Business Automation & Process Optimization
-- Scaling, Growth & Expansion Strategies
-- Negotiation, Sales Psychology & Closing
-- Market Research & Competitive Analysis
-- Business Tools, Templates & Resources
Community & Social
-- Off-Topic Discussions & Community Lounge
-- Off-Topic Discussions & Community Lounge
-- Local Groups & Regional Community Talk
-- Member Projects, Builds & Showcases
-- Forum Feedback, Ideas & Suggestions
-- Contests, Giveaways & Community Events
-- Forum Games & Fun Activities
-- Peer Support, Life Advice & Motivation
-- Special Interest Clubs & Hobby Groups
-- Community Meetups & Social Events
-- Online Communities & Forum Building
-- Social Media Platforms & Usage
-- Digital Communication & Online Behavior
-- Content Creation & Creator Economy
-- Online Trends, Memes & Viral Content
-- Online Privacy, Identity & Digital Footprint
-- Moderation, Rules & Community Management
-- Online Relationships & Social Dynamics
-- Internet Culture, Ethics & Society
-- Crowdsourcing, Collaboration & Open Projects
Creative Arts
-- Graphic Design Tips & Portfolio Reviews
-- Photography Advice, Gear & Photo Sharing
-- Video Editing Tutorials & Software Talk
-- Creative Writing, Stories & Critiques
-- Music Production Tutorials & Audio Mixing
-- Drawing Tips, Art Tutorials & Sketch Sharing
-- Crafts & DIY Project Ideas
-- 3D Modeling Software & Design Talk
-- Animation Techniques & Project Sharing
-- Art Critique & Creative Feedback
-- Digital Art & Illustration
-- Graphic Design & Visual Communication
-- Photography Techniques & Editing
-- Video Creation, Filmmaking & Editing
-- Writing, Storytelling & Creative Expression
-- Music Production, Recording & Sound Design
-- Animation, Motion Design & VFX
-- Art History, Styles & Movements
-- Creative Tools, Software & Resources
-- Creative Careers, Freelancing & Portfolios
Automotive & Transport
-- Car & Motorcycle Discussions and Reviews
-- Electric Vehicle (EV) News & Ownership Tips
-- Car Repair, Maintenance & Mechanic Advice
-- Motorsports Racing News & Discussions
-- Public Transport News & Urban Mobility
-- Trucks, Vans & Commercial Vehicle Talk
-- Aviation Talk, Planes, Pilots & Airlines
-- Boating, Sailing & Marine Equipment Forum
-- Cycling Tips, Bikes & Gear Reviews
-- Driving Tips, Safety & Road Knowledge
-- Car Problems, Errors & Diagnostics
-- Car Maintenance, Service & DIY Repairs
-- Buying a Car: Advice, Checks & Mistakes
-- Electric Vehicles (EVs) & Charging
-- Fuel Economy, Costs & Running Expenses
-- Car Technology, Infotainment & Gadgets
-- Vehicle Insurance, Registration & Legal Topics
-- Motorcycles, Scooters & Two-Wheel Transport
-- Public Transport, Mobility & Urban Travel
-- Logistics, Delivery & Commercial Transport
Gaming (Dedicated Section)
-- PC Gaming Tips, Builds & Discussions
-- Console Gaming News & Community
-- Mobile Gaming Apps & Tips
-- Video Game Reviews & Recommendations
-- Online Multiplayer & Clan Recruitment
-- Game Mods, Tools & Custom Content
-- Retro Gaming Classics & Nostalgia
-- Esports Games, Teams & Tournament News
-- Virtual Reality & Augmented Reality Gaming
-- Game Development Tutorials & Industry Talk
-- Game Errors, Crashes & Fixes
-- Game Performance, FPS & Optimization
-- Game Performance, FPS & Optimization
-- Game Guides, Walkthroughs & Tutorials
-- Multiplayer, Co-Op & Competitive Gaming
-- Mods, Custom Content & Community Creations
-- Game Updates, Patches & Roadmaps
-- Gaming Hardware, Peripherals & Gear
-- Indie Games & Hidden Gems
-- Gaming Platforms, Launchers & Services
-- Upcoming Games, Leaks & Rumors
World & Society
-- Breaking News & World Events Discussion
-- Politics, Government & Public Policy Talk
-- Environment & Climate Change Discussions
-- Human Rights Issues & Global Activism
-- Philosophy Discussions & Deep Thinking
-- Religion, Beliefs & Spirituality
-- Legal Questions, Rights & Law Discussions
-- Global Issues & International Relations
-- Cultural Exchange & Worldwide Traditions
-- Economics, Markets & Global Finance
-- Global Trends & Google Search Insights
-- Breaking News Explained & Context
-- Politics, Elections & Public Policy Explained
-- Economy, Inflation & Cost of Living
-- Conflicts, Crisis & Humanitarian Issues
-- Social Issues, Equality & Human Rights
-- Climate Change, Environment & Society
-- Culture, Traditions & Global Lifestyle
-- Technology Impact on Society
-- Viral Stories, Internet Buzz & Public Reaction
Medicine & Health
-- General Medicine
-- Family Medicine & Primary Care
-- Symptoms & Diagnosis
-- Chronic Diseases
-- Infectious Diseases
-- Cardiology
-- Neurology
-- Mental Health & Psychology
-- Dermatology
-- Gastroenterology
-- Pulmonology
-- Orthopedics & Rheumatology
-- Gynecology & Women’s Health
-- Urology & Men’s Health
-- Pediatrics
-- ENT (Ear, Nose & Throat)
-- Ophthalmology (Eye Health)
-- Medications & Treatments
-- Medical Tests & Lab Results
-- Prevention, Nutrition & Lifestyle
Testing
tasdfsdf
tasdfsdf