Project Overview
Hownd, a promotional marketing platform serving 10,000+ merchants with their FetchRev Method™, faced an existential crisis: their entire IT team resigned during an acquisition due to internal disputes between leadership. We were brought in as an emergency rescue team with just 2 weeks to take over a massively complex, poorly documented system before the last engineer departed—while keeping the platform operational for thousands of paying customers.
The Crisis Situation
The Perfect Storm
- Complete Team Departure: Entire engineering team left due to CTO-CEO conflicts during acquisition
- Two-Week Deadline: Only 14 days to absorb knowledge before last engineer’s departure
- Zero Documentation: Minimal system documentation, tribal knowledge walking out the door
- Production Systems: 10,000+ merchants, 5M+ consumers depending on the platform daily
- Acquisition Deadline: New owners needed operational continuity immediately
- Complex Architecture: Distributed system spanning multiple continents with many moving parts
The Undocumented Infrastructure We Inherited
Application Layer
- Legacy Rails 2.3 application (10+ years old)
- Angular mobile application
- Multiple Go microservices running on ECS
- Kubernetes cluster in Sydney, Australia (for WiFi infrastructure)
Data Layer
- 2 PostgreSQL databases (different purposes, unclear relationships)
- 1 MySQL database (legacy merchant data)
- Redis cache (undocumented key patterns)
- S3 buckets scattered across regions
Infrastructure
- ECS containers with inconsistent naming
- Kubernetes cluster 10,000 miles away
- Multiple AWS accounts with unclear boundaries
- No centralized logging or monitoring
- Hardcoded credentials in various places
- Manual deployment processes
The First 2 Weeks: Emergency Knowledge Transfer
Day 1-3: Rapid Triage
Objective: Identify what’s critical vs. what can fail
- Shadow session with outgoing engineers (16-hour days)
- Document every login credential and access point
- Map critical vs. non-critical systems
- Identify single points of failure
- List all production domains and services
- Capture deployment procedures (however manual)
Day 4-7: System Inventory
Objective: Catalog the entire infrastructure
- 90 distinct services across ECS and Kubernetes
- 23 different databases and data stores
- 15 AWS accounts with overlapping access
- 6 different deployment methods (none automated)
- 200+ environment variables scattered across systems
- 12 third-party integrations with varying documentation
Day 8-14: Emergency Runbooks
Objective: Document enough to survive
- Created emergency restart procedures
- Documented critical system dependencies
- Established on-call rotation protocols
- Set up basic monitoring alerts
- Prepared rollback procedures
- Established communication channels with business stakeholders
Month 1: Stabilization Mode
Week 3-4: Flying Solo
Challenge: Last engineer departed, we’re fully on our own
Critical Incidents:
- Production Outage Day 16: Rails app memory leak (undocumented)
- Database Failover Day 21: PostgreSQL primary crashed, no documented recovery
- Sydney K8s Cluster Day 25: WiFi service down, affecting merchant locations
- Payment Processing Day 28: Stripe webhook failure, manual reconciliation required
Survival Tactics:
- 24/7 on-call rotation
- War room Slack channel
- Daily system health checks
- Gradual system understanding through firefighting
- Building institutional knowledge from production incidents
Month 1 Achievements
- Zero Major Data Loss: Despite chaos, protected customer data
- 99.2% Uptime: Maintained service (vs 99.95% target)
- System Map Created: First complete architecture diagram
- Monitoring Established: CloudWatch dashboards for critical paths
- Team Confidence: From panic to cautious optimism
Months 2-6: Understanding & Stabilization
The Archaeological Dig
We approached the codebase like an archaeological expedition, uncovering layers of technical history:
Legacy Rails 2.3 Application
- 10-year-old Ruby on Rails application
- Gem dependencies frozen in 2014
- Custom SugarCRM integration (undocumented modifications)
- Session management using outdated libraries
- Security vulnerabilities (CVEs unpatched for years)
The Go Microservices Maze
- 15+ Go services on ECS Fargate
- Inconsistent logging formats
- No centralized error tracking
- Service-to-service auth via hardcoded tokens
- Database connections without pooling
The Sydney Mystery
- Kubernetes cluster in Australia for WiFi hotspot infrastructure
- WiFi provisioning for merchant locations
- Why Sydney? Nobody knew (turned out: first developer was Australian)
- 300ms latency to US-based databases
- No disaster recovery plan
Database Archaeology
- PostgreSQL #1: Main merchant and transaction data
- PostgreSQL #2: Analytics and reporting (massive duplication)
- MySQL: Legacy data from original platform (still actively queried)
- No foreign key constraints (referential integrity by hope)
- 40% unused tables (from abandoned features)
Critical Improvements (Months 2-6)
Monitoring & Observability
- Implemented New Relic APM
- Centralized logging with CloudWatch Logs Insights
- Distributed tracing with X-Ray
- Created comprehensive dashboards
- Set up PagerDuty escalations
Security Hardening
- Secrets Manager for all credentials
- Rotated all hardcoded passwords
- Implemented WAF rules
- Added VPC security groups
- Enabled CloudTrail logging
- Passed first security audit post-acquisition
Performance Optimization
- Identified N+1 queries in Rails (dozens)
- Implemented database connection pooling
- Added Redis caching layer
- Optimized ECS task sizing
- Reduced Sydney-US latency with data replication
Documentation
- Complete system architecture diagrams
- Service dependency maps
- Database schema documentation
- Deployment runbooks
- Disaster recovery procedures
- On-call playbooks
Months 7-12: Modernization Begins
Rails 2.3 → Rails 5 Migration
The Challenge: Upgrade 10-year-old Rails app without breaking production
Approach:
- Created parallel Rails 5 environment
- Feature-by-feature migration
- Extensive regression testing
- Gradual traffic shift
- Rollback capability at every step
Obstacles Overcome:
- 200+ deprecated gem updates
- Custom monkey patches (hundreds)
- Breaking changes in ActiveRecord
- Session management overhaul
- Asset pipeline migration
Go Services Consolidation
The Problem: 15 microservices, 8 doing similar things
Solution:
- Merged redundant services
- Standardized logging and errors
- Implemented circuit breakers
- Added health check endpoints
- Containerized with best practices
Database Rationalization
Discovery: 40% of schema unused, massive data duplication
Actions:
- Consolidated reporting to single PostgreSQL
- Archived legacy MySQL data
- Implemented proper foreign keys
- Added database migration scripts
- Set up automated backups
Sydney Kubernetes Migration
Decision: Move WiFi infrastructure to US East
Execution:
- Built identical K8s cluster in US-East-1
- Migrated WiFi provisioning services
- Reduced latency from 300ms to 15ms
- Decommissioned Sydney cluster
- Saved $8K/month in cross-region costs
Months 13-24: Building for the Future
Automated CI/CD Pipeline
- Before: Manual deployments via SSH (hours)
- After: GitHub Actions with automated testing (12 minutes)
- Blue-green deployments
- Automated rollbacks on failure
- Feature flags for gradual rollouts
Infrastructure as Code
- Before: Clickops in AWS console
- After: 100% Terraform-managed infrastructure
- Version-controlled infrastructure
- Reproducible environments
- Dev/QA/Prod parity
Team Growth & Knowledge
- Hired 3 additional engineers
- Comprehensive onboarding documentation
- Eliminated single points of knowledge
- Established code review processes
- Implemented pair programming for complex systems
Business Continuity
- Multi-region disaster recovery
- Automated failover procedures
- Regular DR drills
- RTO: 15 minutes, RPO: 5 minutes
- Comprehensive incident response plan
Results & Impact
Operational Metrics
- Uptime: From 99.2% (crisis) to 99.97% (stable)
- Incident Response: From hours to minutes
- Deployment Frequency: From monthly (manual) to daily (automated)
- Mean Time to Recovery: From 4 hours to 15 minutes
- On-Call Alerts: Reduced by 85%
Cost Optimization
- Infrastructure Costs: 40% reduction through right-sizing
- Cross-Region Costs: Eliminated with Sydney migration
- RDS Optimization: Moved to Aurora Serverless (35% savings)
- Unused Resources: Identified and decommissioned (20% savings)
- Total Annual Savings: $450K
Business Impact
- Zero Customer Churn: During transition period
- Acquisition Completed: Despite technical uncertainty
- New Features: Delivered within 6 months of takeover
- Enterprise Deals: Won 3 major contracts (security confidence)
- Team Morale: From crisis mode to innovation mode
Knowledge & Documentation
- 3,000+ Lines: Of runbook documentation
- 45 Architecture Diagrams: Complete system maps
- 100% Recovery: All systems documented for recovery
- New Engineer Onboarding: From impossible to 2 weeks
- Bus Factor: From 1 (crisis) to team resilience
Technical Deep Dive
The Sydney Kubernetes Mystery Solved
Why was there a K8s cluster in Australia?
After extensive investigation, we discovered:
- Original lead developer was Australian
- WiFi hotspot provisioning started as side project
- Never migrated when team grew
- Became production before anyone questioned it
- Resulted in 300ms latency for database calls
Migration Strategy:
- Built parallel cluster in US-East-1
- Replicated WiFi service configuration
- Deployed canary instances
- Gradual traffic shift over 2 weeks
- Monitored for issues
- Decommissioned Sydney cluster
- Result: 95% latency reduction
Rails 2.3 to Rails 5 Battle Scars
Major Obstacles:
Gem Hell: 50+ gems with no Rails 5 support
- Solution: Fork and updated critical gems
- Replaced deprecated gems with modern alternatives
- Built custom solutions for abandoned gems
Monkey Patches Everywhere: 200+ monkey patches
- Solution: Systematic refactoring
- Replaced with proper inheritance
- Some required Rails core understanding
Test Suite: No tests (seriously, zero)
- Solution: Added tests during migration
- Now at 65% coverage
- Prevented countless regressions
Database Consolidation Discoveries
PostgreSQL Database #2 Mystery:
- Discovered it was 90% duplicate of Database #1
- Created 3 years ago by engineer who “wanted better reporting”
- Never properly synchronized
- Data drift of 15%
- Merchant confusion from conflicting reports
Solution:
- Built proper read replicas from primary
- Migrated all reporting queries
- Validated data consistency
- Decommissioned duplicate database
- Saved $12K/month in RDS costs
Crisis Management Lessons
What Saved Us
- Methodical Approach: Inventory before action
- 24/7 Coverage: Nobody alone during crisis
- Communication: Daily updates to stakeholders
- Prioritization: Critical path first, everything else later
- Documentation: Write everything down immediately
- Humility: “I don’t know” is better than guessing
- Team Support: Psychological safety during chaos
What We’d Do Differently
- Negotiate Longer Handover: 2 weeks wasn’t enough (4-6 weeks minimum)
- Freeze Changes: We should have frozen feature development longer
- External Consultants Earlier: Brought in Rails 2.3 expert month 3 (should have been week 1)
- Load Testing: Should have tested before making changes
- Communication Cadence: Daily standups insufficient, needed twice-daily during crisis
Red Flags We Learned to Spot
- Engineers leaving en masse
- Lack of documentation
- Manual deployment processes
- No centralized logging
- Hardcoded credentials
- “It just works” explanations
- Fear of touching certain code
- Single person knowing critical systems
Client Testimonial
“When our entire engineering team left during the acquisition, we thought the platform was lost. Greicodex came in during our darkest hour and not only kept the lights on—they made the platform better than it ever was. They took a chaotic mess of systems and turned it into a well-oiled machine. Their crisis management, technical expertise, and pure determination saved our business.”
— CEO, Hownd (Post-Acquisition)
Technologies We Wrestled With
- Backend: Ruby on Rails 2.3 → 5.2, Go 1.11 → 1.21
- Frontend: Angular 8, React (introduced)
- Containers: Docker, Amazon ECS Fargate, Kubernetes
- Databases: PostgreSQL 9.6 → 14, MySQL 5.7, Redis 6
- Infrastructure: AWS (ECS, RDS, S3, CloudFront, Lambda)
- Monitoring: New Relic, CloudWatch, X-Ray, PagerDuty
- CI/CD: GitHub Actions, Sentry
- Search: Elasticsearch 6 → 7
The Human Side
Team Mental Health
The first 3 months were brutal:
- Semanas de 80 horas
- Turnos on-call de fin de semana
- Incidentes de producción a las 3 AM
- Síndrome del impostor ("¿Realmente podemos hacer esto?")
- Presión de stakeholders de negocio
- Miedo al fracaso catastrófico
How We Survived:
- Mandatory time off after incidents
- Therapy/counseling support
- Team dinners and bonding
- Celebrating small wins
- Honest communication
- Distributed responsibility
- “It’s a marathon, not a sprint” mindset
The Turning Point
Month 4, Week 2: First week with zero critical incidents
That week changed everything:
- Team confidence skyrocketed
- Business stakeholders relaxed
- We shifted from defense to offense
- Started planning improvements vs. firefighting
- Realized we’d made it through the worst
Ongoing Journey
Current State (Month 24+)
- Platform stable and growing
- Modern development practices
- Happy, confident team
- New features shipping regularly
- Technical debt being systematically addressed
- No more 3 AM pages
Future Plans
- Complete Rails upgrade to 7.x
- Microservices mesh with Istio
- Machine learning for fraud detection
- Real-time analytics pipeline
- International expansion ready
- Mobile app rewrite (React Native)
Facing a Similar Crisis?
We’ve been through the worst-case scenario and came out stronger. If you’re facing:
- Team departures
- Undocumented systems
- Technical chaos
- Acquisition challenges
- Legacy platform risks
We understand the pressure, the uncertainty, and the path forward.